I am trying to do cross shard queries but I am unable to perform it.
Using MySQL Fabric - 1.5.6
Using MySQL Fabric - 1.5.6
↧
How to perform Cross-shard queries to MySQL Fabric? (1 reply)
↧
I am unable to connect to MySQL Workbench 6.3.6 with MySQL Fabric 1.5.6? (1 reply)
I am getting error:-
WBFabricInterface.openConnction(). Lost connection to mysql server at waiting for initial communication packets, system error (10060)
My firewall is disabled
WBFabricInterface.openConnction(). Lost connection to mysql server at waiting for initial communication packets, system error (10060)
My firewall is disabled
↧
↧
Error while creating a shard (1 reply)
Whenever I execute:
mysqlfabric sharding add_table 1 employees.employees emp_no
I get the following error:
DatabaseError: Command (INSERT INTO shard_tables(shard_mapping_id, table_name, column_name) VALUES(%s, %s, %s), ('1', 'employees.employees', 'emp_no')) failed accessing (127.0.0.1:3306). 1452 (23000): Cannot add or update a child row: a foreign key constraint fails (`fabric`.`shard_tables`, CONSTRAINT `fk_shard_mapping_id` FOREIGN KEY (`shard_mapping_id`) REFERENCES `shard_maps` (`shard_mapping_id`)).
mysqlfabric sharding add_table 1 employees.employees emp_no
I get the following error:
DatabaseError: Command (INSERT INTO shard_tables(shard_mapping_id, table_name, column_name) VALUES(%s, %s, %s), ('1', 'employees.employees', 'emp_no')) failed accessing (127.0.0.1:3306). 1452 (23000): Cannot add or update a child row: a foreign key constraint fails (`fabric`.`shard_tables`, CONSTRAINT `fk_shard_mapping_id` FOREIGN KEY (`shard_mapping_id`) REFERENCES `shard_maps` (`shard_mapping_id`)).
↧
Error: Invalid database connection (1 reply)
Hello,
I get this error starting fabric on debian 2.4
[INFO] 1464696852.220653 - MainThread - Initializing persister: user (fabric_store), server (localhost:3306), database (mysql_fabric).
Error: Invalid database connection.
I have been debugging the python script
apparently after the connector has done its handshake, mysql returns an error
"unknown database mysql_fabric"
wich is odd since the script is supposed to create the db after logging to the server
can someone look into this ? thanks
but if I do it manualy, fabric_store has access and can create the db
How to repeat:
mysqlfabric manage start
[INFO] 1464592921.549109 - MainThread - Initializing persister: user (fabric_store), server (localhost:3306), database (mysql_fabric).
Error: Invalid database connection.
mysql -u fabric_store -p*********
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 58
Server version: 5.7.12 MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> create database mysql_fabric;
Query OK, 1 row affected (0.00 sec)
installed packages:
mysql-connector-python_2.1.3-1debian8.2_all.deb
mysql-server_5.7.12-1debian8_amd64.deb-bundle.tar
mysql-utilities_1.5.6-1debian8_all.deb
followed instruction :
https://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric-create-user.html
https://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric-configuration-file.html
regards
I get this error starting fabric on debian 2.4
[INFO] 1464696852.220653 - MainThread - Initializing persister: user (fabric_store), server (localhost:3306), database (mysql_fabric).
Error: Invalid database connection.
I have been debugging the python script
apparently after the connector has done its handshake, mysql returns an error
"unknown database mysql_fabric"
wich is odd since the script is supposed to create the db after logging to the server
can someone look into this ? thanks
but if I do it manualy, fabric_store has access and can create the db
How to repeat:
mysqlfabric manage start
[INFO] 1464592921.549109 - MainThread - Initializing persister: user (fabric_store), server (localhost:3306), database (mysql_fabric).
Error: Invalid database connection.
mysql -u fabric_store -p*********
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 58
Server version: 5.7.12 MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> create database mysql_fabric;
Query OK, 1 row affected (0.00 sec)
installed packages:
mysql-connector-python_2.1.3-1debian8.2_all.deb
mysql-server_5.7.12-1debian8_amd64.deb-bundle.tar
mysql-utilities_1.5.6-1debian8_all.deb
followed instruction :
https://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric-create-user.html
https://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric-configuration-file.html
regards
↧
mysqldbcompare stops processing (1 reply)
Hello
I have a problem with mysqldbcompare
mysqldbcompare fails to complete the comparison of two databases
the outuput is sent to a file, it can not go beyond 20.5MB
$ mysqldbcompare --run-all-tests --difftype=unified --skip-object-compare --skip-diff --skip-row-count --server1=root:***@localhost --server2=root:***@localhost db1_bk:db2 > file_dbcompare.txt
I do not understand if the problem is the database size, which is about 3GB
maybe I should change some my.ini configuration?
there is another tool to compare date database?
I have a problem with mysqldbcompare
mysqldbcompare fails to complete the comparison of two databases
the outuput is sent to a file, it can not go beyond 20.5MB
$ mysqldbcompare --run-all-tests --difftype=unified --skip-object-compare --skip-diff --skip-row-count --server1=root:***@localhost --server2=root:***@localhost db1_bk:db2 > file_dbcompare.txt
I do not understand if the problem is the database size, which is about 3GB
maybe I should change some my.ini configuration?
there is another tool to compare date database?
↧
↧
Connector/J fails if fabric node is not available (1 reply)
Hi,
according to Fabric 1.5 FAQ 8.10.8, if the fabric node becomes unavailable connectors will continue to provide connections to a cached copy of the primary node. The FAQ is available here: http://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric-faq.html#fabric-faq-ha
In my test setup I see that Connector/J does not behave like that, it fails as soon as the fabric node becomes unavailable.
Test setup:
Fabric 1.5 installed on a VM running CentOS 7
Two mysql 5.7 community servers installed on two CentOS 7 VMs
Connector/J is version 5.1.39
The application is deployed on Tomcat 8 running in Windows 7 and it is using Hibernate 4.3.1 to connect to the fabric node.
The JDBC URL looks like this: jdbc:mysql:fabric://192.168.235.4:32274/mydb?fabricServerGroup=mygrp&fabricUsername=admin&fabricPassword=secret
Everything works as expected with the db nodes, I can switch primary and secondary manually, I can shut down the primary node, I can reconnect it as secondary to the HA group etc. But if I shutdown the fabric process on the fabric node I get this exception:
WARN [2016-06-10 14:09:00] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper:144] [nqp5rATecQEwbpCCrfnd] - SQL Error: 0, SQLState: null
ERROR [2016-06-10 14:09:00] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper:146] [nqp5rATecQEwbpCCrfnd] - Unable to establish connection to the Fabric server
If the FAQ is correct, I would expect Connector/J to continue providing connections to the primary of the HA group, using the cached info of which is primary and which secondary.
Any thoughts anyone? Is the FAQ wrong or is some connector/J bug or configuration missing?
Thanks
Lefteris
according to Fabric 1.5 FAQ 8.10.8, if the fabric node becomes unavailable connectors will continue to provide connections to a cached copy of the primary node. The FAQ is available here: http://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric-faq.html#fabric-faq-ha
In my test setup I see that Connector/J does not behave like that, it fails as soon as the fabric node becomes unavailable.
Test setup:
Fabric 1.5 installed on a VM running CentOS 7
Two mysql 5.7 community servers installed on two CentOS 7 VMs
Connector/J is version 5.1.39
The application is deployed on Tomcat 8 running in Windows 7 and it is using Hibernate 4.3.1 to connect to the fabric node.
The JDBC URL looks like this: jdbc:mysql:fabric://192.168.235.4:32274/mydb?fabricServerGroup=mygrp&fabricUsername=admin&fabricPassword=secret
Everything works as expected with the db nodes, I can switch primary and secondary manually, I can shut down the primary node, I can reconnect it as secondary to the HA group etc. But if I shutdown the fabric process on the fabric node I get this exception:
WARN [2016-06-10 14:09:00] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper:144] [nqp5rATecQEwbpCCrfnd] - SQL Error: 0, SQLState: null
ERROR [2016-06-10 14:09:00] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper:146] [nqp5rATecQEwbpCCrfnd] - Unable to establish connection to the Fabric server
If the FAQ is correct, I would expect Connector/J to continue providing connections to the primary of the HA group, using the cached info of which is primary and which secondary.
Any thoughts anyone? Is the FAQ wrong or is some connector/J bug or configuration missing?
Thanks
Lefteris
↧
Fabric connection error - An item with the same key has already been added (1 reply)
Hi All, I have a mysql fabric POC set up, with 6 vmware machines running
windows server 2012 64bit, having 3 groups of 2 servers each. Also have 1 separate fabric server.
I am using the .net connector 6.9.8 and visual studio 15.
Following the example at https://dev.mysql.com/doc/connector-net/en/connector-net-programming-fabric.html
I was actually able to connect from visual studio and run the example creating a database , table and inserting a value, the very first time I ran the program.
The second time I try, I get the following error.
Outer exception
at MySql.Fabric.FabricServerGroup.GetServerList()
at MySql.Fabric.FabricServerGroup.GetServer(Boolean isMaster, MySqlConnectionStringBuilder settings)
at MySql.Data.MySqlClient.Replication.ReplicationManager.GetNewConnection(String groupName, Boolean master, MySqlConnection connection)
at MySql.Data.MySqlClient.MySqlConnection.Open()
at FabricTest.Program.RunFabricTest() in c:\users\dhanabdi\documents\visual studio 2015\Projects\Shard_test1\Shard_test1\Program.cs:line 97
at FabricTest.Program.Main(String[] args) in c:\users\dhanabdi\documents\visual studio 2015\Projects\Shard_test1\Shard_test1\Program.cs:line 28
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
Innerexception - {"An item with the same key has already been added."}
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at MySql.Fabric.FabricServerGroup.GetShards()
at MySql.Fabric.FabricServerGroup.GetServerList()
Some Items I can glean from the exception detail window in Visual Studio.
MySQL.Fabric.FabricException - {"Unable to connect to Fabric server."}
Keys - {System.Collections.ListDictionaryInternal.NodeKeyValueCollection}
Innerexception - {"An item with the same key has already been added."}
Source - MySQL.Fabric.Plugin
I am sure its not authentication related, since I get a invalid username/password if I change id/password.
Looking at "netstat -a -b" I see a connection established at 32275, so looks like something it does after the connection is established but before the connect call is a success.
Appreciate any thoughts , inputs.
windows server 2012 64bit, having 3 groups of 2 servers each. Also have 1 separate fabric server.
I am using the .net connector 6.9.8 and visual studio 15.
Following the example at https://dev.mysql.com/doc/connector-net/en/connector-net-programming-fabric.html
I was actually able to connect from visual studio and run the example creating a database , table and inserting a value, the very first time I ran the program.
The second time I try, I get the following error.
Outer exception
at MySql.Fabric.FabricServerGroup.GetServerList()
at MySql.Fabric.FabricServerGroup.GetServer(Boolean isMaster, MySqlConnectionStringBuilder settings)
at MySql.Data.MySqlClient.Replication.ReplicationManager.GetNewConnection(String groupName, Boolean master, MySqlConnection connection)
at MySql.Data.MySqlClient.MySqlConnection.Open()
at FabricTest.Program.RunFabricTest() in c:\users\dhanabdi\documents\visual studio 2015\Projects\Shard_test1\Shard_test1\Program.cs:line 97
at FabricTest.Program.Main(String[] args) in c:\users\dhanabdi\documents\visual studio 2015\Projects\Shard_test1\Shard_test1\Program.cs:line 28
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
Innerexception - {"An item with the same key has already been added."}
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at MySql.Fabric.FabricServerGroup.GetShards()
at MySql.Fabric.FabricServerGroup.GetServerList()
Some Items I can glean from the exception detail window in Visual Studio.
MySQL.Fabric.FabricException - {"Unable to connect to Fabric server."}
Keys - {System.Collections.ListDictionaryInternal.NodeKeyValueCollection}
Innerexception - {"An item with the same key has already been added."}
Source - MySQL.Fabric.Plugin
I am sure its not authentication related, since I get a invalid username/password if I change id/password.
Looking at "netstat -a -b" I see a connection established at 32275, so looks like something it does after the connection is established but before the connect call is a success.
Appreciate any thoughts , inputs.
↧
mysqlfailover 1.6.1 and mysql 5.7.12 ERROR: Query failed. 1054 (42S22): Unknown column 'password' in 'field list' (10 replies)
Is there a version of mysqlfailover that works with mysql 5.7, since they removed the password column? Or is this utility assumed to use passwordless accounts?
Thanks,
Aaron
Thanks,
Aaron
↧
Pick the wrong master and fails to failover (2 replies)
Hello,
following this : https://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric-quick-start-replication.html
I created one slave, one master, and replication is working between them
I install fabric on a separate box , create a group and add both machines to that group
now 2 things strike me
first fabric set the primary on the slave instead of the master
mysqlfabric group lookup_servers my_group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
server_uuid address status mode weight
------------------------------------ ------------- --------- ---------- ------
24e83dc5-3848-11e6-b605-0050569958d0 slave:3306 PRIMARY READ_WRITE 1.0
acb3a95e-3847-11e6-b25a-005056996666 master:3306 SECONDARY READ_ONLY 1.0
second, as I power off the slave(considered as master by fabric) to check the failover, the group health command hangs, and once I power back on the slave, I get this
mysqlfabric group health my_group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------------------------------------------------------------------------------------- ---------
24e83dc5-3848-11e6-b605-0050569958d0 0 FAULTY 0 0 0 0 False False
acb3a95e-3847-11e6-b25a-005056996666 1 SECONDARY 0 0 1 0 error connecting to master 'fabric_server@slave:3306' - retry-time: 60 retries: 12 False
this all dont make any sense...
- how do I force fabric to take te correct master ?
- why the failover fails so miserably ?
thanks
following this : https://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric-quick-start-replication.html
I created one slave, one master, and replication is working between them
I install fabric on a separate box , create a group and add both machines to that group
now 2 things strike me
first fabric set the primary on the slave instead of the master
mysqlfabric group lookup_servers my_group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
server_uuid address status mode weight
------------------------------------ ------------- --------- ---------- ------
24e83dc5-3848-11e6-b605-0050569958d0 slave:3306 PRIMARY READ_WRITE 1.0
acb3a95e-3847-11e6-b25a-005056996666 master:3306 SECONDARY READ_ONLY 1.0
second, as I power off the slave(considered as master by fabric) to check the failover, the group health command hangs, and once I power back on the slave, I get this
mysqlfabric group health my_group
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------------------------------------------------------------------------------------- ---------
24e83dc5-3848-11e6-b605-0050569958d0 0 FAULTY 0 0 0 0 False False
acb3a95e-3847-11e6-b25a-005056996666 1 SECONDARY 0 0 1 0 error connecting to master 'fabric_server@slave:3306' - retry-time: 60 retries: 12 False
this all dont make any sense...
- how do I force fabric to take te correct master ?
- why the failover fails so miserably ?
thanks
↧
↧
Had been building mysqlfabric using official document, using official test shard data script error. (no replies)
Mysqlfabric throughout the build process is to use the official installation manual, mysql server version is5.7.13 for Linux - glibc2.5 (X86_64), mysql fabric version is 1.5.6, mysql connector - python version is 2.1.3-1. E16. X86_64. Python version is 2.6.6) does not appear in the process of building an error.In the use of official test the fragmented data fragmentation test script, and appear the error:
Traceback (most recent call last):
File "fabric_tetst.py", line 95, in <module>
add_employee(conn, emp_no,first_names[emp_no % len(first_names)],last_names[emp_no % len(last_names)],gtid_executed)
File "fabric_tetst.py", line 28, in add_employee
cur.execute("USE employees")
File "/usr/lib/python2.6/site-packages/mysql/connector/cursor.py", line 515, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/usr/lib/python2.6/site-packages/mysql/connector/fabric/connection.py", line 1608, in cmd_query
self.handle_mysql_error(exc)
File "/usr/lib/python2.6/site-packages/mysql/connector/fabric/connection.py", line 1569, in handle_mysql_error
raise exc
mysql.connector.errors.ProgrammingError: 1049 (42000): Unknown database 'employees'
I looked at the library of employees in global group in the DB, however, employees in library not synchronized to the other two shard group.Why is this?
here is my profile:
import random
import mysql.connector
from mysql.connector import fabric
def prepare_synchronization(cur):
# We need to keep track of what we have executed so far to guarantee
# that the employees.employees table exists at all shards.
gtid_executed = None
cur.execute("SELECT @@global.gtid_executed")
for row in cur:
gtid_executed = row[0]
return gtid_executed
def synchronize(cur, gtid_executed):
# Guarantee that a slave has created the employees.employees table
# before reading anything.
cur.execute(
"SELECT WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS('%s', 0)" %
(gtid_executed, )
)
cur.fetchall()
def add_employee(conn, emp_no, first_name, last_name, gtid_executed):
conn.set_property(tables=["employees.employees"], key=emp_no,mode=fabric.MODE_READWRITE)
cur = conn.cursor()
synchronize(cur, gtid_executed)
print emp_no,first_name,last_name,gtid_executed
cur.execute("USE employees")
print "777777777777777"
cur.execute(
"INSERT INTO employees VALUES (%s, %s, %s)",
(emp_no, first_name, last_name)
)
def find_employee(conn, emp_no, gtid_executed):
conn.set_property(tables=["employees.employees"], key=emp_no,
mode=fabric.MODE_READONLY)
cur = conn.cursor()
synchronize(cur, gtid_executed)
cur.execute("USE employees")
for row in cur:
print "Had to synchronize", row, "transactions."
cur.execute(
"SELECT first_name, last_name FROM employees "
"WHERE emp_no = %s", (emp_no, )
)
for row in cur:
print row
def pick_shard_key():
shard = random.randint(0, 2)
shard_range = shard * 100000
shard_range = shard_range if shard != 0 else shard_range + 1
shift_within_shard = random.randint(0, 99999)
return shard_range + shift_within_shard
# Address of the Fabric, not the host we are going to connect to.
conn = mysql.connector.connect(
fabric={"host" : "localhost", "port" : 32274,
"username": "admin", "password" : "admin"
},
user="fabric_server", password="wxy@123..", autocommit=True
)
conn.set_property(tables=["employees.employees"], scope=fabric.SCOPE_GLOBAL,
mode=fabric.MODE_READWRITE)
print "111111111111111111"
cur = conn.cursor()
cur.execute("CREATE DATABASE IF NOT EXISTS employees")
print "222222222222222222"
cur.execute("USE employees")
print "333333333333333333"
cur.execute("DROP TABLE IF EXISTS employees")
cur.execute(
"CREATE TABLE employees ("
" emp_no INT, "
" first_name CHAR(40), "
" last_name CHAR(40)"
")"
)
print "44444444444444444444"
gtid_executed = prepare_synchronization(cur)
print gtid_executed
print "55555555555555555555"
conn.set_property(scope=fabric.SCOPE_LOCAL)
print "6666666666666666666"
first_names = ["John", "Buffalo", "Michael", "Kate", "Deep", "Genesis"]
last_names = ["Doe", "Bill", "Jackson", "Bush", "Purple"]
print last_names
list_emp_no = []
for count in range(10):
emp_no = pick_shard_key()
print emp_no
list_emp_no.append(emp_no)
add_employee(conn, emp_no,first_names[emp_no % len(first_names)],last_names[emp_no % len(last_names)],gtid_executed)
for emp_no in list_emp_no:
print emp_no
find_employee(conn, emp_no, gtid_executed)
Traceback (most recent call last):
File "fabric_tetst.py", line 95, in <module>
add_employee(conn, emp_no,first_names[emp_no % len(first_names)],last_names[emp_no % len(last_names)],gtid_executed)
File "fabric_tetst.py", line 28, in add_employee
cur.execute("USE employees")
File "/usr/lib/python2.6/site-packages/mysql/connector/cursor.py", line 515, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/usr/lib/python2.6/site-packages/mysql/connector/fabric/connection.py", line 1608, in cmd_query
self.handle_mysql_error(exc)
File "/usr/lib/python2.6/site-packages/mysql/connector/fabric/connection.py", line 1569, in handle_mysql_error
raise exc
mysql.connector.errors.ProgrammingError: 1049 (42000): Unknown database 'employees'
I looked at the library of employees in global group in the DB, however, employees in library not synchronized to the other two shard group.Why is this?
here is my profile:
import random
import mysql.connector
from mysql.connector import fabric
def prepare_synchronization(cur):
# We need to keep track of what we have executed so far to guarantee
# that the employees.employees table exists at all shards.
gtid_executed = None
cur.execute("SELECT @@global.gtid_executed")
for row in cur:
gtid_executed = row[0]
return gtid_executed
def synchronize(cur, gtid_executed):
# Guarantee that a slave has created the employees.employees table
# before reading anything.
cur.execute(
"SELECT WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS('%s', 0)" %
(gtid_executed, )
)
cur.fetchall()
def add_employee(conn, emp_no, first_name, last_name, gtid_executed):
conn.set_property(tables=["employees.employees"], key=emp_no,mode=fabric.MODE_READWRITE)
cur = conn.cursor()
synchronize(cur, gtid_executed)
print emp_no,first_name,last_name,gtid_executed
cur.execute("USE employees")
print "777777777777777"
cur.execute(
"INSERT INTO employees VALUES (%s, %s, %s)",
(emp_no, first_name, last_name)
)
def find_employee(conn, emp_no, gtid_executed):
conn.set_property(tables=["employees.employees"], key=emp_no,
mode=fabric.MODE_READONLY)
cur = conn.cursor()
synchronize(cur, gtid_executed)
cur.execute("USE employees")
for row in cur:
print "Had to synchronize", row, "transactions."
cur.execute(
"SELECT first_name, last_name FROM employees "
"WHERE emp_no = %s", (emp_no, )
)
for row in cur:
print row
def pick_shard_key():
shard = random.randint(0, 2)
shard_range = shard * 100000
shard_range = shard_range if shard != 0 else shard_range + 1
shift_within_shard = random.randint(0, 99999)
return shard_range + shift_within_shard
# Address of the Fabric, not the host we are going to connect to.
conn = mysql.connector.connect(
fabric={"host" : "localhost", "port" : 32274,
"username": "admin", "password" : "admin"
},
user="fabric_server", password="wxy@123..", autocommit=True
)
conn.set_property(tables=["employees.employees"], scope=fabric.SCOPE_GLOBAL,
mode=fabric.MODE_READWRITE)
print "111111111111111111"
cur = conn.cursor()
cur.execute("CREATE DATABASE IF NOT EXISTS employees")
print "222222222222222222"
cur.execute("USE employees")
print "333333333333333333"
cur.execute("DROP TABLE IF EXISTS employees")
cur.execute(
"CREATE TABLE employees ("
" emp_no INT, "
" first_name CHAR(40), "
" last_name CHAR(40)"
")"
)
print "44444444444444444444"
gtid_executed = prepare_synchronization(cur)
print gtid_executed
print "55555555555555555555"
conn.set_property(scope=fabric.SCOPE_LOCAL)
print "6666666666666666666"
first_names = ["John", "Buffalo", "Michael", "Kate", "Deep", "Genesis"]
last_names = ["Doe", "Bill", "Jackson", "Bush", "Purple"]
print last_names
list_emp_no = []
for count in range(10):
emp_no = pick_shard_key()
print emp_no
list_emp_no.append(emp_no)
add_employee(conn, emp_no,first_names[emp_no % len(first_names)],last_names[emp_no % len(last_names)],gtid_executed)
for emp_no in list_emp_no:
print emp_no
find_employee(conn, emp_no, gtid_executed)
↧
what is going on for PHP ? (6 replies)
hello,
I got a fabric server installed, from it's config it is listening on 127.0.0.1:32274
- how is this default setup even put in the doc ? nothing can access it that way
- all examples suing php try to acces fabric doing RPC XML over http but fabric does not evel listen to 80/8080 port
and all python/java examples directly acces port 32274
can someone enlight me on this?
thanks
I got a fabric server installed, from it's config it is listening on 127.0.0.1:32274
- how is this default setup even put in the doc ? nothing can access it that way
- all examples suing php try to acces fabric doing RPC XML over http but fabric does not evel listen to 80/8080 port
and all python/java examples directly acces port 32274
can someone enlight me on this?
thanks
↧
COME ON !! please give me a real answer (2 replies)
Hello,
Mysql Fabric does XML/RPC on port 32274
PHP examples refer to XML/RPC over HTTP on port 8080
WHAT IS THE GLUE BETWEEN THESE TOO ?
Mysql Fabric does XML/RPC on port 32274
PHP examples refer to XML/RPC over HTTP on port 8080
WHAT IS THE GLUE BETWEEN THESE TOO ?
↧
MySQL Fabric HA (1 reply)
What happens if the master is "dead" but failover did not happen yet ?
In this case all current and new transactions are failed ?
In this case all current and new transactions are failed ?
↧
↧
MySQL Fabric HA (1 reply)
What happens if all slaves are down? Can we take reads from the master?
If yes, is this something that needs to be configure manually ?
If yes, is this something that needs to be configure manually ?
↧
mysqlslap (1 reply)
Hi,
We have two tables (A=38 million rows) & (B=20 millions of rows), when we run count query after join both table (join column indexed) and we provide one where clause ( also indexed ). The query examine 200 thousand rows and return count. Query took 200 milliseconds to execute.
My first question is that is this time is OK! or we need to reduce this time ?
Second question when I use same query with mysqlslap with 100 concurrent value it took four minute to complete, why it took so long? A simple count query is not blocking on Innodb engine.
The testing instance has 15 GB RAM and 2VCPU (AWS RDS).
Thank you
We have two tables (A=38 million rows) & (B=20 millions of rows), when we run count query after join both table (join column indexed) and we provide one where clause ( also indexed ). The query examine 200 thousand rows and return count. Query took 200 milliseconds to execute.
My first question is that is this time is OK! or we need to reduce this time ?
Second question when I use same query with mysqlslap with 100 concurrent value it took four minute to complete, why it took so long? A simple count query is not blocking on Innodb engine.
The testing instance has 15 GB RAM and 2VCPU (AWS RDS).
Thank you
↧
Not able to connect mysql fabric remotely (1 reply)
Hi, I have set up Mysql fabric. But not able to connect to it remotely. Below is my config.
[DEFAULT]
prefix =
sysconfdir = /etc
logdir = /var/log
[storage]
address = linux08:3306
user = fabric
password =
database = fabric
auth_plugin = mysql_native_password
connection_timeout = 6
connection_attempts = 6
connection_delay = 1
[servers]
user = fabric
password =
backup_user = fabric
backup_password =
restore_user = fabric
restore_password =
unreachable_timeout = 5
[protocol.xmlrpc]
address = 10.0.0.208:32274
threads = 5
user = admin
password =
disable_authentication = no
realm = MySQL Fabric
ssl_ca =
ssl_cert =
ssl_key =
[protocol.mysql]
address = linux08:32275
user = admin
password =
disable_authentication = no
ssl_ca =
ssl_cert =
ssl_key =
[executor]
executors = 5
[logging]
level = INFO
url = file:///var/log/fabric.log
[sharding]
mysqldump_program = /usr/bin/mysqldump
mysqlclient_program = /usr/bin/mysql
prune_limit = 10000
[statistics]
prune_time = 3600
[failure_tracking]
notifications = 300
notification_clients = 50
notification_interval = 60
failover_interval = 0
detections = 3
detection_interval = 6
detection_timeout = 1
prune_time = 3600
[connector]
ttl = 1
[DEFAULT]
prefix =
sysconfdir = /etc
logdir = /var/log
[storage]
address = linux08:3306
user = fabric
password =
database = fabric
auth_plugin = mysql_native_password
connection_timeout = 6
connection_attempts = 6
connection_delay = 1
[servers]
user = fabric
password =
backup_user = fabric
backup_password =
restore_user = fabric
restore_password =
unreachable_timeout = 5
[protocol.xmlrpc]
address = 10.0.0.208:32274
threads = 5
user = admin
password =
disable_authentication = no
realm = MySQL Fabric
ssl_ca =
ssl_cert =
ssl_key =
[protocol.mysql]
address = linux08:32275
user = admin
password =
disable_authentication = no
ssl_ca =
ssl_cert =
ssl_key =
[executor]
executors = 5
[logging]
level = INFO
url = file:///var/log/fabric.log
[sharding]
mysqldump_program = /usr/bin/mysqldump
mysqlclient_program = /usr/bin/mysql
prune_limit = 10000
[statistics]
prune_time = 3600
[failure_tracking]
notifications = 300
notification_clients = 50
notification_interval = 60
failover_interval = 0
detections = 3
detection_interval = 6
detection_timeout = 1
prune_time = 3600
[connector]
ttl = 1
↧
Not able to connect mysql fabric remotely (no replies)
I have installed Mysql Fabric, but not able to access it remotely. Please see my config below.
[DEFAULT]
prefix =
sysconfdir = /etc
logdir = /var/log
[storage]
address = linux08:3306
user = fabric
password =
database = fabric
auth_plugin = mysql_native_password
connection_timeout = 6
connection_attempts = 6
connection_delay = 1
[servers]
user = fabric
password =
backup_user = fabric
backup_password =
restore_user = fabric
restore_password =
unreachable_timeout = 5
[protocol.xmlrpc]
address = 10.0.0.208:32274
threads = 5
user = admin
password =
disable_authentication = no
realm = MySQL Fabric
ssl_ca =
ssl_cert =
ssl_key =
[protocol.mysql]
address = linux08:32275
user = admin
password =
disable_authentication = no
ssl_ca =
ssl_cert =
ssl_key =
[executor]
executors = 5
[logging]
level = INFO
url = file:///var/log/fabric.log
[sharding]
mysqldump_program = /usr/bin/mysqldump
mysqlclient_program = /usr/bin/mysql
prune_limit = 10000
[statistics]
prune_time = 3600
[failure_tracking]
notifications = 300
notification_clients = 50
notification_interval = 60
failover_interval = 0
detections = 3
detection_interval = 6
detection_timeout = 1
prune_time = 3600
[connector]
ttl = 1
[DEFAULT]
prefix =
sysconfdir = /etc
logdir = /var/log
[storage]
address = linux08:3306
user = fabric
password =
database = fabric
auth_plugin = mysql_native_password
connection_timeout = 6
connection_attempts = 6
connection_delay = 1
[servers]
user = fabric
password =
backup_user = fabric
backup_password =
restore_user = fabric
restore_password =
unreachable_timeout = 5
[protocol.xmlrpc]
address = 10.0.0.208:32274
threads = 5
user = admin
password =
disable_authentication = no
realm = MySQL Fabric
ssl_ca =
ssl_cert =
ssl_key =
[protocol.mysql]
address = linux08:32275
user = admin
password =
disable_authentication = no
ssl_ca =
ssl_cert =
ssl_key =
[executor]
executors = 5
[logging]
level = INFO
url = file:///var/log/fabric.log
[sharding]
mysqldump_program = /usr/bin/mysqldump
mysqlclient_program = /usr/bin/mysql
prune_limit = 10000
[statistics]
prune_time = 3600
[failure_tracking]
notifications = 300
notification_clients = 50
notification_interval = 60
failover_interval = 0
detections = 3
detection_interval = 6
detection_timeout = 1
prune_time = 3600
[connector]
ttl = 1
↧
↧
Mysql Fabric hash sharding not balanced in shards (1 reply)
MySQL Fabric 1.5.6
MySQL version: 5.6.31
O/S version: Centos 6.8
group_id description failure_detector master_uuid
-------- ----------- ---------------- ------------------------------------
global None 1 103383e3-542d-11e6-b7c0-066c5b481b6b
s1 None 1 605fe051-5389-11e6-b394-06fbcce39c6f
s3 None 1 b3689258-5389-11e6-b396-06a644a10425
s2 None 1 ff024a40-5386-11e6-b385-0661de27c503
mapping_id type_name table_name global_group column_name
---------- --------- -------------- ------------ -----------
1 HASH silo_db.Member global id
I'm using python mysql connector to load the data.
mysql-connector-python-2.1.3
When I load the data it don't balance over 3 shards. It's skewing one shard a lot.
Issue is when I query fabric to locate the record, it does say that it's on different shard but it's not loading on the shard. In other words, connector is not loading in the right shard as per fabric. When I ask fabric to locate the record it's not there on that shard. See below
mysqlfabric sharding lookup_servers silo_db.MemberMME 2000
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
server_uuid address status mode weight
------------------------------------ ----------- ------- ---------- ------
ff024a40-5386-11e6-b385-0661de27c503 s2-mme:3306 PRIMARY READ_WRITE 1.0
This record is not on that shard. It's on shard s1-mme. This is skewing data a lot and totally imbalanced over the shards.
Any help would be highly appreciate that why hash is not working or it's a connector issue while loading.
Best,
MySQL version: 5.6.31
O/S version: Centos 6.8
group_id description failure_detector master_uuid
-------- ----------- ---------------- ------------------------------------
global None 1 103383e3-542d-11e6-b7c0-066c5b481b6b
s1 None 1 605fe051-5389-11e6-b394-06fbcce39c6f
s3 None 1 b3689258-5389-11e6-b396-06a644a10425
s2 None 1 ff024a40-5386-11e6-b385-0661de27c503
mapping_id type_name table_name global_group column_name
---------- --------- -------------- ------------ -----------
1 HASH silo_db.Member global id
I'm using python mysql connector to load the data.
mysql-connector-python-2.1.3
When I load the data it don't balance over 3 shards. It's skewing one shard a lot.
Issue is when I query fabric to locate the record, it does say that it's on different shard but it's not loading on the shard. In other words, connector is not loading in the right shard as per fabric. When I ask fabric to locate the record it's not there on that shard. See below
mysqlfabric sharding lookup_servers silo_db.MemberMME 2000
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
server_uuid address status mode weight
------------------------------------ ----------- ------- ---------- ------
ff024a40-5386-11e6-b385-0661de27c503 s2-mme:3306 PRIMARY READ_WRITE 1.0
This record is not on that shard. It's on shard s1-mme. This is skewing data a lot and totally imbalanced over the shards.
Any help would be highly appreciate that why hash is not working or it's a connector issue while loading.
Best,
↧
Mysql fabric is not found (6 replies)
Hi,
i am new to mysql utilities. i am using Red Hat Enterprise Linux 7 and i have installed mysql-utilities-1.6.4-1.el7.noarch.rpm.
some how mysqlfabric command is not found, however i am able to find mysqlauditadmin, mysqldiff , mysqlfrm ..etc
any idea why mysqlfabric is not there?
i am new to mysql utilities. i am using Red Hat Enterprise Linux 7 and i have installed mysql-utilities-1.6.4-1.el7.noarch.rpm.
some how mysqlfabric command is not found, however i am able to find mysqlauditadmin, mysqldiff , mysqlfrm ..etc
any idea why mysqlfabric is not there?
↧
Fabric creates multiple DB connections (5 replies)
I have setup of 1 global group, 3 HA groups and 1 fabric node
The below piece of code ends up creating thousands of physical DB connections. The code errors out once all the max connection on DB is reached. According to my understanding, 1 fabric connection will contain map of single database connections to each physical DB servers.
Then why is below code trying to create thousands of physical connection on my HA1 server?
The range grouping is setup as 1-33000 to HA1, 33001-66000 to HA2 and 66001 onwards to HA3
Connection rawConnection = DriverManager.getConnection("jdbc:mysql:fabric://<fabricServer>:<port>/employees?fabricUsername=<fabricUser>&fabricPassword=<fabricPassword>&fabricShardTable=employees.employees", "<db_user>", "<db_password>");
FabricMySQLConnection fabricConnection = (FabricMySQLConnection) rawConnection;
for(int emp_id=1; emp_id<100000; emp_id++) {
fabricConnection.setShardKey(emp_id);
PreparedStatement ps = fabricConnection.prepareStatement(INSERT_EMPLOYEE_SQL);
ps.setInt(1, emp_id);
ps.setString(2, "Name"+i);
ps.setString(3, "Surname"+i);
ps.executeUpdate();
ps.close();
ps == null;
}
PS: i am using PreparedStatement inside the loop and not outside because i want to simulate real world scenareo where multipl user requests will run in parallel and they will create their own PreparedStatements
The below piece of code ends up creating thousands of physical DB connections. The code errors out once all the max connection on DB is reached. According to my understanding, 1 fabric connection will contain map of single database connections to each physical DB servers.
Then why is below code trying to create thousands of physical connection on my HA1 server?
The range grouping is setup as 1-33000 to HA1, 33001-66000 to HA2 and 66001 onwards to HA3
Connection rawConnection = DriverManager.getConnection("jdbc:mysql:fabric://<fabricServer>:<port>/employees?fabricUsername=<fabricUser>&fabricPassword=<fabricPassword>&fabricShardTable=employees.employees", "<db_user>", "<db_password>");
FabricMySQLConnection fabricConnection = (FabricMySQLConnection) rawConnection;
for(int emp_id=1; emp_id<100000; emp_id++) {
fabricConnection.setShardKey(emp_id);
PreparedStatement ps = fabricConnection.prepareStatement(INSERT_EMPLOYEE_SQL);
ps.setInt(1, emp_id);
ps.setString(2, "Name"+i);
ps.setString(3, "Surname"+i);
ps.executeUpdate();
ps.close();
ps == null;
}
PS: i am using PreparedStatement inside the loop and not outside because i want to simulate real world scenareo where multipl user requests will run in parallel and they will create their own PreparedStatements
↧