The deadlocks can be rightly called the “show stoppers” for the production environments where the application transactions can get errors without any details being logged in the database systems. For instance, in the lower versions of MySQL (below 5.6), capturing the deadlock information is not available with the MySQL provided log files. This blog will help users/MySQL DBAs to capture such information by making use of a simple shell script and thus providing the important details regarding the deadlock errors.
InnoDB Diagonostics:
The principal source of information for InnoDB diagnostics is the output of SHOW ENGINE INNODB STATUS but there are some sections that are not very useful. For example, LATEST DETECTED DEADLOCK only shows, as the name implies, the latest error detected. If we have 100 deadlocks per minute we will be able to see only the latest one and that is incomplete information when you need to debug your application code.
We had resolved a similar issue for one of our esteemed clients, who have a huge MySQL Instance hosted with us.
This MySQL instance serves as an Online Transaction Processing (OLTP) Instance, implying that there is a high number transactions floating in the MySQL Instance.
The application running on top of this MySQL instance needs to have all the transactions logged successfully in case any issues are observed in the application’s transactions, the same needs to be attended immediately on the MySQL DB instance side.
As the deadlocks are internally resolved by MySQL instance by rolling-back the failed transactions, it becomes critical to log such actions and report such issues to dilute their impact on the applications and minimize the deadlock from happening again.
The issue was a tricky one as We only had access to the Database and no access to the application logs.
As the MySQL instance version was an older version the deadlock was not easily identifiable. We had to resolve this issue differently and was able to do so. The steps below cover the methodology adopted to help the client.
Inventory provided by client for this Issue:
1) Application logs showing the deadlock information and the transaction failures.
2) Timing of the deadlock occurrence.
3) Timing of the failed transaction.
Diagnosis & Analysis done to find where the Issue was:
The first two steps taken towards resolving the issue were:
1) We internally checked the MySQL instance for the deadlock information.
2) We rediscovered that it was the MySQL Instance being monitored in this case was not good enough to log its deadlock information in the alert files.
Resolution:
1) We created a deadlock capture script to monitor the deadlock information from the MySQL Instance.
2) We implemented the deadlock capture script in crontab of the MySQL DB host to query the MySQL instance every 10 minutes and report any future deadlock occurrence in the MySQL Instance due to application transactions.
Script used to capture the Deadlocks:
Below is the detailed script that was implemented:
#!/bin/ksh
a=`hostname`
LOG=/var/lib/mysql/FAKEPATH/deadlocks_status.log;export LOG
mysql -Bse "SHOW ENGINE INNODB STATUS\G" -u **** –p***** | awk '/LATEST DETECTED DEADLOCK/{f=1} /WE ROLL BACK TRANSACTION /{f=0;print} f' > $LOG
#------------Checking Log file for any errors----
valdump=`cat /var/lib/mysql/Fakepath/deadlocks_status.log |wc -l`
if [ $valdump = 0 ]
then
echo "No Issues"
else
errors=$(cat /var/lib/mysql/FAKEPATH/deadlocks_status.log)
echo "$errors" > /var/lib/mysql/FAKEPATH/current-deadlocks_status.log
if [ -e "/var/lib/mysql/FAKEPATH/prior-deadlocks_status.log" ]
then echo "prior-deadlocks_status.log Exists" > /dev/null
else
touch /var/lib/mysql/FAKEPATH/prior-deadlocks_status.log | echo "" > /var/lib/mysql/FAKEPATH/prior-deadlocks_status.log
fi
newentries=$(diff --suppress-common-lines -u /var/lib/mysql/FAKEPATH/prior-deadlocks_status.log /var/lib/mysql/FAKEPATH/current-deadlocks_status.log | grep '\+[0-9]')
if
test "$newentries" != "" && test "$errors" = ""
then echo "No New Errors" > /dev/null
elif
test "$newentries" != ""
then echo "$errors" | mailx -s "TRI-ALT:`hostname`:MySQL Instance Has New Deadlock Entries"
client.mysql.mail@FAKEemail.com
echo "$errors" > /var/lib/mysql/FAKEPATH/prior-deadlocks_status.log
fi
fi
Crontab Entry Used:
00,10,20,30,40,50 * * * * /var/lib/mysql/FAKEPATH/check_deadlocks.ksh > /tmp/check_deadlocks.log 2>&1
Conclusion: Where the lower versions of MySQL instances do not have the capabilities to report or print all the deadlock information, the operating systems utilities can be efficiently utilized to capture them at regular intervals. Hence, providing a robust monitoring system to the users.
Note: - All above deadlock capture efforts were done in MySQL Instance version 5.0.45