QMT Failover replication Setup
QMT Failover replication Setup
Craig Smith - 26th October 2006 - craig@doc-net.com
Thanks to Jake for taking the time to review this for me before posting. It always helps to have a sounding board and Jake was kind enough to be that board for me.
This page gives you a procedure to configure a backup qmt server that will be available for failover in the event of primary server failure. The backup server will only ever be 10 minute out from the primary.(depending on cronjob timing)
Please note initial replication (the first run) will take some time, so schedule this for off peak hours. Once the first run has finished and unison has a db of what it is working with subsequent runs are pretty quick. So enable the cron job settings at a time that you can manage the traffic for initial replication.
Also this setup is based on 2 servers where the port used is internal and not visible publically. If you cannot do this on a private network, then read up on using ssh for replication as this is not a secure transport and should not be used on open networks.
This was setup and tested on Fedora core 5 on both servers, and it works without any hiccups.
The details are pretty much cut and paste.
I would test this installation on a similar setup first as I only have my system to compare with, but given the ease of installation and use of unison, I don't forsee any major problems.
This setup assumes that QMT is installed and configured on both servers. The backup servers version is to match the primary's.
I have also setup a script that will test your server availability by pinging it. This is posted at the bottom of this page, but still needs work. The mail commands are purely for testing. If the primary server really were off, the emails wouldn't go out until you switch to the backup which doesn't help. I think replacing these mail commands to sms text messages would certainly help. A quick and easy monitoring system, which can be added to cron to run every x minutes.
This is a first draft and also my fist proper linux/unix scripts so please forgive any errors and let me know if you have any problems. If the primary server fails and the switch is made to backup in a short time, clients will probably not even notice the downtime.
- Unison is a great replication utility. http://www.cis.upenn.edu/~bcpierce/unison/
- We use the command line version available from http://www.cis.upenn.edu/~bcpierce/unison/
- This is based on unison-2.13.16-linux-text.bz2 which is the current stable release as of 10/10/2006
- To modify files/folders either replicated or ignored edit the unison profile (qmail.prf).
- This setup is for servers that are in a lan environment with the unison port not visible to the public world. Therefore the security overhead of ssh is not needed. However unison can be run with ssh if needed.
- To configure unison and ssh see http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html#remote. The procedure stays largely the same, but the socket server is not needed.
Comments and Notes
Please feel free to leave comments about your experience with this procedure.
Craig Smith - 26th October 2006 On our setup of 2 x fedora core 5 boxes, whenever I make a change on primary I check on backup to make sure it has replicated, and so far everytime it works a charm. I've not run into anything strange.
Also I'm not sure how people prefer the logs. I had it initially set to log each run, but that was changed to keep logging for 20MB which is roughly 24 hours worth of logging. Times are logged so finding a specific run should be fairly easy. However it's quite easy for me to change back to log each run in a seperate file. I will go with preferance really, please let me know.
- * Note about non FC versions.
One of the variables in the script obtains file size by cutting the relevant field from an ls listing. This field it turns out is not the same as on Fedora, so if your script runs into errors, the problem lies here more than likely. To fix it run this (ls -l $LOG|cut -d ' ' -f 5)portion in the command line and change the f no. until it correctly displays the size field and then change the number in the script and this should fix any script errors.
It is the following variable.
size=`ls -l $LOG|cut -d ' ' -f 5`
I believe for Centos it is f6 so that variable would be changed to
size=`ls -l $LOG|cut -d ' ' -f 6`
Primary Server (Client) setup
- Primary server is the main server being replicated. This has the unison program and the client side script to replicate data based on the profile.
- All scripts assume a default path of /unison.
- The script calls unison to run as follows
/unison/unison -force / -batch qmail
- -force / is very important, as it specifies this root as the primary and will default all conflicts to this root. NB IF THIS IS NOT INCLUDED ANY CHANGES MADE ON THE BACKUP SERVER WILL REPLICATE TO PRIMARY.
- -batch tells unison to run without promts and qmail refers to the qmail.prf file. (see below)
- To configure unison on the client side take the following steps as root/su
mkdir /unison cd /unison wget https://svn.cis.upenn.edu/svnroot/unison-contributed-binaries/linux/unison-2.13.16-linux-text.bz2 bzip2 -d unison-2.13.16-linux-text.bz2 mv unison-2.13.16-linux-text unison chmod 755 unison ./unison *This will create the initial /root/.unison database folder vi qmail-replicatec *Once in the editor, copy and paste the qmail-replicatec script from below :wq vi /root/.unison/qmail.prf *Once in the editor, copy and paste the qmail.prf details from below :wq
Client script gets added to cron to run every 10 minutes.
crontab -e */10 * * * * /unison/qmail-replicatec
*Below is the full script for the client side, script name qmail-replicatec
#!/bin/sh #Version 1.0 - Oct 11th 2006 #Script created by Craig Smith - craig@doc-net.com #This script is the client (Primary mail server) side script to replicate based on the #qmail.prf file located under /root/.unison. The Mysql variables and dump were taken from #qmailbackup script written initially by Nate Davis. #To add to or change files paths edit /root/.unison/qmail.prf accordingly. #This script assumes unison and scripts are placed in /unison #To keep logs from building too excessively as this script will run fairly regularly #they will be moved to the /unison/oldlogs folder after 10. The oldlogs folder will #be emptied after each group of 10. You can increase this if needed. See comment below. #With default settings the last 20 runs are available. Add email address below for notifications of log rotation. #starting with mysql so the dumped file can replicate #Checking for mysql.dump and oldlogs folder folder1=/unison/mysql.dump folder2=/unison/oldlogs lock=/unison/replicate.lock email=(email address)
if [ ! -d $folder1 ]; then mkdir -p $folder1 fi if [ ! -d $folder2 ];then mkdir -p $folder2 fi if [ ! -f $lock ]; then touch $lock elif [ -f $lock ]; then echo "Lock file still in place, investigate."|mail -s "unison script problems please investiagate server" $email exit 0 fi
# MYSQL variables
mysqlfile="/home/vpopmail/etc/vpopmail.mysql";
mysqlhost=`cut -d\| -f1 < $mysqlfile`;
mysqlport=`cut -d\| -f2 < $mysqlfile`;
mysqluser=`cut -d\| -f3 < $mysqlfile`;
mysqlpswd=`cut -d\| -f4 < $mysqlfile`;
mysqldb=`cut -d\| -f5 < $mysqlfile`;
if [ $mysqlport == "0" ]; then
mysqlport=""
else
mysqlport="-P$mysqlport"
fi
#echo "Backing up MYSQL Data"
mysqldump -u$mysqluser -p$mysqlpswd -h$mysqlhost $mysqlport $mysqldb > /unison/mysql.dump/vpopmail
#script for log clearup
#Only last 10 log files are kept in /unison/oldlogs.
#The full log will be replaced every 20MB and moved to /unison/oldlogs.
LOG=/unison/unisonlog.full
LOGCOUNT=`ls $folder2 |wc -l`
size=`ls -l $LOG|cut -d ' ' -f 5`
LOGSAVE=/unison/oldlogs/unisonlog.`date +%Y%m%d%H%M`
#echo $size
if [ $size -gt 20000000 ];then
echo "this is bigger than 20MB, moving">>$LOG
mv $LOG $LOGSAVE
echo "" >$LOG
fi
#LOG COMMENT : if you want to increase the saved logs change the 10 below.
if [ $LOGCOUNT -gt 15 ];
then
echo "more than 15 logs exist, moving to folder /unison/oldlogs/previous"
#keep previous batch in previous
rm -f /unison/oldlogs/previous/*
mv /unison/oldlogs/* /unison/oldlogs/previous
clear
echo "`date` : log files have been moved to oldlogs" >>$LOG
echo "Unison Log files moved, please check folders on server"|mail -s"Unison Logile Rotation : `date`" $email
fi
echo "" >>$LOG
echo "`date` ***STARTING REPLICATION RUN**** " >>$LOG
#Begin unison replication using /root/.unison/qmail.prf file for folder locations
#Log file location and format
/unison/unison -force / -batch qmail >> $LOG 2>&1
echo "Deleting lock file" >> $LOG 2>&1
rm -f $lock
echo "Done `date`" >> $LOG 2>&1
;*Below is the contents of the qmail.prf file. This file points to the backup server ip and port and specifies the folders to be replicated. It will only replicate what is specified by Path. If you don't include path, then it will replicate the whole root. If you want to add details e.g. squirrellmail prefs etc, then add path = path to folder. note no trailing / add the correct ip address and port no when you paste this into the qmail.prf file. eg. root = socket://10.0.0.1:1234//
#Root and path setup for qmail backup root = / root = socket://xxx.xxx.xxx.xxx:port no.// path = home/vpopmail path = var/qmail/control path = var/qmail/users path = etc/mail/spamassassin path = etc/tcprules.d path = unison/mysql.dump ignore = Name *.lock
Backup Server (Server) Setup
- The backup server runs the unison program with the -sockets option. This creates a listening socket that the client will try and connect to. There is also a script that will deal with qmail cleanup and configuration as unison does not replicate ownerships.
*To configure Unison on the Server side take the following steps. The same unison application as above is used.
mkdir /unison cd /unison wget https://svn.cis.upenn.edu/svnroot/unison-contributed-binaries/linux/unison-2.13.16-linux-text.bz2 bzip2 -d unison-2.13.16-linux-text.bz2 mv unison-2.13.16-linux-text unison chmod 755 unison ./unison *This will create the initial /root/.unison database folder vi qmail-replicateb *Once in the editor, copy and paste the script from below :wq chmod 755 qmail-replicateb vi unison-run *Once in the editor, copy and paste the unison-run script from below :wq chmod 755 unison-run vi qmail-switch *Once in the editor, copy and paste the qmail-switch script from below. :wq chmod 755 qmail-switch cp unison-run /etc/init.d ln -s /etc/init.d/unison-run /etc/rc3.d/S50unison-run *This will configure unison to start on boot up.
- To manually start or stop the socket do the following
/unison/unison-run start
or
/unison/unison-run stop
- The server side script uses a qmail queue repair tool. This needs to be configured before running the script
cd /unison mkdir queue-repair wget http://pyropus.ca/software/queue-repair/queue-repair-0.9.0.tar.gz tar -xzf queue-repair-0.9.0.tar.gz mv queue-repair-0.9.0 queue-repair
- Run the client script if you need to switch to the backup server
;Below is the script for the server called qmail-replicateb
#!/bin/sh
#Version 1.0 - Oct 11th 2006
#Script created by Craig Smith - craig@doc-net.com
#This script is the server (backup mail server) side script to monitor
#changes and import the changes and fix qmail accordingly.
#It also sets the correct ownership on the vpopmail folders as unison doesn't
#replicate ownership.
#This script assumes unison and scripts are placed in /unison
#The mysql commands,qmail-newu and queue repair were taken from Jake Vickers' qmail
#restore scripts.
#Put this script in cron job to finalize changes based on the main server
#Check the qmail-replicate.log file for each run to see if changes were made.
#Log file will be adjusted to similarly match client side script.
#Currently qmail is left off to prevent unecessary direct mailings to the
#backup server. This can be uncommented to leave it running.
#Please add your mysql root password below
mysqlrootpass="mysql password"
#This portion compares the vpopmail files between replication for changes and
#acts accordingly. If you want this to run with every cron job, comment out
#the next 4 script lines as well as the last 3 script lines.
#Checking for mysql.dump and oldlogs folder
folder1=/unison/mysql.dump
folder2=/unison/oldlogs
if [ ! -d $folder1 ]; then
mkdir -p $folder1
fi
if [ ! -d $folder2 ]; then
mkdir -p $folder2
fi
#script for log clearup
#Only last 10 run log files are kept in /unison. The rest are moved to
#/unison/oldlogs and cleared out each time 10 are exceeded to prevent excessive
#buildup.
#Log file location and format
LOG=/unison/unisonlog.`date +%Y%m%d%H%M` 2>&1
LOGCOUNT=`ls /unison/unisonlog.* |wc -l`
#mysql file to be imported
FILE=/unison/mysql.dump/vpopmail
#LOG COMMENT : if you want to increase the saved logs change the 10 below.
if [ $LOGCOUNT -gt 10 ]
then
# echo "more than 10 logs exist, moving to folder /unison/oldlogs"
rm -f /unison/oldlogs/*
mv /unison/unisonlog.* /unison/oldlogs
clear
echo "`date` : log files have been moved to oldlogs" >$LOG
elif [ $LOGCOUNT -le 10 ]
then
echo "`date` : Log count is fine, nothing to do" >>$LOG
fi
mysqladmin -f -uroot -p$mysqlrootpass drop vpopmail >>$LOG
mysqladmin -uroot -p$mysqlrootpass flush-tables >>$LOG
mysqladmin -uroot -p$mysqlrootpass refresh >>$LOG
mysqladmin -uroot -p$mysqlrootpass reload >>$LOG
mysqladmin create vpopmail -uroot -p$mysqlrootpass >>$LOG
mysql -uroot -p$mysqlrootpass vpopmail < $FILE
#Set the file permission for vpopmail.vchkpw on vpopmail folder
cd /home
chown -R vpopmail:vchkpw vpopmail
#cd /home/vpopmail/
#chown -R vpopmail:vchkpw *
#Run queue repair
echo "Running qmail-newu for any other loose ends....">>$LOG
/var/qmail/bin/qmail-newu >>$LOG
#Reload mysql data and restart httpd
echo "Reloading (and refreshing) MySQL and Apache...." >>$LOG
mysqladmin -uroot -p$mysqlrootpass reload >>$LOG
mysqladmin -uroot -p$mysqlrootpass refresh >>$LOG
/sbin/service httpd reload >>$LOG
#Rebuild qmail cdb's to accomadate any replicated changes
echo "Rebuilding QMail's CDB and starting the mail server...." >>$LOG
echo "If qmail is not running the svc errors are normal and can be ignored." >>$LOG
qmailctl stop >>$LOG 2>&1
sleep 3
qmailctl cdb >>$LOG
#Leaving qmail stopped to prevent unecessary direct mailing etc.
#qmailctl start
echo "`date` The database has been imported into MYSQL. Verify user details." >>$LOG
;Below is the script for the unison control file called unison-run
!/bin/sh
#Script created by Craig Smith - craig@doc-net.com
#Script for Unison Start up shutdown
#Set the socket number in the /unison/unison -socket # line.
case "$1" in
start)
# have a silent kill in case someone tries to start the service when it
# is already running
/unison/unison-run stop >/dev/null 2>&1
echo -n "Starting Unison Socket Server"
/unison/unison -socket xxxx &
echo $! > /var/run/unison.pid
echo "."
;;
stop)
echo -n "Stopping Unison Socket Server"
kill `cat /var/run/unison.pid`
rm -rf /var/run/unison.pid
;;
help)
HELP
;;
*)
echo "Usage: unison-run{start|stop}"
exit 1
;;
esac
Below is the script to use, when you need to switch to your backup server. Change the backup server's ip to match the primary and then run this. By changing the ip you don't have to worry about dns updates by changing the host record. This script will do a final clearup and then start qmail. /unison/qmail-switch to run this file.
Below is the qmail-switch script
#!/bin/sh #Version 1.0 - Oct 11th 2006 #Script created by Craig Smith - craig@doc-net.com #Please add your mysql root password below mysqlrootpass="mysql rootpass" LOG=/unison/qmailswitch.log FILE=/unison/mysql.dump/vpopmail mysqladmin -f -uroot -p$mysqlrootpass drop vpopmail >>$LOG mysqladmin -uroot -p$mysqlrootpass flush-tables >>$LOG mysqladmin -uroot -p$mysqlrootpass refresh >>$LOG mysqladmin -uroot -p$mysqlrootpass reload >>$LOG mysqladmin create vpopmail -uroot -p$mysqlrootpass >>$LOG mysql -uroot -p$mysqlrootpass vpopmail < $FILE #Set the file permission for vpopmail.vchkpw on vpopmail folder cd /home chown -R vpopmail:vchkpw vpopmail #Run queue repair echo "Running Charles Cazabon's queue_repair python utility to fix any loose ends...." >>$LOG sleep 3 cd /unison/queue-repair chmod 777 queue_repair.py ./queue_repair.py -r >>$LOG echo echo "Running qmail-newu for any other loose ends....">>$LOG /var/qmail/bin/qmail-newu >>$LOG #Reload mysql data and restart httpd echo "Reloading (and refreshing) MySQL and Apache...." >>$LOG mysqladmin -uroot -p$mysqlrootpass reload >>$LOG mysqladmin -uroot -p$mysqlrootpass refresh >>$LOG /sbin/service httpd reload >>$LOG #Rebuild qmail cdb's to accomadate any replicated changes echo "Rebuilding QMail's CDB and starting the mail server...." >>$LOG echo "If qmail is not running the svc errors are normal and can be ignored." >>$LOG qmailctl stop >>$LOG 2>&1 sleep 3 qmailctl cdb >>$LOG qmailctl start echo "`date` The database has been imported into MYSQL. Verify user details." >>$LOG echo "QMAIL has been fixed and started. It should now be live and workings as primary mail. please verify." >>$LOG
Base State Check script
This script will ping your host for uptime, and will create a fail based on failure. To use, vi filename e.g. statecheck and paste the contents from below. Then chmod 755 filename. At this time it's not really that helpful without a method of notifying you. if you have an email address that you monitor regularly that is not on your primary server, then uncomment all email options below and you will be notified on the various levels of failure.
#!/bin/sh #Created by Craig Smith 16/10/2006 #Script file to check for up time on hosts. Will ping at x min intervals, and email after each consequtive failure. #To increase failure numbers, add more variables i.e. fouth, fifth etc. Add to cron to specify check rate. #There are probably other ways of doing this, but for now, this is what came about. HOST=host to be pinged #EMAIL=mail for testing FOLDER=/unison/uptime first=/unison/uptime/fail1 second=/unison/uptime/fail2 third=/unison/uptime/fail3 fine=/unison/uptime/fine if [ ! -d $FOLDER ];then mkdir /unison/uptime fi #If increasing error count, extend these counts. if [ -f $first ];then file=$second elif [ -f $second ];then file=$third elif [ ! -f $first ];then file=$first fi rm -f $FOLDER/* ping -c 4 $HOST >/dev/null if [ $? -eq 0 ]; then echo "$HOST is up and visible" > $fine elif [ $? -eq 1 ];then echo "failure `date`" >$file fi #used for testing. Should be replaced with sms based commands. #if [ -f $first ];then #echo "$HOST has failed to respond for the 1st time on `date`"| mail - s "Please perform first time $HOST state check" $EMAIL #elif [ -f $second ];then #echo "$HOST has failed to respond for the 2nd time on `date`"| mail -s "Please perform second time $HOST state check!!" $EMAIL #elif [ -f $third ];then # echo "$HOST has failed to respond for the 3rd time on `date`"| mail -s "$HOST state!!! OFFLINE 3RD FAILURE" $EMAIL #elif [ -f $fine ]; then #echo "$HOST is alive and well"| mail -s "No problems were detected `date`." $EMAIL; #fi