Heartbeat is already installed, so cd into the "/dev/ha.d" directory to do the configuration. We'll be creating an older "R1" style Heartbeat cluster. The new "R2" style clusters allow for more than 2 nodes and allow for resource level monitoring. Since we'll be doing neither of those, R1 style is a lot easier to use. (See here for more information about the differences.) First we'll create a shared secret file that our two nodes will use to make sure they are talking to each other. Run the following commands at the terminal on one of your nodes:echo "auth 1" >/etc/ha.d/authkeys echo -n "1 sha1 " >> /etc/ha.d/authkeys dd if=/dev/urandom bs=512 count=1 | openssl md5 >> /etc/ha.d/authkeys chmod 600 /etc/ha.d/authkeys Then we need to create the Heartbeat configuration file, "ha.cf". Here's the contents: node node1 node2 autojoin none auto_failback off crm off logfacility daemon bcast eth1 eth2 udpport 695 serial /dev/ttyS0 baud 19200 keepalive 2 warntime 6 deadtime 12 ping_group internal 10.10.1.1 10.10.1.5 10.10.1.7 ping_group external 1.2.3.4 1.2.3.5 1.2.3.6 deadping 12 respawn hacluster /usr/lib/heartbeat/ipfail apiauth dopd gid=haclient uid=hacluster respawn hacluster /usr/lib/heartbeat/dopd The "definitive" guide to Heartbeat configuration options is here. Consult that if you have any questions. But here's an explanation of our values. node node1 node2 autojoin noneWe explicitly define the nodes in this cluster. Only these two nodes are part of this cluster. You can also set "autojoin" to "other/any" and let nodes join automatically. But since we run multiple two-node clusters on the same internal network, we don't want any chance of the wrong nodes joining together. (It recommended that you put the internal IP addresses of these two nodes into your "/etc/hosts" file so that a DNS failure won't cause any headaches.) auto_failback offIn another configuration file, "haresources", we'll specify which node should be Primary when both nodes first start up. If "auto_failback" is set to "on", then whenever both nodes are up, Heartbeat will make the node specified in the "haresources" file to be the Primary. We don't want that because both nodes are identical. We don't care which is Primary. (And we don't want our vservers switching over more than necessary.) crm offThis creates an older "R1" style Heartbeat cluster. logfacility daemonLog to syslog with the "daemon" facility. The other option is the "logfile" directive which specifies a file to log to. The Heartbeat docs say that using a separate log file can block under high loads, so syslog is "safer" from that perspective. bcast eth1 eth2 udpport 695This is one of the ways Heartbeat on the two nodes will communicate with each other. Here we will "broadcast" (bcast) from each node to the other on eth1 and eth2. On our systems eth1 is our internal network, and eth2 is the cross-over cable between these two nodes. The broadcast will be UDP on port 695. If you don't specify the "updport" option then Heartbeat uses UDP port 694. Since we have multiple clusters on the same internal network, we explicitly state a separate port for each cluster so there's no "collisions". serial /dev/ttyS0 baud 19200Here's another communication channel. We have a null modem cable plugged into a serial port (/dev/ttyS0) on both nodes to allow them to communicate in case both NICs go down. It's not necessary, but it can prevent "split-brain" in case of network failure. If both NICs were to go down, then each side of the cluster would think it was up and the other was down and each side would try to become "Primary". Then when the network came back up, they would fight over who was really Primary. We recommend you have a serial connection between the two nodes as well as network connections. keepalive 2This is how often Heartbeat on each side checks the connection to the other via the "channels" we set up above. Every 2 seconds we check over the three channels - eth1, eth2, and /dev/ttyS0 warntime 6 deadtime 12This is how long Heartbeat waits until it decides that the other node is dead based on its checks listed above. After 6 seconds of no replies, Heartbeat will log a warning. After 12 seconds of no replies it will assume the other node is dead and make this node Primary if it isn't. ping_group internal 10.10.1.1 10.10.1.5 10.10.1.7 ping_group external 1.2.3.4 1.2.3.5 1.2.3.6These are directives that we use to make sure this node can see both the internal and the external networks. If for some reason this node doesn't die, but does lose connection to the internet, or to the shared internal services like DNS, we want the other node to take over. (Assuming it still has a good connection to those networks) So we define two "ping_group" options, one called "internal", which has the IP addresses of several internal machines on our network, and another called "external" which has the IP addresses of several machine on our external network. (In reality, the "external" group has the IP address of our ISP's gateway to the internet and DNS servers.) If any of the IP addresses in a group can be reached via ping, then this group is "up". Only if none of the addresses can be reached that it will "fail". You can also use a "ping" directive and a single IP address if you want to check only one address. deadping 12This is how many seconds a ping_group has to fail before we consider this node as "failed". So here after 12 seconds, if this node can't see either the internal or the external group and the other node can, the other node will become Primary. respawn hacluster /usr/lib/heartbeat/ipfail"ipfail" is the external program that the "ping_group" directive uses to actually ping the listed IP addresses. So this directive tells Heartbeat to start that program and keep it up (running it as user "hacluster"). apiauth dopd gid=haclient uid=hacluster respawn hacluster /usr/lib/heartbeat/dopdThese are the Heartbeat equivalents to the "drbd-peer-outdater" line in our "drbd.conf" file. Heartbeat will use the "dopd" program to mark DRBD data as "Outdated" if the connection goes down. So we need Heartbeat to start (and keep running) the "dopd" program. We also have to specify the user and group for these programs to run as. Since the dopd program needs root access to make this change, we also need to make a "chmod" to some of the DRBD programs themselves. So run the following commands to set that up. chgrp haclient /sbin/drbdmeta chgrp haclient /sbin/drbdsetup chmod 4754 /sbin/drbdmeta chmod 4754 /sbin/drbdsetup This will allow the Heartbeat group (and only the Hearbeat group) to run the appropriate commands to mark DRBD data as "Outdated" if necessary. Now we'll create the "haresources" file which actually tells Heartbeat what to do when it changes a node between Secondary and Primary. Here's the contents: node1 drbddisk::vs Filesystem::/dev/drbd0::/home/drbd0::ext3::defaults,tag Filesystem::/home/drbd0/vservers/root::/var/lib/vservers::ext3::defaults,rbind,tag Filesystem::/home/drbd0/vservers/conf::/etc/vservers::ext3::defaults,rbind util-vserver This is all one line which specifies what Heartbeat should do when promoting or demoting nodes. The first item, "node1" is the node that should be primary when Heartbeat first starts on this cluster. After that, all the following items are init scripts which will be run in order, left-to-right when promoting a node to Primary, and right-to-left when demoting a node to Secondary. The scripts need to be located in "/etc/init.d" or "/etc/ha.d/resource.d". Each script is separated from the others by spaces. When promoting a node to Primary, the scripts will be passed "start" as the first parameter; when demoting a node to secondary, each script will be passed "stop". Two colons "::" are used to pass additional parameters to the script. So "drbddisk::vs" means that Heartbeat will look for a script called "drbddisk" in "/etc/init.d" and "/etc/ha.d/resource.d". Once it finds the script, it will pass it "start" or "stop" and then the parameter "vs". The drbddisk script (which is part of the Heartbeat package) will then promote or demote the DRBD named resource depending on whether it was passed "start" or "stop". The "Filesystem" script mounts and unmounts resources. You pass it the mount device, the mount point, the filesystem type, and the mount options. So we are mounting the DRBD device "/dev/drbd0" to the directory "/home/drbd0". Then we remount the appropriate directories from the DRBD device to the correct mount points using the "rbind" option to mount. (This is the same thing that we did manually when we set up the DRBD resource. Now we are having Heartbeat automate it.) The last item, "util-vserver" is the standard vserver start up script. For Heartbeat to stop and start your vservers in this fashion, you need to configure them to stop and start automatically when util-vserver is run. In Debian Lenny, that's already configured, you just need to echo "default" into the "/etc/vservers/<vserver name>/app/init/mark" file of any vserver you want to start automatically. Then any vservers with the word "default" in the "mark" file will be started when the "util-vserver" script is run, and ALL running vservers will be stopped when the "util-vserver" script is stopped. Since Heartbeat now handles this script, we should remove it from the startup scripts. The right way (see first comment) to do this is to runupdate-rc.d util-vserver disable so the startup scripts are disabled and they will stay disabled even after updates. Unfortunately, Lenny doesn't have a version of 'update-rc.d' which includes the 'disable' option. So we'll have to do it the hard way by running the following line.rename 's/S20util-vserver/K80util-vserver/' /etc/rc*/*That will do what "update-rc.d disable" would have done and disable the "util-vserver" start-up scripts. Now Heartbeat will run the script when needed, and your vservers will start and stop properly when a node becomes Primary or Secondary. The last step is to copy the three files we created, "authkeys", "ha.cf", and "haresources" to the "/etc/ha.d" directory on the other node. These three files should be identical on both nodes. (Make sure you set "authkeys" permissions to "600" or there will be an error.) Now restart Heartbeat on both nodes ("/etc/init.d/heartbeat restart" on Lenny) and everything should run on node1. Once you see everything running properly on node1, then on node1 run "/usr/lib/heartbeat/hb_standby" (or on node2 run "/usr/lib/heartbeat/hb_takeover") to move all the vservers over to node2. Heartbeat should stop all the vservers on node1, then umount everything, then demote the DRBD resource to "Secondary" on node1. Then it will promote it to "Primary" on node2, mount everything, and start the vservers. That's it; now you are running an HA cluster and your vservers should be up all the time. :-)