Internet Services from KKoncepts

Recently I wrote my own start/stop script for our vservers. The "util-vserver" package gives you a little control over the order that the servers are stopped and started, but I wanted to specify the exact order that virtual machines started and stopped in the event of a heartbeat takeover.

So I took the basic "Preforking Server" script from the Perl Cookbook (a truly wonderful book - I have lugged it over 4 continents and never regretted it) and adapted it to start/stop the vservers in a particular order. The script forks off 6 (or 8 or as many as you set) "vserver start/stop" processes and then waits for them to finish. As soon as one finishes, another is fired off, so that we've always got 6 going in parallel until all the virtual machines are taken care of.

In order to know when one process has stopped, you have to "reap your children". If not you get "zombies". (Who comes up with these terms??!!) That's pretty simple in perl. The example script from the cookbook had this (simplified a bit):sub REAPER {
   $SIG{CHLD} = \&REAPER;
   my $waitpid = wait;
   $children--;
}
$SIG{CHLD} = \&REAPER;
So the SIGCHLD signal is "trapped" and the "REAPER" routine will run each time the script receives that signal. Elsewhere in the script, the "$children" variable is increased each time a child is created (up to the limit set in the script) and here it is decreased every time a child exits. Once we forked off all the processes we need, then there's the linesleep while $children > 0;So we "sleep" while there are still child processes running. Once all the kids have returned, the script exits.

My problem was that it didn't work! I frequently ended up with "zombies", children who were not properly accounted for by the parent. And because they weren't accounted for, the "$children" variable never got back to 0. So my script just sat there "sleeping" forever until I manually killed it. And it wasn't consistent - sometimes I'd fork off 10 children and all would be properly "reaped", and other times 1 to 6 out of the 10 would end up as zombies.

In the end I found this page. It recommends this for reaping child processes: use POSIX ":sys_wait_h";
sub REAPER {
   my $child;
   while (($child = waitpid(-1,WNOHANG)) > 0) {
      $Kid_Status{$child} = $?;
   }
   $SIG{CHLD} = \&REAPER;
}
Here's their explanation.If a second child dies while in the signal handler caused by the first death, we won't get another signal. So must loop here else we will leave the unreaped child as a zombie. And the next time two children die we get another zombie. And so on.Sure enough, that's exactly what I was seeing. So I changed my "REAPER" code to:sub REAPER {
   my $waitpid;
   while (($waitpid = waitpid(-1,WNOHANG)) > 0){ $children-- ; }
   $SIG{CHLD} = \&REAPER;
}and that did it. No more zombies. By looping over the "waitpid" command (and testing to see if the result is greater than 0; the result will be -1 in the children themselves) the script accurately accounts for all the children. Voila - no more zombies. :-)