Devlog

Today I want to talk about daemonizing PHP scripts on Linux servers -- creating scripts that are meant to run in the background for long periods of time. After we've covered that, I'll explain how I like to use supervisord to manage these long-running PHP processes.

Creating long-running scripts

A daemon is nothing but a script that goes on forever. When you get right down to it, a daemon is simply an infinite loop.

class MyDaemon
{
	public function run()
	{
		while (1) {
			// Like the energizer bunny
		}
	}
}

$daemon = new MyDaemon();
$daemon->run();

The work you do within that infinite loop is the whole purpose of your daemon. Maybe it's listening for a network connection, or maybe it's waiting for an item to enter a queue. For example, how about a hypothetical website that needs to process new users using a database queue, you might end up with something like this:

(Note: using a database for a queue isn't very efficient, hence the use of sleep() here to prevent hundreds of count queries per second being run in the loop. In practice you'd want to use a real queue, like beanstalkd.)

class MyDaemon
{
	public function run()
	{
		while (1) {
			if ($this->isNewUsers()) {
				foreach ($this->getNewUsers() as $user) {
					$this->setupNewUser($user);
				}
			} else {
				// If there's no work to do, let's just
				// wait 10 seconds before hitting the DB to
				// count again
				sleep(10);
			}
		}
	}

	// ...
	// Imagine we defined isNewUsers, getNewUsers, setupNewUser
}

Note: In this post I'm creating simple daemons that generally perform one task at a time. "True" daemons, like a web server for example, fork off children so multiple tasks can be run in parallel.

Shutting down safely

So we have a long-running script now. But you might have noticed there's no way to actually stop it. The only way we could ever stop this script from running is to issue a kill command to kill the process. But this is dangerous because what if the script was in the middle of processing something? Using our example above, what if we were only half-finished processing a user?

You can sometimes mitigate problems like this. For example, if you use database transactions to ensure an all-or-nothing commit. But what if you need to interact with systems that don't offer such transactional safety?

So the only way a PHP daemon is really worth it is if we can shut it down safely. That means we need some way to tell the script it's time to go; a way to tell it to break the infinite loop. Something like this:

class MyDaemon
{
	public function run()
	{
		while (1) {
			// Check to see if we should break out
			if ($this->shouldStop()) break;

			if ($this->isNewUsers()) {
				foreach ($this->getNewUsers() as $user) {

					// Check inside the loop too, to make sure we
					// dont continue processing a big batch of users
					if ($this->shouldStop()) break;

					$this->setupNewUser($user);
				}
			} else {
				sleep(10);
			}
		}
	}

	/**
	 * Returns true when the main loop should finally stop
	 */
	public function shouldStop()
	{
		return false;
	}
}

But what can we put inside that shouldStop() method?

Checking an external value

Perhaps the simplest thing to do is to check to see if there is some external value. Maybe a database record that says "should_stop=1" or the presence of a file called "should_stop.txt".

	/**
	 * Returns true when the main loop should finally stop
	 */
	public function shouldStop()
	{
		clearstatcache();
		if (file_exists('should_stop.txt')) {
			unlink('should_stop.txt'); // delete the file for next time
			return true;
		}

		return false;
	}

But these methods aren't very efficient, and they build up a dependance on some other system.

Handling POSIX signals

The best way for us to handle clean shutdowns is to handle standard POSIX signals like TERM and INT. When you're using a shell and press CTRL+C, Linux sends the application an INT signal. When you the kill command on a process, it will send the TERM signal by default. So intercepting these two signals is perfect for us. If we can intercept them, then we can make sure we shut down gracefully.

Enabling Process Control in PHP

PHP has process control features (PCNTL) that we need to interpret POSIX signals. But they are not enabled by default. You must compile PHP with the --enable-pcntl configuration option.

Listening for signals

There are two steps we need to do:

1. Declare ticks. Ticks are a little known feature of PHP that allows you to run a callback function every X statements. This can be useful for memory profiling or debugging, but it's not very useful otherwise. But we aren't actually going to use ticks ourselves -- the PHP PCNTL features just uses it internally.
2. Assign callbacks to listen for signals using pcntl_signal().

Here's what we end up with:

declare(ticks = 1);

function sig_handler($signo)
{
	switch ($signo) {
		case SIGTERM:
		case SIGINT:
			global $_MYDAEMON_SHOULD_STOP;
			$_MYDAEMON_SHOULD_STOP = true;
			break;
	}
}
pcntl_signal(SIGTERM, 'sig_handler');
pcntl_signal(SIGINT, 'sig_handler');

class MyDaemon
{
	// ...

	public function shouldStop()
	{
		global $_MYDAEMON_SHOULD_STOP;
		if (isset($_MYDAEMON_SHOULD_STOP) AND $_MYDAEMON_SHOULD_STOP) {
			return true;
		}

		return false;
	}
}

So in this example we've created a callback function that listens for TERM and INT signals, and assigns a value of 'true' to a variable. Then within the actual daemon class code, we check this variable each time to determine when it's time to call it quits.

Note that it's important to use a basic function callback with pcntl_signal and not an object method. There is some weirdness when using using $this in callbacks. That's why in the example above I've opted to use a simple function and a global variable.

Problems with long-running PHP scripts

The biggest problem with long-running PHP scripts is memory usage. PHP isn't exactly designed for long-running programs. There are memory leaks that don't usually matter in websites because requests start and finish so quickly, but matter a lot in long-running processes. These leaks are usually to do with circular object references and PHP's garbage collection. But with recent versions of PHP it's not so much of a problem anymore.

Even so, you have to be careful about memory usage in your actual scripts as well. Not many libraries are designed for long-running processes. So maybe your emailing library saves some information in an array after you send each email. In a web script, it doesn't matter -- the array is lost after the request finishes. But now you might have a process that runs for hours on end. This little array of log messages can become huge!

It's problems like that that you must be weary of. You need to consciously think about things like that while designing your PHP daemons. What I would recommend is to plan for your PHP daemons, if possible, to shut down and restart periodically. I'll explain the best way to do this in the next section.

Handling processes (startup, monitoring, detaching etc)

So we've got our long-running PHP scripts now, but we still have a few more things to tackle. First is how do we detach a script so it runs in the background? That is, if you run one of these scripts from your shell for example, you'll just get an empty screen that does nothing until you quit the process. You want to run the script so it detaches and runs in the background -- aka, you get your command prompt back. You can do this in straight PHP -- it involves forking so the parent process can die, but the children live on.

But there are other things to consider. How do we handle output? How do we handle fatal errors and exceptions (like a database disconnection) -- how do we get the process to restart? How do we make sure the correct user owns the process?

The easiest way to do all of this is to use a process manager like supervisord. This software runs on your server and is responsible for starting your long-running PHP scripts in the background (so you don't have to worry about detaching yourself), and also sports features like monitoring so you can automatically restart processes that've quit unexpectedly or when they are using too much memory.

With supervisord, you add some configuration for each script you want to run. Here's an example that runs a script, logs output to a file, and will TERM the script when it uses over 50MB of memory (where it'll be restarted again).

[program:mydaemon]
command=/usr/bin/php /home/john/mydaemon.php
directory=/home/john
startsecs=5
exitcodes=2
user=john
stdout_logfile=/home/john/logs/mydaemon.out
stderr_logfile=/home/john/logs/mydaemon.err

[eventlistener:memmon]
command=memmon -p mydaemon=50MB
events=TICK_60

There are other good features of supervisord and other configuration options -- be sure to check it out, it's insanely useful.

That's all, folks

So that's how I write long-running PHP scripts. Don't fuss around with minutely cron jobs to run PHP scripts anymore. Do it properly. Create a daemon.