The basic information I gathered from the MySQL manual, but of course I needed to restore the account, not just reset the password. It's a simple matter of replacing that init file with the proper SQL to create the user.

root# /etc/init.d/mysqld stop
root# su mysql
mysql$ pwd
/home/mysql
mysql$ echo "GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY 'my_pass' WITH GRANT OPTION;"> mysql-init
mysql$ mysqld_safe --init-file=/home/mysql/mysql-init &
mysql$ rm mysql-init
mysql$ mysql -uroot -p   # woot, works

(Note that it's important that you switch to the same user that MySQL uses when running mysqld_safe. This might be 'mysql' or something else, you'll need to check out your init files to see.)

In a nutshell, this policy prevents pages from two different domains from modifying each others properties, using XMLHttpRequest, setting cookies etc. For instance, Example.com and OtherExample.com can't get references to each others document properties and can't set cookies on each other. Additionally, Example.com can't use XMLHttpRequest (aka AJAX) to load a resource from OtherExample.com. This last bit is probably the biggest issue for developers today -- in todays world of open web services and mashups. How do you consume a web service with Javascript if you can't load the data properly?

Solution 1: Use a server-side proxy

The first way around this problem is to use a very simple server-side script that acts as a proxy to the web service. So instead of requesting AJAX from the remote site, you request it locally on your own domain through the proxy. The proxy itself has it's programming logic to send your request off to the remote site, gather the data, and then serve it back to you.

.________________________.
| Client on YourSite.com |
'-\/---------/\----------'
  ||         ||
  ||_________||__.
  | AJAX Request |
  '---\/-----/\--'
      ||     ||
      ||_____||______________.
      | YourSite.com         |
      | (Ex. ajax_proxy.php) |
      '---\/-----/\----------'
          ||     ||
          ||_____||__________.
          | OtherSite.com    |
          '------------------'

Since your server-side script has no qualms about fetching data from another domain, you can successfully proxy all AJAX like this without running into any trouble. And then since the client browser is fetching the data from the local domain (even though you're making another request behind the scenes), it doesn't violate the same origin policy.

An example proxy script might look something like this (maybe a little more complex if you're handling POST data too):

$url = 'http://othersite.com/someservice?' . http_build_query($_GET);
echo file_get_contents($url);

Remember, the sole purpose of the proxy is just so the client browser can load the data locally on the same domain so the same origin policy isn't violated.

Drawbacks

This method works well but has two obvious drawbacks. First is of course that you need a server-side script at all. Especially if you are providing a service, this raises the barrier for entry and makes it harder for "noobs" to use your widget -- it's not a simple "paste this code into your footer" instruction; you also need to explain how to install the proxy service.

The second drawback is that your own servers are making these requests which makes the whole process slower, but also eats up your resources -- both your processing power and even just simple bandwidth.

Solution 2: JSONP

The second solution is to use JSONP -- "JSON with Padding." This is a technique that lets you get around the same origin policy. In order to use JSONP, the service you are requesting needs to support it.

So in the last solution, the consumer (the user using the service) required an ajax_proxy.php or whatever server-side proxy script. In this solution, the provider (Digg, Amazon, Yahoo -- whatever) needs to support JSONP themselves. Thankfully, these days the idea is catching on and it's likely JSONP is an option. And if you're a service provider, then you'll want to build it in to your service for sure.

How it works

A normal JSON request is sent using XMLHttpRequest and the reply looks something like this:

{'uid': 23, 'username': 'Chroder', 'name': 'Christopher Nadeau'}

Now the truth is, JSONP isn't AJAX at all technically speaking because it does not use XMLHttpRequest. JSONP requests are made by dynamically inserting a <script> tag into the DOM. For example, you'd insert this:

<script type="text/javascript" src="http://othersite.com/service?all=your&params=as&usual=gohere"></script>

Then that remote Javascript file is loaded and contains an actual Javascript function call. For example:

handleJsonReply({'uid': 23, 'username': 'Chroder', 'name': 'Christopher Nadeau'});

In other words, instead of reading AJAX data directly into a variable in Javascript, you define a callback function that is called when the data arrives, and the first parameter is the data itself as an object literal. If you were a provider implementing JSON on your service you might have something like:

// A service provider accepts a 'callback' parameter
// that it uses to wrap an AJAX reply

// Imagine we did some work here
$json = json_encode($mydata);

// Using JSONP
if ($_GET['callback']) {
	echo $_GET['callback'] . "($json);"; // somefunction({data here});

// Normal JSON
} else {
	echo $json;
}

The Javascript source code result of that request is nothing but a real executable function call with a Javascript object literal. It's as if you added the tag yourself and it called some functions -- the difference being that the JS is written by the producer on-the-fly and embeds the data you want right in the code. This is why it all works; it's not actually AJAX at all, it's just loading a remote Javascript file that happens to have some useful data in it.

Using a library like jQuery that supports JSONP, these details of inserting the special script tag and creating the special callback function are all taken care of automatically. Using a JS library, usually the only difference between JSONP and real AJAX is that you enable a 'jsonp' option.

Drawbacks

The first limitation to this method is that you have to rely on the provider to implement JSONP. The provider needs to actually support JSONP -- they need to wrap their JSON data with that callback function name.

Then the next limitation -- and this is a big one -- is that JSONP doesn't support POST requests. Since all data is passed in the query string as GET data, you are severely limited if your services require the passing of long data (for example, forum posts or comments or articles). But for the majority of consumer services that fetch more data than they push, this isn't such a big problem.

JSONP with jQuery

Step 1: Make sure the provider supports JSONP.
Step 2: Set the dataType option to jsonp, and if the provider uses a different GET param other than 'callback', specify the jsonp option to that parameter name.

$.ajax({
    // ... Use the AJAX utility as you normally would
    dataType: 'jsonp',
    // ...
});

jQuery will generate a unique callback name for this request (something like json1268267816). Thus, the reply from a web service would be something like:

json1268267816({'uid': 23, 'username': 'Chroder', 'name': 'Christopher Nadeau'});

But jQuery handles it all seamlessly, so you as the developer just handle it like a normal AJAX request using the same jQuery success/failure/complete callback hooks.

Creating long-running scripts

A daemon is nothing but a script that goes on forever. When you get right down to it, a daemon is simply an infinite loop.

class MyDaemon
{
	public function run()
	{
		while (1) {
			// Like the energizer bunny
		}
	}
}

$daemon = new MyDaemon();
$daemon->run();

The work you do within that infinite loop is the whole purpose of your daemon. Maybe it's listening for a network connection, or maybe it's waiting for an item to enter a queue. For example, how about a hypothetical website that needs to process new users using a database queue, you might end up with something like this:

(Note: using a database for a queue isn't very efficient, hence the use of sleep() here to prevent hundreds of count queries per second being run in the loop. In practice you'd want to use a real queue, like beanstalkd.)

class MyDaemon
{
	public function run()
	{
		while (1) {
			if ($this->isNewUsers()) {
				foreach ($this->getNewUsers() as $user) {
					$this->setupNewUser($user);
				}
			} else {
				// If there's no work to do, let's just
				// wait 10 seconds before hitting the DB to
				// count again
				sleep(10);
			}
		}
	}

	// ...
	// Imagine we defined isNewUsers, getNewUsers, setupNewUser
}

Note: In this post I'm creating simple daemons that generally perform one task at a time. "True" daemons, like a web server for example, fork off children so multiple tasks can be run in parallel.

Shutting down safely

So we have a long-running script now. But you might have noticed there's no way to actually stop it. The only way we could ever stop this script from running is to issue a kill command to kill the process. But this is dangerous because what if the script was in the middle of processing something? Using our example above, what if we were only half-finished processing a user?

You can sometimes mitigate problems like this. For example, if you use database transactions to ensure an all-or-nothing commit. But what if you need to interact with systems that don't offer such transactional safety?

So the only way a PHP daemon is really worth it is if we can shut it down safely. That means we need some way to tell the script it's time to go; a way to tell it to break the infinite loop. Something like this:

class MyDaemon
{
	public function run()
	{
		while (1) {
			// Check to see if we should break out
			if ($this->shouldStop()) break;

			if ($this->isNewUsers()) {
				foreach ($this->getNewUsers() as $user) {

					// Check inside the loop too, to make sure we
					// dont continue processing a big batch of users
					if ($this->shouldStop()) break;

					$this->setupNewUser($user);
				}
			} else {
				sleep(10);
			}
		}
	}

	/**
	 * Returns true when the main loop should finally stop
	 */
	public function shouldStop()
	{
		return false;
	}
}

But what can we put inside that shouldStop() method?

Checking an external value

Perhaps the simplest thing to do is to check to see if there is some external value. Maybe a database record that says "should_stop=1" or the presence of a file called "should_stop.txt".

	/**
	 * Returns true when the main loop should finally stop
	 */
	public function shouldStop()
	{
		clearstatcache();
		if (file_exists('should_stop.txt')) {
			unlink('should_stop.txt'); // delete the file for next time
			return true;
		}

		return false;
	}

But these methods aren't very efficient, and they build up a dependance on some other system.

Handling POSIX signals

The best way for us to handle clean shutdowns is to handle standard POSIX signals like TERM and INT. When you're using a shell and press CTRL+C, Linux sends the application an INT signal. When you the kill command on a process, it will send the TERM signal by default. So intercepting these two signals is perfect for us. If we can intercept them, then we can make sure we shut down gracefully.

Enabling Process Control in PHP

PHP has process control features (PCNTL) that we need to interpret POSIX signals. But they are not enabled by default. You must compile PHP with the --enable-pcntl configuration option.

Listening for signals

There are two steps we need to do:

1. Declare ticks. Ticks are a little known feature of PHP that allows you to run a callback function every X statements. This can be useful for memory profiling or debugging, but it's not very useful otherwise. But we aren't actually going to use ticks ourselves -- the PHP PCNTL features just uses it internally.
2. Assign callbacks to listen for signals using pcntl_signal().

Here's what we end up with:

declare(ticks = 1);

function sig_handler($signo)
{
	switch ($signo) {
		case SIGTERM:
		case SIGINT:
			global $_MYDAEMON_SHOULD_STOP;
			$_MYDAEMON_SHOULD_STOP = true;
			break;
	}
}
pcntl_signal(SIGTERM, 'sig_handler');
pcntl_signal(SIGINT, 'sig_handler');

class MyDaemon
{
	// ...

	public function shouldStop()
	{
		global $_MYDAEMON_SHOULD_STOP;
		if (isset($_MYDAEMON_SHOULD_STOP) AND $_MYDAEMON_SHOULD_STOP) {
			return true;
		}

		return false;
	}
}

So in this example we've created a callback function that listens for TERM and INT signals, and assigns a value of 'true' to a variable. Then within the actual daemon class code, we check this variable each time to determine when it's time to call it quits.

Note that it's important to use a basic function callback with pcntl_signal and not an object method. There is some weirdness when using using $this in callbacks. That's why in the example above I've opted to use a simple function and a global variable.

Problems with long-running PHP scripts

The biggest problem with long-running PHP scripts is memory usage. PHP isn't exactly designed for long-running programs. There are memory leaks that don't usually matter in websites because requests start and finish so quickly, but matter a lot in long-running processes. These leaks are usually to do with circular object references and PHP's garbage collection. But with recent versions of PHP it's not so much of a problem anymore.

Even so, you have to be careful about memory usage in your actual scripts as well. Not many libraries are designed for long-running processes. So maybe your emailing library saves some information in an array after you send each email. In a web script, it doesn't matter -- the array is lost after the request finishes. But now you might have a process that runs for hours on end. This little array of log messages can become huge!

It's problems like that that you must be weary of. You need to consciously think about things like that while designing your PHP daemons. What I would recommend is to plan for your PHP daemons, if possible, to shut down and restart periodically. I'll explain the best way to do this in the next section.

Handling processes (startup, monitoring, detaching etc)

So we've got our long-running PHP scripts now, but we still have a few more things to tackle. First is how do we detach a script so it runs in the background? That is, if you run one of these scripts from your shell for example, you'll just get an empty screen that does nothing until you quit the process. You want to run the script so it detaches and runs in the background -- aka, you get your command prompt back. You can do this in straight PHP -- it involves forking so the parent process can die, but the children live on.

But there are other things to consider. How do we handle output? How do we handle fatal errors and exceptions (like a database disconnection) -- how do we get the process to restart? How do we make sure the correct user owns the process?

The easiest way to do all of this is to use a process manager like supervisord. This software runs on your server and is responsible for starting your long-running PHP scripts in the background (so you don't have to worry about detaching yourself), and also sports features like monitoring so you can automatically restart processes that've quit unexpectedly or when they are using too much memory.

With supervisord, you add some configuration for each script you want to run. Here's an example that runs a script, logs output to a file, and will TERM the script when it uses over 50MB of memory (where it'll be restarted again).

[program:mydaemon]
command=/usr/bin/php /home/john/mydaemon.php
directory=/home/john
startsecs=5
exitcodes=2
user=john
stdout_logfile=/home/john/logs/mydaemon.out
stderr_logfile=/home/john/logs/mydaemon.err

[eventlistener:memmon]
command=memmon -p mydaemon=50MB
events=TICK_60

There are other good features of supervisord and other configuration options -- be sure to check it out, it's insanely useful.

That's all, folks

So that's how I write long-running PHP scripts. Don't fuss around with minutely cron jobs to run PHP scripts anymore. Do it properly. Create a daemon.

The prevailing misconception seems to be that having GET and POST variables mixed together is insecure. While this is true under some specific circumstances (see Cross-Site Request Forgeries for a weak example) -- it isn't the main reason why $_REQUEST is insecure. In fact, combining GET and POST data is often a very useful feature. In many cases you don't really care if a variable comes from GET or POST and so using $_REQUEST in those situations can help reduce the verbosity of your code while increasing flexibility.

The real problem is that COOKIE data is also mixed into $_REQUEST. The implications of this may not be readily apparent until you start thinking like a malicious user. Any cookie ever set on a users machine will always override any GET or POST data -- creating "sticky" variable data.

For instance, imagine if your application uses "action=" or "do=" switches to control page commands. If an attacker some how managed to set a cookie -- perhaps through some Javascript injection -- he could set a cookie named "action" and give it the value "logout." Any users that fell victim to the cookie set would never be able to use your application because "action" would be permanently set to "logout."

So to sum it up:

Question: Why is $_REQUEST insecure?
Answer: Because it combines COOKIE as well as GET and POST, and the COOKIE data always takes precedence -- creating the possibility for dangerous "sticky" variables.

Solutions

If you're already using PHP 5.3, then you can edit your php.ini and check the request_order variable -- make sure 'C' removed. This 'C' is removed by default, but if you use a shared host it might've been put back for backwards-compatibility. Make sure to check!

Otherwise, you can easily just create your own $_REQUEST array by merging $_GET and $_POST manually as part of your global initialization routines.

$_REQUEST = array_merge($_GET, $_POST);

And finally, best practice is to abstract your input reading out (see Stop using superglobals!) so you can define yourself exactly how variables are read and from where.

1. High Coupling

All software you write depends on some other software. And parts of your own software depends on other parts of your software. Coupling is how much one part of your software depends on another part. In a highly coupled system, lots of parts all depend on each other. In a lowly coupled system, parts are independent from each other as much as possible.

Why It's Bad

• Changes made to one part of your system affect other parts of your system. This means making changes becomes difficult and error prone.
• It becomes difficult or impossible to use even functionally unrelated modules separately because they depend on each other so heavily.
• Due to the previous point, it is nearly impossible to write unit tests. This compounds the problem of making changes because you can't properly test to see if your changes break things in other parts of your system.

Solution

Stop coupling! Write classes that are as separate from each other as you can. This may mean building utility classes, or classes that help "connect" bits of your system. The point is that Part A of your system should be completely separate from Part B of your system.

Further Reading

• Strive for low coupling and high cohesion: What does that even mean?

2. Premature Optimization

We've all heard the famous quote by Donald Knuth, the author of The Art of Computer Programming: Premature optimization is the root of all evil.

Developers often worry about the performance of their software before performance should be considered at all.

Why It's Bad

• A developer who writes code with performance as a first priority will produce code that is hard to use and maintain.
• Code that is "optimized" prematurely rarely ever results in software that actually runs faster. Without proper testing/benchmarking, it is often hard to know which parts of your system is the bottleneck. Saving 15 milliseconds on a calculation routine does nothing for the data-fetching routine that takes 1500 milliseconds.
• Due to the lack of proper testing, developers will spend too much time on menial performance optimizations while missing the important ones.

Solution

Write your code with performance in your mind -- but don't sacrifice usability and maintainability. Write your code first, and then only once you have identified real bottlenecks you can start optimizing.

3. Abusing Inheritance

Inheritance is great, but some developers try to force their software to fit into rigid inheritance trees. Inheritance is a tool, but it's not the only way to write reusable software.

Why It's Bad

• Abusing inheritance results in classes that have functionality that they shouldn't.
• Subclasses make assumptions about their parent classes. This couples children to their parents. As we know, coupling is bad. The deeper our class tree gets, the more coupling, and the worse the design is.
• You end up with huge families of classes. For example: A parent Animal class. Then a Dog child. Then a BlackDog, and a BlackBigDog, and BlackSmallDog, and BlackBigLoudDog and BlackSmallLoudDog... You can see how it quickly gets out of control!
• You tie functionality/algorithms to a specific family of classes, making it harder to reuse. The only way to access such functionality is through inheritance -- even when inheritance doesn't exactly fit the bill.

Solution

The old maxim: Favor composition over inheritance.

Composition adds functionality at runtime -- giving you much flexibility. Inheritance forces you to define all functionality at code-time, limiting possibilities.

Composition means your classes are built up (composed) of other classes for additional functionality. This reduces needless children, and decouples functionality into separate reusable classes of their own. Using the example mentioned previously, a Dog class might be made up of separate classes to define it's appearance (AnimalSize class?) and loudness of it's bark (AnimalSound class?). An example calls might then be, $dog->getSize()->display() and $dog->getSound()->makeSound().

In other words, you want "HAS A" (composition) relationships rather than "IS A" (inheritance) relationships.

4. Failure to separate responsibilities

Some developers fall into the trap of creating classes that do way too much -- so called God Objects. Classes should do one thing only (see the Single responsibility principle).

Why It's Bad

• Other parts of your system depend too much on a single class, creating very high coupling. Any changes made will ripple out to all it's dependants.
• It is hard to reuse code written in these classes because they come with so much "baggage". Instantiating such a monster object wastes resources with all of the unneeded functionality that has been added.

Solution

Refactor. The God Object needs to be properly deconstructed into smaller, more specialized classes that all do a single job.

5. Failure to refactor

Refactoring is the activity of switching around your softwares internal design, while keeping it's functionality the same. For example, renaming a method is a small example of refactoring. A more intense example might be splitting one class into two separate classes (maybe you're tearing apart a God Object?).

Why It's Bad

• As your product grows older and you add new features, classes will start to gain functionality they did not before. Over time, classes may become too big resulting in monster or God Objects.
• As your product grows and you add features, some classes may start to become complex. Without refactoring to simplify the inner workings of your system, this complexity may start to get out of control. For example, a class constructor that takes 20 arguments.
• The longer refactoring is put off, the worse the problem becomes and the harder the task will be in the future.

Solution

Refactor as soon as you encounter smelly code. The earlier you refactor, the easier it will be.

6. Complexity

Code should be simple to read and maintain. Functions that take 20 arguments, or classes that require too many helper objects are examples of complex designs that can be fixed.

Why It's Bad

• Complex designs are hard to understand and thus hard to maintain.
• Complex designs will result in hard to find bugs.

Solutions

Simplify. If a function takes 20 arguments, maybe some of those arguments can be encapsulated into a class. Or maybe the one function can be replaced by an easier to use class. If a class requires lots of setup, consider using the Builder pattern. If you have a system of classes that all work together towards some goal, consider simplifying the interface by using a Facade.

7. Reinventing The Wheel

Some developers, especially amateurs, like to create everything from scratch and ignore all of the available free library code that exists on the internet.

Why It's Bad

• You waste time writing components when you could be actually building your product.
• You get none of the free testing done against a popular library. Anything you write from scratch will contain bugs.
• Not always true (but may especially apply to amateurs): Many libraries are written by developers who may be more experienced than you. These coders produce better organized code with more features.
• Not always true: Developers of libraries are often experts in whatever task the library is meant to solve. For example, database abstraction. Because of this, the library developers will produce a better product than you could.

Solutions

Realize that writing everything in-house is almost always a bad thing. There will of course be times where there exists no suitable library. But almost everything has been written before, and written well. A good developer knows the tools at his disposal, and uses them.

Every time you consider writing something yourself, you should search for a good library first. Common difficult tasks such as database abstraction, web services, image manipulation, validation and parsing are examples of components where an abundance of free library code exists. Less well-known problems, or new problems, may not have many pre-built solutions.

8. Failure to apply known design patterns

Design patterns are well known solutions to architectural problems in software. I've mentioned a couple in this article already (Builder, Facade).

Why It's bad

• A developer who doesn't know design patterns is more likely to design his software incorrectly.
• Not knowing about design patterns will decrease your efficiency. Every problem has been solved before. By knowing which pattern to apply to a problem, half of your work is done for you. Otherwise, you will go through many revisions before you end up at the same solution anyway.
• Design patterns help a developer communicate with other developers. It's the language of the profession. You will not be able to convey ideas effectively if you don't understand the terminology.

Solution

Read all material you can find on the topic of design patterns (see next section). Amateur programmers may not appreciate the value of these patterns at first -- or, indeed, may not fully understand them -- but as time goes on, this knowledge is priceless.

Further Reading

• Head First Design Patterns
• Design Patterns: Elements of Reusable Object-Oriented Software
• Patterns of Enterprise Application Architecture

As you know, timestamps represent the number of seconds elapsed since January 1, 1970 00:00:00 UTC. The UTC timezone there is important. Basically all unix timestamps are in UTC -- there is no "EST timestamp" for example, there is only UTC.

Let's be clear about that: If we both compared our timestamps on our computers right now, regardless of where in the world we were, the timestamps would match. A timestamp is a timestamp is a timestamp (how's that for poetic geekery :-).

But confusion ensues when you want to create timestamps. For example, using PHP's mktime() function. The problem is that to specify a time you have to know what timezone it is. Imagine if I were to say, "It is 19:00". That statement would make no sense; it's ambiguous. To make any sense at all, I need to specify where it is 19:00 -- and our world does this via timezones.

The thing to keep in mind is that "human time" is different in every timezone. For humans, my 19:00 might be your 16:00, which might be someone else's 22:00. But in "computer time" of timestamps, all of these times are the same -- they are all timestamps. Because a point in time is the same point in time everywhere, only when we "render" time to "human time" do we introduce the concept of reference points (timezones).

How do I make a timestamp that is my 19:00? To do this, all I need to do is create a timestamp that is 19:00 UTC, and then calculate my timezone offset. Why does this work? Remember that 19:00 is the exact same point in time everywhere in the world -- the only thing that changes is how we humans reference that time. Thus, "my" 19:00 is not UTC 19:00. To get "my" 19:00, I get UTC's 19:00 and offset it for my timezone, so that the timestamp correctly represents 19:00 from my point of view.

Timezones with PHP

The point of this post was originally to talk about confusions with PHP. Because it seems a little known fact that PHP's mktime() converts input from local time into UTC. That is, it does the offset calculations for you.

So when I do this with PHP (create a time where it is 19:00):

mktime(19);

PHP uses your systems timezone and offsets the time for you. If the systems timezone is GMT-5, then it'll offset 5 hours to create the correct UTC representation of that time. (PHP will use the system defined timezone for your server unless you use date_default_timezone_set)

What about when we do these:

time();
mktime();

These mean "the current timestamp". As we discussed before, a point in time is the exact same no matter where you are in the world -- so the timestamp for "right now" requires no offset calculations (there is no ambiguity for "right now"). Timezones only matter when you're referencing a date and time specifically (aka, using mktime() with arguments).

If you want to make a timestamp with time elements that are already in UTC time, then you should use the gmmktime() function instead. Alternatively, you could set PHP's internal timezone to UTC.

Best Practice: Cast to what you want

The first part of this post is all about how important casting incoming data is. As you know, security rule number one is never trust the user. So if you are expecting an Article ID which is an integer, you should make sure that you actually got an integer.

The most common approch is to cast user input when you get it. So if an attacker tries to supply some arbitrary string, PHP will cast it to an integer which usually results in a harmless 0. Sure, you have precautions in place against SQL injection or XSS but by casting data to a specific type, you can greatly simplify code because you can start working under assumptions ("I know for sure that $id is a harmless integer").

Here are a few examples:

// ID's (the use of max to get rid of negatives; Usually ID's are unsigned)
$article_id = max(0, (int)$_GET['id']);

// Simple strings
$username = preg_replace('#^[a-zA-Z0-9\-_\.]$#', '', $_GET['show_user']);

// Booleans
$show_drafts = (bool)$_GET['show_drafts'];

Note that this isn't really about form validation. If someone is filling out a form, you still need to make sure what they input is valid. For example, the "Simple strings" snippet above is useless to users when they're registering because you're not telling them that their input is invalid.

These strict casts are useful only when you don't expect users to fiddle with values. So things like view.php?id=34 where the URL is generated by your app, or <select> values where the user shouldn't change values etc.

Getting rid of the superglobals

Well, sort of. We obviously can't completely stop using $_GET/$_POST/$_COOKIE for reading incoming data because there's no replacement. What I'm talking about is ridding your code of these superglobals -- pushing them into a special input class. A code snippet is worth a thousand words; here's what I'm talking about:

// Bad
$article_id = $_GET['id'];

// Better
$article_id = $input->get('id');

Why

The above snippet doesn't really show off anything other than the concept. So let's talk about what this mysterious input class is meant to do, and why it's better than using superglobals.

Control

You don't have much control over superglobals. They are simply arrays. You can't do anything special before trying to fetch values. You might be thinking about what kind of processing you would want to do. But think about this for a moment.

How many times have you done something like this:

$article_id = (int)$_GET['id'];

// Or
if (isset($_GET['id']) {
    $article_id = (int)$_GET['id'];
}

Since you control the input class, you can add a bunch of features to make gathering and casting input really easy.

$article_id = $input->getInt('id');
$search_user = $input->getSimpleString('search_user');
$delete_ids = $input->getArrayOfInts('delete_ids');
$article_content = $input->getString('content');

UTF-8 Handling

One place where this sort of functionality is really useful is when you've switched to using UTF-8. As you know, it is possible for UTF-8 strings to be malformed -- and this is a security risk. So every time you read strings you must make sure they're valid. Without an input class, this would be a lot of work. With an input class, you can just modify your 'getString' method to add UTF-8 checking.

Stripping magic quotes

Another common task PHP programmers routinely need to do is handle magic quotes. Most of us simply test to see if magic quotes is enabled, and run the entire superglobal array family through a function that strips the slashes -- basically undoing this devil-feature.

But by doing this, you're affecting the entire app (because, well, you're modifying a superglobal of course!). This might not matter to you, but it gets a bit dangerous if you use third-party libraries.

A friend of mine creates Facebook applications. The Facebook PHP library is clever enough to test for magic quotes and strips slashes out for values it needs. But what if you're a clever PHP programmer, and you stripped out the slashes as part of your "global.php" file? FB doesn't know that, so it just strips them again! This is an example of how modifying global data is a bad idea.

By moving input-reading into your own class, you can do whatever the hell you want to your data and can rest assured knowing you didn't affect anything else.

Abstraction

By creating this new class, you abstract the details of how input is gathered. You might be wondering why this is important -- isn't there just get/post/cookie? No! While the vast majority of your webpages will only use these types of input, there are certainly others. Here are two that jump to my mind.

Command-Line

If you make command-line scripts, getting arguments may be a perfect use for your input class.

Friendly-URL's

$_GET only works for values encoded in the query string. But if you are a creating an app with friendly URL's, values are sometimes embedded right into the URL.

For example: /article/42/edit

This URL might mean to "edit" the article with ID of "42". Most frameworks, like Zend Framework's MVC components, make creating URL's like these very easy. In ZF, you define a route with placeholders of the values:

$router->addRoute('article_action', new Zend_Controller_Router_Route(
    'article/:article_id/:action',
    array('controller' => 'article', 'action' => 'view')
));

The framework provides a way to get the value of 'article_id' and 'action' that you could plug right into your input class. It might look something like this:

$article_id = $input->getIntFromUrl('article_id');
$action = $input->getStringFromUrl('action');

Building an input class

Building your own input class is an excersice you can complete yourself. But I'll get you started.

First of all, what is our goal?

• Ability to clean data
• Ability to get data from multiple sources

I think a well-designed system calls for a handful of classes:

• Cleaner takes values and cleans/casts them to a correct data type. We're using a separate class because it's not directly tied to input -- we might reuse this functionality elsewhere.
• InputSource_* are classes that read raw data from some source. We have one reader for each source. For example, InputSource_Array for reading information from an array (like supergloabls) or InputReader_Url for reading information from a friendly URL.
• InputReader is the main class that ties the Cleaner and the InputSource_* classes together.

Here's how it might work:

$cleaner = new Cleaner();
$input = new InputReader($cleaner);
$input->addSource('req', new InputSource_Array($_REQUEST));
$input->addSource('get', new InputSource_Array($_GET));
$input->addSource('post', new InputSource_Array($_POST));
$input->addSource('cookie', new InputSource_Array($_COOKIE));
$input->setDefaultSource('req');

$input->getInt('id'); // Get from the 'req' source, a default
$input->getInt('id', 'get'); // Get from 'get' source

// $input->getInt() calls a corresponding $cleaner method to
// clean an integer.

About Scalability

A scalable system is a system that is easily maintainable and can easily grow to accommodate increased usage.

Scalability is not about performance. Though you should aim towards a good performing system, performance and scalability are completely separate topics. Scalability is about maintaining the same level of performance as the usage grows.

Scalability is not about specific technologies (PHP, Ruby, Python, .NET, Java etc). Any language can be used to create a scalable system. In the realm of web applications, a website that is written in a scripting language like Python can be just as fast and scalable as one written in C. Large parts of Yahoo! for example are written in PHP -- and we know that scales. The language makes no difference because the speed of execution is not the trouble with scalable systems, it's the data. We'll come back to this idea in a bit.

Vertical Scaling or "scaling up"

Vertical scaling is achieving better performance by simply adding better hardware. If your system runs well with 1000 users but not so well at 5000 users, then you can simply upgrade your server with the fastest hardware and you're set.

Vertical scaling is easy. You don't need to think about scaling at all when writing your applications because it's essentially always running in one environment, on one server. If things get too busy, you just upgrade your server.

The problem with scaling up is that it is expensive. Low-end hardware is very cheap, but high end hardware is exponentially more expensive. There comes a time where the cost of the hardware is simply too much, or, that the hardware with the required resources simply does not exist.

So, vertical scaling is a type of scaling that you usually get "for free" without any work. But it is not practical if you want to build a site that serves millions.

Summary:

• Better hardware = better performance
• Pros: Easy
• Cons: Expensive; there will always be a technological limit to how powerful a machine you can possibly buy.

Horizontal Scaling or "scaling out"

Scaling out is achieving better performance by adding more servers. Instead of constantly upgrading to more powerful machines, we can buy a bunch of less-powerful machines cheaply. It is much cheaper to buy many commodity servers than it is to buy one really powerful server. And unlike vertical scaling which has a "limit" to how powerful of a machine you can possibly buy, horizontal scaling can go on forever.

Basically instead of 1 really powerful machine with 32GB of memory and 4 processors, you have 8 commodity machines with 1 processor and 4GB of memory. Each machine handles a portion of the load instead of one machine handling it all. When you need more resources, it's as simple as adding a new (and cheap) commodity machine.

The cost of scaling out comes in two new flavors. The first is the cost of administering these new machines. Instead of say 1 monster server you now have to babysit 8 smaller servers. The real-world cost of doing this though is (fortunately) relatively low. Each machine you add to your system should be identical, and so, system administration tasks can be largely automated.

The second cost comes in design difficulty. You have to think a lot more while designing your system to make sure it can scale out. That is, how does your system work accross 8 separate servers?

Summary:

• More hardware = better performance
• Pros: Cheaper; infinitely scalable
• Cons: Harder to design, a bit harder to manage because you have more machines to watch

Database Replication

Database replication is about automatically copying data from one database server to another database server. There is a master database that sends out data to it's slaves who copy it to their own database. The idea is that you can spread load to all of the slaves instead of relying on a single database server to handle the load of every user.

There are few types of database replication, but the most popular is called master-slave replication. This is where you have a single master database, and any number of slave databases. All write queries (queries that modify the database) go to the master. All read queries (SELECT type queries) are spread amongst the slaves. This works well in sites where the number of reads far outweigh the number of writes (which is just about every website online!).

Designing for Database Replication

For master-slave replication, your application must know which server to send queries to. This is fairly simple because it is easy to interpret plain-text SQL and know which kinds of queries are writes (UPDATE, INSERT, REPLACE) and which are reads (SELECT). For example, in your database abstraction layer you could modify your query method to read something like:

public function query($sql)
{
    if (preg_match('#^\s*SELECT#i', $sql)) {
        $conn = $this->slave_connection;
    } else {
        $conn = $this->master_connection;
    }

    // Send query to $conn database
    // ...
}

The other thing you need to do is decide how you will load-balance the slave servers. There are hardware solutions or software solutions. The simplest method, and the one I personally like to use, is to simply choose one at random when your application starts up.

Replication Lag

Replication lag is the time it takes for all slaves to execute the write queries that happened on the master. This is usually very very fast (milliseconds) and thus not usually something to worry about. But if you are overloading your cluster, then the time it takes may increase and your users will start to see the affects. Things like saved changes not appearing when they reload the page.

One potential cause is that the slaves themselves are being overloaded -- they are serving too many read queries so they are falling behind. This can be fixed by adding new slaves to help spread the load some more.

Database Engines

One thing worth noting is that a slave can use a different database engine than its master. For example, your master can use MySQL InnoDB tables and your slaves can use MySQL MyISAM tables.

I mention this because InnoDB is required for transactions, but is much slower than MyISAM. So it seems natural to use InnoDB for the master (write) database and MyISAM for the slave (read) database. It means your slaves can handle more load.

Database Partitioning

Database partitioning is about splitting up your database so that all of the data is spread across multiple databases.

Clustering

Clustering is splitting your database up at the table level. So you have your users, usergroups and profiles on one database, and then you have your forums, topics and replies on another database.

This method is pretty straightforward but it is severely limited. It means you can't join on tables used in a different database, and it adds a certain overhead because each page now requires connections for each cluster. So if you have split your database into 4, one request now needs 4 connections.

Also, each cluster will likely have different requirements. For example, there will likely be many many more replies than there are user profiles. So this makes it more difficult to manage.

Sharding

Sharding is splitting data up at the row level. This means you have a bunch of databases with identical tables, but the data they contain is split up. For example, users 0-10000 are stored on shard 1, 10001-20000 are stored on shard 2 etc.

This method is infinitely scalable because when one database becomes overworked, you just add another shard.

The difficulty comes when you need to join with data that exists on different shards. The best way is to simply ensure that all related data exists together in a single shard -- but this can be pretty difficult. The other way is to denormalize your database so that you don't need to join with other tables.

Database Denormalization

Going against all best-practices with database design -- denormalization basically means copying data from one table into another to eliminate the need for table joins. Joins are fairly expensive, so it is sometimes worth the horrible icky feeling in your stomach when you denormalize.

So as an example, you might have a 'threads' table that has a field 'last_poster_id' that links to a 'users' record. If you wanted to get the username for the last poster, you'd need to join with the 'users' table. But if you denormalize your schema, the 'threads' table would have a new field called 'last_poster_username'.

The downside of denormalization is that you need to make sure all records stay consistent. With the example above, it means that if we change the username for the user we must also change every 'last_poster_username' in the 'threads' table. In situations where data is read many more times than it is written, this is usually okay.

Caching

The single most important aspect to scalable websites is caching. Every website, when you get right down to it, reads data more than it writes data. So, instead of reading data from a database -- which means reading from disk -- you should read data from a memory cache. With very popular websites, reading from disk is complete and utter disaster! You must cache to memory if you want to become the next YouTube.

Cache what? How to cache?

You should, in short, cache everything. The database is only there for persistent storage, and to refresh caches when the caches become stale. So that means your User::getUserinfo() method looks something like:

public function getUserinfo($user_id)
{
    if (cache_exists("user:$user_id")) {
        return cache_get("user:$user_id");
    }

    // Else, you need to fetch the data from the database
    // ...

    // And then cache the result for subsequent requests
    cache_save("user:$user_id", $userinfo);

    return $userinfo;
}

There are many different technologies that you can use to cache data. But the best of them (or at least most widely used) is called memcached. PHP even has a memcached module.

Handling stale data

When you are caching data, you need to be careful that the data in the cache is updated when the data in your database is updated. So this might mean a simple update to your User::save() method for example. Other times, you might add some logic so that cached data is automatically removed after a certain period (so that the next time its requested, it will be fetched and then recached once again).

Where to cache

Above, I mentioned that you should cache to memory. But I should clarify that memory is not the only way to cache things; and might be superfluous in many situations. Indeed, before you even need to think about memory caching you should identify a disk IO bottleneck so you know for sure that a memory cache will even improve the situation.

Before you move to a memory cache, the easiest way to implement a caching system is through files in the filesystem. You might even cache data in the database. The whole point in caching is to reduce load. So, for example, if you cached an entire page in the database all you need to do is fetch one row and you can present it to the user quickly (versus say, 10 queries and HTML-building routines you might normally need to do).

The point I'm trying to make is that caching isn't always about memory. It's about storing data (possibly pre-processed data) in a location that is inexpensive to access -- or at least inexpensive relative to the original source.

It just so happens that memory is the only source for a very popular site to read cache from, because all other disk-bound caches (including files and databases) become too slow.

Separate Components

You should separate components where you can so you can create specialized machines that are very good at certain tasks. For example, Apache is great for serving PHP pages but is really overkill if you need to serve up static CSS/Javascript files. Try adding a new server that runs Lighttpd to serve up those kinds of things. Additionally, don't put your MySQL server on the machine as your memcached server. You get the idea.

In Closing...

So there's a very brief overview of the topics you need to read about. I hope you now understand the sorts of things that go into a scalable website. My goal was to demystify the ideas of scalability so you can better research specifics; and of course in coming days I'll try and talk about specific techniques.

What are stream wrappers?

PHP 4.3.0 introduced the idea of streams. The PHP manual has a perfect explanation:

Streams were introduced with PHP 4.3.0 as a way of generalizing file, network, data compression, and other operations which share a common set of functions and uses. In its simplest definition, a stream is a resource object which exhibits streamable behavior. That is, it can be read from or written to in a linear fashion, and may be able to fseek() to an arbitrary locations within the stream.

Streams are identified by a particular scheme in the URI. For example, with http://someurl the scheme is 'http' and when no scheme is provided like in /file.php, the scheme is defaulted to 'file' and is expected to be a file in the filesystem (thus file:///file.php would mean the same thing).

Stream wrappers are bits of code that tell PHP how to handle certain schemes. The interesting thing is that you can create your own custom stream wrappers written in PHP with little effort.

For example, you may have done file_get_contents('file.txt'); before. But say you wanted to have the same ease of use but with data from a different source. You could create a custom stream wrapper to get data from wherever you want and use all of the normal PHP functions as usual: file_get_contents('mystream://whatever');.

Almost any function that works on files will work with custom stream wrappers. That includes fopen, fwrite, fseek, is_readable, is_writable, stat, unlink and more. In essence you can create a virtual filesystem in your applications.

Why create custom stream wrappers?

So why in the world would you want to create custom stream wrappers? Basically it comes down to decoupling filesystem-coupled interfaces from the filesystem (or any stream, for that matter). That is, make software think it's talking to files when it's talking to your custom PHP class. It's all about abstraction!

You could also use stream wrappers to simplify an interface, though this is less common. If you are using stream wrappers to simplify an interface, you can just as easily create a normal facade to simplify the interface and leave streams out of it.

For example

I wanted to use a newish PHP templating engine called Dwoo for an upcoming project. The problem was it expects files everywhere. Template files were read from the filesystem and compiled to the filesystem, and caches were written to the filesystem. There was no easy way to change this.

For this particular project I need to read templates from a string (loaded from an arbitrary source decided at runtime), and I can't use the filesystem for caches, I need to use memcached. The reason being that the application will be run in a cluster. Using shared disks is not practical with this many machines, and disk IO would eventually become a bottleneck. True, we could use RAM disks, but it's a lot easier and portable to manage the data layer from our application instead of pushing it to the OS.

Using custom stream wrappers I created two new schemes for 'tpl://' (for reading) and 'tplc://' (for writing compiled templates). I was able to fill the requirements for the project while still using the awesome templating engine with little modification. The library still thinks it's reading and writing to files but behind the scenes it's reading from a cache of template sources and writing to memcached. Perfect!

Creating custom stream wrappers

To create your own custom stream wrappers you need to write a class that implements a set of methods. After you write the class, you can register it with stream_wrapper_register. Refer to that manual page for details on the methods you need.

The only problem I encountered was how to get my custom stream working with is_readable() and is_writable(). The trick is to implement the 'url_stat' method and set the 'mode' correctly. The mode is a number that represents what type of file the file is (file or directory etc) and the file permissions -- the usual chmod values. Here's a code snippet that accurately defines how the number should add up:

$dir = 040000;
$file = 0100000;

// Now you can add any permissions you want using the usual chmod octal notation:
$file += 0777; // A file that is readable
$dir += 0222; // A directory that is writable

Why use Unicode

I covered this in my previous article on Unicode, but let's recap. The reason to use Unicode is so you can create multilingual applications.

Without Unicode you basically have two ways to serve a multilingual site. I'll briefly explain each here.

Encoded HTML Entities

The first way is to store all data in one character set (for example, iso-8895-1 for English) but encode all other foreign characters as HTML entities. Most web developers have probably done this at one time or another -- for example, using &copy; for the copyright symbol. But you can also encode any character this way by specifying the Unicode code point like so: &#0960; for π.

Common Uses:

• When you have a page in one language, but sometimes need to use a character from another (perhaps you use the word über a lot ;-)).

Advantages

• Easy to use.
• Works with any character set you already have in place.

Disadvantages

• It's not "correct". Instead of solving the problem, you're just patching it up.
• Building systems that do not know about HTML or entities becomes troublesome. For example, sending emails or building a search engine.
• As just normal text (not rendered HTML), the entity codes mean nothing. This becomes a problem if you have some way for users to consume your content other than through a web-browser (as already mentioned, email is a big one).

Multiple Encodings

The second way is to store data in multiple encodings. Store English in ISO-8859-1, store Russian in ISO-8859-5 etc.

Common Use:

• These days you see this kind of thing in places where systems are not ready to handle Unicode. For example, years ago some email providers or email clients did not handle Unicode at all. So a reason to mix encodings is for compatibility with legacy systems.
• Another reason this is in use today is due to legacy code. It might be a big job to switch to using Unicode.

Advantages:

• Legacy systems usually have no problem working with any of the popular character sets.
• Content is portable -- unlike using HTML entities which requires a parser to convert the entities.

Disadvantages:

• Impossible to mix characters from two different languages. For example, imagine you have a site that contains both Japanese and Russian articles. If you were to have some global site-map that lists all articles, the titles would be mangled because you can't represent both Cyrillic and Japanese characters with any single character set.
• You introduce some complexity. Along with each piece of content, you need to store which character set was used. You need to worry about outputting the correct character set header when serving web pages. If you have some sort of service-based application, you need to worry about which character set requests are sent in and which one to send results back in.

Using Unicode

Using Unicode suffers from none of the disadvantages describes above. Indeed, the very reason Unicode was created was to overcome those problems.

The most often used encoding for Unicode is UTF-8. This variable-length encoding stores Unicode characters in 1 to 4 bytes. There are other encodings, like UTF-16 or the depreciated UTF-7 -- but we'll only be talking about UTF-8 today. In fact, UTF-8 will probably be all you ever need.

Before we continue, I want to at least mention a few "problems" associated with using Unicode.

• Unicode will sometimes use more storage space. While normal ASCII characters will use the same amount of storage (1 byte), most other characters will use 2-4 bytes. Storage is cheap nowadays so this often isn't a problem.
• Unicode is "new". The idea isn't new, but the widespread adoption is. (At least, relative to older character sets like ISO-8895-1 et all). This means that if your app interacts with very old systems a lot, then you might have some trouble with Unicode.

  You'll likely have no problems, but I mention this just to make sure you test any systems and devices your app needs to work with. For example, cell phones are notorious for their slow Unicode adoption rate.

PHP and UTF-8

Declaring UTF-8

To start serving UTF-8 web pages, you need to send a content-type header. Web servers will automatically send a default header if you don't (which is why you may never have had to do this before). Most of the time the default header says the content is HTML, and is encoded in ISO-8895-1. Since you output HTML and your web pages are English -- this is almost always okay.

To start using Unicode, however, you need to either change the server config and change the default encoding or output it yourself from within PHP:

header('Content-Type: text/html; charset=utf-8');

Also note that your HTML content might also be spouting a 'http-equiv' meta tag that features the encoding. You should make sure it has UTF-8 there, too:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>

A browser will always take the HTTP header over the one in the meta tag, but it doesn't hurt to be consistent.

Handling UTF-8

It's all good to declare your pages as UTF-8, but that's useless if you're data isn't actually encoded in UTF-8. So let's talk about how to work with UTF-8 strings.

First of all, let me mention that browsers will submit forms in the same encoding as the page. So since you declared you pages as being encoded in UTF-8, all user data will be in UTF-8 too. The real work comes with using PHP to validate and manipulate user input.

The Problem

PHP is pretty much completely unaware of UTF-8 except in some specific circumstances. Most of the time, PHP treats all text as a string of 1 bytes -- that is, ASCII. This has a number of consequences due to the way UTF-8 uses a variable-length encoding scheme, and PHP's lack of functionality to properly handle it.

PHP's string functions are made to work on 1 byte characters. So counting the length of a string is as simple as checking how many bytes the string consumes, or getting a specific character by providing an offset is as simple as returning the byte at the offset. But stick in a UTF-8 encoded string, and now you have single characters that take up 2 (or 3 or 4) bytes. The result: PHP's string functions don't see a 2 byte character, they see 2 separate 1 byte characters (which is, of course, incorrect).

The solution is to write your own string functions that know all about UTF-8. You need to think about which functions will require characters to be 1 byte -- those are the dangerous ones. So things like strlen() or strpos(). Others, like explode(), only search for a series of bytes (which can be UTF-8) and use it -- so they are okay.

Here's a list of some unsafe UTF-8 functions:

• ord
• str_ireplace
• str_pad
• str_split
• strcasecmp
• strcspn
• stristr
• strlen
• strpos
• strrpos
• strrev
• strspn
• strtolower
• strtoupper
• substr_replace
• substr_replace
• [l|r]trim - Only unsafe when using the second $charlist argument, otherwise it's safe
• ucfirst
• ucwords

Libraries and Extensions

You're pretty insane if you want to go ahead and re-write all of the string functions yourself! Instead, there are a few ways you can safely handle UTF-8 strings without all the headbanging.

• iconv which is default in PHP5, is mostly used when converting various character sets to UTF-8. However, there are a few string functions like iconv_strlen() that are UTF-8 aware. Not the holy grail..
• mbstring is another PHP extension that is intended to be the holy grail. But it's not in PHP by default, which can be a deal-breaker if you produce mass-market products that'll be used on shared servers. This extension has redefined all string functions so they are UTF-8 aware, and also offers some other features.
• phputf8 (download) is a rewrite of the string functions written in PHP. It will auto-detect if iconv or mbstring are available and use those string functions where possible (since compiled string functions will be much faster than PHP string functions).

I like to the phputf8 library in all my projects. It's simple and clean and it works!

Well Formedness

It is possible that UTF-8 input can be ill-formed. This is because of the variable-length encoding. To remove any ambiguity when parsing UTF-8 there are certain bytes that are invalid. These invalid ranges are a design feature, they make it impossible for the computer to "mix-up" a two-byte character for two separate one byte characters (or visa-versa).

You don't need to really understand what I just said since it has to do with how UTF-8 is represented on your computer (read the Wikipedia article if you want to know more). All you need to understand is that UFT-8 text can be malformed, and thus, it's something you need to validate.

So when you are accepting form input, you should always make sure it's valid or else there's a possibility of various attacks. For example, imagine a UTF-F byte sequence is "AB", but an attacker modifies the request so only the first byte "A" is sent. Due to the way UTF-8 is encoded, your computer knows there should be a second byte after "A". So when you blindly output the single "A" into your HTML page, it's possible the next byte (whatever it may be) is "eaten up". Imagine if the next character is a double-quote intended to close off an HTML attribute or something like that. Possibility for XSS is introduced.

One very simple way to check for well formedness is to use preg_match() with the 'u' modifier:

if (strlen($str) AND !preg_match('/^.{1}/us', $str)) {
    die('Invalid UTF-8');
}

Since any invalid UTF-8 string will result in the match failing, all you need to do is match a single character.

If you're using phputf8, you can use the utf8_compliant() or utf8_is_valid() functions. utf8_compliant() uses the method described above, but will also pass 5 and 6 byte sequences which aren't technically valid UTF-8 (5 and 6 bytes are not unsafe, however). utf8_is_valid() will make sure the string is actually valid UTF-8 but will take longer.

Regular Expressions

When using regex with UTF-8, you need to use the 'u' modifier. That's all it takes!

if (preg_match('/myregex/u', $str)) {

}

Stick with ASCII when...

When possible. There are still a lot of places where Latin letters and numbers are the only valid input. For example, URL's and email addresses.

Also, working with ASCII will often be better performing than using UTF-8. Especially if you are using a PHP-based library like phputf8 which will be slower still then compiled libraries like mbstring at certain operations.

UTF-8 and MySQL

If you use MySQL out of the box, and PHP's mysql_* functions, then you can INSERT and SELECT data from MySQL and you're safe almost all the time. That is, you can INSERT a string of UTF-8 characters into a table that is marked as ISO-8895-1. This is because MySQL simply sees a string of 1 byte characters -- it doesn't actually know that your text is UTF-8.

You'll immediately see a problem if you use a tool like phpMyAdmin, though. Because phpMyAdmin will output the results to you in the same charset as the database table. You'll end up seeing garbled text.

Two other problems will quickly become apparent:

1. Collation. Collation is how a language is sorted. For example, in English we go from A-Z.
2. String functions. Using MySQL's string functions may garble your strings if the table charset and the data charset are different.

What you should do is make sure your tables are encoded in UTF-8, and that the default collation is UTF-8, and that your MySQL client connection is being interpreted as UTF-8.

To set the global character set and collation you can edit the MySQL configuration file:

default-character-set=utf8
default-collation=utf8_general_ci

This sets the charset to UTF-8 and the collation to case-insensitive UTF-8.

When creating a database or table, you can also supply these values:

CREATE DATABASE `test` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci
CREATE TABLE `test` (...) CHARACTER SET utf8 COLLATE utf8_general_ci

Note that specific fields can also have their own charset and collation using the same syntax.

Once you connect to the MySQL server, you should issue a query to indicate that your queries will be in UTF-8:

mysql_query("SET NAMES 'utf8'");

After that -- you're good to go.

Converting content to UTF-8

First, a note: Make sure you have enough storage for the new strings. As explained earlier, UTF-8 strings can possibly use up to 4 times the amount of storage space. This usually isn't much of a problem with content saved to disk. However, if you are storing content in a database, your fields like VARCHAR(255) may be much too small.

If you need to convert simple strings on the go, you should use iconv. For example, perhaps you need to convert an email message from Windows-1251 (common with Russian) to UTF-8:

$content = iconv('Windows-1251', 'UTF-8', $old_content);

If you are using MySQL, you can convert entire tables of content quite easily by issuing ALTER TABLE queries. Note that your existing charset and collation set on the tables need to be correct or else the translation will garble your text! (Going back to my note above about how MySQL won't know what charset you INSERT text in).

ALTER TABLE `table` CHARACTER SET utf8 COLLATE utf8_general_ci
ALTER TABLE `table` CHANGE `table` `table` [TYPE] CHARACTER SET utf8 COLLATE utf8_general_ci

Converting UTF-8 content into another character set

Sometimes you might need to convert content back into another set. For example, if you've decided to switch to using UTF-8 but your app still communicates with a legacy system that only understands ISO-8895-1.

Again, you should use iconv which is perfectly suited for this kind of thing:

$content = iconv('UTF-8', 'ISO-8895-1', $old_content);

Note that the characters represented in UTF-8 may not be able to be represented in the character set you want to convert to. For example, if you're content contained the copyright symbol, ISO-8895-1 won't have that symbol.

You have two options: You can either transliterate the characters that can't be converted or simply discard them.

Transliteration is the process of replacing the character with one that looks sort-of like it. For example, "é" might be replaced with "e". To achieve this, you append "//TRANSLIT" to the out-charset:

$content = iconv('UTF-8', 'ISO-8895-1//TRANSLIT', $old_content);

On the other hand if you just want to get rid of the offending characters, simply append "//IGNORE" to the out-charset.

Conclusion

I hope you've learned a bit about using UTF-8 with PHP. It's really not all that bad once you take in all of the information -- definitely not as scary as you thought it'd be, I bet!

Additional reading:

• WACT: Character Sets / Character Encoding Issues
• UTF-8: The Secret of Character Encoding
• Wikipedia on UTF-8 and Unicode