I’ll start off by clarifying things that were NOT the problem.

1) The problem wasn’t that we don’t have staff on hand to look at things. We have 24/7/365 support. This account is flagged such that if there is a service down issue, either myself or Daniel Greene (another of our engineers) is to be located no matter what time of day it is. This is because the two of us are very familiar with the server set up. The amount of time it takes me to wake up at 4am, walk over to my computer, and shell in is less time than it would take for an engineer who is not familiar with the site to look around as see what is going on, so that’s why I got the phone call.

2) The problem was not hardware related. The site has more than enough hardware for the traffic it is currently getting.

3) The problem was not Wordpress. There were a few issues that came up because of a couple of Wordpress plugins that are really badly written and were beating up the database, but they were identified and disabled. Wordpress itself did not cause any issues.

So, now moving on to what the issues actually were. Essentially, what was happening is that in circumstances where you had the combination of a traffic burst (say from Digg) and any slowdown in any part of the system (say from widgets) there would be a snowball effect with the apache processes. In the forking model that apache is running under, when a widget is slow to load, an apache child stays open waiting for the request to complete. During that time, the child is unable to accept any new requests, so when the next request came in apache would fork another child to handle it. Under a traffic burst situation, this would rather quickly run apache up to the maximum number of allowed children. At that point, the next connection would appear to just hang and the site would appear non-responsive or very slow. Increasing the amount of children apache was allowed to spawn to a number high enough to handle all the requests ate up all the RAM on the box so that wasn’t a viable solution.

The thing that did fix the situation was switching techcrunch.com to being served by lighttpd. I had been thinking about doing that for a while, but I hadn’t run it before on a site with these operating parameters. And, I didn’t understand its blocking model as well as I do now, so I wasn’t sure that switching to lighttpd would have fixed things.

Fortunately, the great Matt Mullenweg shot an email over saying “if you’re running apache and mod_php, you’re likely to see behavior like [exactly what we were seeing] … you should run something like Lightspeed or lighttpd.” Now, since Matt is one of the smartest guys I’ve ever talked to in this industry, I was more than happy to believe him and we switched over to lighttpd that evening.

That solved the issue because lighttpd has an event driven architecture and therefore doesn’t block on things the same way apache does. We have not had any material issues with the setup since then.

So, I hope this clears things up for those parties that are interested. If you’d like to know more about anything in particular, I can be reached at chl@mediatemple.net.

It’s never easy, and it always takes more time and money than you think it will. We’re doing this with a number of larger sites right now and everyone’s happy about it. It lets you just worry about blogging instead of all the other crap.

What a joke…I’m sorry to say!