As TechCrunch traffic continues to grow, a problem is popping up more and more often – the traffic we send to a site when we write about it on its launch day can (and often does) take it down. It’s not that TechCrunch traffic is that massive, but it’s enough that if there’s a bug somewhere in application that wasn’t noticed with small traffic testing, it can be exploited and quickly take the site down. The last week, we’ve averaged one site down per day.
Examples: We wrote about Spotplex and it went down fast, as did Amie Street and Kegulator tonight (Kegulator is more of a toy, so it doesn’t really count).
Another problem is that the traffic doesn’t last. See this Alexa chart for Spotplex as an example. There’s a spike, and then most of the people never come back. Hopefully a few stick around, register and tell their friends, but building an application to scale to handle a TechCrunch post is a long term solution to a short term problem.
Startups are noticing, too. Occasionally a new startup tells us they don’t want us to write about them yet, because they aren’t ready for too many eyeballs. We comply, of course. But I hate not being able to write about something that I like.
I’m not sure what to do about this. One solution is to start a new blog to focus on really young startups that haven’t had to deal with any traffic load, and write about them on TechCrunch later. But I like to write about things on TC before anyone else has heard of them, so it isn’t a great solution.
Anyone have any ideas?
I don’t think that I have solution for TechCrunch but I wanted to say something to the startups that their site is down because of massive traffic. First lets start with the fact that you do send MASSIVE traffic to new services that you wrote about Mike. 100K-300K a day can kill any good application. From our experience, I think it’s better (if your site is down) to write something to your visitors about it to make sure they will come back after the storm. Startups can even take down the site themselves, upload a page that says, “comeback soon” or add the button “add me to del.icio.us”, maybe even add a box to enter an email to let the users know when the site is alive. We chooses to write to people that the site is having some traffic issues because of techcrunch
and if you want to continue you can try now or come back later.
Anyway, for my opinion, if you want TC to write about you, You still need to own a good server that will handle some huge traffic.
Maybe you can gather some new startups (that can’t handle massive traffic) to one post. Every post can have 4-5 services. In that case the traffic could be more balanced.
Hi, I think a new sub-blog would be good, especially if it means more sites being featured. Some sort of intermediate blog for smaller sites would be great. Perhaps a “quick first look” at smaller sites?
To support another site would you:
a] review more sites?
b] spread the existing number of reviews over the two sites? or
c] find another writer?
Quite frankly I’m looking forward to the day when too much traffic is a problem
http://bla.st/
None of the businesses you write about need TechCrunch-scale capacity except on the day that you write about them, so it’s not viable for them to build to that scale just for one day. But you need that scale every day, because otherwise you’re writing about sites that have stopped working. Could there be such a thing as a TechCrunch temporary hosting solution that you lend/hire/make available to the sites you mention? CruchS3 or similar.
(Of course, that solves the bandwidth problem but not bugs that only come out under massive loads.)
Well, you’re talking about two different problems.
1) The spike that doesn’t stay. I don’t see this as a problem. It is only normal that at first, people are going to check the sites, but except for some very few cases, only a small percentage are actually going to become loyal from day 1. Your job by delivering the news is done, and you shouldn’t feel bad that the traffic you generate doesn’t stay up.
Also, no startup should feel bad to see the spike go down. Sure there are exceptions, but a steady large audience doesn’t get built up in a day, nor even in a month or six. I should know. I launched a site in 2000. The first year it grew to about 30,000 users. Then the second year it already had a couple of million. Today it’s thriving on an 8 digits number of *active* registered users. By today’s standards I probably should have given up after 12 months, so I don’t buy the idea that if you don’t get big quickly your idea sucks.
2) The “TechCrash” effect. Indeed I don’t think a startup should deploy 12 industrial servers for a launch when it will probably do just fine with a couple of them for the first 12 months after that first spike (not to mention those nasty bugs under heavy load that went undetected during the closed beta or whatever). But it also has to be expected. And I don’t mean by the startup but also by the visitors. And I think people are starting to understand, so even it doesn’t look good to go visit a site featured in TC or any other service that can send your way thousands of people very quickly, it doesn’t mean that the startup sucks or won’t scale. And I think that’s they key: people need to understand that.
This is not the same as building a site that won’t scale. Ideally every site nowadays needs to be acchitected so that all it should require in order to scale is to add more hardware (and at a reasonable pace too) without having to go back to the drawing board, database redesign, etc.
Bur personally, I don’t think this is something *you* need to fix.
Do I make sense?
Mike – I presume you speak with most start-ups before you give them the TC treatment, so why not advise them to offer up an additional (mirrored) version of their site, sans dynamic interface, so that TechCrunch readers who only intend to look at the site can do so via a stripped down version? It would look the same, navigation would be the same, but they wouldn’t overload the host because the dynamic interface has been cut out. The average start-up post would include two links; one for those who think they’ll use the site, and one for those who just want to take a look around, read the spiel, see the interface, but not register or use the service (I’d bet that’s a fair percentage of your readers).
Secondly – it sounds like there’s an opportunity for a ‘TC rush’ hosting service that can accommodate a start-up during their 15 minutes of fame. Not all of the down-time will be due to dynamic scripting errors, some of it will be due to unprepared hosts, so why doesn’t someone take the chance to build a business around your traffic shepherding powers?
Thirdly – why not devise a specification for TC start-ups, so at least they not what they are in for. There are hundreds of start-ups that could advise on this.
A blog for young start-ups would only end up with the same readership as TechCrunch, and in turn create the same problems. Your readers want to hear about the latest and greatest, so they’ll swamp a new blog in no time at all. The only effect this would have is diluting the TechCrunch brand.
A warning before a post is made might help. This would give the site a chance to move images to a different server for example.
Leave everything as it is. Techcrunching a service that might be a winner one day is one of the first tests a serious company must pass with grace (even if this means that the service will have to go down for some time).
TC audience are mostly early adopters that visit the site, register, take a quick look, blog about it (mainly copying Mike’s opinion) and then never come back! So for the service to go down, it does not mean that it will loose a valuable audience!
By the way… Handling TC with grace is the necessary step for all serious services, considering they would have to deal with digg or NYtimes (…or other heavy traffic sites like engadget) one day. They have plenty of time to prepare for that after the TC post! No need to start with a dozen servers!
As per Tim, *most* cases of site meltdown are too many spawned apache processes usually because of images hosted on the same server. Bandwidth saturation is another problem.
Here’s a good article that every site should employ:
http://www.codinghorror.com/blog/archives/000807.html
This sounds like a new metric to me. If you crash on “techcrunch” day, how much does that increase a site’s odds of hitting the deadpool.
Jedi ninja designers (with enough warning) should be able to keep a site up even if it is just the user/signup system. Sites that are not optimized well from the beginning may never scale well.
I am not knocking anybody that has recently crashed. My company’s sites have certainly been knocked off the planet by nasty slashdot/digg combos in the past.
Still if a project well handles the techcrunch stress test, I bet they’ll have a better chance of survival.
IMHO.
I don’t think a new blog is needed to staunch the flow of noob blood over your hands, Mike. I agree with TechCrunching being a worthy test: if a startup fails it then it’s the founders’ fault, not yours. However, a quiet word in the shell-like of the founders before you post the story wouldn’t go astray.
When appropriate, send your early betas to Scripting News, where our readers tend to get more involved with the sites they discover, and the numbers are much smaller and easier to work with. Some sites would do better to be launched there, for example the new My.Netscape, that got a cold reception here and elsewhere, but was greeted as a familiar friend on Scripting News. I look at a much smaller number of sites, over a much longer period of time.
How about doing a screencast as you play around with it and posting that so people can get a feel for the site even if it goes down? Or just take more screenshots? Since most people don’t go back to a site after it’s crunched, a preview would satisfy most people’s curiosity. If they like what they see, they’ll revisit the site later when it’s back up.
Michael,
I can understand the frustration of the effect you blog bring to any new startup, But you do need to remember that most of the visitors you send to theses sites are early adopters, most of them wont remember or stay in the site.
Many times when i end up using a site or a service, I remember the first time i was introduced to this service/site and i can say that sometime, it’s been a long time before i use it. Therefore i dont think that the effect has a negative effect, most of the site would lose only a tiny fraction of registered users/visitors when their site is going down.
On the positive effect of the techcrunch effect, they experience their first load on the servers, and they do find bugs that would only appear when their services would start to become successful. It would be better for them to crash at the begining while being unknown than crash when they start to be a widely used service.
Maybe a smaller blog with a different approach such as giving them advices, build a community around in order to provide the advices and the experience some startup need at earlier stages might be a good idea.
I know for myself that i started to work on a project of myself 4-5 months ago, we are launching some kind of pre-alpha in 20 days limited to 20 users and we wont open it to more people till we feel that it has a working interface and user-friendly. I do think that looking at the past attempt i had, the first 2 months were a waste and i could have progressed faster if i had the right mentorship.
I did read lot of good stuff on the Internet, and one of them was to start reading daily your blogs amongst other (john battelle, orly’s, guy kawazaki and 20-30 others) in order to have a perception of whats going on in the world since i live in Israel.
Anyway,
Thanks for the blogging experience.
I would say you can still write about them, but do so without linking directly to them. Taking away that easy link is going to deter a lot of readers, perhaps enough to give the site a chance to stay up. You can write about them and include screenshots, but stay away from including the hyperlink. That way, people who really want to check it out will find the link from a search engine, while that effort will be enough to deter a lot of people from visiting the site. They’ll get a jump in traffic, but hopefully not enough to kill the application.
Coming from a startup founder himself – you can’t have your cake and eat it too. Don’t change a thing.
Crashing sites is what has made Slashdot, Digg, and even Techcrunch, popular sources of information. It’s part of the game.
If you know a site can’t handle the traffic, do a pre-launch/early write up showing screen shots and features. Then you can link to a static page for signups or not link to them at all. If it sounds interesting enough, I’ll type in url myself.
Mirror the home page, or at the very least post a full size screen grab of the home page.
Personally I have no interest in 99% of the sites reviewed because, let’s be honest, 99% of them are ugly.
I’ve just no interest in ugly sites. Not to mention, ugly or not, I tend to judge the usefulness of any site off merely a passing glance.
I’d be willing to bet that alot of your readers are the same way. As such, I’d bet that alot of the traffic you’re sending to these sites doesn’t make it past the home page.
So easy solution is to be consistent about screen shots. This is something that Cashmore has done well that I really appreciate. It should at least lessen the amount of “junk” traffic that these sites are getting.
Don’t change anything! If you put your stuff on the web, you’re inviting comments/traffic. If they don’t want the traffic, they should not make the site public. Internet = public.
Hey Mike,
If you change anything at all, why not ask the start up if they can handle a link from your post? If they can’t handle typical TC traffic, still write about them, but avoid hyperlinking. Make TC readers interested in the product type out the URL. That would cut traffic, but not all of it.
Speaking as a company that was recently mentioned on your site I have a couple of comments. I don’t think this is a huge problem, but it could be handled better with some effort on both parties.
A little more coordination would be nice. Naturally, this won’t work for sites like Digg, but this could work for TC and /.. Knowing that the rush is coming before the posting would be helpful in several ways. For us, the posting came the afternoon of the Super Bowl, so you might image we might have been away from the net. As it turned out, we don’t watch football so we all happened to be online. Also, knowing that a flood of traffic is coming would be helpful in terms of releasing new code. If we knew a bunch of new traffic was heading our way, we might hold off that next release of green code until the flood subsides.
The sites themselves need to take some responsibility for getting their sites into decent enough shape before opening up to the general public. We kept our signup process very limited until we had some confidence that we could handle a flood of users. If you can’t handle a bunch of users, what’s the point to having a site? Especially one that thrives on traffic! Having a backup plan is important as well. You should have a safety valve that allows you to switch your site back to ‘invite only’ or something similar. It’s not as good as having a live site, but it is way better than a 404. Plan like a business, not like a hobby.
As far as the traffic spike is concerned, we were glad to see it. We know that it’s a special crowd, mostly early adopters where few will stick – we surf the same sites. But for a new site, that spike is very valuable. First, you get to see how your site does under some load with a pretty forgiving audience. This was interesting for us because we were able to see how much bandwidth per user was consumed vs our projections. And second, we were happy to have the eyeballs, even if they weren’t all that sticky. We were hoping to retain 5% of the spike, I think we’ve done better than that.
How about a follow-up post? Mabye 1-2 months after the initial post. Maybe have a special “where are they now” section where you could do a quick blurb with the link and a top level wrap up of what has happened since the initial post. You can start with us. After our next release.
As the founder of a site that was Crunched and crashed, I’ll weigh in.
Feed Crier was featured before we were ready for the traffic. We’d done some early testing with a few users and opened up the service to the public quietly hoping to pick up a new user a day or so. The slow initial growth would help us gauge what still needed doing. We were in a learning phase, still figuring out how to scale (there’s not much info out there on how to scale instant messaging bots).
TechCrunch found us about 12 hours after we took the wraps off and wrote us up on a Saturday night. I wasn’t concerned — we were only picking up about 15 visitors an hour from TC. Very early Monday morning I got on a plane for Seattle, and when I got off, I found that the bot had crashed several hours previously and all that TC traffic was for nothing. Bummer.
In our case, it was a bug that crashed the bot. It didn’t handle subscriptions to URLs that didn’t exist very well. We knew that, but our early tests were controlled enough that it wasn’t a big concern right then.
I don’t know the right solution for your end. A separate site won’t help — you’ll end up with a similar readership and similar traffic patterns. Asking or warning each site before you write about them may help a bit, but until the traffic hits they probably have no idea if they’ll be able to handle it. Trial by fire. If you take the time to contact each site first, you also may lose out on scoops due to the delays you’ll be imposing in your publishing.
My advice to startups is to grow your load carefully if you are uncertain that you can handle it. Manage the signup process by placing limits on active users, or even blocking public signups altogether. Take names and contact info of interested parties and then give them access in smaller groups. Putting up that gatekeeper will reduce your signups, but most of the users you’ll lose are the drive-by signups anyway. Most of the seriously interested parties will stick around.
And if you were one of the ones that tried Feed Crier when it launched, come by and check us out again. We’re stable and have a lot of new features, including support for Google Talk, Jabber, and MSN IM networks in addition to the AIM support we launched with.
Why not reveal the story to different user locales at different times? Or ip ranges? The zones wouldn’t even need to be that much out of sync; it is the simultaneous load that is the killer.
Mike, have you considered giving startups couple hrs heads up? After we were TC’d, we had another box setup and running two three later.
It is a SUPER bummer for a startup to get TC and not make use full use of it.
As for user retention, for sites that run a consumer service(not b2b), they shouldn’t depend on TC for their core user-base. TC helped us most with biz dev leads that continue to come even two months later.
Not sure about starting another site for startups. There is a wierd attraction with the TechCrunch.com domain and the mix of stories featured on it.
-Zaid
Mike,
Don’t change anything!
People read TechCrunch because you get the first scoop. If you start waiting for companies to tell you when they’re ready then 1) you could lose the scoop to a less scrupulous blog and 2) you become little more than their startup PR company!
Getting “Tech-crashed” is a great problem to have. I don’t lose any sleep feeling sorry for those companies!
Jon
I would definitely follow TechCrunch Jr. or a Pre-Crunch blog.
Encourage startups to build, where possible, on a combination of Amazon Web Services which will scale.
Thus if they have a bug which means that database performance is much lower than it should be, they will only pay more. Their site won’t go down.
A TC baby blog isn’t a great idea – TC is supposed to cover early stage startups.
What if you offered something of a testing service to startups who aren’t quite ready for the big time…some sort of mailing list. If a company is nervous about traffic, you could offer to write about them to a special, smaller group of people who would be willing to hit the site a few times, dig around, and help see if anything breaks. Once they get all the kinks out, you could re-post what you sent out to the mailing list to the main blog.
You still risk someone forwarding the email about a new site on before they are ready for prime time, but if the group is small enough and targeted enough, perhaps you could get compliance.
the ideal solution, of course, is to be able to switch the app load to any number of servers, dynamically – a la VMWare – maybe in concert with Amazon/etc. how cool would that be?
sounds like a startup to me.
After the /. effect, now the “TC effect”. Great
http://lucafiligheddu.blogspot.com/2007/03/tc-effect.html
ciao
Do posts late at night so that only people certain timezones are likely to see it first. This way the traffic hits a site over a matter of hours and not a single hour…
I echo the earlier suggestion that writing about a company without providing a real link would actually help in some of these cases by cutting down on the visits from people who are not interested enough to type in the url.
Otherwise, aside from warning the sites, there’s nothing you can do. Sites that are warned in advance could put up some information about themselves but shut off user registrations and then gradually send invitations to people who left an e-mail address over the next week.
1. KEEP it as is.
2. AVOID any kind of paid “mirror service”: it will sound like a blackmail: “I’ll write about you, if you don’t wanna crash, pay me the mirror”.
3. AVOID any kind of free “mirror service”: it’s a good test for the company and a huge hit for your bills.
4. ADD SCREENSHOTS. Screens are very useful to have a first glance to a website, even if it’s down. Make them a first-grade citizen of your reviews.
5. ADD a “mail me when it’s back online” feature. This is what I prefer. Sometimes I read an interesting review, I find the web down and then I forgot about it in a few minutes.
It will be interesting to have this feature, that designed in this form will also allow some form of throttling and statistics even for the startup you mention.
The check of the up status could be done both automatically (by a bot) or manually by the site owners, through a panel that “releases” a # of mails.
…mmm… I think I’ve to suggest this one to Digg too
The TechCrunch Reference Architecture
Build using Amazon EC2 and S3.
Use a load-balanced architecture
Add EC2 nodes when you go live – as many as you can…
Alert TechCrunch
Wait for mention (pray for mention)
2 days later start reducing nodes…
It might cost you a few hundred dollars for the week of servers (an EC2 node is $72/month – you can get 13 for less than $1000).
Perhaps someone will build a TC simulator you can run for $100 before launching to see how it scales (maybe something from Keynote or similar service).
You need a new TC site feature. A “Subscribe to this company” button. Fill out a TC form once and TC will autosubscribe you to that company’s alert’s for six months. Get a custom rss feed blending alerts (“Hi, we’re out of alpha”) from all your subscribed firms. This gives TC a source of good data on what people are following, TC’d companies a chance to communicate on their own timetable, and lets folks scan TC without wasting three minutes on a site before it’s ready.
I’d say maintain your attention on your readers first and foremost. Breaking fresh startups is an important part of TC, so relegating it to a corner seems imprudent. And getting into the hosting business seems like an incredible distraction from journalism.
You can be courteous and give notice to start-ups as early as feasible. If startups want you to wait on a story, fine, then secure what you can. But if you add more screenshots to the bottom of your articles, readers will feel informed even if they can’t check out the site.
Serve the reader. My two cents anyways (and I’m a start-up guy, for what it’s worth)…
i agree with George above. it is a right of passage
Yes, serve the reader. In the end they still get a permanent link from and write up from TC, which is pretty important!
Give them a warning if you can, but don’t delay your post for them. This is life on the net.
I think this is an excellent idea by Phil Wolff,
“You need a new TC site feature. A “Subscribe to this company” button. Fill out a TC form once and TC will autosubscribe you to that company’s alert’s for six months.”
Exactly as I said above, Geoff.
But, from my point of view, I don’t want to be informed for 6 months, for a service
I want to be informed when it’s back ONLINE.
Then, if I’m interested, I’ll subscribe to the company, not to the TC website.
Six months of updates from a website that, after my first visit, I’ve evaluated as unuseful to me?
A simple, one line form for just the e-mail will be enough. With the assurance that my email will be deleted after I’ve received the only update I wanted.
keeping users is the startup’s problem.
dealing with “TC effect” could be handled several ways:
1) don’t do anything. they need to learn to scale
2) use/setup a service to send 1/x of traffic to the site, and direct (x-1)/x to page that says “please come back in (10*x) minutes to check out the site”
3) tell startup to setup a page that does equivalent of #2, basically using time / email invite to do load balancing.
1 is easiest. 2 or 3 could be done by startup with a basic service setup, or a home page with an email invite list that spreads out the impact over time.
- dave
With our site going live within 10 days, this article is very timely. Our site is a weekly internet video show, that is utilizing flash video. Our content is quite unique and hopefully will get strong initial viewership to scale our backbone.
We dropped the coin and have a kick ass server and VM mirrors ready for the abuse.
We have our signup options read…
PS if you stop linking to sites (as some people have suggested) someone will write, within much less than 24 hours, a Firefox plug-in that takes people to the sites you review in one click. Just saying.
A couple of people (luca and Phil Wolff) have mentioned using Amazon Web Services’ EC2 to solve this problem. We have found that EC2 is a viable solution. In addition, solving this problem has led us to an effective solution to scale any of our server requirements for cost efficient use on an hourly basis. No longer do we have to buy servers in anticipation for spiking traffic, or even routine cyclical traffic, for EC2 offers a truly scalable solution for optimizing the number of servers we need on a minute to minute basis.
We had to overcome some critical issues such as dynamic IP addressing coupled with a lack of a 24/7 service level agreement (losing an AMI and all web services also means you must also repopulate the global DNS tables with the new “A” record). However, we have solved these issues for our own application at Weogeo, and on Monday we will begin to offer this EC2 management AMI, called WeoCEO, as a private beta product for developers of AWS applications.
The WeoCEO application, working within the EC2 environment, eliminates the above mentioned limitations, and maximizes the power of EC2 by providing automatic and instantaneous scaling, load balancing, and fail-safe supports, including a stable IP environment.
I referenced luca and Peter Wolff in the previous comment. I misread the authors. Peter and Crunchback mentioned an EC2 solution.
What is very telling is the fact that the traffic is so fleeting and ‘does not stick about’. This is not at all surprising given that the vast majority of TC readers are startups also and are interested solely in the competition and personal site exposure.
I’m sure most sites are quoted an amount of bandwidth they would never use up and therefore the web host does not need to actually support this maximum, it is really a choice of the right host in this situation
-
http://www.windowsvistauserguide.com/glossary_extension/?q=web_hosting
Ok, firstly, I havent had time to read all of the comments above, and that is rude.
Ok, hows this for an idea. Have a section where you put these articles, and they are randomly shown (it will show 1 of the top 5 articles), so that not everyone sees them at once, and new people continue to see them for the next few days, evening out the load.