Preparing for the Unexpected

This past week, Michael Jackson showed how the Internet has evolved over the past ten years into a truly global communications medium. News of his untimely passing last week drove millions not to their televisions or radios, but rather, their keyboards and cell phones. In a way Jackson’s status as a global celebrity, combined with the power of the Internet, ‘flattened’ the world, and on June 25, we, as a global community, together experienced the news of his passing .

The effect on the Internet infrastructure was experienced far and wide. Biz Stone, co-founder of Twitter, told the L.A. Times that “this particular news about the passing of such a global icon is the biggest jump in tweets per second since the U.S. presidential election.” And Google Trends rated the relevance of the Jackson story as volcanic.

For Limelight Networks’ customer TMZ.com, being the first to break news meant a flash crowd of millions of new readers, as well as a web of referring links and traffic from sites referencing the original story. In this extreme situation, their site had to perform its best and serve the story to all.

UPDATE 1: Hitwise is reporting that traffic to TMZ reached a 3 year high, with visits increasing 5x in volume from the previous day. TMZ was 60th among all websites on Thursday when ranked by the market share of visits, up from 305th on Wednesday, June 24th. Click here to see traffic charts.

In a June 29 radio interview with Ryan Seacrest, TMZ.com founder and CEO Harvey Levin explains: “It seemed to me that social media [users] were keying into what was real, in some ways faster than traditional media…And so, the story itself brought down part of the Internet…Twitter and Facebook were having huge technical problems. In pop culture, it’s a whole different world now.”

What kept TMZ up and serving their story to the world was a result of their site’s ability to scale on-demand to absorb peak traffic, thanks to proper planning beforehand by the TMZ IT team. There are lessons here from which all site operators can learn and benefit.  These lessons are put into practice for our customers everyday by our professional services organization. As General Manager of this group, Dave Burkhardt and team have helped customers prepare their infrastructure for some of most extreme, high-trafficked events on the Internet, including the 2008 Beijing Summer Games and the Obama Inauguration. We asked Dave to share some best practices for making your site available during extreme conditions, and here are five points he provided:

Build a capacity plan. “Creating one in your early stages will save you time and money later on. Hence, having a thoroughly vetted capacity plan will help expedite technology expenditures when your traffic starts to expand. When building out your plan, make sure you include all layers of your infrastructure. Start from understanding the limitations of your data center and then work your way up the technology stack, while focusing on the most common bottlenecks like storage I/O, CPU, or databases,” says Dave.

Do a health check of your systems. Burkhardt explains: “Ensure your origin architecture is performing optimally. If you are using a CDN, a poor performing origin can limit the full benefits and the ROI associated with having an outsource partner.”

Offload popular content from your origin. “When traffic spikes occur, some of the ways outages can be avoided is by ensuring that your most popular content is not being served directly from your origin, “ explains Dave. “Additionally, many sites during high growth stages will experience difficulties around delivery of their images or thumbnails. Offloading services to a CDN can assist you greatly with your ability to scale and absorb traffic spikes.”

Remember the end-user. “Many companies we work with optimize their internal systems, but still hear complaints from their end-users. Why? They forgot to look at the last mile. Using third-party performance measurements to understand your end-user’s experience is critical, as is using those tools to tune ongoing performance.”

Make monitoring and reporting a constant, ongoing discipline. “Continually track, review, and update your systems and application instrumentation. These efforts will help to ensure that you have a meaningful and accurate capacity plan developed, and show you how you are tracking to that plan,” notes Dave.

Limelight Networks offers a full suite of professional services that can help companies plan for unexpected traffic spikes. If you’d like more information, please email us.

As a side note, there were some erroneous reports that the TMZ site went down, but those were corrected later on in the news cycle. For example, an AOL representative told TechCrunch (see Update 5 at the bottom of the page) “our internal records show that the site didn’t experience any interruption due to traffic. It’s possible that some people may have had trouble accessing the site due to local network issues, but TMZ was not down.”

This entry was posted on Monday, June 29th, 2009 at 3:51 pm and is filed under Content Delivery Network. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.