The Web 2.0 World Built By Memcached

Whether you like the term "Web 2.0" or not, everything it implies — the Web as a platform for user-generated content, information sharing, collaboration — would not be possible without one cornerstone technology: Memcached.

Think about it.

Memcached was released in October 2003 by Djanga Interactive to scale LiveJournal. Tim O'Reilly and John Battelle presented their definition of Web 2.0 at the first Web 2.0 conference a year later (their definition is published here).

Every Web business started since then has been built for Web 2.0, and they've done it en masse on the back of the LAMP+Memcached stack (or RUBY+Memcached or JAVA+Memcached... you get the picture), which provided a massive cost advantage over the old "Web 1.0" way; that is, with big, monolithic, expensive databases architectures and transaction servers from Sun, Oracle, BEA, and so on. LAMP (and its siblings) provided a low-cost, efficient development foundation for Web 2.0 but did not free us from the monolithic, vertically oriented, "scale-up" platforms. Memcached provided the heavy lifting in terms of horizontally scaling ("scale-out") on non-monolithic SMP server architectures from Intel and AMD. Put simply, Memcached is a foundation for economically scaling Web 2.0.

Which brings us back to my initial premise. Without Memcached, Web 2.0 would not be possible. The original Memcached developer, Brad Fitzpatrick, noted in a post back in 2003 that "Most of the site [LiveJournal] now runs from memory and doesn't get anywhere near a database with slow, moving parts (disks)." More recently, Evan Weaver of Twitter echoed,"Everything runs from memory in Web 2.0." Now, as all eyes turn towards the cloud, I think it's fair to say that "Everything runs from memory in the cloud." (Check out our upcoming webinar on using Memcached in the cloud: "Memcached: Making the Cloud Scalable for Web 2.0")

Here's what I mean by "runs from memory": Very high read loads typically seen by dynamic web applications are not possible to sustain if the application has to go back and forth between the database, the application and the disk before sending a personalized result back across the internet. Well... they could be sustained but it would be way beyond the point that you get bored and simply close your browser. Without Memcached, users would have to wait many tenths to tens of seconds just to view the status or thumbnails of their friends on any reasonably sized social network (e.g. Facebook, myYearbook etc.). And the Web 2.0 developer can't rely on Akamai (or pick your favorite CDN) to rescue her either! CDNs are great for large or static information that are accessed by millions of users globally. However, everyone's most recent status and profile are not going to be cached on a CDN in Kuala Lumpur (especially given the number of status updates I see). Personalized (i.e. "long tail" in Web 2.0 parlance) dynamic content is not the type of content that gets delivered via a CDN.

What makes Memcached the linchpin in the Web 2.0 architecture? Memcached is the most effective, language agnostic and widely used on-ramp to memory there is. You can build a perfectly good site on LAMP without Memcached and run it smoothly -- as long as your customers aren't doing too much with it. But if your site is dynamic and its users depend on user-generated personalized content, without Memcached your site will begin to experience severe performance degradation and eventual downtime as it attempts to keep up with user requests. The only other alternative is to build on a monolithic architecture using big iron. That constraint RULES OUT 90% of all Web 2.0 sites. There would be no Twitter, Flickr, Digg, myYearbook, Glam or Facebook because the developers that started these sites could not have afforded the big iron these sites would have required without Memcached. VCs would not have funded the majority of these sites before they had hundred of thousands to millions of users. Memcached allowed these sites grow organically on industry standard servers and LAMP.

Let's look at some Web 2.0 sites: Craigslist keeps all its pages in Memcached so that all page render times are instantaneous. Flickr uses Memcached to offload, simplify and scale their MySQL cluster. Twitter and Glam use Memcached to scale their messaging and event tracking services. Fotolog cached everything from non-deterministic data to session data to scale its site from 300k users to 11m users in less than three years. Without Memcached, these sites could not have scaled without breaking the bank.

Memcached has proved the quickest and least expensive onramp to memory over and over again. That simple fact has allowed an explosion of Web 2.0 sites and services we could never have imaged. It has allowed creativity and diversity to rule in Web 2.0 before having to worry about monetization (look no further than Twitter's example for that).