Twitter's Data Center Migration Story on its 5th Birthday

There is lots of news out there on Twitter's 5th Bday. Twitter launched a new website and video.

Twitter Launches New Website & Video To Celebrate Its 5th Birthday [VIDEO]

In celebration of its fifth birthday, Twitter has launched a new website and a new video featuring some of its most prominent users.

Twitter employees have started tweeting links to discover.twitter.com, a website that features 16 different users from a variety of backgrounds, including entertainment, political, business and even astronautics. The list includes tennis star Serena Williams, Virgin Group founder Richard Branson, rapper Snoop Dogg, U.S. Speaker of the House John Boehner and astronaut Paolo Nespoli, among other recognizable celebrities and institutions.

What will be missed by most is the data center story by the Twitter engineering team.

Twitter Engineering

MONDAY, MARCH 21, 2011

The Great Migration, the Winter of 2011

If you look back at the history of Twitter, our rate of growth has largely outpaced the capacity of our hardware, software, and the company itself. Indeed, in our first five years, Twitter's biggest challenge was coping with our unprecedented growth and sightings of the infamous Fail Whale.
These issues came to a head last June when Twitter experienced more than ten hours of downtime. However, unlike past instances of significant failure, we said at the time that that we had a long-term plan.

Here are a few nuggets that jump out.

Once all the data was in place we began serving live traffic from the second data center for end-to-end testing and to continue to shed load from our primary data center. Confident that our strategy for replicating Twitter was solid, we moved on to the final leg of the migration, building out and moving all of Twitter from the first and second data centers to the final nesting grounds. This essentially required us to move Twitter a second time. Yes, that’s right, we moved all of Twitter twice!

Twitter discusses three data centers.

First, our engineers extended many of Twitter’s core systems to replicate Tweets to multiple data centers. Simultaneously, our operations engineers divided into new teams and built new processes and software to allow us to qualify, burn-in, deploy, tear-down and monitor the thousands of servers, routers, and switches that are required to build out and operate Twitter. With hardware at a second data center in place, we moved some of our non-runtime systems there – giving us headroom to stay ahead of tweet growth. This second data center also served as a staging laboratory for our replication and migration strategies. Simultaneously, we prepped a third larger data center as our final nesting ground.

Another released pieced of information is Twitter's expected employee growth from 350 to 3,000 in 2 years.

Now headquartered in the South of Market, Twitter is eyeing a move to Brisbane. Twitter CFO Ali Rowghani informed city officials last week that if the proposed tax exemption were approved, the company would move into the vacant Furniture Mart on Market Street just east of Ninth Street. It projects growing its current workforce of 350 to 3,000 by 2013.

Visitors Guide to Silicon Valley

Silicon Valley has the highest concentration of data center decision makers and innovation, and I am sure most of your take regular trips to Silicon Valley.  I am heading down today, and I’ll be back for The Green Grid conference the following week.  Going to the bay area is almost as easy as a bus ride, a 2 hr flight.  This trip I am taking my kids to see grandma. Being raised in Cupertino/Saratoga getting around is not a problem.

Steve Blank writes a post that would be useful for those who want to see some highlights of Silicon Valley.  One area that Steve hits the mark and why I go to the bay area so much for clients and friends.

The Valley is about the Interactions Not the Buildings
Like the great centers of innovation, Silicon Valley is about the people and their interactions. It’s something you really can’t get a feel of from inside your car or even walking down the street. You need to get inside of those building and deeper inside those conversations. Here’s a few suggestions of how to do so.

…

Steven has humor in his post.

Hackers’ Guide to Silicon Valley
Silicon Valley is more of a state of mind than a physical location. It has no large monuments, magnificent buildings or ancient heritage. There are no tours of companies or venture capital firms. From Santa Clara to South San Francisco it’s 45 miles of one bedroom community after another. Yet what’s been occurring for the last 50 years within this tight cluster of suburban towns is nothing short of an “entrepreneurial explosion” on par with classic Athens, renaissance Florence or 1920’s Paris.

 

And, pokes fun at many aspects of the valley.

A Visitors Guide to Silicon Valley

Posted on February 22, 2011 by steveblank

If you’re a visiting dignitary whose country has a Gross National Product equal to or greater than the State of California, your visit to Silicon Valley consists of a lunch/dinner with some combination of the founders of Google, Facebook, Apple and Twitter and several brand name venture capitalists. If you have time, the President of Stanford will throw in a tour, and then you can drive by Intel or some Clean Tech firm for a photo op standing in front of an impressive looking piece of equipment.

The “official dignitary” tour of Silicon Valley is like taking the jungle cruise at Disneyland and saying you’ve been to Africa. Because you and your entourage don’t know the difference between large innovative companies who once were startups(Google, Facebook, et al) and a real startup, you never really get to see what makes the valley tick.

If you didn’t come in your own 747, here’s a guide to what to see in the valley (which for the sake of this post, extends from Santa Clara to San Francisco.) This post offers things to see/do for two types of visitors: I’m just visiting and want a “tourist experience” (i.e. a drive by the Facebook / Google / Zynga / Apple building) or “I want to work in the valley” visitor who wants to understand what’s going on inside those buildings.

 

Did coal power in SLC scare the Twitter bird away?

DataCenterKnowledge reports on Twitter looking at Sacramento.

Twitter Scouting Sites in Sacramento

December 15th, 2010 : Rich Miller


What’s the latest on Twitter’s data center expansion? The company’s not saying. But we’re hearing that Twitter has been scouting data center space in Sacramento, Calif. What does that mean for Twitter’s announced plans to open a new facility in Salt Lake City? It’s not entirely clear.

Rich Miller tried to ask more.

Opening “Later This Year”?
In July Twitter said that its new facility would be located in Salt Lake City and open “later this year.” With 2010 drawing to a close, we touched base with Twitter spokesman Matt Graves and asked whether the Salt Lake City data center project was on schedule, or whether the expansion had been postponed or shifted to another location. “We’re still not commenting on our data center,” Graves wrote in an email.

That’s consistent with Twitter’s practice with most data center inquiries. So we don’t know for certain whether the company’s interest in Sacramento represents an additional expansion, or a rethinking of its announced plans.

The Data Center World is so small it is hard to keep secrets.  A little bird told me four months ago that Twitter was looking at Sacramento and the reasons we speculated on the possible reasons for change.  Given it has been over 4 months I would expect Twitter has already made its decisions by now even though there is now a whole lot of activity base on DataCenterKnowledge breaking the news.

One thing we'll probably never know for sure is whether the 100% coal power in SLC part of what scared Twitter to look at other locations.

Read more

So a blogger walks into a bar with a group of Super Angel investors

Last night I met a friend at Rosewood on Sand Hill Road and it was entertaining watching the VCs and social networking.

The bar area was packed with the folks from Sand Hill Road - VCs, I Bankers and a few cougars....

For those of you who don't know what a cougar is.

What is a Cougar?

Cougars are Women in their PRIME: independent, sexy and wildly successful. They enjoy men that are youthful, fit with the same zest for life. Cougars are classy, confident women that already possess many of the finer things in life — but now want the young, hot guy to go with it.

What is a Cub?

To snare a true Cougar a man needs to be youthful, fit, unintimidated and of course sexually driven! These men can range from athletes to intellectuals, and from technologists to entrepreneurs and all points in between; they can come in all shapes and sizes, but one thing they have in common is the desire to possess a sexually charged older woman.

But, to get serious discussions done, you don't want Cougars in the scene.  Michael Arrington of Techcrunch writes on a discovery he made going to a bar where the super elite angel investors were meeting.

So A Blogger Walks Into A Bar…

Michael Arrington

alt

Yesterday I was tipped off about a “secret meeting” between a group of “Super Angels” being held at Bin 38, a restaurant and bar in San Francisco. “Do not come, you will not be welcome,” I was told.

So I did what any self respecting blogger would do – I drove over to Bin 38, parked my car and walked in.

And here is what he discovered.

This group of investors, which together account for nearly 100% of early stage startup deals in Silicon Valley, have been meeting regularly to compare notes. Early on it was mostly to complain about a variety of things. But the conversation has evolved to the point where these super angels are actually colluding (and I don’t use that word lightly) to solve a number of problems, say multiple sources who are part of the group and were at the dinner. According to these souces, the ongoing agenda includes:

  • Complaints about Y Combinator’s growing power, and how to counteract competitiveness in Y Combinator deals
  • Complaints about rising deal valuations and they can act as a group to reduce those valuations
  • How the group can act together to keep traditional venture capitalists out of deals entirely
  • How the group can act together to keep out new angel investors invading the market and driving up valuations.
  • More mundane things, like agreeing as a group not to accept convertible notes in deals (an entrepreneur-friendly type of deal).
  • One source has also said that there is a wiki of some sort that the group has that explicitly talks about how the group should act as one to keep deal valuations down.

At least two people attending were extremely uneasy about the meetings, and have said that they are only there to gather information, not participate.

So what’s wrong with this?

Collusion and price fixing, that’s what. It is absolutely unlawful for competitors to act together to keep other competitors out of the market, or to discuss ways to keep prices under control. And that appears to be exactly what this group is doing.

There are many things that occur behind the scenes.

Back to the Rosewood story my friend says the valets are tipped to check out the bar call/text the Cougars what VCs (cubs) are in the bar, and they know whether it was worth the trip.  When I left the bar after my meeting at 7:30p the bar was 50% women.  Doesn't sound like the data center conference bar scene.  :-)  Off to Data Center Dynamics Chicago and Data Center World LV in the next two weeks.  But, one more trip to a VC today for lunch.

BTW, my friend and I are not Cougar targets (cubs) as we are both over 50, our wives are good friends, I am sure I'll hear it when my wives friends read this post, and the dumbest thing I could do if I wanted to keep this secret is to blog about a VC cougar bar.  But, it does make for entertaining reading.

Read more

Data Center Hunter or Harvester/Farmer, looking for customers

In the data center industry there are many people who enjoy game hunting.

And a dominant method to find customers follows a hunting methodology as opposed to a harvester/farmer approach.

Here is an article that talks about the Hunter vs. Harvester approach.

In working with business owners and entrepreneurs over the years, I’ve noticed that when it comes to acquiring new customers, most of them are hunters. They pounce on new leads, chase the prospects, make themselves readily available to the prospect and then bend over backwards to land the new customer.

On the other hand, I’ve noticed that the most successful business owners and entrepreneurs take a different approach to customer acquisition: they are harvesters. They gather in all their leads, work hard to prevent any from slipping through the cracks, cultivate those leads and then harvest them when the time is right for the customer.

The most interesting thing about these two styles is that the hunter usually gets tired, a bit humiliated and ends up getting small margins. On the other hand, the harvester stays fresh, confident and usually earns higher margins.

I just saw this post on DataCenterKnowledge post on Data Center planning, and got me thinking about hunting vs. harvesting.

How to Avoid Data Center Planning Mistakes

September 8th, 2010 : Kevin Normandeau

Why do so many data center build outs and expansion projects fail? This white paperfrom Lee Technologies addresses this question by revealing the top nine mistakes organizations make when designing and building new data center space. It also examines an effective way to achieve success through the Total Cost of Ownership (TCO) approach.

One person may think this is harvesting, but I think it is more like hunting.

Here is an example of what of what I think as a harvesting/farmer approach. 

I've been watching my top 5 data center construction companies post.  I get about 50 hits a week - every week for the past 8 months.  I am amazed there are 470 keywords that point to my post.

image

And here is an example of this last week.

image

Looking at the ISPs the list shows the following companies - Capital One, GM, global crossing, JP Morgan Chase, Network Appliances, and Yahoo besides a long list of ISPs.

Here are the top 10 cities for this week.  Note, the ability to look at ranges of time to see what cities the customers are in.  If I looked for 8 months, I would get the every major city.

image

A group of people in Cleveland are looking for a data center.  Who?  This is obvious one.

Capital One Partners

Place page

1300 East 9th Street
Cleveland, OH 44114-1506

Pretty cool I can do this research from a blog post and Google Analytics!

Here are the top google keywords used to find my post.

image

If you were thinking like a Farmer/Harvester you would be figuring out how to reach the customers who are looking for these keywords in the cities I listed.

It is common for data center vendors to pay over $10K for a booth at a conference and  maybe buy a speaking spot in front of as few as 2 dozen people.  But few people think like a harvester and prefer hunting.

Not only that, but hunting for business is tough stuff, even for those who manage to make a living at it.  When you’re in “hunting” mode, you’re dialing for dollars; you feel resistance at every turn; rejection is common; you get “price shopped” against competitors so margins are thin; and you waste tons of time working with prospects who simply aren’t ready to buy.

Seems more efficient to be a Harvester.

On the other hand, when you’re in “harvesting” mode, you’re working smart and scooping up sales left and right. You’re like the fisherman with the irresistible bait, drawing your prospects to you. You can spend your time closing deals on the phone with hot leads or go out on the golf course because you know your prospects will call you when they’re ready to move forward.

And, now that I think about it, the data center people I enjoy talking are Harvesters, and funny enough many of them enjoy game hunting.

Lead warming is about communicating with your prospects from the moment they express interest and then if they don’t buy right away, that’s OK because you then don’t let them slip away and instead breadcrumb them with information they’ll find valuable about your product, service or company

If you think you want to be a Harvester and want to leverage my post you can drop me an e-mail dave@greenm3.com.  The easiest thing to do is to drop an inline advertisement in my post, and you'll have 50 eyeballs a week.  :-)  But, there are many more interesting things to try to be a data center harvester/farmer.

Read more