Why Twitter dedicated servers for Justin Bieber, 3% of the load

There are 247 news articles about Twitter having dedicated servers for Justin Bieber which is 3% of Twitter traffic.

image

CNET broke the news with citing a tweet.

Report: Justin Bieber is 3 percent of Twitter

by Chris Matyszczyk

If you believe that Twitter is full of inane, immature narcissism, here's one in your solar plexus.

For an allegation has reached my eyes and baffled them into blindness. The allegation is that, at any given moment, at any given movement of your lungs and toes, 3 percent of Twitter's infrastructure is dedicated solely to the one person who most defines our hopes and our times.

I am not speaking of Kim Kardashian, nor of Rep. Jack Kimble. I am speaking of the one person who can unite men and women, young and old, sane and slightly less so: Justin Bieber.

A fascinatingly hopeful post at Gizmodo offers that Dustin Curtis, a designer of some repute, was told by an employee at Twitter that 3 percent of the company's infrastructure is dedicated to the little man with the unreal voice and the even more unreal hair.

(Credit: Screenshot: Chris Matyszczyk/CNET)

Read more: http://news.cnet.com/8301-17852_3-20015781-71.html#ixzz0ywhpUh60

What I haven't seen anyone attempt to explain is why would there be Racks of Servers dedicated?

This is common sense with no information from Twitter folks.

When you want speed in your servers you want to keep your whole search index in RAM in a server. Google, eBay, and Amazon are good examples of companies who use this method in their search servers to support keyword lookup and result match-up.  Any time you go to disk or another server you will slow down the search results by magnitudes.

The costs to put your whole search index in RAM is too high, so break down the problem.  Analyzing the flow of tweets, Twitter figured out many people are focused on a few areas, and probably stick to that topic watching the tweets.

Second Life was started with the idea of virtual reality and avatars, but many people are too lazy to keep up their 3D appearance.  Tweets are much more efficient to support a virtual presence.

In the same way that Saudi Arabia wanted to get access to RIM's encrypted data the information and analytics available in Justin Bieber's servers can be useful.  And, Twitter is probably figuring out ways to monetize access to analytics.

Which brings up an interesting point is Twitter the next service which governments want access to crawl?  For many people, e-mail and blogging is in the past.  And, Twitter is their main method to communicate.

There was a rumor of part of the reason why Twitter chose Salt Lake City is to be near the NSA's new data center in Salt Lake City. 

Room to grow: a Twitter data center

Wednesday, July 21, 2010

Later this year, Twitter is moving our technical operations infrastructure into a new, custom-built data center in the Salt Lake City area. We're excited about the move for several reasons.
First, Twitter's user base has continued to grow steadily in 2010, with over 300,000 people a day signing up for new accounts on an average day. Keeping pace with these users and their Twitter activity presents some unique and complex engineering challenges (as John Adams, our lead engineer for application services, noted in a speech last month at the O'Reilly Velocity conference). Having dedicated data centers will give us more capacity to accommodate this growth in users and activity on Twitter.
Second, Twitter will have full control over network and systems configuration, with a much larger footprint in a building designed specifically around our unique power and cooling needs. Twitter will be able to define and manage to a finer grained SLA on the service as we are managing and monitoring at all layers. The data center will house a mixed-vendor environment for servers running open source OS and applications.
Importantly, having our own data center will give us the flexibility to more quickly make adjustments as our infrastructure needs change.
Finally, Twitter's custom data center is built for high availability and redundancy in our network and systems infrastructure. This first Twitter managed data center is being designed with a multi-homed network solution for greater reliability and capacity. We will continue to work with NTT America to operate our current footprint, and plan to bring additional Twitter managed data centers online over the next 24 months.

The probability of the NSA talking to Twitter at some point of time is almost certainty.

Is Twitter's dedicated servers a help for the NSA?  There are Twitter servers dedicated to President Obama.  And Oprah too. :-)  Are the number of dedicated servers the new status symbol and measure of popularity?  It used to be records sold and movie revenue.  Is it now the billions of tweets?

Read more

Thought Patterns for High Performance

I am spending the next 2 days in a seminar at The Pacific Institute on Thought Patterns for High Performance.

image

The Nick Saban is a great story of how The Pacific Institute helped the Alabama Football team.  if this good enough for the #1 football team, there is something there to learn.

Psychology of the Data Center, learning from the Science of Football - Alabama’s Nick Saban

I just had a conversation with Tom Roth who introduced me to some of the work that The Pacific Institute does.  Tom and I had an interesting discussion on data centers as he is familiar with real estate development in Eastern Washington’s recent data center build out that has used up hydroelectric power which has little employment impact the original dam developers intended.

image

One of the points well made is on what Adult Learners Learn and Retain:

  1. 10% of what we HEAR
  2. !5% of what we SEE
  3. 20% of what we both SEE and HEAR.
  4. 40% of what we DISCUSS with OTHERS.
  5. 80% of what we EXPERIENCE.
  6. 90% of what we ATTEMPT to TEACH OTHERS.

This is an excellent point on why some of the top data center industry people are quite social and interact with others.

For those of you who lock your data center people in the building, consider letting them out to let them achieve a higher performance.

Read more

Spies use of Social Networks

The FBI's arrest of 11 Russian spies is hot news and most of the attention is going to Anna Chapman.

Alleged Russian agent Anna Chapman could have warmed up any Cold War night

GALLERY

Anna Chapman, alleged Russian spy, posts photos online

Alleged Russian spy, Anna Chapman, became an instant Web sensation following the release of photos posted on the Russian social-networking Web site "Odnoklassniki," or Classmates.

By Monica Hesse

Washington Post Staff Writer
Thursday, July 1, 2010

There were 11 alleged Russian agents arrested this week, under accusations that they'd been living as Americans while reporting back to the mother country.

THIS STORY

  • But mostly we care about the hot one.

Ever since photos of Anna Chapman began circulating online late Tuesday, the Internet at large has been foaming, frothing, fanatic for details about the reported 28-year-old secret agent/Maxim model look-alike who specialized in sultry-eyed, pouty-lipped, come-hither stares. Da, da,da!

CNET news has a post on Facebook effect and spies.

A Cold War tale reheated for Facebook

by Caroline McCarthy

A time traveler from the 1960s would find many of today's headlines completely befuddling. Something called theiPhone? Threats from an amorphous, stateless band of terrorists? Reality television? A lot has changed, for sure.

But accused spies for the Kremlin--that's something our unstuck-in-time explorer would thoroughly recognize. Except, of course, when he heard everybody talking about foreign concepts like Facebook and LinkedIn.

Should a spy be on Facebook?

But for a young secret agent like Anna Chapman or Mikhail Semenko, the absence of a Facebook profile could trigger suspicion. If they're going to be like everybody else, of course they're going to use social networks.

Why spying is important.

Espionage is a fact of international life and always has been. The first spy manual, The Art of War, was written by Sun Tzu some 2,500 years ago. Espionage fills a vital niche; a successful operation can provide insight into intentions, plans, and human dynamics that cannot be gleaned from intercepted communications or pictures from space.

It is safe to assume that since the end of World War II there has never been a day that the Soviet Union or Russia was not spying on the United States, or vice versa. Espionage will continue, even as the United States and Russia work out a new modus vivendi.

Data Centers is a competitive industry, and as much as people try to keep things secret. With public disclosures it is not hard to piece together information.  Rich Miller posted on Facebook's 60,000 server count on June 28, 2010.

Facebook Server Count: 60,000 or More

June 28th, 2010 : Rich Miller

It’s said that a picture is worth a thousand words. Sometimes a PowerPoint slide can tell the story of thousands of servers.

That was the case with a presentation from Facebook’s Tom Cook at last week’s  Velocity 2010 conference, which depicted the growth of the company’s server footprint. Designed to illustrate Facebook’s insatiable need for more servers to support its 400 million users, the chart didn’t include any numbers, seeking not to reveal the actual server count.

Dates Provide A Clue
But the chart included dates, which allows us to do some math to fill in the blanks. In a presentation in November 2009, Facebook vice president of technology Jeff Rothschild disclosed that the company had more than 30,000 servers. Cook’s chart shows a brief plateau in Facebook’s server growth at about that time, followed by a sharp upward spike in the growth line through the first quarter of 2010 that effectively doubles the total number of servers.

That suggests that Facebook now has 60,000 or more servers. The sharp acceleration in Facebook’s server growth in late 2009 also helps explain the company’s move to lease large chunks of data center space in northern Virginia and Silicon Valley in March. The growth spurt occurred after Facebook announced plans to construct its own data center in Prineville, Oregon.

Putting together the pieces that are public is what I wrote about in this post.

Mar 09, 2010

A different interpretation of “Open Source” in an Intelligence Analysis scenario that defines how GreenM3 works public data

I ran across the term Open Source Intelligence.

Open source intelligence (OSINT) is a form of intelligence collection management that involves finding, selecting, and acquiring information from publicly available sources and analyzing it to produce actionable intelligence.

This description fits what I have been telling others about the various data center sources of information.

“If there is a public publication of information, we are open to look at and provide feedback on the value we see in the information.”

Which is a pretty good description of how this blog has been run, commenting on public available information.

BTW, the use of social networks is not anything new.  Three Days of the Condor is about a group of CIA researchers who did Open Source Intelligence - "I just read books"

Joe Turner (Robert Redford) is a CIA employee who works in a clandestine office in New York City. He is not a field agent, and indeed bristles at Agency discipline; among other things, he wonders why he can't tell people what he does for a living and notes "I trust some people ... that's a problem." His job is in the OSINT field: he has to read books, newspapers, and magazines from around the world, looking for hidden meanings and new ideas. As part of his duties, Turner files a report to CIA headquarters on a low-quality thriller novel his office has been reading, pointing out strange plot elements therein, and the unusual assortment of languages in which the book has been translated (Turkish but not French, Arabic but not Russian or German, Dutch, and Spanish).

Robert Redford's character says.

I work for the CIA. I'm not a spy. I just read books. We read everything that's published in the world, and we-- we feed the plots-- dirty tricks, codes into a computer, and the computer checks against actual CIA plans and operations. I look for leaks, new ideas. We read adventures and novels and journals. I-- I can-Who'd invent a job like that? I-- Listen! People are trying to kill me!

Here is a video clip from Three Days of the Condor.


Behind all of this is the technique of using public information with a bunch of people who are thinking of questions others don't ask.  There are tons of experts in the CIA.  There are tons in the data center industry.  Yet can the experts anticipate the unexpected?

Taking a bunch of people who look at what is going on in the world allows you to be more creative in figuring out the questions that should be asked.

If you are a spy you need cover your tracks.

There are perils to the process. One source here said that analysts who engage in searches without masking their origin can lead to foreign governments or companies cutting off access to web sites or to people involved. The problem? Some analysts at NSA, CIA and other alphabet soup agencies forget to mask their IP addresses and the times at which they are searching. Chinese, Russian and other savvy operators can check time stamps, for example. If a search occurs during American working hours, it’s a pretty good bet that it’s an American source looking for the information.
Read more: http://www.dodbuzz.com/2009/10/20/open-source-intel-use-soars/#ixzz0svCx1zvG

This technique makes it easier to think of what is a green data center.  Which is more than a low PUE.

Read more

Who will be next for Government Regulation? Google, Facebook or Apple

One side affect of the Microsoft anti-trust action is the governments of world feel good about taking on technology companies with regulation.  You go throughout history and technologies at first were not regulated - auto mfg, power generation, oil & gas, and healthcare.  And, there is still people arguing there needs to be more legislation in these areas.

Microsoft fought the battle with the DOJ and EU and besides paying lots of money, is government regulated.  SEC has a degree on financial regulation, but doesn't touch on the latest issues with Google, Apple, and Facebook.

You look in the news at Google's Wifi.

Privacy group to go to police over Google Wi-Fi data collection

Privacy International likens mistake to tapping phone without consent

By Carrie-ann Skinner, PC Advisor UK
June 10, 2010 11:26 AM ET

Privacy International has revealed it will approach the Police regarding Google's recent admission it had mistakenly collected data from unsecured Wi-Fi networks for the past three years.

The search enginer's error came to light after the German data protection authority audited the Wi-Fi data collected by Street View cars for use in location-based products such as Google Maps for mobile.

Apple's ad policy.

Earlier today we reported that Apple (NASDAQ:AAPL) had amended itsiPhone Developer Agreement, specifically clause 3.3.9 related to advertising networks. The amendment was written in such a way that it allowed Steve Jobs to make good on a promise he made at D8 to open up the SDK for third parties to collect data for services such as advertising. But at the same time it was worded very carefully so as to exclude Google’s AdMob mobile advertising network from the iPhone eco-system.

This was met yesterday with lots of grumbling from Google, and gnashing of teeth from Admob’s CEO. Whilst the wording of the clause excludes any rival smart phone maker, Google is the only real threat on the horizon that Apple probably cares about right now.

And bloggers like danah boyd make the point on Facebook being a utility, and utilities get regulated as they are monopolies.  Being a Microsoft employee she knows what is like to be a monopoly.

Facebook is a utility; utilities get regulated

From day one, Mark Zuckerberg wanted Facebook to become a social utility. He succeeded. Facebook is now a utility for many. The problem with utilities is that they get regulated.

Yesterday, I ranted about Facebook and “radical transparency.” Lots of people wrote to thank me for saying what I said. And so I looked many of them up. Most were on Facebook. I wrote back to some, asking why they were still on Facebook if they disagreed with where the company was going. The narrative was consistent: they felt as though the needed to be there. For work, for personal reasons, because they got to connect with someone there that they couldn’t connect with elsewhere. Nancy Baym did a phenomenal job of explaining this dynamic in her post on Thursday: “Why, despite myself, I am not leaving Facebook. Yet.”

danah goes on, and makes points about society, utilities and choice.

I hate all of the utilities in my life. Venomous hatred. And because they’re monopolies, they feel no need to make me appreciate them. Cuz they know that I’m not going to give up water, power, sewage, or the Internet out of spite. Nor will most people give up Facebook, regardless of how much they grow to hate them.

Your gut reaction might be to tell me that Facebook is not a utility. You’re wrong. People’s language reflects that people are depending on Facebook just like they depended on the Internet a decade ago. Facebook may not be at the scale of the Internet (or the Internet at the scale of electricity), but that doesn’t mean that it’s not angling to be a utility or quickly becoming one. Don’t forget: we spent how many years being told that the Internet wasn’t a utility, wasn’t a necessity… now we’re spending what kind of money trying to get universal broadband out there without pissing off the monopolistic beasts because we like to pretend that choice and utility can sit easily together. And because we’re afraid to regulate.

And here’s where we get to the meat of why Facebook being a utility matters. Utilities get regulated. Less in the United States than in any other part of the world. Here, we like to pretend that capitalism works with utilities. We like to “de-regulate” utilities to create “choice” while continuing to threaten regulation when the companies appear too monopolistic. It’s the American Nightmare. But generally speaking, it works, and we survive without our choices and without that much regulation. We can argue about whether or not regulation makes things cheaper or more expensive, but we can’t argue about whether or not regulators are involved with utilities: they are always watching them because they matter to the people.

The political win to regulate Google, Apple, or Facebook is huge.  You can argue whether it is right or not, but the fact is a group of  politicians see regulating one of the companies as a career milestone that will set them for life, and allow them to break into private industry making more money or guaranteeing them more influence.

Read more

Open Leadership - can it work in the data center where control leadership is the norm?

I was having an interesting e-mail conversation with a data center and how well a local monthly data center event is working to build better relationships with the community of data center operators.  The openness of data center discussion flies in the face of secrecy typical of data centers.  Data Center events try to stimulate open conversation, but given the irregularity of events can you build a relationship by attending?

image

Open Leadership is a new book being released by Charlene Li, co-author of Groundswell.  Josh Bernoff the other co-author of Groundswell writes.

I found some parts of the book a lot more useful or interesting than others. Here are three good parts.

  1. Sandbox covenants. These are the rules organizations set up to determine what sorts of limits and conventions there are on openness. The book includes a link to social media policies of a bunch of corporations, not yet live, but I am looking forward to seeing that. This discussion, in Chapter 5, goes a long way to helping bridge the gap between social media backers within companies and corporate policymakers.
  2. Organizational models for openness. Charlene describes three types of organization: organic, centralized, and coordinated, and shows when each one makes sense. Given all the questions I get these days about organization for social, this is quite relevant.
  3. Leadership mindsets and traits. Chapter 7 classifies leaders according to whether they are optimistic or pessimistic, and whether they are independent or collaborative. Anyone who has ever had a boss will find this instructive. This is a fascinating way to look at leadership.

There are people who are realizing an Open approach is powerful, but difficult for most.  The book will be available on May 24.

As Li explains, openness requires more—not less—rigor and effort than being in control. Open Leadership reveals step-by-step, with illustrative case studies and examples from a wide range of industries and countries, how to bring the precision of this new openness both inside and outside the organization. The author includes suggestions that will help an organization determine an open strategy, weigh the benefits against the risk, and have a clear understanding of the implications of being open. The book also contains guidelines, policies, and procedures that successful companies have implemented to manage openness and ensure that business objectives are at the center of their openness strategy.

I'll post later on the data center social event that is taking an Open approach to Data Center Networking.  And,  I plan on adding a trip to an event to see for myself how it works.

Read more