See the Cargo Cult Science in Data Centers

Now you may assume you that the data center world is all built on science and facts.  But, the Data Center World is no different than the rest of the world, and unfortunately those who benefit from managing your perception may have little interest in science.

Richard Feynman presented a talk on Cargo Cult Science.

I think the educational and psychological studies I mentioned are examples of what I would like to call cargo cult science. In the South Seas there is a cargo cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they've arranged to imitate things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas--he's the controller--and they wait for the airplanes to land. They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they're missing something essential, because the planes don't land.

If you want a way to detect the Cargo Cult Science in data centers, a pretty good indicator is whether you can find what Richard Feynman tell the graduates to do to not be Cargo Cult Scientists.  How are they credible.

It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty--a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid--not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked--to make sure the other fellow can tell they have been eliminated.

Details that could throw doubt on your interpretation must be
given, if you know them. You must do the best you can--if you know
anything at all wrong, or possibly wrong--to explain it. If you
make a theory, for example, and advertise it, or put it out, then
you must also put down all the facts that disagree with it, as well
as those that agree with it. There is also a more subtle problem.
When you have put a lot of ideas together to make an elaborate
theory, you want to make sure, when explaining what it fits, that
those things it fits are not just the things that gave you the idea
for the theory; but that the finished theory makes something else
come out right, in addition.

An example of the Cargo Cult Science is IO Data Centers Last Snowflake that 2013 will be the last non-standardized snowflake.  There are many others, but this is one of the more timely ones that just got picked up.

Uptime: CEO of IO reaffirms prediction of the “snowflake’s” last day

Slessman sticks by prediction of end of non-standardized data centers
The world will not see a single non-standardized “snowflake” data center built after May 2013.

There may be people excited about this, but where is the data to support this claim?

When i looked at Slessman's transcript from 2011 where he made this claim.  There is not a single mention of quality or errors in the talk.  What data is shown that every IO data center is the same.   Huh. IO data centers has redefined physics where there are no errors or quality issues with any of their build outs, so every data center is the same.

So, let's go with a brand that prides themselves on quality and german engineering - BMW.

BMW builds 450 cars a day in its South Carolina plant and I am going to point to a bunch of things that make me believe BMW more than IO Data Centers.

BMW has an analysis center to figure out what is wrong.

Analysis Center

analysis

At the Analysis Center, we ignore the BMW mystique, look past the dazzling lines and impeccable paint job. We strip down the BMW and take an honest look at our work. The naked truth in all its beauty is revealed.

The 60,000-square-foot Analysis Center is a fully functioning laboratory that allows us to examine and test every weld, every dimension, and every component on vehicles as they come off the production line. The Analysis Center covers three key areas of vehicle development: Functional Analysis, Manufacturing Analysis and Customer Feedback.

BMW has a 1MW data center to support its operations and analysis.

NewImage

The BMW South Carolina factory is described in this National Geographic Video.  What is not shown in this short video is the camera inspection equipment and many others to detect quality and errors in manufacturing. 

The laws of physics are tough to beat and it is really hard to make hundreds of complex products be exactly the same.  The products are not the same, they are all a bit different.  The issue is whether the products perform within specifications and meet quality standards.

You can believe the Cargo Cult Science that 2013 will be the last snowflake and they will all be the same after that, but you may be like the villagers doing all the those things to bring the cargo planes back.  Wearing wooden headhphones, bulding runways and putting wooden planes out.

NewImage

Or you can go with the view that every data center is a bit different and I want an engineering science backed team who can adapt the manufacturing process to give a quality performing product. 

King of Human Error influences Checklist Manifesto

Vanity Fair has an article by Michael Lewis on Daniel Kahneman's Thinking Fast and Slow book.

LETTER FROM BERKELEY
December 2011

The King of Human Error

Billy Beane’s sports-management revolution, chronicled by the author inMoneyball, was made possible by Israeli psychologists Daniel Kahneman and Amos Tversky. At 77, with his own new book, Thinking, Fast and Slow, the Nobel Prize-winning Kahneman reveals the built-in kinks in human reasoning—and he’s Exhibit A.

Related: “The Quiz Daniel Kahneman Wants You to Fail.”

THINKING MAN Daniel Kahneman outside his Berkeley, California, home. “He [is] more alert and alive than most 20-year-olds,” writes Lewis.

We’re obviously all at the mercy of forces we only dimly perceive and events over which we have no control, but it’s still unsettling to discover that there are people out there—human beings of whose existence you are totally oblivious—who have effectively toyed with your life.

One of the data center executives turned me on to the Checklist Manifesto by Atul Gawande, and guess what.  Atul was influenced by the same King of Human Error.

When you wander into the work of Kahneman and Tversky far enough, you come to find their fingerprints in places you never imagined even existed. It’s alive in the work of the psychologist Philip Tetlock, who famously studied the predictions of putative political experts and found they were less accurate than predictions made by simple algorithms. It’s present in the writing of Atul Gawande (Better, The Checklist Manifesto), who has shown the dangers of doctors who place too much faith in their intuition.

One of the patterns that is interesting to investigate is where the judgement errors are made in the data center.

Why is this important for a green data center?  Because there are judgement errors all over the place.

What I am working on next? Product Life Cycle, Supply Chain, Asset Management

I ran into Kevin Heslin and Rich Miller at the Open Compute Project hosted by Facebook.  I figured I would run into Kevin and Rich given the NYC was in their backyard.  Luckily i had been planning a trip with some data center executives to the NYC area for months and it just so happened we were planning on being in NYC at the same time as the Open Compute Project which is also very convenient as many of us had gone to the Open Compute Project in Palo Alto.

Many people know me through my blogging role as I have now reached 4 years writing on the green data center topic.  Prior to that I spent 14 years at Microsoft working on mostly Windows from Win3.1, NT3.1, NT3.5, NT4.0, Windows 2000, and Windows XP on a whole bunch of different OS features.  TrueType font technology is where I spent the most time on a technology cumulatively between Microsoft and Apple.  Fonts is where I made some great friends I still work with today.

At Apple I re-engineered the distribution system to handle triple the capacity with a 1/3 of the labor by analyzing batches of orders to process them as a group instead of individually.  Some ideas stick and can be re-used over and over. After distribution logistics I moved on to OEM supply chain for Apple products, with the best being the Mac II, and even though it was a market flop it was fun traveling all over to get the parts for the Mac Portable.  I cut my teeth on operating systems for the Mac II and Portable, and moved to the OS group to work on KanjiTalk, then work on System 7.

Kevin Heslin asked me what I am busy working on.  I told him I am working on supply chain and asset management systems.  Kevin said, "I thought you were working on Green IT."  There are huge ways to Green IT addressing the supply chain in the data center ecosystem.  Analyzing the overall hardware flowing through the product life cycle will allow you to see things you can't see when staring at servers in a rack which pretty don't move for 3-4 years.  PUE and green technologies can reduce the carbon impact required in the power and cooling systems.  But, there is probably at least a 3X improvement potential in how the IT systems are put together in the data center

After 4 years of blogging on green data centers it is easy for me to write on the topic, spending on average an hour a day and I can write 3 posts.

There are other ideas i am working on like what does a location service look like in a supply chain system.  This stuff takes hundreds of hours over months to figure out, talking to some really smart people.

4 years ago I started working on green data center ideas, and it has been a great experience.  Now, I am spending more time on greening the IT systems, and solving the problems in different way with systems, knowledge, and data analytics.  The data center is a critical part of the solution, and I will still spend time on data centers, but I am more excited to think of new information systems for the data center.

I don't know if this answers the question Kevin asked, "what I am busy working on?"  And, most likely in 6 months there will be something new to study.  But, for now I am thinking about Supply Chain Management, Asset Management and Inventory Control.

PG&E fires four inspectors - just another example of why you need to audit work in the data center

I am constantly amazed when critical infrastructure in the data center work is not audited.  Domenic Alcaro's talk at 7x24 about submarine maintenance and ideas that apply to the data center, the inspection and auditing of work is standard procedure.

NewImage

SJ Mercury news discusses the situation PG&E has found when they audited their inspectors.

PG&E fires four inspectors

Updated: 09/01/2011 02:54:21 PM PDT

PG&E has fired four inspectors after an internal investigation determined they falsely claimed to have inspected the company's underground electrical gear, and four others were disciplined.

The probe, which PG&E launched in November after receiving an employee's tip about the phony inspections, was first disclosed publicly in June. Over the past two months, the company reinspected nearly 25,000 underground enclosures containing various electrical equipment and found 25 where inspectors had falsely claimed to have inspected them, according to PG&E spokesman Andrew Souvall.

...

"In the four instances where employees were terminated, we found substantial evidence that they had not properly conducted their inspections and had falsified records," Souvall said.

Some of my friends and I have been thinking about how to solve this type of problem in the data center, and we are circulating the solution to some early adopters.

Checklist for Checklists and a website for medical checklists

I found a checklist for checklists on Project Check website.

Welcome!  Project Check is a website designed to provide the public with easy access to a number of life saving medical checklists.  To view and download one of our currently available checklists, please visit the "Checklists" page.  If you do not see the checklist you are looking for, we encourage you to develop your own.  We have created a "Checklist for Checklists" to aid you in this process.  If you have questions along the way, feel free to either contact us or pose your question to the forum.  Once your checklist is complete, please submit it to us and we will post it for others to access.

image

image

image

image

Picture

And, here is a Surgical Safety Checklist on the site which is from http://www.safesurg.org/

Picture

I've got the e-mail of Boeing's Checklist god, Dan Boorman and a collection of data center executives who will attend a future meeting.  Part of the homework for the people who will join us is to spend some time thinking of data center checklists.