Amazon's James Hamliton shares Overall Data Center Costs, except taxes

James Hamilton has a nice post on the Overall Data Center Costs

Overall Data Center Costs

A couple of years ago, I did a detailed look at where the costs are in a modern , high-scale data center. The primary motivation behind bringing all the costs together was to understand where the problems are and find those easiest to address. Predictably, when I first brought these numbers together, a few data points just leapt off the page: 1) at scale, servers dominate overall costs, and 2) mechanical system cost and power consumption seems unreasonably high. Both of these areas have proven to be important technology areas to focus upon and there has been considerable industry-wide innovation particularly in cooling efficiency over the last couple of years.

I posted the original model at the Cost of Power in Large-Scale Data Centers. One of the reasons I posted it was to debunk the often repeated phrase “power is the dominate cost in a large-scale data center”. Servers dominate with mechanical systems and power distribution close behind. It turns out that power is incredibly important but it’s not the utility kWh charge that makes power important. It’s the cost of the power distribution equipment required to consume power and the cost of the mechanical systems that take the heat away once the power is consumed. I referred to this as fully burdened power.

This post supports one of the ideas I threw out there on Amazon's financial discipline.

At many companies the rigors of getting approval a data center construction project approved are from the business units, technical people, and the CFO.  I would expect at Amazon.com, everyone wants to see the numbers, and they spend much more time on financial modeling.

Should we lease or build?

What is the best use of Amazon's capital and cash?

What is the overall operating expense of a leased vs. owned facility?

Latest decision in the East Coast. Lease.

The one variable that is difficult to put into an  excel document is the taxes you pay.  Believe it or not the tax incentives are one of the biggest things that gets the big builders to be in one state vs. another.

James's data is a good starting point, but don't forget to get your tax department involved in the data center project.

Read more

Site Selection process means choosing a vendor network

The site selection process is one of the top ways to influence your ability to run a green data center.  Power generation composition defines your carbon footprint.  The site environmental condition defines your choice of cooling systems. Mike Manos adds his opinion on this topic.

So you might be thinking, ‘Great, I just need to find the areas that have cheap power and a good Carbon Emission Factor right?’  The answer is no.  Many Site Selection processes that I see emerging in the generic space start and stop right at this line.   I would however advocate that one takes the next logical step which is to look at the relationship of these factors together and over a long period of time.

But, what most people miss focusing on these technical details is the network you choose when you start down the process.  One of the insider data center discussions I have had is how people don't understand the vendor networks that exist.  Almost every vendor has a network of people who they work with as preferred vendors.  The relationships are not always transparent and can have referral fees for diverting new business.  Customers may think they get around the requirement for competitive bidding requirements, but the bidding may not be as competitive as you think as many vendors cannot get into a bid if they don't have an established relationship with the group controlling the bid process.

The ones to be most suspicious of are the ones who make repeatedly claims of being objective and independent.  It is analogous to someone who says "trust me."  But it sounds like "don't trust me" to me.

How do you get out of this mess, as the data center owner you need to be in control of your own destiny and open your eyes to complexity of the relationships. 

Also,  following others can make your life easier as someone with more resources has run a more thorough analysis.  Mike Manos discusses how others are following Microsoft to Chicago.

We have seen many examples like Quincy, Washington and San Antonio, Texas where the site selection process has led to many Data Center providers locating in the same area to benefit from this type of analysis (even if not directly exposed to the criteria).  There is a story (that I don’t know if its true or not) that in the early days when a new burger chain was looking to expand where it would place its restaurants, it used the footprint of its main competitor as its guide. The thinking was that they probably had a very scientific method for that selection and they would receive that same ancillary benefit without the cost and effort.   Again, not sure if that is true or not, but its definitely something likely to happen in our industry.

In many markets these types of selections are in high demand.   Ascent Corporation out of St. Louis is in the process of building a modern facility just down the street from the Microsoft Mega-Facility near Chicago.   While Ascent was a part of the original Microsoft effort to build at that location, there has been an uptick in interest for being close to that facility for the same reasons as I have outlined here.  The result is their CH2 facility is literally a stones throw from the Microsoft Behemoth.  The reasons? Proximity to power, fiber, and improved water infrastructure are already there in abundance.  The facility even boasts container capabilities just like its neighbor.   The Elmhurst Electrical Substation sits directly across the highway from the facility with the first set of transmission poles within easy striking distance. 

Elmhurst Electrical Yard

The Generation mix of that area has a large nuclear component which has little to no carbon impact, and generates long term stability in terms of power cost fluctuations.   According to Phil Horstmann, President of Ascent, their is tremendous interest in the site and one of the key draws is the proximity of its nearby neighbor.  In the words of one potential tenant ‘Its like the decision to go to IBM in the 80s.  Its hard to argue against a location where Microsoft or Google has placed one of its facilities.’

This essentially dictates that there will be increasing demand on areas where this analysis is done or has been perceived to be done.   This is especially true where co-location and hosting providers can align their interests with those commercial locations where there is market demand.  While those that follow first movers will definitely benefit from these decisions (especially those without dedicated facility requirements), first movers continue to have significant advantage if they can get this process correct.

But, as Mike points out you rarely get as good a deal as those who are the first movers.

While those that follow first movers will definitely benefit from these decisions (especially those without dedicated facility requirements), first movers continue to have significant advantage if they can get this process correct.

Read more

Changing the Site Selection game, being in Control, leveraging Mike Manos's post

Mike Manos is extremely busy now and doesn't blog as much as he used to, but when he does post he still gets lots of traffic.  Mike and I were laughing once when a corporate data center blog was discussing proudly how many people they reached in a month with their blog. Mike said, "I get that much in less than a week."  Which brings up a good point of how your view changes if you knew what you don't know.  If they had known Mike gets 10x more traffic than them in a month, they'd wonder how influential they really were, and whether they are successful or not.

I always enjoy reading Mike's post, figuring out ways to use less words than he does, but also reading what Mike is trying to say, but hasn't put it down in words.  Luckily leveraging multiple discussions with Mike, I can take a pretty stab at what he was thinking of when he was writing.

Mike posted July 14 on Site Selection.

Site Selection,Data Center Clustering and their Interaction

July 14, 2010 by mmanos

I have written many times on the importance of the site selection for data centers and its growing importance when one considers the regulatory and legislative efforts underway globally.   Those who make their living in this space know that this is going to have a significant impact on the future landscape of these electronic bit factories.   The on-going long term operational costs over the life of the facility,  their use of natural resources (such as power) and what they house and protect (PII data or Personally Identifiable Information) are even now significantly impacting this process for many large global firms, and is making its way into the real estate community.  This is requiring a series of crash courses in information security, power regulation and rate structures, and other complex issues for many in the Real Estate community.

What I think Mike is trying to say is it is much easier to build a better performing low cost data center with the right site.  But, few understand the complex relationships that affect data center performance.  I've always found it naive and over simplistic when companies and consultants use a long list of weighted criteria as the method to pick a site, assuming the highest score is the best site.  This works for those who the most complex math they are comfortable with is multiplication and addition, but think about this hundreds of millions of dollars of CAPex, OPex, and IT equipment will be spent over a data center lifetime, and you are going to make the decision based on addition and multiplication?

I believe modeling techniques should be applied to ask the question "what is the right site?"  And, Mike has his own mental models of what is right and wrong.

The right site for what?  Pick 3 - 5 data center designs that you think you would want to build and use them as models to represent what you intend to build.  If you have built the model with enough detail you should see the relationships that Mike talks about.

Tying into the power conversation is that of water.  With the significant drive for economization (whether water based or air-based)  water continues to be a factor.  What many people don’t understand is that in many markets the discharge water is clean to dump into the sewage system and to ‘dirty’ to discharge to retention ponds.  This causes all kinds of potential issues and understanding the underlying water landscape is important.   The size of the metropolitan sewage environments, ability to dig your own well efforts, the local water table and aquifer issues, your intended load and resulting water requirements, how the local county, muncipality, or region views discharge in general and which chemicals and in what quantities is important to think about today.  However, as the use of water increases in terms of its potential environmental scrutiny – water is quickly rising on the site selection radar of many operators and those with long term holds.

I have friends who designed a waterless cooling system in Australia due to the drought conditions.  There was a cost associated for this data center design vs. cooling towers, but when you looked at the total system it was the right design.

If you really want to be advanced you can use semantic models.

Savanna is a model-driven analysis solution for solving complex problems. The magic of Savanna is in defining models that address what’s relevant to your problem at any given point in the analysis process. Savanna’s semantic models are driven by the Thetus Publisher architecture, enabling information synthesis by offering users the unique ability to derive meaning from information sets and to bridge the gaps in information. Savanna’s innovative, mind-mapping interface provides intuitive tools for approaching analysis from a point that frames the problem rather than one that starts from the information out.

Warning this technique works, but few have the capabilities to operate in this way. Using semantic models change the game as you focus on the problems and questions to ask, and enables you to see things others cannot.

To take control of site selection you need to have data center designs in mind for what you are building.  If you are Mike Manos you can see the relationships of the site to the data center designs and how the system will operate.

If you can't do what Mike Manos does, then be prepared to make lots of mistakes even if you hire experts.  Because you are not in control and are being told what to do.  Do you think you can be a good cook by hiring a bunch of experts to tell you what to do?  You need to be in control.  Use data center designs to take control.

If you walk into a site selection consultant and say here are five data center designs I am looking at find me sites that support these designs.  When you find me a site tell me which design works best and worst on the site. You'll find out whether the consultant can do more than addition and multiplication, and whether they really understand what a good data center site is.

Read more

Complexity of Data Center Construction Projects requires challenging the status quo, site selection example

I met with a friend who has worked in the high tech start-up environment and currently works in a company that has a data center presence, and he is thinking of a career change over to data centers.  We spent a couple of hours discussing many topics, a who's who in the industry, dynamics of how the industry works, and systemic problems in the current ways.

A systemic problem is a problem due to issues inherent in the overall system,[1][2] rather than due to a specific, individual, isolated factor. Contrast with pilot error, user error or mistake.

A change to the structure, organization or policies in that system could alleviate the systemic problem. On an Ishikawa diagram (fishbone diagram) of cause-and-effect links, the source of the problem can be said to be a common cause, rather than a special cause.

My friend showed me some jobs he was thinking of applying for, and some ideas started to gel together on problems the data center industry have.

One, data centers are computers.  Google threw this idea out there.

Google: The Data Center Is the Computer

By Stacey Higginbotham Jun. 15, 2009, 11:47am PDT 1 Comment

61

Server room and devices

As folks increasingly store and access information online, the data centers powering cloud services need to be managed more like a single computing entity rather than a bunch of servers, according to a Google white paper (Google calls it a mini-book) released today.

The paper lays out the concept of warehouse-scale computers (which we have previously referred to as both web-scale computing and mega data centers), specifically how to build out the infrastructure to support Internet services managed across thousands of servers. Google’s Luiz Barroso, a distinguished engineer, and Urs Hölzle, SVP of operations, both of whom help manage and build out Google’s data center, lay out their definition of WSCs:

Most data center projects are projects where fiefdoms battle for control of budget and resources - facilities, real estate, IT ops, application groups, and various users.

Most projects start with a data center project approval at the executive level and an approval of a budget. Then the real estate group takes control on site selection.  Consultants are hired for the site selection process to figure out the right data center site.  Lots of hours are spent interviewing people making the project more complex and more expensive.

Each group maneuvers to be the owner of a piece, and it sets you down a path without an overall design.

How many site selection companies are thinking like Mike Manos's post on site selection?

Site Selection,Data Center Clustering and their Interaction

July 14, 2010 by mmanos

I have written many times on the importance of the site selection for data centers and its growing importance when one considers the regulatory and legislative efforts underway globally.   Those who make their living in this space know that this is going to have a significant impact on the future landscape of these electronic bit factories.   The on-going long term operational costs over the life of the facility,  their use of natural resources (such as power) and what they house and protect (PII data or Personally Identifiable Information) are even now significantly impacting this process for many large global firms, and is making its way into the real estate community.  This is requiring a series of crash courses in information security, power regulation and rate structures, and other complex issues for many in the Real Estate community.

...


Elmhurst Electrical Yard

The Generation mix of that area has a large nuclear component which has little to no carbon impact, and generates long term stability in terms of power cost fluctuations.   According to Phil Horstmann, President of Ascent, their is tremendous interest in the site and one of the key draws is the proximity of its nearby neighbor.  In the words of one potential tenant ‘Its like the decision to go to IBM in the 80s.  Its hard to argue against a location where Microsoft or Google has placed one of its facilities.’

How can a data center site be picked without a data center design?  Do you pick a data center design and then figure out sites?  If you have not picked your cooling system design, how do you know the affect of the water supply is and the impact of water on your design.

Tying into the power conversation is that of water.  With the significant drive for economization (whether water based or air-based)  water continues to be a factor.  What many people don’t understand is that in many markets the discharge water is clean to dump into the sewage system and to ‘dirty’ to discharge to retention ponds.  This causes all kinds of potential issues and understanding the underlying water landscape is important.   The size of the metropolitan sewage environments, ability to dig your own well efforts, the local water table and aquifer issues, your intended load and resulting water requirements, how the local county, muncipality, or region views discharge in general and which chemicals and in what quantities is important to think about today.  However, as the use of water increases in terms of its potential environmental scrutiny – water is quickly rising on the site selection radar of many operators and those with long term holds.

The industry standard is you pick a site, spent million dollars plus on site selection consultants, buy a site after a long evaluation period.  Then, the rest of the system never says a word about you picking a bad site.

I get a good laugh whenever people criticize another person's site selection.

One of the ways you can change site selection is to bring in the top data center construction companies to evaluate your site before you make your final selection. "here are the 2 - 3 sites I am looking at to build a data center.  Can you tell me the pros and cons for the sites?"

I got this idea talking to data center construction executive when he said they had won a new construction project and thank god the customer didn't pick the site in another state that was a crappy site.  "Why don't you get involved before a site is selected?"  After more chatting, figured out the site selectors want to close the deal with a site selection process they control.  Bringing in data center construction experts decreases their control and shifts the control to the customer.  This is bad for the commercial real estate process.  We need to remove conflicts on interest by keeping suppliers who would influence the decision process isolated.  BS!!!

If you take the idea to the next step, why not invite the electrical contractors to the site as well.  Electrical systems are the dominant cost in a data center project.  Site characteristics will affect the electrical systems and the costs.  Why not get the electrical contractors input too.  Cooling and water would be easy to add.

Hope this gets you thinking on how to change your site selection process.

A holistic approach thinking of the data center as a computer is better than following the status quo. A process of hiring a bunch of experts who have limited knowledge on how data centers are computers.

Read more

Breaking the rules for Data Center Site Selection, HP discusses Farm Waste as energy supply

One of the smart people I get to have regular conversations with is Pat Kennedy, Founder and CEO of OSIsoft.  Pat is the one who got me thinking about green data centers when he asked a simple question three years ago, "how do you measure the power consumption of an application in a data center?"  This got me started down a whole path of monitoring and modeling.

One of the latest topics Pat and I have discussed is MicroGrids.  Google thinks about this too.  See this Google video, I can see some of the Google data center team in the audience.

HP is making news today with their paper on a microgrid for data centers powered by cow manure.

image

ABSTRACT
In this paper, we design a supply-side infrastructure for data centers that runs primarily on energy from digested farm waste. Although the information technology and livestock industries may seem completely disjoint, they have complementary characteristics that we exploit for mutual benefit. In particular, the farm waste fuels a combined heat and power system. The data center consumes the power, and its waste heat feeds back into the combined system. We propose a resource management system to manage the resource flows and effluents, and evaluate the direct and indirect economic benefits. As an example, we explain how a hypothetical farm of 10,000 dairy cows could fulfill the power requirements of a 1MW data center.

Pat Kennedy long ago was making the point that data centers could be a lot efficient if sites were chosen to be next to power generation, biomass, and/or other large consumer of power.  But, this idea is controversial in that a standard practice for data center risk reduction to place data centers far away from hazardous materials.  I think a large methane store would typically get classified as a risk to a data center.  So, if you are totally risk averse and don't pay for the power bill, why not skip over the site with methane.  Most would.

Plus there are risks that HP doesn't mention in their brief statement on financial and associated risks.

Financial cost and associated risks are perhaps the most
important consideration. Existing farms that have invested in
supply-side infrastructure often do so only if a power-purchase
agreement can be signed. Otherwise, the return could be too
speculative to justify the capital investment. A data center has substantial, continuous, and long-term power needs. Thus the data center owner could sign the power purchase agreement and provide the assured return desired by the farmer.

You are now dependent on a Farm.  What is the #1 risk to your manure production?  Water!!!  When there is a drought there is an impact to agriculture production and cattle need a lot of water.  This article says it takes 2,000 gallons of water to make a gallon of milk.

It can take up to 2,000 gallons of water to produce one gallon of milk. The cow needs water to perform basic biological functions from day to day, and only a fraction of the water the cow consumes is actually converted into milk. The fact that it takes so much water to produce cow's milk means that anytime you or any consumer chooses to drink milk, the burden you place on the natural environment is a thousand times greater than if you were to consume water itself. Drinking one gallon of milk is like pouring 1,999 gallons of fresh water down the drain.

Actually putting a data center in operation using a Farm has these risks like water and methane gas.  There are a bunch of other issues that can be addressed like water. 

Mike Manos and I regularly discuss that water is the next scarce resource for data centers.  Be careful not thinking about the secondary and tertiary affects of a change in the water supply.

I congratulate the guys at HP for creating more awareness that a microgrid data center strategy has merit.

Read more