Below is a lot of different parts in what Univ of Illinois’s NCSA facility is building to host the IBM Blue Waters Super Computer. I’ve seen lots of people talk about energy efficiency and cost savings. But, the things that got my attention is the fact is this facility dropped the UPS feature and it is built for $3mil per mW for a 24 mW facility.

How can this be done? I think a key contributor is IBM’s computer architects were involved to help make sure the building was designed to Blue Waters needs.

Maybe one of these days I can visit the facility in ~~Chicago~~Urbana-Champaign, but I can learn a lot just from the knowing where to look for information on the web.

Cnet news has an article IBM’s Blue Water super computer at University of Illinois National Center for Supercomputing (NCSA). But this article doesn’t have much details about the building. I’ve had a few discussions with IBM’s supercomputing folks and I knew they have put a lot of work into the buildings, but it can be sometimes hard to get the information. The good thing is given the project is run by Univ of Illinois there is public information you can get to like here.

William Kramer, Deputy Project Director, Blue Waters

By William Kramer
Deputy Project Director, Blue Waters

The computational science and engineering community requires five attributes from the systems they use and the facilities that provide those systems. These attributes deliver systems that efficiently and productively enhance the scientists' ability to achieve novel results. They are performance, effectiveness, reliability, consistency, and usability (which I refer to as the PERCU method). This is a holistic, user-based approach to developing and assessing computing systems, in particular HPC systems. The method enables organizations to use flexible metrics to assess the features and functions of HPC systems and, if they choose to purchase systems, assess them against the requirements negotiated with the vendor.

Here is a video of the raised floor above being built out.

But wanting more details I dug around for details about the site. Here are details about the site. Note the last paragraph. No UPS.

Energy efficiency is an integral part of the Blue Waters project and the Petascale Computing Facility. The facility will:

Achieve LEED Silver certification, with LEED Gold as the goal.

Rely heavily on more efficient water cooling for the systems it houses.

Take advantage of an on-site tower to chill water for cooling the compute systems. This will reduce energy consumption by using the outside air to chill water during the cold winter months.

Take advantage of the campus' highly reliable electricity supply, avoiding the need for the standard back-up Uninterruptible Power Supply (UPS). Eliminating the UPS saves equipment costs, minimizes floor space used, and increases energy efficiency because systems that employ a UPS convert AC to DC and back, incurring substantial energy losses.

Also, Blue Water uses water directly to the IT equipment.

And how does IBM keep this dense collection of ultrafast processors cool? In a word, water. "We actually went a bit further environmentally," said Ed Seminaro, an IBM Fellow who is involved with the University of Illinois project. "We took a lot of the infrastructure that's typically inside of the computer room for cooling and powering and moved the equivalent of that infrastructure right into that same cabinet with the server, storage, and interconnect hardware."

Seminaro continued: "The whole rack is water-cooled. We actually water-cool the processor directly to pull the heat out. We take it right to water, which is very power efficient," he said.

John Melchi in the video below discusses the building and how it was designed to have efficient power and cooling systems. Here is a transcript of his conversation.

One of the things you don’t think about when you look at a facility like this is the fact that the computer architect has been involved in the design of the building. So IBM has just been a tremendous partner and collaborator in helping Illinois and NCSA ensure that the Petascale Computing Facility will meet the needs of Blue Waters.

Specifically, we’ve made sure there’s enough space, power, and cooling. At the level of Blue Waters, you’re talking about substantial amounts of infrastructure to make a computer and a project like this work.

From the beginning the U of I and NCSA intended to build a data center that was a multi‐use facility. We have the ability to provide 5,400 tons of chilled water to the building. We have 24 megawatts of power coming in. That’s substantially more than the Blue Waters system is going to need. So we’re very well positioned to bring in new air‐cooled systems to the Petascale Computing Facility that will enable U of I researchers and researchers across the country to do their science.

But not just not the building is changed to accommodate Blue Water. the applications are as well.

The Blue Waters staff is now working with about 20 large science teams to start revising their application codes to take full advantage of the Blue Waters features. Much of the work will enable codes to run well and at large scale on Blue Waters, but the work can also be applied to other systems in the future. We are doing this with simulation of the machine itself, application and system performance modeling with premier modeling groups, and early access to prototype systems and software. Over time, we will engage with other science areas as they are allocated time on Blue Waters.

CNET news’s article.

IBM: Envisioning the world's fastest supercomputer

IBM will release a radical new chip next year that will go into a University of Illinois supercomputer in a quest to build what may become the world's fastest supercomputer.

That university's supercomputer center is a storied place, home to both famous fictional and real supercomputers. The notorious HAL 9000 sentient supercomputer in "2001: A Space Odyssey" was built in Urbana, Illinois, presumably on the University of Illinois Urbana-Champaign campus.

The Power7 chip die.

(Credit: IBM)

Though not aspiring to artificial intelligence, the IBM Blue Waters project supercomputer, like the HAL 9000 series, will be able to do massively complex calculations in an instant and, like HAL, be built in Urbana-Champaign. It is being housed in a special building on the Urbana-Champaign campus specifically for the computer that will theoretically be capable of achieving 10 petaflops, about 10 times as fast as the fastest supercomputer today. (A petaflop is 1 quadrillion floating point operations per second, a key indicator of supercomputer performance.)

Part of the National Center for Supercomputing Applications (NCSA) at the University of Illinois, it will be the largest publicly accessible supercomputer in the world when it's turned on sometime in 2011.

The data center for this will look like this

Artist rendering of University of Illinois center that will house IBM's Blue Waters supercomputer

(Credit: University of Illinois)