Five Fascinating Power Facts About Supercomputers

Supercomputers do more than predict the weather, model nuclear reactions and help develop new drugs.

Thought you knew everything about supercomputers - those almost unimaginably powerful machines that predict the weather, model nuclear reactions, and help develop new drugs? Read on to find out a few cool things about their power systems you might not be aware of and amaze your geeky friends & family.

They use lots of power, and take up lots of space 

OK - you might know this first one. But we're talking about some serious power and space here. Tianhe-2, the current top dog on the TOP500 list , which ranks machines by how fast they burn through the LINPACK benchmark, draws an eye-popping 17.8MW of power, enough to supply some 5100 homes, and sprawls over 720 square meters - about 7750 square feet.

Tianhe-2: the world's fastest supercomputer
(Source: cnmeonline)

What does it take to keep the lights on? For the No. 2 machine Titan, power is provided to each cabinet at 480 Volts. This voltage requires thinner cables than the US standard 208 V, saving $1 million in copper. At its peak, the relatively parsimonious Titan draws a mere 8.2 MW - 1.2 MW more than Jaguar, its predecessor, but giving almost 10 times the floating point performance.

In the event of a power failure, carbon fiber flywheel power storagecan keep the networking and storage infrastructure running for up to 16 seconds. After 2 seconds without power, diesel generators kick into action. They can provide power indefinitely, but are designed only to keep the networking and storage components powered so that a reboot is much quicker; the generators aren't capable of powering the processing infrastructure.

Cooling system design is key to success

Needless to say, at these power levels getting rid of the heat produced is also a major concern. Over the years, numerous approaches have been used – the Cray-2 was completely immersed in Fluorinert, an inert liquid - but liquid cooling with water is the current method of choice.

This has led to some innovative implementations. IBM's liquid-cooled Aquasar runs its cooling fluid through heat exchangers to provide heat for surrounding buildings at Swiss university ETH Zurich. Added bonus? It lowers the carbon footprint an estimated 85%.

Liquid cooled blade used for the Aquasar system
(source: ETH)

Google pipes in sea water through tunnels direct from the Bay of Finland to help cool its Hamina data center. To minimize the environmental impact, they cool the returning water again before returning it to the bay so that it's the same temperature as its surroundings.

None of this cooling infrastructure comes for free power-wise, of course. For example, the Tianhe-2 cooling system, which uses city water, consumes another 6 MW of power.

Off-the-shelf components are a key part of helping supercomputers achieve their speed

Unlike earlier custom designs, current supercomputers attain petaflop-level performance through massively parallel architectures using mostly off-the-shelf components, including power supplies.

For example, the IBM Blue Gene/Q family of machines, which forms the basis for Sequoia (No. 3 on the TOP500 list) uses the Vicor Factorized Power architecture to supply 2.8kW to each card, with primary processor power being 0.8V at 130A.

For the cores themselves, many machines rely heavily on graphics processing units (GPUs) originally developed for video gaming. In fact, 75 of the top 100 systems use GPUs from either NVIDIA, ATI Radeon, or Intel (Xeon Phi). Nvidia has a nine-year studyconducted on 22 systems at Los Alamos National Laboratory found failure rates as high as 1100 per year.

This awareness has led to the emergence of other ranking methods, including the Green500 list which ranks the top 500 supercomputers in the world by energy efficiency.

As you might expect, only one machine -- Piz Daint -- appears in the top 10 of both the TOP500 and Green500 lists. The No. 1 machine on the Green500 list achieves 5,271 MFLOPS/W. Contrast that with Tianhe-2, No. 1 on the TOP500. Its 2077 MFLOPS/W is only good enough for 57th place on the Green500.

Fundamental changes are ahead for the next generation of machines

Supercomputing performance levels have ramped up steadily over the last 20 twenty years in lock step with advances in semiconductor technology, but storm clouds are on the horizon. The next big target in performance is exascale computing -- that's 1018 FLOPS, around 30 times that of the Tianhe-2.

But that level is unlikely to be reached by a straight-line extrapolation of current techniques. Such a machine would require around 540MW of power and consume 100% of the output of a gas-fired power plant.

Early hopes for exascale computing as early as 2018 have faded; current predictions are 2022 or even later. A 2008 study for DARPAconcluded that a whole new way of thinking about architectures, programming, and energy would be needed to meet the goal.

The problems to be solved are fundamental, both at the CMOS transistor level and with the von Neumann architecture itself. There are several promising lines of research aimed at both reducing power consumption and eliminating the von Neumann bottleneck, though, including: lower-power memory; assortment of rock stars: anyone feel like programming "U2", "Joplin", "Crosby" or "Nash"...??

Makes you wonder why Neil Young didn't make the list. Perhaps the CCR (yes, really) director was a Lynyrd Skynyrd fan.

The inspiration for the current TOP500 list of the world's most powerful machines runs the gamut: “Tianhe“ ("Milky Way" in Chinese), “Titan,“ “Sequoia“, “Stampede“ (at the University of Texas, naturally - they previously had "Longhorn"), “Mira“ (a red giant star), “Vulcan“, “Piz Daint“ (a mountain in the Swiss Alps)... and the enigmatic K Computer, installed at the RIKEN Institute in Kobe, Japan. Although there is a train station named after it.

The trend towards personalizing supercomputers includes dressing them up in some pretty nifty custom graphics, too. Take a look hereand here.

How many fascinating facts did you know already? Before writing this, I came in at about 20%. But then, I have trouble programming my toaster.

—Paul Pickering writes about power management, and occasionally supercomputers, for EE Times.