IBM Roadrunner

Infobox One-of-a-kind computers


Caption=Roadrunner components
Website=http://www.lanl.gov/roadrunner/
Dates=operational 2008, final completion 2009
Location=Los Alamos National Laboratory, nobreak|USA
Sponsors=IBM, nobreak|USA
Operators=National Nuclear Security Administration, nobreak|USA
Architecture=12,960 IBM PowerXCell 8i CPUs, 6,480 AMD Opteron dual-core processors, Infiniband, Linux
Memory=103.6 TiB
Storage=
Speed=1.7 petaflops (peak)
Power=2.35 MW
Space=296 racks, convert|6000|sqft|m2|abbr=on
Cost=US$133M
ChartName=TOP500
ChartPosition=1
ChartDate=June 2008
Purpose=Modeling the decay of the U.S. nuclear arsenal.
Legacy=First TOP500 Linpack sustained 1.0 petaflops, May 25, 2008
Emulators=
Sources=

Roadrunner is a supercomputer built by IBM at the Los Alamos National Laboratory in New Mexico, USA. Currently the world's fastest computer, the US$133-million Roadrunner is designed for a peak performance of 1.7 petaflops, achieving 1.026 on May 25, 2008, [cite web
url=http://www.computerworld.com/action/article.do?command=viewArticleBasic&taxonomyName=hardware&articleId=9095318&taxonomyId=12&intsrc=kc_top
title=IBM's Roadrunner smashes 4-minute mile of supercomputing
accessdate=2008-06-10
author=Sharon Gaudin
date=2008-06-09
publisher=Computerworld] [cite web
url=http://news.cnet.com/Military-supercomputer-sets-record/2100-1010_3-6241145.html?tag=nefd.top
title=Military supercomputer sets record - CNET News.com] [cite web
url=http://news.bbc.co.uk/1/hi/technology/7443557.stm
publisher=BBC
title=Supercomputer sets petaflop pace
accessdate=2008-06-09
date=2008-06-09 ] and to be the world's first TOP500 Linpack sustained 1.0 petaflops system. It is a one-of-a-kind supercomputer, built from commodity parts, with many novel design features.

Overview

IBM built the computer for the U.S. Department of Energy's (DOE) National Nuclear Security Administration. [cite web
url=http://www-03.ibm.com/press/us/en/pressrelease/20210.wss
publisher=IBM
title=IBM to Build World's First Cell Broadband Engine Based Supercomputer
accessdate=2008-05-31
date=2006-09-06 ] [cite web
url=http://nnsa.energy.gov/news/1015.htm
publisher=NNSA
title=IBM Selected to Build New DOE Supercomputer
accessdate=2008-05-31
date=2006-09-06 ] It is a hybrid design with 12,960 IBM PowerXCell [ [http://top500.org/blog/2008/06/09/international_supercomputing_conference_host_first_panel_discussion_breaking_petaflop_s_barrier International Supercomputing Conference to Host First Panel Discussion on Breaking the Petaflop/s Barrier] ] 8i CPUs and 6,480 AMD Opteron dual-core processors in specially designed server blades connected by Infiniband. The Roadrunner uses Red Hat Enterprise Linux along with Fedora as its operating systems and is managed with xCAT distributed computing software. It occupies approximately convert|6000|sqft|m2|abbr=off [cite web
url=http://www.ibm.com/news/us/en/2008/06/2008_06_09.html
publisher=IBM
title=Los Alamos computer breaks petaflop barrier
accessdate=2008-06-12
date=2008-06-09 ] and became operational in 2008.

The DOE plans to use the computer for simulating how nuclear materials age in order to predict whether the USA's aging arsenal of nuclear weapons is safe and reliable. Other uses for the Roadrunner include the sciences, financial, automotive and aerospace industries.

Hybrid design

Roadrunner differs from many contemporary supercomputers in that it is a hybrid system, using two different processor architectures. Usually supercomputers only use one, since such a design is easier to design and program for. To realise the full potential of Roadrunner, all software will have to be written specially for this hybrid architecture. The hybrid design consists of dual-core Opteron server processors manufactured by AMD utilizing the standard AMD64 architecture. Attached to each Opteron core is a Cell processor manufactured by IBM using Power Architecture technology. As a supercomputer, the Roadrunner is considered an Opteron cluster with Cell accelerators, as each node consists of a Cell attached to an Opteron core and the Opterons to each other.

The Opterons are very good general purpose processors, popular in supercomputer clusters and easy to program. However building a 1 petaflops system solely from Opteron processors would require at least 20 times as many processors as Roadrunner uses and would be extremely expensive to house, build, power and cool. For mathematical operations the Cell processors are nearly 30 times more powerful than the Opterons, but are not general purpose processors, so not well suited for general operating system tasks. This makes it impractical to design a supercomputer of this scale using only Cell processors.

Development

Roadrunner has been in development since 2002, and went online in 2006. Due to its novel design and complexity it is constructed in three phases and became fully operational in 2008.

Phase 1

The first phase of the Roadrunner was building a standard (albeit quite large) Opteron based cluster, while evaluating the feasibility to further construct and program the future hybrid version. This Phase 1 Roadrunner reached 71 teraflops and has been in full operation at Los Alamos National Laboratory doing advanced weapons simulations since 2006. Even if Roadrunner had not been greenlit for Phase 2, the Phase 1 form would still be a formidable supercomputer and would have ranked, at its time, in the top 10 list of the world's fastest computers.

Phase 2

Phase 2 known as “AAIS” (Advanced Architecture Initial System) included building a small hybrid version of the finished system using an older version of the Cell processor. This phase was used to build prototype applications for the hybrid architecture. It went online in January 2007.

Phase 3

The goal of Phase 3 was to reach sustained performance in excess of 1 petaflops. Additional Opteron nodes and new PowerXCell processors were added to the design. These PowerXCell processors are five times as powerful as the Cell processors used in Phase 2. It was built to full scale at IBM’s Poughkeepsie, New York facility, where it broke the 1 petaflops barrier during its fourth attempt on May 25th, 2008. The complete system will be moving to its permanent location in New Mexico in the summer of 2008, where fine tuning of the applications will continue until final completion in 2009.

Technical specification

Processors

The Roadrunner uses many processors in the system, some for computation (solving the problem at hand) and some for operation (helping the humans, storing data, monitoring, passing data around, etc). Roadrunner is unusual in the respect that it uses two different processors for solving the problems. Usually a supercomputer only uses one processor kind for the complete system or one for operations and another for the problem solving.

Opteron

AMD Opteron 2210, running at 1.8 GHz. These are processors with two general purpose cores each. Opterons are used both in the computational nodes feeding the Cells with useful data and in the system operations and communication nodes passing data between computing nodes and helping the operators running the system. Roadrunner has a total of 6912 Opteron processors (6480 computation, 432 operation), for a total of (12960+864) 13824 cores.

PowerXCell

IBM PowerXCell 8i, running at 3.2 GHz. These processors have one general purpose core (PPE), and eight special performance cores (SPE) for floating point operations. Roadrunner has a total of 12960 PowerXCell processors, with 12960 PPE cores and 103680 SPE cores, for a total of 116640 cores.

Number of cores

On the Top500 list, Roadrunner is said to have 122400 cores. It is important to know which core is counted.
* 13824 Opteron cores + 116640 Cell cores = 130464 cores for both the computing nodes and the operation nodes.This is a number larger than the one mentioned on Top500. It turns out that the Roadrunner only used 17 Connected Units while doing the LINPACK benchmark, and it was not counting the cores in the operations and communication nodes (they didn't run the benchmark). [cite web
url=http://versi.edu.au/downloads/SuperCGrice.pdf
publisher=Los Alamos National Laboratory and IBM
title=Roadrunner: Science, Cell and a Petaflop/s
accessdate=2008-09-22
date=2008-06-18 ]
* 6120 Opteron (2 cores) + 12240 PowerXCell 8i (9 cores) = 122400 cores

TriBlade

Logically, a TriBlade consists of two dual-core Opterons with 16 GB RAM and four PowerXCell 8i CPUs with 16 GB Cell RAM.cite web|url=http://www.lanl.gov/orgs/hpc/roadrunner/pdfs/Koch%20-%20Roadrunner%20Overview/RR%20Seminar%20-%20System%20Overview.pdf|publisher=Los Alamos National Laboratory|title=RR Seminar - System Overview|accessdate=2008-05-31|date=2008-03-13]

Physically, a TriBlade consists of one LS21 Opteron blade, an expansion blade, and two QS22 Cell blades. The LS21 has two 1.8 GHz dual-core Opterons with 16 GB memory for the whole blade, providing 8GB for each CPU. Each QS22 has two PowerXCell 8i CPUs, running at 3.2 GHz and 8GB memory, which makes 4 GB for each CPU. The expansion blade connects the two QS22 via four PCIe x8 links to the LS21, two links for each QS22. It also provides outside connectivity via an Infiniband 4x DDR adapter. This makes a total width of four slots for a single TriBlade. Three TriBlades fit into one BladeCenter H chassis.

Connected Unit (CU)

A Connected Unit is 60 BladeCenter H full of TriBlades, that is 180 TriBlades. All TriBlades are connected to a 288-port Voltaire ISR2012 Infiniband switch. Each CU also has access to the Panasas file system through twelve System x3755 servers.cite web|url=http://www.lanl.gov/orgs/hpc/roadrunner/pdfs/Koch%20-%20Roadrunner%20Overview/RR%20Seminar%20-%20System%20Overview.pdf|publisher=Los Alamos National Laboratory|title=RR Seminar - System Overview|accessdate=2008-05-31|date=2008-03-13] .

CU system information:.
* 360 dual-core Opterons with 2.88 TiB RAM.
* 720 PowerXCell 8i cores with 2.88 TiB RAM.
* 12 System x3755 with dual 10-GBit Ethernet each.
* 288-port Voltaire ISR2012 switch with 192 Infiniband 4x DDR links (180 TriBlades and twelve I/O nodes).

Roadrunner cluster

The final cluster is made up of 18 connected units, which are connected via eight additional (second-stage) Infiniband ISR2012 switches. Each CU is connected through twelve uplinks for each second-stage switch, that makes a total of 96 uplink connections.

Overall system information:
* 6,480 Opteron processors with 51.8 TiB RAM (in 3,240 LS21 blades)
* 12,960 Cell processors with 51.8 TiB RAM (in 6,480 QS22 blades)
* 216 System x3755 I/O nodes
* 26 288-port ISR2012 Infiniband 4x DDR switches
* 296 racks
* 2.35 MW power [cite web|url=http://www.top500.org/list/2008/06/100|publisher=TOP500 Supercomputer Sites|title=TOP500 List - June 2008|accessdate=2008-08-12|date=2008-06-15]

ee also

*Multi-core
*Computer architecture
*Central processing unit

References

External links

*cite web
url=http://www.lanl.gov/roadrunner
publisher=Los Alamos National Laboratory
title=Los Alamos National Laboratory Roadrunner Home Page
accessdate=2008-05-31
date=2007-03-30
*cite web
url=http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9085021
publisher=Computerworld
title=In Pictures: A look inside what may be the world's fastest supercomputer
accessdate=2008-05-31
date=2008-05-13