BlueGene/L failure analysis and prediction models

Yinglung Liang, Yanyong Zhang, Morris Jette, Anand Sivasubramaniam, Ramendra Sahoo

Research output: Chapter in Book/Report/Conference proceedingConference contribution

227 Scopus citations


The growing computational and storage needs of several scientific applications mandate the deployment of extreme-scale parallel machines, such as IBM's BlueGene/L which can accommodate as many as 128K processors. One of the challenges when designing and deploying these systems in a production setting is the need to take failure occurrences, whether it be in the hardware or in the software, into account. Earlier work has shown that conventional runtime fault-tolerant techniques such as periodic checkpointing are not effective to the emerging systems. Instead, the ability to predict failure occurrences can help develop more effective checkpointing strategies. Failure prediction has long been regarded as a challenging research problem, mainly due to the lack of realistic failure data from actual production systems. In this study, we have collected RAS event logs from BlueGene/L over a period of more than 100 days. We have investigated the characteristics of fatal failure events, as well as the correlation between fatal events and non-fatal events. Based on the observations, we have developed three simple yet effective failure prediction methods, which can predict around 80% of the memory and network failures, and 47% of the application I/O failures.

Original languageEnglish (US)
Title of host publicationProceedings - DSN 2006
Subtitle of host publication2006 International Conference on Dependable Systems and Networks
Number of pages10
StatePublished - 2006
EventDSN 2006: 2006 International Conference on Dependable Systems and Networks - Philadelphia, PA, United States
Duration: Jun 25 2006Jun 28 2006

Publication series

NameProceedings of the International Conference on Dependable Systems and Networks


OtherDSN 2006: 2006 International Conference on Dependable Systems and Networks
Country/TerritoryUnited States
CityPhiladelphia, PA

All Science Journal Classification (ASJC) codes

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications


Dive into the research topics of 'BlueGene/L failure analysis and prediction models'. Together they form a unique fingerprint.

Cite this