TY - JOUR
T1 - IOPro
T2 - a parallel I/O profiling and visualization framework for high-performance storage systems
AU - Kim, Seong Jo
AU - Zhang, Yuanrui
AU - Son, Seung Woo
AU - Kandemir, Mahmut
AU - Liao, Wei keng
AU - Thakur, Rajeev
AU - Choudhary, Alok
N1 - Publisher Copyright:
© 2014, Springer Science+Business Media New York.
PY - 2015/3
Y1 - 2015/3
N2 - Efficient execution of large-scale scientific applications requires high-performance computing systems designed to meet the I/O requirements. To achieve high-performance, such data-intensive parallel applications use a multi-layer layer I/O software stack, which consists of high-level I/O libraries such as PnetCDF and HDF5, the MPI library, and parallel file systems. To design efficient parallel scientific applications, understanding the complicated flow of I/O operations and the involved interactions among the libraries is quintessential. Such comprehension helps identify I/O bottlenecks and thus exploits the potential performance in different layers of the storage hierarchy. To profile the performance of individual components in the I/O stack and to understand complex interactions among them, we have implemented a GUI-based integrated profiling and analysis framework, IOPro. IOPro automatically generates an instrumented I/O stack, runs applications on it, and visualizes detailed statistics based on the user-specified metrics of interest. We present experimental results from two different real-life applications and show how our framework can be used in practice. By generating an end-to-end trace of the whole I/O stack and pinpointing I/O interference, IOPro aids in understanding I/O behavior and improving the I/O performance significantly.
AB - Efficient execution of large-scale scientific applications requires high-performance computing systems designed to meet the I/O requirements. To achieve high-performance, such data-intensive parallel applications use a multi-layer layer I/O software stack, which consists of high-level I/O libraries such as PnetCDF and HDF5, the MPI library, and parallel file systems. To design efficient parallel scientific applications, understanding the complicated flow of I/O operations and the involved interactions among the libraries is quintessential. Such comprehension helps identify I/O bottlenecks and thus exploits the potential performance in different layers of the storage hierarchy. To profile the performance of individual components in the I/O stack and to understand complex interactions among them, we have implemented a GUI-based integrated profiling and analysis framework, IOPro. IOPro automatically generates an instrumented I/O stack, runs applications on it, and visualizes detailed statistics based on the user-specified metrics of interest. We present experimental results from two different real-life applications and show how our framework can be used in practice. By generating an end-to-end trace of the whole I/O stack and pinpointing I/O interference, IOPro aids in understanding I/O behavior and improving the I/O performance significantly.
UR - http://www.scopus.com/inward/record.url?scp=84925488709&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84925488709&partnerID=8YFLogxK
U2 - 10.1007/s11227-014-1329-0
DO - 10.1007/s11227-014-1329-0
M3 - Article
AN - SCOPUS:84925488709
SN - 0920-8542
VL - 71
SP - 840
EP - 870
JO - Journal of Supercomputing
JF - Journal of Supercomputing
IS - 3
ER -