TY - GEN
T1 - Parallel Read Partitioning for Concurrent Assembly of Metagenomic Data
AU - Rengasamy, Vasudevan
AU - Kandemir, Mahmut T.
AU - Medvedev, Paul
AU - Madduri, Kamesh
N1 - Funding Information:
This research is supported in part by NSF awards #1453527, #1356529, and #1439057. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231 and the Extreme Science and Engineering Discovery Environment (XSEDE) [39], which is supported by National Science Foundation grant number ACI-1548562. We thank Chita Das for providing access to the Ganga cluster.
Funding Information:
ACKNOWLEDGMENT This research is supported in part by NSF awards #1453527, #1356529, and #1439057. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231 and the Extreme Science and Engineering Discovery Environment (XSEDE) [39], which is supported by National Science Foundation grant number ACI-1548562. We thank Chita Das for providing access to the Ganga cluster.
Publisher Copyright:
© 2018 IEEE.
PY - 2018/7/2
Y1 - 2018/7/2
N2 - We present MetaPartMin and MetaPart, two new lightweight parallel metagenomic read partitioning strategies. Metagenomic data partitioning can aid the concurrent de novo assembly of partitions. Prior read partitioning methods tend to create a giant component of reads. We avoid this problem with new heuristics amenable to statically load-balanced parallelization. Our strategies require enumerating and sorting k-mers and minimizers from the input read sequences, and traversing an implicit graph to identify components. MetaPartMin uses minimizers to significantly lower aggregate main memory use, thereby enabling the processing of massive datasets on a modest number of compute nodes. All steps in our strategies exploit hybrid multicore and distributed-memory parallelism. We demonstrate scaling and efficiency on a collection of large-scale datasets. MetaPartMin can process a 1.25 terabase soil metagenome in 6 minutes on just 32 Intel Skylake nodes (48 cores each) of the Stampede2 supercomputer, and a 252 gigabase soil metagenome in 54 seconds on 16 Stampede2 Skylake nodes. The source code is available at https://github.com/vasupsu/MetaPart.
AB - We present MetaPartMin and MetaPart, two new lightweight parallel metagenomic read partitioning strategies. Metagenomic data partitioning can aid the concurrent de novo assembly of partitions. Prior read partitioning methods tend to create a giant component of reads. We avoid this problem with new heuristics amenable to statically load-balanced parallelization. Our strategies require enumerating and sorting k-mers and minimizers from the input read sequences, and traversing an implicit graph to identify components. MetaPartMin uses minimizers to significantly lower aggregate main memory use, thereby enabling the processing of massive datasets on a modest number of compute nodes. All steps in our strategies exploit hybrid multicore and distributed-memory parallelism. We demonstrate scaling and efficiency on a collection of large-scale datasets. MetaPartMin can process a 1.25 terabase soil metagenome in 6 minutes on just 32 Intel Skylake nodes (48 cores each) of the Stampede2 supercomputer, and a 252 gigabase soil metagenome in 54 seconds on 16 Stampede2 Skylake nodes. The source code is available at https://github.com/vasupsu/MetaPart.
UR - http://www.scopus.com/inward/record.url?scp=85063138815&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85063138815&partnerID=8YFLogxK
U2 - 10.1109/HiPC.2018.00044
DO - 10.1109/HiPC.2018.00044
M3 - Conference contribution
AN - SCOPUS:85063138815
T3 - Proceedings - 25th IEEE International Conference on High Performance Computing, HiPC 2018
SP - 324
EP - 333
BT - Proceedings - 25th IEEE International Conference on High Performance Computing, HiPC 2018
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 25th IEEE International Conference on High Performance Computing, HiPC 2018
Y2 - 17 December 2018 through 20 December 2018
ER -