Wrangling distributed computing for high-throughput environmental science: An introduction to HTCondor

PLOS Computational Biology
By:

Links

Abstract

Biologists and environmental scientists now routinely solve computational problems that were unimaginable a generation ago. Examples include processing geospatial data, analyzing -omics data, and running large-scale simulations. Conventional desktop computing cannot handle these tasks when they are large, and high-performance computing is not always available nor the most appropriate solution for all computationally intense problems. High-throughput computing (HTC) is one method for handling computationally intense research. In contrast to high-performance computing, which uses a single "supercomputer," HTC can distribute tasks over many computers (e.g., idle desktop computers, dedicated servers, or cloud-based resources). HTC facilities exist at many academic and government institutes and are relatively easy to create from commodity hardware. Additionally, consortia such as Open Science Grid facilitate HTC, and commercial entities sell cloud-based solutions for researchers who lack HTC at their institution. We provide an introduction to HTC for biologists and environmental scientists. Our examples from biology and the environmental sciences use HTCondor, an open source HTC system.

Additional publication details

Publication type Article
Publication Subtype Journal Article
Title Wrangling distributed computing for high-throughput environmental science: An introduction to HTCondor
Series title PLOS Computational Biology
DOI 10.1371/journal.pcbi.1006468
Volume 14
Issue 10
Year Published 2018
Language English
Publisher PLOS
Contributing office(s) Upper Midwest Environmental Sciences Center
Description e1006468; 8 p.
First page 1
Last page 8