Wrangling distributed computing for high-throughput environmental science: An introduction to HTCondor
Links
- More information: Publisher Index Page (via DOI)
- Open Access Version: Publisher Index Page
- Download citation as: RIS | Dublin Core
Abstract
Biologists and environmental scientists now routinely solve computational problems that were unimaginable a generation ago. Examples include processing geospatial data, analyzing -omics data, and running large-scale simulations. Conventional desktop computing cannot handle these tasks when they are large, and high-performance computing is not always available nor the most appropriate solution for all computationally intense problems. High-throughput computing (HTC) is one method for handling computationally intense research. In contrast to high-performance computing, which uses a single "supercomputer," HTC can distribute tasks over many computers (e.g., idle desktop computers, dedicated servers, or cloud-based resources). HTC facilities exist at many academic and government institutes and are relatively easy to create from commodity hardware. Additionally, consortia such as Open Science Grid facilitate HTC, and commercial entities sell cloud-based solutions for researchers who lack HTC at their institution. We provide an introduction to HTC for biologists and environmental scientists. Our examples from biology and the environmental sciences use HTCondor, an open source HTC system.
| Publication type | Article |
|---|---|
| Publication Subtype | Journal Article |
| Title | Wrangling distributed computing for high-throughput environmental science: An introduction to HTCondor |
| Series title | PLOS Computational Biology |
| DOI | 10.1371/journal.pcbi.1006468 |
| Volume | 14 |
| Issue | 10 |
| Year Published | 2018 |
| Language | English |
| Publisher | PLOS |
| Contributing office(s) | Upper Midwest Environmental Sciences Center |
| Description | e1006468; 8 p. |
| First page | 1 |
| Last page | 8 |
| Google Analytic Metrics | Metrics page |