Labeling poststorm coastal imagery for machine learning: Measurement of interrater agreement

Evan B. Goldstein; Daniel D. Buscombe; Eli D. Lazarus; Somya Mohanty; Shah N. Rafique; K A Anarde; Andrew D Ashton; Tomas Beuzen; Katherine A. Castagno; Nicholas Cohn; Matthew P. Conlin; Ashley Ellenson; Megan Gillen; Paige A. Hovenga; Jin-Si R. Over; Rose V. Palermo; Katherine Ratlif; Ian R Reeves; Lily H. Sanborn; Jessamin A. Straub; Luke A. Taylor; Elizabeth J. Wallace; Jonathan Warrick; Phillipe Alan Wernette; Hannah E Williams

doi:10.1029/2021EA001896

Labeling poststorm coastal imagery for machine learning: Measurement of interrater agreement

Earth and Space Science

By: Evan B. Goldstein, Daniel D. Buscombe, Eli D. Lazarus, Somya Mohanty, Shah N. Rafique, K A Anarde, Andrew D Ashton, Tomas Beuzen, Katherine A. Castagno, Nicholas Cohn, Matthew P. Conlin, Ashley Ellenson, Megan Gillen, Paige A. Hovenga, Jin-Si R. Over, Rose V. Palermo, Katherine Ratlif, Ian R Reeves, Lily H. Sanborn, Jessamin A. Straub, Luke A. Taylor, Elizabeth J. Wallace, Jonathan Warrick, Phillipe Alan Wernette, and Hannah E Williams

https://doi.org/10.1029/2021EA001896

Links

More information: Publisher Index Page (via DOI)
Open Access Version: Publisher Index Page
Download citation as: RIS | Dublin Core

Abstract

Classifying images using supervised machine learning (ML) relies on labeled training data—classes or text descriptions, for example, associated with each image. Data-driven models are only as good as the data used for training, and this points to the importance of high-quality labeled data for developing a ML model that has predictive skill. Labeling data is typically a time-consuming, manual process. Here, we investigate the process of labeling data, with a specific focus on coastal aerial imagery captured in the wake of hurricanes that affected the Atlantic and Gulf Coasts of the United States. The imagery data set is a rich observational record of storm impacts and coastal change, but the imagery requires labeling to render that information accessible. We created an online interface that served labelers a stream of images and a fixed set of questions. A total of 1,600 images were labeled by at least two or as many as seven coastal scientists. We used the resulting data set to investigate interrater agreement: the extent to which labelers labeled each image similarly. Interrater agreement scores, assessed with percent agreement and Krippendorff's alpha, are higher when the questions posed to labelers are relatively simple, when the labelers are provided with a user manual, and when images are smaller. Experiments in interrater agreement point toward the benefit of multiple labelers for understanding the uncertainty in labeling data for machine learning research.

Study Area

Additional publication details
Publication type	Article
Publication Subtype	Journal Article
Title	Labeling poststorm coastal imagery for machine learning: Measurement of interrater agreement
Series title	Earth and Space Science
DOI	10.1029/2021EA001896
Volume	8
Issue	9
Year Published	2021
Language	English
Publisher	American Geophysical Union
Contributing office(s)	Pacific Coastal and Marine Science Center, Southwest Biological Science Center, Woods Hole Coastal and Marine Science Center
Description	e2021EA001896, 18 p.
Google Analytic Metrics	Metrics page