thumbnail

A guide to creating an effective big data management framework

Journal of Big Data
By: , , , , and 
https://doi.org/10.1186/s40537-023-00801-9

Links

Abstract

Many agencies and organizations, such as the U.S. Geological Survey, handle massive geospatial datasets and their auxiliary data and are thus faced with challenges in storing data and ingesting it, transferring it between internal programs, and egressing it to external entities. As a result, these agencies and organizations may inadvertently devote unnecessary time and money to convey data without existing or outdated standards. This research aims to evaluate the components of data conveyance systems, such as transfer methods, tracking, and automation, to guide their improved performance. Specifically, organizations face the challenges of slow dispatch time and manual intervention when conveying data into, within, and from their systems. Conveyance often requires skilled workers when the system depends on physical media such as hard drives, particularly when terabyte transfers are required. In addition, incomplete or inconsistent metadata may necessitate manual intervention, process changes, or both. A proposed solution is organization-wide guidance for efficient data conveyance. That guidance involves systems analysis to outline a data management framework, which may include understanding the minimum requirements of data manifests, specification of transport mechanisms, and improving automation capabilities.

Publication type Article
Publication Subtype Journal Article
Title A guide to creating an effective big data management framework
Series title Journal of Big Data
DOI 10.1186/s40537-023-00801-9
Volume 10
Year Published 2023
Language English
Publisher Springer
Contributing office(s) NGTOC Rolla, Center for Geospatial Information Science (CEGIS)
Description 146, 22 p.
Google Analytic Metrics Metrics page
Additional publication details