A Data Intensive Statistical Aggregation Engine: A Case Study for Gridded Climate Records

Satellite derived climate instrument records are often highly structured and conform to the “Data-Cube” topology. However, data scales on the order of tens to hundreds of Terabytes make it more difficult to perform the rigorous statistical aggregation and analytics necessary to investigate how our climate is changing over time and space. It is especially cumbersome to supply the full derivation (provenance) of this analysis, as is increasingly required by scientific conferences and journals. In this paper, we address our approach toward the creation of a 55 Terabyte decadal record of Outgoing Longwave Spectrum (OLS) from the NASA Atmospheric Infrared Sounder (AIRS), and describe our open source data-intensive statistical aggregation engine “Gridderama” intended primarily for climate trend analysis, and may be applicable to other aggregation problems involving large structured datasets.