Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets
📜 Abstract
Today’s scientific simulations require a significant reduction of the data size because of extremely large volumes of data they produce and the limitation of storage bandwidth and space. If the compression is set to reach a high compression ratio, however, the reconstructed data are often distorted too much to tolerate. In this paper, we explore a new compression strategy that can effectively control the data distortion when significantly reducing the data size. The contribution is threefold. (1) We propose an adaptive compression framework to select either our improved Lorenzo prediction method or our optimized linear regression method dynamically in different regions of the dataset. (2) We explore how to select them accurately based on the data features in each block to obtain the best compression quality. (3) We analyze the effectiveness of our solution in details using four real-world scientific datasets with 100+ fields. Evaluation results confirm that our new adaptive solution can significantly improve the rate distortion for the lossy compression with fairly high compression ratios. The compression ratio of our compressor is 1.5X∼8X as high as that of two other leading lossy compressors (SZ and ZFP) with the same PSNR, in the high-compression cases. Parallel experiments with 8,192 cores and 24 TB of data shows that our solution obtains 1.5X dumping performance and 1.72X loading performance compared with the second-best lossy compressor, respectively.
✨ Summary
The paper presents an error-bounded lossy compressor designed for scientific datasets at high compression ratios, where conventional predictors can produce severe artifacts and poor reconstruction quality. The method divides multidimensional data into independent blocks and adaptively chooses among three predictors: the classic Lorenzo predictor, a mean-integrated Lorenzo predictor, and a linear-regression predictor.
The mean-integrated Lorenzo predictor replaces clustered values within a dense error-bound interval with the interval’s mean, reducing systematic prediction bias and visual artifacts. The regression predictor fits a linear hyperplane to each block and avoids the error-propagation limitation caused by using previously decompressed values. A lightweight sampling-based cost model selects the predictor expected to produce the smallest prediction error for each block. Quantized prediction residuals are Huffman encoded, while regression coefficients are compressed separately using a controlled lossy representation.
Experiments covered 104 fields from CESM-ATM, Hurricane ISABEL, NYX, and SCALE-LETKF simulations. The proposed method maintained the specified pointwise error bounds and, in the reported high-compression regimes, achieved compression ratios 1.5–8 times higher than SZ and ZFP at comparable PSNR. At 8,192 cores and 24 TB of NYX data, the paper reports improved parallel dumping and loading performance relative to the competing compressors evaluated.
A later study explicitly identifies this work’s adaptive selection of the mean-integrated Lorenzo and linear-regression predictors as an improvement incorporated into SZ 2.0, indicating that the paper’s predictor-selection strategy influenced subsequent development of the SZ compressor family. (par.nsf.gov) The work is also listed in subsequent scientific-compression surveys and research literature as part of the development of prediction-based, error-bounded compression for high-performance computing. (doi.org)