Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization
📜 Abstract
Today’s HPC applications are producing extremely large amounts of data, such that data storage and analysis are becoming more challenging for scientific research. In this work, we design a new error-controlled lossy compression algorithm for large-scale scientific data. Our key contribution is significantly improving the prediction hitting rate (or prediction accuracy) for each data point based on its nearby data values along multiple dimensions. We derive a series of multilayer prediction formulas and their unified formula in the context of data compression. One serious challenge is that the data prediction has to be performed based on the preceding decompressed values during the compression in order to guarantee the error bounds, which may degrade the prediction accuracy in turn. We explore the best layer for the prediction by considering the impact of compression errors on the prediction accuracy. Moreover, we propose an adaptive error-controlled quantization encoder, which can further improve the prediction hitting rate considerably. The data size can be reduced significantly after performing the variable-length encoding because of the uneven distribution produced by our quantization encoder. We evaluate the new compressor on production scientific data sets and compare it with many other state-of-the-art compressors: GZIP, FPZIP, ZFP, SZ-1.1, and ISABELA. Experiments show that our compressor is the best in class, especially with regard to compression factors (or bit-rates) and compression errors (including RMSE, NRMSE, and PSNR). Our solution is better than the second-best solution by more than a 2x increase in the compression factor and 3.8x reduction in the normalized root mean squared error on average, with reasonable error bounds and user-desired bit-rates.
✨ Summary
The paper introduces SZ-1.4, an error-bounded lossy compressor for multidimensional scientific floating-point data. Its main technical contributions are a multilayer multidimensional predictor, selection of the prediction depth using decompressed rather than original values, adaptive error-controlled quantization, and Huffman-based variable-length encoding of the resulting nonuniform quantization codes. The resulting algorithm has overall linear time complexity and supports parallel processing by independently compressing data partitions.
On the evaluated climate, X-ray, and hurricane data sets, the authors report higher compression factors and lower RMSE/NRMSE than GZIP, FPZIP, ZFP, SZ-1.1, and ISABELA under the tested error bounds. The principal trade-offs are lower sequential throughput than ZFP and application-dependent autocorrelation of compression errors; the paper also reports near-linear parallel scaling through 1,024 processes, with efficiency declining to approximately 90% at the largest tested scale.
The work influenced subsequent scientific-compression research primarily through the SZ-1.4 design and implementation. Later work used SZ-1.4 as a reference compressor when developing SIMD CPU acceleration, reporting improvements in rate-distortion performance and speed relative to SZ-1.4. (arxiv.org) Z-checker subsequently treated SZ-1.4 as a principal compressor in a broader framework for evaluating compression quality across scientific applications and metrics. (journals.sagepub.com) Later research on unstructured-mesh compression explicitly redesigned the prediction stage of SZ and describes SZ-1.4’s Lorenzo-prediction, quantization, and encoding pipeline as a baseline. (onlinelibrary.wiley.com) Survey literature likewise places SZ-1.4 among the foundational prediction-based, error-bounded scientific compressors that motivated later developments in prediction, quantization, and quality control. (doi.org)