In-Depth Exploration of Single-Snapshot Lossy Compression Techniques for N-Body Simulations

by   DingDingwen Tao, et al.

In situ lossy compression allowing user-controlled data loss can significantly reduce the I/O burden. For large-scale N-body simulations where only one snapshot can be compressed at a time, the lossy compression ratio is very limited because of the fairly low spatial coherence of the particle data. In this work, we assess the state-of-the-art single-snapshot lossy compression techniques of two common N-body simulation models: cosmology and molecular dynamics. We design a series of novel optimization techniques based on the two representative real-world N-body simulation codes. For molecular dynamics simulation, we propose three compression modes (i.e., best speed, best tradeoff, best compression mode) that can refine the tradeoff between the compression rate (a.k.a., speed/throughput) and ratio. For cosmology simulation, we identify that our improved SZ is the best lossy compressor with respect to both compression ratio and rate. Its compression ratio is higher than the second-best compressor by 11 Experiments with up to 1024 cores on the Blues supercomputer at Argonne show that our proposed lossy compression method can reduce I/O time by 80 with writing data directly to a parallel file system and outperforms the second-best solution by 60 have the best rate-distortion with reasonable compression errors on the tested N-body simulation data compared with state-of-the-art compressors.


page 1

page 2

page 3

page 4


Polynomial data compression for large-scale physics experiments

The new generation research experiments will introduce huge data surge t...

Understanding GPU-Based Lossy Compression for Extreme-Scale Cosmological Simulations

To help understand our universe better, researchers and scientists curre...

CEAZ: Accelerating Parallel I/O via Hardware-Algorithm Co-Design of Efficient and Adaptive Lossy Compression

As supercomputers continue to grow to exascale, the amount of data that ...

Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP

With ever-increasing volumes of scientific data produced by HPC applicat...

A Parallel Data Compression Framework for Large Scale 3D Scientific Data

Large scale simulations of complex systems ranging from climate and astr...

Time-universal data compression and prediction

Suppose there is a large file which should be transmitted (or stored) an...

Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling

Extreme-scale cosmological simulations have been widely used by today's ...

Please sign up or login with your details

Forgot password? Click here to reset