Beyond the Data Bottleneck: Physics AI Training with Compressed Data Streaming

Physics AI places enormous demands on data infrastructure. From automotive neural surrogates to climate modelling, training datasets can reach multiple petabytes, yet the field still lacks a common data layer. Staging and preprocessing can take days before a training experiment even begins.

As models expand to time-dependent simulations and foundation models spanning multiple physics domains, these challenges become harder to manage. Infrastructure costs rise, experiments are delayed, and expensive GPUs sit underutilised while waiting for data. Scaling Physics AI requires rethinking the entire pipeline, from how simulation data is stored and accessed to how it reaches the model.

Simulation caches can be 3–4 orders of magnitude larger than other media datasets, such as video. Moving them off a simulation cluster and creating multiple copies during preprocessing quickly becomes expensive and impractical. Video streaming offers a familiar example of a much better approach. Watching a film once meant downloading a large file. Today, we choose a title, press play and receive only the compressed content needed for playback. Scientific workflows should offer the same convenience: open a dataset, request the region, field or timestep needed for an experiment, and start working while the rest stays in storage.

ZibraXYZ brings these requirements together in a common data layer for scientific datasets. It combines configurable lossless, lossy and hybrid compression with random access and GPU-native decompression, so applications can retrieve selected data without unpacking the complete dataset. Crucially, that access extends to unstructured meshes: a request for a region retrieves the compressed pages containing the relevant cells, their physical fields, and the coordinates and connectivity needed to reconstruct a usable selection. Built on Zarr, the stack can use Icechunk for consistent, versioned snapshots, allowing teams to return to the dataset used for a particular analysis or experiment. Integrations with post-processing tools such as ParaView and AI training workflows in PyTorch let those applications consume selected data from the same underlying store, reducing the need for separate task-specific copies.

Your dataset library becomes a queryable database

Physics AI teams often work with tens or even hundreds of simulation datasets. These may use different file formats and require substantial preprocessing before they can be used for training.

ZibraXYZ turns this collection into a single queryable database. Request a specific timestep, a spatial region within it, or a field within that region. This works for both grids and meshes, and you can request point data without downloading the full geometry.

This enables what we call a Training Recipe: a description of the fields, spatial regions and timesteps needed for an experiment, applied across multiple datasets. The recipe produces a data stream integrated with PyTorch. You can train directly from that stream or cache the selected data for repeated production training runs.

Instead of extracting points and fields into separate training files during preprocessing, you request exactly what each experiment needs.

This workflow improves data access for Physics AI training and becomes especially critical for multi-physics models. Training data may come from several solvers, use different file formats and evolve as simulations are added or updated. Keeping datasets versioned while repeatedly preprocessing them into separate training copies quickly becomes difficult to manage. ZibraXYZ provides a common access layer, allowing training recipes to query the required data across dataset versions without rebuilding the entire training collection for each experiment.

Infrastructure relief

A single pre-trained Physics AI model requires at least hundreds of terabytes of simulation data. As a lab develops models for different use cases, storage can quickly reach petabyte scale. Time-dependent simulations and multi-physics models accelerate this growth. Climate modelling already operates at a scale of hundreds of petabytes, and automotive and aerospace teams face similar challenges as their simulation programmes expand. This creates another bottleneck: scaling Physics AI is economically challenging given the infrastructure costs.

At this stage, it is clear that advanced scientific data compression should be a part of the solution. But it has to preserve much more than visual fidelity. Scientific datasets contain geometry, connectivity and physical fields whose numerical accuracy affects engineering calculations and model training. Different representations and downstream tasks also need different precision settings.

ZibraXYZ addresses this through a combination of lossless and lossy compression strategies developed by Zibra over the past three years. A compression algorithm is selected according to the representation and statistical characteristics of the data. Sensitive arrays can remain lossless, while other fields use configurable error budgets. Compression settings are then evaluated against the quantities that matter for the application, including physical diagnostics and model performance.

Current implementations deliver a 10× compression for unstructured data and over 30× for structured grids. For compression benchmarks and quality metrics, read ZibraXYZ: Accelerating Scientific Data Access by 100×.

Remove the data loading bottleneck

With 10×–30× compression, 10–30 times less input data needs to cross PCIe. ZibraXYZ transfers the compressed payload to the GPU and decompresses it there, reducing the time spent moving each batch from host memory to the device.

This helps keep the GPU supplied with training data while reducing CPU-side loading and decompression work. Combined with prefetching, data transfer and GPU decompression can overlap with model computation, helping the next batch arrive before the model needs it.

On an NVIDIA A100, measured decompression throughput is approximately 60 GB/s for mesh data and 600 GB/s for structured grid data. In our turbulence super-resolution experiment on four NVIDIA A100 GPUs, decompression accounted for just 0.4% of the measured runtime. Average training-step time fell by 27.7%, with the same validation behaviour and the same number of steps to convergence.

The benefit depends on how much time the existing pipeline spends waiting for data. A workload with substantial loading and transfer delays has more to gain than one already dominated by model computation.

Try ZibraXYZ with your own training pipeline

Start with a dataset and training job your team already uses. We’ll help you compress the data, connect it to your PyTorch pipeline and compare it with your current setup.

You keep the same model architecture. ZibraXYZ decompresses the requested data before passing it to the model, so you can evaluate the benefits without redesigning your network.

Together, we’ll measure how much storage you save, how quickly batches reach the GPU and whether training runs faster. We’ll also check the model’s performance on validation data to find a compression setting that meets your quality requirements.