ZibraXYZ: accelerating scientific data access by 100×

Even after a scientific simulation finishes, turning its results into usable insights often takes time. The simulation is cached but an engineer still has to copy it to a workstation, a researcher has to extract a smaller region, and an AI team has to prepare another version for training. Each task starts with moving and reshaping data before the useful work can begin. As datasets reach petabytes, those steps become prohibitively expensive.

This is data gravity in practice. The size and location of a dataset start to dictate where people work, which tools they use and how much they can explore. A team may have the compute to run a larger simulation or train a more capable model, yet struggle to deliver the resulting data to the next application.

We have already seen a different experience emerge in other media. Watching a film once meant downloading a large file. Today, we choose a title, press play and receive the compressed content needed to start watching. Scientific workflows would benefit from the same convenience: open a dataset, ask for a region or timestep and begin working while the rest stays in storage.

The scale makes this challenging. An uncompressed 1080p RGB frame contains about 6.2 MB. A 1024 by 1024 by 1024 grid with eight float32 variables contains 32 GiB per timestep. A thousand timesteps add up to 31.25 TiB before additional metadata or geometry. Volumetric data is 3–4 orders of magnitude larger than other media modalities and it requires a new solution for the data movement problem.

To solve this problem, a universal 3D/4D compression approach has to handle more than a sequence of images. A structured grid, an unstructured mesh and a point cloud describe space differently. Achieving a high compression ratio is only part of the problem. The compressed data must also remain accessible in the way applications need it.

ZibraXYZ brings these requirements together in a common data layer for scientific datasets. It combines configurable lossless, lossy and hybrid compression with random access and GPU-native decompression, so applications can retrieve selected data without unpacking the complete dataset. Crucially, that access extends to unstructured meshes: a request for a region retrieves the compressed pages containing the relevant cells, their physical fields, and the coordinates and connectivity needed to reconstruct a usable selection. Built on Zarr, the stack can use Icechunk for consistent, versioned snapshots, allowing teams to return to the dataset used for a particular analysis or experiment. Integrations with post-processing tools such as ParaView and AI training workflows in PyTorch let those applications consume selected data from the same underlying store, reducing the need for separate task-specific copies.

The practical benefit is that a large dataset can remain available for several kinds of work without every user first bringing the whole thing to their machine. The sections below explain how representation support, precision settings and selective access make that possible, and how to measure the difference in a real workflow.

Different representations need a common way to access them

ZibraXYZ supports structured grids, unstructured meshes and point clouds, including associated physical fields and time series. This unified representation creates an interoperability layer between post-processing and visualisation solutions. Through the relevant application integrations, the same underlying dataset can be accessed in ParaView for quantitative analysis, Omniverse for collaborative inspection, or Unreal Engine for interactive visualisation. Integration with OpenUSD brings that scientific data into scene-based workflows, where simulation results can be combined with geometry, materials and other assets.

Choose precision for the job

Some data must survive bit for bit. Other data can tolerate small, controlled losses. ZibraXYZ supports lossless and lossy compression, and a hybrid approach that combines them within a dataset. Mesh connectivity and other arrays remained lossless throughout the mesh benchmarks below. Coordinates and physical fields can receive different precision settings according to how they will be used.

The settings below express an absolute error bound as a fraction of each channel's value range. The error relative to each individual value can be larger, especially near zero. We therefore check the calculations that consume the data as well as the reconstructed values.

Mesh data and the engineering result

For a mesh, quality includes the geometry, its fields and the quantities calculated from them. Across eight automotive cases, the preprocessed dataset totals 12.81 GB, covering meshes and fields, volume points and fields, and signed-distance fields with their gradients. The table pairs overall dataset compression with the largest absolute drag and lift coefficient changes across those cases, calculated from the decoded surface face data.

Volumetric grids

For structured volumes, we tested velocity and pressure in a 1024 by 1024 by 1024 cutout of JHTDB Channel5200 turbulence data, using a lossless reference from one timestep. Pressure and velocity have different compression behaviour, so the table reports them separately.

Averages can remain extremely close while differentiation amplifies small reconstruction errors. In this test, gradients and vorticity are much more sensitive than mean energy or pressure variance, especially near the wall. If the task depends on the smallest resolved structures, test a tighter setting: at 0.0001, the wall-normal gradient error falls to 0.45% and the Q-criterion error to 0.20%, with velocity still compressed 8.03×. These are compression-induced differences on one cutout, not a validation of the simulation itself.

The useful question is whether a setting preserves the result you care about. For an engineer, that might mean forces or a pressure distribution. For a learning system, it means model performance. We examine those checks in Streaming simulation results into ParaView for faster analysis and Beyond the Data Bottleneck: Physics AI Training with Compressed Data Streaming.

Read the part you need and decompress it where it is used

Compressed storage becomes much more useful when it supports selective access. An archive that must be unpacked in full before every operation still leaves a large loading step in the workflow.

ZibraXYZ organises payloads into indexed pages, manageable chunks of compressed data that can be addressed independently. Metadata identifies the pages needed for a request. The application retrieves their compressed bytes and decodes them into the selected output. For a mesh, that selection also includes the coordinates and connectivity required to reconstruct usable cells.

Random access and GPU-native decompression address different costs. Selection reduces how much data has to be read. Compression reduces the bytes carried through storage and the network. GPU decompression can keep the reconstructed output close to a GPU consumer, avoiding a full-sized CPU-to-GPU transfer.

Decompression adds virtually no overhead to the workload

Compression is useful only if the application can recover the data quickly enough. In our GPU-resident volumetric-codec benchmark, the decompressor sustained over 800 GB/s on an NVIDIA A100. This is decoder throughput, measured as reconstructed data per second; storage reads, network transfers and application processing have their own costs.

The training result shows what that speed means in practice. In the four-A100 turbulence super-resolution experiment, using a 300 GB JHTDB subset at 27× compression, the decompression kernel accounted for just 0.4% of the measured runtime. Average training-step time fell from 19.523 to 14.119 seconds, a 27.7% reduction. In that workload, decompression added negligible computation while the smaller data transfers helped the complete pipeline run faster.

Build on Zarr and add versioned storage where it helps

Zarr provides a useful foundation: multidimensional arrays, metadata, chunked storage and an extensible codec pipeline. Its sharding support can group multiple chunks into a storage object while keeping the inner chunks independently accessible.

Our approach is to build on Zarr with ZibraXYZ's compression and runtime capabilities, representation conventions and indexes for scientific access. Those conventions describe how geometry, connectivity and physical fields relate. Clients need the relevant ZibraXYZ codec or adapter to decode its payloads, giving applications a defined integration path.

Icechunk addresses the storage-management side. It adds transactions, consistent snapshots, branches and tags to Zarr-backed data. A team can publish a complete update atomically or pin a particular dataset version for an experiment. Those capabilities complement the codec; their value is consistency and reproducibility as datasets evolve.

Measure the benefits

A smaller stored dataset is the first saving. Selective reads can also reduce transferred bytes, staging time and repeated copies. The size of a decoded working set still depends on the selected data and the operations performed on it.

A useful evaluation therefore measures an actual workflow: ingest a case, retrieve a defined selection and complete an analysis or training step. Record stored bytes, transfer volume, latency, memory use and scientific quality, including the cost of preparation and compression.