Interactive dataset network streaming: eliminate the wait between a completed simulation and post-processing insights.
Post-processing large simulations usually requires transferring results from a compute cluster, staging them locally and loading them into an analysis tool. Inspecting a pressure field or comparing designs may require only a small part of the dataset, yet the application needs to load the entire data sample.
These transfer and loading steps delay analysis and repeat each time the data is accessed. This is the practical effect of data gravity: the size and location of simulation results constrain how quickly engineers can work with them.
We have already seen a different experience emerge in other media. Watching a film once meant downloading a large file. Today, we choose a title, press play and receive the compressed content needed to start watching. Scientific workflows would benefit from the same convenience: open a dataset, ask for a region or timestep and begin working while the rest stays in remote storage.
Simulation results require additional features: an engineer may jump to an arbitrary region, inspect several numerical fields and calculate a quantity that must remain accurate. A universal 3D/4D codec has to preserve those quantities across grids, meshes, points and time.
ZibraXYZ brings these requirements together in a common data layer for scientific datasets. It combines configurable lossless, lossy and hybrid compression with random access and GPU-native decompression, so applications can retrieve selected data without unpacking the complete dataset. Crucially, that access extends to unstructured meshes: a request for a region retrieves the compressed pages containing the relevant cells, their physical fields, and the coordinates and connectivity needed to reconstruct a usable selection. Built on Zarr, the stack can use Icechunk for consistent, versioned snapshots, allowing teams to return to the dataset used for a particular analysis or experiment. Integrations with post-processing tools such as ParaView and AI training workflows in PyTorch let those applications consume selected data from the same underlying store, reducing the need for separate task-specific copies.
Our ParaView workflow covers structured grids, unstructured meshes, point data and time series. The reader uses metadata and spatial indexes to identify the relevant compressed pages. It retrieves those pages via network, resolves the geometry dependencies and decompresses the selection at the point of use. ParaView receives data as a standard VTK object.
Region of interest selection and compression reinforce each other. The first reduces the portion of the dataset requested while the second reduces the bytes needed to deliver it. Fast decoding then determines how quickly those bytes become a useful field or mesh. In combination we get a 10–100× faster data access which enables interactive visualisation while streaming data over the network.
The underlying compression rates and precision choices are covered in ZibraXYZ: Accelerating Scientific Data Access by 100×.
Our first demonstration is a Johns Hopkins Turbulence Database channel-flow sample containing more than 120 billion grid points. The raw frame is approximately 1.9 TB and is compressed at an average ratio of 33×. With ZibraXYZ an engineer can visualize the entire data sample immediately while streaming it from a remote location.
We open the entire data sample through the ZibraXYZ ParaView plugin and begin with a surface view to establish its shape. The reader requests and decodes the compressed pages needed for that view. We also support a volume rendering that reveals the internal structure of the turbulent flow.
As the camera moves, level-of-detail loading prioritises data needed for the current view. Detail is refined progressively. The application can build a view of the full frame while requesting and reconstructing only the pages and levels of detail that the view requires. The stored data remains compressed throughout the session, with requested pages downloaded and decoded as they are needed. A large archive becomes something the user can explore through a sequence of data queries.
Interactive viewing is only the beginning of post-processing. In the volume demonstration, we extract a slice through the flow. ZibraXYZ resolves the pages intersecting the plane and supplies the selected data to ParaView. The reader does not have to reconstruct the complete field to produce that slice.
The selection also feeds ParaView filters. In the demonstration, a dense vector visualisation reveals local flow direction and magnitude. Slicing, filtering, recolouring and camera interaction become part of the same analysis session, with compressed remote data supplying the working set.
Unstructured meshes add another challenge: retrieving a field is useful only if the application can reconstruct the associated geometry and cell connectivity. Our automotive demonstration shows this on a 50 GB source dataset.
The Zibra loader produces the complete point-cloud view in about two and a half minutes. The native loader is still processing after eleven minutes. Reducing the bytes read and the memory footprint brings the usable view forward.
But in many cases an engineer does not need to have access to the entire geometry or field data. ZibraXYZ provides a random access data loading to enable the region of interest use cases.
For a closer investigation, we define a region of interest around the front wheel. The vehicle surface remains visible for context while ZibraXYZ retrieves and decodes the volumetric cells inside the selected box, together with their required geometry and connectivity. Moving or resizing the box changes the data request, allowing the engineer to follow local flow structures without materialising the complete volume.
The region behaves like normal ParaView data. We can isolate it, switch its representation, change the colour mapping, zoom into its internal structure and reposition the selection. This makes ROI access different from clipping a case after everything has loaded: the spatial selection determines what is downloaded and reconstructed in the first place.
The 50 GB source remains available for the next question, while the engineer works with the portion needed for the current decision. Selection happens at page granularity, so the working set can include neighbouring data and mesh dependencies beyond the exact boundary of the box.
Our JHTDB wind-farm demonstration extends the browser workflow to time-dependent volumetric data. Users can move through recorded timesteps and inspect how flow structures evolve around the turbines, adding a temporal dimension to the exploration of a large simulation.
ZibraXYZ keeps the sequence compressed in remote storage. The browser requests the pages needed for the selected timestep and view, then decompresses the requested data on the GPU for visualisation. Moving through time updates that request, allowing users to explore the sequence without downloading and reconstructing every frame in advance.
Selective access lets the application build a working set around the current investigation, while the rest of the sequence remains available remotely. The demonstration shows how the same data layer can support spatial exploration and navigation through time in a browser, making simulation results easier to inspect and share.
The value of faster access becomes clearer when the same team opens data repeatedly. Consider an illustrative planning scenario with 20 engineers, five accesses per engineer per workday, 12 minutes of waiting per access in the existing workflow and 30 seconds with the proposed streaming workflow.
| Waiting time | Existing workflow | Streaming scenario |
|---|---|---|
| Per dataset access | 12 min | 30 sec |
| Per engineer per day | 60 min | 2.5 min |
| Across 20 engineers per day | 20 hours | 50 min |
Under those assumptions, the team avoids 19 hours and 10 minutes of aggregate waiting each workday. Across 220 workdays, that is about 4,217 engineer-hours. This is a scenario calculation; a pilot establishes the actual access latency and frequency for a particular team.
The same pilot should record transferred bytes, peak memory, cold and warm access times, and the time required to finish the analysis. Smaller transfers and fewer staging steps can make repeated exploration easier, even when the complete case is much larger than the selected working set.
A practical starting point is one representative case and a few tasks engineers perform every week. Establish the current access time, choose an ROI and a quantitative check, then repeat the tasks through the ZibraXYZ reader.
Simulation data can also feed machine learning workflows. Beyond the Data Bottleneck: Physics AI Training with Compressed Data Streaming explains the loading and quality considerations for that use, including why training samples may need a different physical layout.
A 30-minute call with our team on Google Meet.
Loading available times
(your time)
A calendar invitation with the Google Meet link is on its way to .
All displayed trademarks are the property of their respective owners. All company, product and service names are used for identification purposes only. Use of these names and trademarks does not imply endorsement
