Datasets & instrument files

What Quodra does with the files you import — which formats the parsers accept, how a dataset is stored and named by its own checksum, what the slide keeps versus what it re-reads on demand, and what happens when the network is not there.

Every Matter Lab import brings a real file with it. Quodra keeps that file: the bytes are archived exactly as they arrived, addressed by their own checksum, and can be handed back to you unchanged. The document itself carries a bounded preview plus a reference — which is what keeps decks fast to open and safe to merge while the full run stays available.

What the parsers accept#

Tabular files are read without assumptions: the delimiter is decided by a majority vote across the first lines rather than guessed from the extension, so one corrupt row cannot veto what the other thousand agree on; a decimal comma is detected and reported; and rows that cannot be read are counted and named in the warnings rather than dropped in silence. Nothing is smoothed, scaled or sorted on the way in.

Parsing runs off the main thread, so importing a large dump does not freeze the slide you are presenting. Where a worker is unavailable, the same parser runs inline — the answers and the error messages are identical either way.

Importing#

Every import is one action in the inspector: Import… on the relevant row, then a file picker. Warnings appear in the inspector and stay there until your next action, rather than in a toast that vanishes before it can be read. Import and the property change it causes land as a single step, so one undo takes back the whole import.

Checksums, and why re-importing is free#

The moment a file lands, its raw bytes are hashed with SHA-256 and stored under that hash:

• locally in your browser's IndexedDB database quodra-matter-datasets, so an offline deck can still hand you the original file;
• in the cloud as a matter_datasets row plus the raw file in your existing assets bucket, under a per-user folder that the storage rules already protect.

Because the address is the content, importing the same file twice costs nothing. The inspector says so plainly: these exact bytes are already in the dataset store — reused, not copied. Two people importing the same dump on different machines produce the same address, so their references resolve to the same bytes.

The cloud copy is written in the background and never waited on. Nothing about presenting, editing or exporting requires a round trip — if the sync has not happened yet, or cannot, the deck carries on working from what is in front of it.

What a reference records#

A dataset reference in the document is small and says only what it can prove: the file's name, its kind — tabular, structure, spectrum or image — its checksum, how many rows or frames it held, and, for anything derived, the parent it came from and the transform that produced it. The bytes themselves live in the archive, not in the deck.

That is deliberately a different thing from provenance, which is what you say about the data: Source, Method, Citation and Sample, edited under Data & methods in the same inspector. One is a fact about bytes, the other is a claim about an experiment, and Quodra keeps them apart. Import fills in what it can — the file name, and a method line such as the frame count, atom count or grid size — and leaves the rest to you, warning about gaps without ever blocking on them.

The Datasets section#

Every Matter inspector has a Datasets section listing what this object's data is made of. Each row shows the file name, a chip with the first eight characters of its checksum, and a Save button that hands the original file back — from the local cache when you are offline, from the cloud copy otherwise. Hovering a row shows the full name, the complete SHA-256, and the transform for derived rows.

Derived data is real data. Smoothing or baseline-correcting a spectrum writes the result back out as a two-column CSV and archives it as its own dataset, carrying the method that produced it and — when there is exactly one candidate, never a guess — a link to the parent it came from. Those rows are marked with a ↳.

If neither copy can be reached, Save says so rather than failing quietly: "…" is not cached on this machine and no cloud copy was reachable.

What the slide keeps, and what it re-reads#

This is the honest part, and worth reading before you import a hundred-megabyte run.

The document stores a capped preview, not the whole file. The caps are per kind:

Thinning is always reported. A long scan warns that its points were thinned and states that the peaks were kept exactly, because min/max bucketing keeps the highest and lowest sample in every bucket. A strided run warns how many frames were kept and every how-manyth. Atom counts past the trajectory limit are refused outright rather than sampled, because a thinned set of atoms is a different material.

Full runs, restored from the archive#

Where the source file held more frames than the preview kept, the viewer restores the full run from the archive: the bytes come back from the local cache or the cloud copy, are re-parsed in the worker under a display budget, and are cached once per checksum so the stage, the rail thumbnail, the presenter view and a capture all share one copy. When that happens the object shows a badge — full run · N frames from the archive — and the transport range grows to match.

Display budgets are an order of magnitude past the inline caps and still finite: up to 2,000 frames, up to 4,000,000 field values, up to 3,000,000 trajectory coordinates.

Three things are worth knowing about this:

• Nothing is written back. A hydrated run is runtime state; the document is unchanged.
• The preview never stops working. While the full run is loading, or offline with a cold cache, or if the archive cannot be reached, the object plays exactly what it shipped with.
• Bound sliders survive. A frame slot ranges over the source run's frame count, so a slider built against the full run keeps its range whether or not hydration has happened yet.

Across devices#

Sign in and the archive follows you: the checksum in the document resolves against your cloud copy, which is downloaded on first use and cached locally on the way through. Downloaded bytes are re-hashed and checked against the checksum before they are trusted — a stored path that claims to be a set of bytes is not the same thing as being them.

Working signed out, or offline, is a supported way to work. The local archive answers, and the inline preview covers whatever the archive cannot.

The compute cache#

Derived results are cached on what was computed from what: an input fingerprint, the method, and the parameters. A diffraction pattern simulated from a crystal is therefore paid for once and reused by the slide, the rail thumbnail and the filmstrip rather than recomputed by each. The cache holds the most recent 64 results and evicts the oldest.

The fingerprint used for in-memory arrays is deliberately cheap and is not a security claim — identity belongs to the dataset archive's SHA-256. The cost of a fingerprint collision is a stale plot, not a wrong file.

Next#

• The object types — which import row belongs to which object.
• Matter Lab — provenance, refusals and the workspace.
• Import & export — getting whole decks in and out.