Icechunk and virtual stores¶

These need the cloud extra (pip install "disp_xr[cloud]").

Write and read DISP datasets in Icechunk repositories (cloud extra).

An Icechunk repository is a versioned Zarr v3 store: every write is a commit with a snapshot id, and tags name snapshots that must not move (a Cal/Val run). to_icechunk wraps :func:icechunk.xarray.to_icechunk, which, unlike ds.to_zarr on a session store, keeps writes made under dask.distributed or multiprocessing.

Provenance (disp-xr and package versions, product, frame, input granules and the history of operations) goes in the commit metadata as well as in the dataset attributes. A stack built by appending one ministack per commit uses the "ministack" chunk preset, so an append writes new shards instead of rewriting old ones. Library code never amends a commit or expires snapshots.

to_icechunk ¶

to_icechunk(ds: Dataset, repo: RepoLike, *, branch: str = 'main', message: Optional[str] = None, append_dim: Optional[str] = None, tag: Optional[str] = None, preset: Optional[Preset] = None, mode: Literal['w-', 'w'] = 'w-', keep_bits: Optional[int] = None, compression_level: int = 3, target_mb: Optional[float] = None) -> str

Write ds to an Icechunk repository as one commit.

Parameters:

Name	Type	Description	Default
`ds`	`Dataset`	DISP dataset with `(time, y, x)` and `(y, x)` variables.	required
`repo`	`(Repository, str or Path)`	Repository, or a local directory (created if missing).	required
`branch`	`str`	Branch to commit to. A missing branch is created from `main`.	`'main'`
`message`	`str`	Commit message; defaults to a summary of the write.	`None`
`append_dim`	`'time'`	Append along time. The dataset must be on the store's grid, with the same variables and dtypes, and start after the store's last epoch. On an empty branch this creates the store with the `"ministack"` layout.	`"time"`
`tag`	`str`	Tag the new snapshot (for example a Cal/Val run). Tags never move.	`None`
`preset`	`('timeseries', 'maps', 'ministack')`	Chunk layout of a new store; ignored when appending to an existing one. Defaults to `"timeseries"`, or `"ministack"` with `append_dim`. `"timeseries"` cannot be appended to, since each shard spans the whole time axis and every append would rewrite it.	`"timeseries"`
`mode`	`('w-', 'w')`	Without `append_dim`: fail if the branch has data (default), or replace it in a new commit (older snapshots stay readable).	`"w-"`
`keep_bits`	`Optional[int]`	As in :func:`disp_xr.to_zarr`.	`None`
`compression_level`	`Optional[int]`	As in :func:`disp_xr.to_zarr`.	`None`
`target_mb`	`Optional[int]`	As in :func:`disp_xr.to_zarr`.	`None`

Returns:

Type	Description
`str`	Snapshot id of the commit. Record it in downstream provenance; :func:`open_icechunk` does so as `attrs["source_snapshot_id"]`.

Examples:

>>> dxr.to_icechunk(ds, "results/", message="first epochs")
>>> dxr.to_icechunk(
...     new_epochs, "results/", append_dim="time", tag="v1"
... )

open_icechunk ¶

open_icechunk(repo: RepoLike, *, branch: Optional[str] = None, tag: Optional[str] = None, snapshot_id: Optional[str] = None, chunks: Any = None, mask_and_scale: bool = False) -> xr.Dataset

Open a DISP dataset from an Icechunk repository, read-only.

Give at most one of branch (default "main", its latest commit), tag or snapshot_id. The snapshot read is recorded in attrs["source_snapshot_id"], which exports carry as provenance.

Parameters:

Name	Type	Description	Default
`repo`	`(Repository, str or Path)`	Repository, or a local directory holding one.	required
`branch`	`str`	Version to read.	`None`
`tag`	`str`	Version to read.	`None`
`snapshot_id`	`str`	Version to read.	`None`
`chunks`	`optional`	Passed to :func:`xarray.open_zarr`; by default the stored chunks.	`None`
`mask_and_scale`	`bool`	Decode fill values to NaN. Off by default, so integer layers keep their dtype, as :func:`disp_xr.open_disp` does.	`False`

Examples:

>>> ds = dxr.open_icechunk("results/", tag="v1")
>>> earlier = dxr.open_icechunk("results/", snapshot_id=sid)

Virtual stores¶

Virtual DISP frames: Icechunk stores that point at the granules (cloud extra).

A virtual store holds no pixels. Each layer chunk is a reference to a byte range in a granule, read through VirtualiZarr's HDF parser, so a whole frame opens as one Zarr dataset without copying data. time, reference_time, x, y and spatial_ref are loaded into the store, because their encoding differs between granules; the /identification and /metadata groups are not read.

Displacement stays relative to each ministack reference, as in the granules: chain and re-reference on top of :func:open_virtual. A virtual store is an index for ingest and exploration; for per-pixel work write a native store (:func:disp_xr.to_zarr with the "timeseries" preset).

The store registers a virtual chunk container for the granule location (the S3 bucket, or the local directory). Credentials for it are given when the repository is opened and are never saved with it.

build_virtual_frame ¶

build_virtual_frame(granules: GranulesLike, repo: RepoLike, *, branch: str = 'main', layers: Optional[Sequence[str]] = None, max_workers: Optional[int] = None, credentials: Optional[AWSCredentials] = None) -> list[str]

Write a frame of granules to an Icechunk repository as virtual references.

One commit per ministack, in reference-date order. A branch that already holds a frame is extended: every new ministack must start after its last epoch and be on the same grid, with the same layers and dtypes.

Parameters:

Name	Type	Description	Default
`granules`	`str, Path, iterable of paths, or DataFrame`	Local or `s3://` granules of one frame and one production, or the output of :func:`disp_xr.find_products`.	required
`repo`	`(Repository, Storage, str or Path)`	Repository to write to. For a storage or a local path, the repository is created if needed and a virtual chunk container for the granule location is registered. A `Repository` must already have one and be opened with access to it.	required
`branch`	`str`	Branch to commit to.	`'main'`
`layers`	`sequence of str`	Layers to reference. Defaults to all displacement and quality layers.	`None`
`max_workers`	`int`	Processes that parse granules; `1` parses in this process.	`None`
`credentials`	`AWSCredentials`	For `s3://` granules. Defaults to `AWS_*` environment variables, then ASF temporary credentials.	`None`

Returns:

Type	Description
`list of str`	Snapshot id of each commit, one per ministack.

Raises:

Type	Description
`MixedProductionError`	If the granules mix productions (dtype or encoding differ).
`ValueError`	If a ministack is on another grid or does not follow the branch's last epoch.

Examples:

>>> snapshots = dxr.build_virtual_frame(
...     "F11116/", "F11116-virtual/"
... )
>>> dxr.build_virtual_frame(new_granules, "F11116-virtual/")

open_virtual ¶

open_virtual(repo: RepoLike, *, branch: Optional[str] = None, tag: Optional[str] = None, snapshot_id: Optional[str] = None, chunks: Any = None, credentials: Optional[AWSCredentials] = None) -> xr.Dataset

Open a virtual frame read-only, with access to its granules.

As :func:disp_xr.open_icechunk, but the repository is opened with read access to every virtual chunk container in its config: local directories, or S3 with credentials (default: environment, then ASF).

Examples:

>>> ds = dxr.open_virtual("F11116-virtual/")
>>> before = dxr.open_virtual("F11116-virtual/", snapshot_id=sid)

validate_virtual ¶

validate_virtual(repo: RepoLike, granules: GranulesLike, *, n_samples: int = 10, seed: int = 0, branch: str = 'main', credentials: Optional[AWSCredentials] = None) -> dict[str, Any]

Check a virtual frame against the granules, read with h5py.

Compares the time axis, n_samples random pixel series of every layer, and one random epoch in full.

Returns:

Type	Description
`dict`	`n_granules`, `pixels` (`(row, col)` pairs), `epoch` and `layers` checked.

Raises:

Type	Description
`ValueError`	If any value or the time axis differs, naming what differs.

Examples:

>>> report = dxr.validate_virtual("F11116-virtual/", "F11116/")
>>> report["n_granules"]