Settings and chunks¶

Settings¶

disp_xr.config holds the runtime settings. Change them for a block of code with config.set, or for a batch job with DISP_XR_* environment variables (nested keys use a double underscore).

from disp_xr.config import config

with config.set({"chunks.target_mb": 50}):
    ...

Key	Default	Meaning
`chunks.target_mb`	100	Uncompressed size of one dask chunk (and Zarr shard) for the `timeseries` and `maps` presets, in MB
`chunks.read_block`	1024	Spatial block read from each granule; the `native` chunk size. 2048 gives 3.5 times fewer tasks when writing a full-frame store (`performance.md`)
`max_load_bytes`	2 GiB	Largest array export and plotting load into memory
`cache.dir`	`None`	Downloaded static layers; `None` means `$XDG_CACHE_HOME/disp_xr` or `~/.cache/disp_xr`
`threads`	`None`	Threads per worker for compiled kernels
`num_workers`	32	Dask threads when disp-xr computes (plots, exports), unless dask's own `num_workers` is set
`numba_threads`	1	Threads of numba kernels (numbagg) inside each dask task, set as `NUMBA_NUM_THREADS` on import unless that variable is set or numba is already loaded; numba's default of one per core made every dask thread open 224 threads on a large node and crash

Chunks¶

Dask chunk sizes for DISP stacks.

Chunks are sized from a byte target (config["chunks.target_mb"]), not a fixed shape, and aligned to the 256 x 256 HDF5 chunks of DISP granules so a dask chunk never reads part of a file chunk. Three presets:

"native" (the :func:disp_xr.open_disp default) One epoch and one read block (config["chunks.read_block"], 1,024 x 1,024) per chunk: the layout of the files. Every selection reads only the epochs and blocks it needs, so maps, points and chaining stay cheap. "timeseries" The whole time axis in each chunk, for per-pixel work (fits, filters). Functions that need it rechunk to it themselves (:func:time_contiguous, which merges time within the input's spatial blocks instead of reshuffling them). "maps" One epoch per chunk, sized to the byte target, for whole-map work.

:func:rechunk switches a dataset between them when one kind of access will be repeated, for example ds.disp.rechunk("timeseries") before many point reads of a store that fits in memory.

Clamping is adapted from opera-utils (src/opera_utils/disp/_utils.py, _clamp_chunk_dict, commit 97f77d9), Copyright (c) 2022 California Institute of Technology, BSD-3-Clause. See THIRD_PARTY_NOTICES.md.

rechunk ¶

rechunk(obj: Any, chunks: Optional[ChunkSpec] = None) -> Any

Return obj (Dataset or DataArray) with dask chunks for one kind of work.

Parameters:

Name	Type	Description	Default
`obj`	`Dataset or DataArray`	DISP data with `time`, `y`, `x` dimensions (any subset).	required
`chunks`	`('native', 'timeseries', 'maps')`	A preset (see the module docstring) or explicit `{"time", "y", "x"}` sizes, `-1` for a whole axis. Required: leaving it out still means `"timeseries"` but warns, and is an error from 1.0 (so every call says which layout it asks for).	`"native"`

Returns:

Type	Description
Same type as ``obj``, lazily rechunked. Nothing is read.

Examples:

>>> import xarray as xr, numpy as np
>>> da = xr.DataArray(np.zeros((4, 8, 8)), dims=("time", "y", "x")).chunk(1)
>>> rechunk(da, "timeseries").chunks[0]
(4,)