dandi.files

Added in version 0.36.0.

dandi.files defines functionality for working with local files & directories (as opposed to remote resources on a DANDI Archive server) that are of interest to DANDI. The classes for such files & directories all inherit from DandiFile, which has two immediate subclasses: DandisetMetadataFile, for representing dandiset.yaml files, and LocalAsset, for representing files that can be uploaded as assets to DANDI Archive.

class dandi.files.BIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: ReferenceType[BIDSDatasetDescriptionAsset])

Bases: LocalFileAsset

Added in version 0.46.0.

Base class for non-dataset_description.json assets in BIDS datasets

property bids_dataset_description: BIDSDatasetDescriptionAsset

The dataset_description.json file for the containing dataset

bids_dataset_description_ref: ReferenceType[BIDSDatasetDescriptionAsset]

A weak reference to the dataset_description.json file for the containing dataset.

Users are advised to use bids_dataset_description to access the dataset_description.json file instead.

property bids_path: str

/-separated path to the asset from the root of the BIDS dataset

property bids_root: Path

The directory on the filesystem in which the BIDS dataset is located

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset

Return the DANDI metadata for the asset

get_validation_bids_version() → str | None

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Attempt to validate the file and return a list of errors encountered

Parameters:

missing_file_content (MissingFileContent | None) – When set (not None), the file’s content is known to be unavailable (e.g. broken symlink). Validators should adjust behaviour accordingly — for only-non-data they should skip content-dependent checks and emit a WARNING.

class dandi.files.BIDSDatasetDescriptionAsset(filepath: ~pathlib.Path, dandiset_path: ~pathlib.Path | None, path: str, dataset_files: list[~dandi.files.bids.BIDSAsset] = <factory>, _dataset_errors: list[~dandi.validate._types.ValidationResult] | None = None, _asset_errors: ~collections.defaultdict[str, list[~dandi.validate._types.ValidationResult]] | None = None, _asset_metadata: ~collections.defaultdict[str, ~dandischema.models.BareAsset] | None = None, _bids_version: str | None = None)

Bases: LocalFileAsset

Added in version 0.46.0.

The dataset_description.json file for a BIDS dataset, used to perform operations on the dataset as a whole

property bids_root: Path

The directory on the filesystem in which the BIDS dataset is located

property bids_version: str | None

The version of BIDS used for in validation

Note

This value is not necessarily the same as the value of the "BIDSVersion" field in the represented dataset_description.json file.

dataset_files: list[BIDSAsset]

A list of all other assets in the dataset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Return all validation results for the containing dataset per the BIDS standard

class dandi.files.DandiFile(filepath: Path, dandiset_path: Path | None)

Bases: ABC

Abstract base class for local files & directories of interest to DANDI

dandiset_path: Path | None

The path to the root of the Dandiset, if there is one

filepath: Path

The path to the actual file or directory on disk

abstract get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → CommonModel

Return the DANDI metadata for the file

abstract get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Attempt to validate the file and return a list of errors encountered

Parameters:

missing_file_content (MissingFileContent | None) – When set (not None), the file’s content is known to be unavailable (e.g. broken symlink). Validators should adjust behaviour accordingly — for only-non-data they should skip content-dependent checks and emit a WARNING.

property modified: datetime

The time at which the file was last modified

property size: int

The size of the file

class dandi.files.DandisetMetadataFile(filepath: Path, dandiset_path: Path | None)

Bases: DandiFile

Representation of a dandiset.yaml file

as_readable() → LocalReadableFile

Added in version 0.50.0.

Returns a Readable instance wrapping the local file

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → Dandiset

Return the Dandiset metadata inside the file

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Attempt to validate the file and return a list of errors encountered

Parameters:

missing_file_content (MissingFileContent | None) – When set (not None), the file’s content is known to be unavailable (e.g. broken symlink). Validators should adjust behaviour accordingly — for only-non-data they should skip content-dependent checks and emit a WARNING.

class dandi.files.GenericAsset(filepath: Path, dandiset_path: Path | None, path: str)

Bases: LocalFileAsset

Representation of a generic regular file, one that is not of any known type

class dandi.files.GenericBIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: ReferenceType[BIDSDatasetDescriptionAsset])

Bases: BIDSAsset, GenericAsset

Added in version 0.46.0.

An asset in a BIDS dataset that is not an NWB file, a Zarr directory, or a dataset_description.json file. Note that, unlike the non-BIDS classes, this includes video files.

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Attempt to validate the file and return a list of errors encountered

Parameters:

missing_file_content (MissingFileContent | None) – When set (not None), the file’s content is known to be unavailable (e.g. broken symlink). Validators should adjust behaviour accordingly — for only-non-data they should skip content-dependent checks and emit a WARNING.

class dandi.files.LocalAsset(filepath: Path, dandiset_path: Path | None, path: str)

Bases: DandiFile

Representation of a file or directory that can be uploaded to a DANDI Archive as an asset of a Dandiset

abstract get_digest() → Digest

Calculate a DANDI etag digest for the asset using the appropriate algorithm for its type

abstract get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset

Return the DANDI metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Attempt to validate the file and return a list of errors encountered

Parameters:

missing_file_content (MissingFileContent | None) – When set (not None), the file’s content is known to be unavailable (e.g. broken symlink). Validators should adjust behaviour accordingly — for only-non-data they should skip content-dependent checks and emit a WARNING.

abstract iter_upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) → Iterator[dict]

Upload the asset with the given metadata to the given Dandiset, returning a generator of status dicts.

Parameters:

• dandiset (RemoteDandiset) – the Dandiset to which the asset will be uploaded

• metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s path attribute if no such field is already present.

• jobs (int) – Number of threads to use for uploading; defaults to 5

• replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset

Returns:

A generator of dicts containing at least a "status" key. Upon successful upload, the last dict will have a status of "done" and an "asset" key containing the resulting RemoteAsset.

path: str

The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) → RemoteAsset

Upload the file as an asset with the given metadata to the given Dandiset and return the resulting asset. Blocks until the upload is complete.

Parameters:

• dandiset (RemoteDandiset) – the Dandiset to which the file will be uploaded

• metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s path attribute if no such field is already present.

• jobs (int) – Number of threads to use for uploading; defaults to 5

• replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset

Return type:

RemoteAsset

class dandi.files.LocalDirectoryAsset(filepath: Path, dandiset_path: Path | None, path: str)

Bases: LocalAsset, Generic[P]

Representation of a directory that can be uploaded to a DANDI Archive as a single asset of a Dandiset. It is generic in P, bound to dandi.misctypes.BasePath.

abstract property filetree: P

The path object for the root of the hierarchy of files within the directory

iterfiles(include_dirs: bool = False) → Iterator[P]

Yield all files within the directory

property size: int

The total size of the files in the directory

class dandi.files.LocalFileAsset(filepath: Path, dandiset_path: Path | None, path: str)

Bases: LocalAsset

Representation of a regular file that can be uploaded to a DANDI Archive as an asset of a Dandiset

as_readable() → LocalReadableFile

Added in version 0.50.0.

Returns a Readable instance wrapping the local file

get_digest() → Digest

Calculate a dandi-etag digest for the asset

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset

Return the DANDI metadata for the asset

iter_upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) → Iterator[dict]

Upload the file as an asset with the given metadata to the given Dandiset, returning a generator of status dicts.

Parameters:

• dandiset (RemoteDandiset) – the Dandiset to which the file will be uploaded

• metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s path attribute if no such field is already present.

• jobs (int) – Number of threads to use for uploading; defaults to 5

• replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset

Returns:

A generator of dicts containing at least a "status" key. Upon successful upload, the last dict will have a status of "done" and an "asset" key containing the resulting RemoteAsset.

class dandi.files.LocalZarrEntry(parts: tuple[str, ...], zarr_basepath: Path)

Bases: BasePath

A file or directory within a ZarrAsset

exists() → bool

True iff the resource at the given path exists

property filepath: Path

The path to the actual file or directory on disk

get_digest() → Digest

Calculate the DANDI etag digest for the entry. If the entry is a directory, the algorithm will be the DANDI Zarr checksum algorithm; if it is a file, it will be MD5.

is_dir() → bool

True if the resource at the given path exists and is a directory

is_file() → bool

True if the resource at the given path exists and is a file

iterdir() → Iterator[LocalZarrEntry]

Returns a generator of the paths under the instance, which must be a directory

property modified: datetime

The time at which the entry was last modified

property parent: LocalZarrEntry

The parent path of the object. The parent of the root of a path hierarchy is itself.

property size: int

The size of the entry. For a directory, this is the total size of all entries within it.

zarr_basepath: Path

The path to the root of the Zarr file tree

class dandi.files.NWBAsset(filepath: Path, dandiset_path: Path | None, path: str)

Bases: LocalFileAsset

Representation of a local NWB file

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset

Return the DANDI metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Validate NWB asset

If schema_version was provided, we only validate basic metadata, and completely skip validation using nwbinspector.inspect_nwbfile.

If missing_file_content is only-non-data, content-dependent validators (pynwb, nwbinspector) are skipped and only path-layout validation is performed.

class dandi.files.NWBBIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: ReferenceType[BIDSDatasetDescriptionAsset])

Bases: BIDSAsset, NWBAsset

Added in version 0.46.0.

An NWB file in a BIDS dataset

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset

Return the DANDI metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Validate NWB asset

If schema_version was provided, we only validate basic metadata, and completely skip validation using nwbinspector.inspect_nwbfile.

If missing_file_content is only-non-data, content-dependent validators (pynwb, nwbinspector) are skipped and only path-layout validation is performed.

class dandi.files.VideoAsset(filepath: Path, dandiset_path: Path | None, path: str)

Bases: LocalFileAsset

class dandi.files.ZarrAsset(filepath: Path, dandiset_path: Path | None, path: str)

Bases: LocalDirectoryAsset[LocalZarrEntry]

Representation of a local Zarr directory

property filetree: LocalZarrEntry

The LocalZarrEntry for the root of the hierarchy of files within the Zarr asset

get_digest() → Digest

Calculate a dandi-zarr-checksum digest for the asset

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset

Return the DANDI metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Attempt to validate the file and return a list of errors encountered

Parameters:

missing_file_content (MissingFileContent | None) – When set (not None), the file’s content is known to be unavailable (e.g. broken symlink). Validators should adjust behaviour accordingly — for only-non-data they should skip content-dependent checks and emit a WARNING.

iter_upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) → Iterator[dict]

Upload the Zarr directory as an asset with the given metadata to the given Dandiset, returning a generator of status dicts.

Parameters:

• dandiset (RemoteDandiset) – the Dandiset to which the Zarr will be uploaded

• metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s path attribute if no such field is already present.

• jobs (int) – Number of threads to use for uploading; defaults to 5

• replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset; if the old asset is a Zarr, the Zarr will be updated & reused for the new asset

Returns:

A generator of dicts containing at least a "status" key. Upon successful upload, the last dict will have a status of "done" and an "asset" key containing the resulting RemoteAsset.

stat() → ZarrStat

Return various details about the Zarr asset

class dandi.files.ZarrBIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: ReferenceType[BIDSDatasetDescriptionAsset])

Bases: ZarrAsset, BIDSAsset

Added in version 0.46.0.

A Zarr directory in a BIDS dataset

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset

Return the DANDI metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False, missing_file_content: MissingFileContent | None = None) → list[ValidationResult]

Attempt to validate the file and return a list of errors encountered

Parameters:

missing_file_content (MissingFileContent | None) – When set (not None), the file’s content is known to be unavailable (e.g. broken symlink). Validators should adjust behaviour accordingly — for only-non-data they should skip content-dependent checks and emit a WARNING.

class dandi.files.ZarrStat(size: int, digest: Digest, files: list[LocalZarrEntry])

Bases: object

Details about a Zarr asset

digest: Digest

The DANDI Zarr checksum of the asset

files: list[LocalZarrEntry]

A list of all files in the asset in unspecified order

size: int

The total size of the asset

dandi.files.dandi_file(filepath: str | Path, dandiset_path: str | Path | None = None, bids_dataset_description: BIDSDatasetDescriptionAsset | None = None) → DandiFile

Return a DandiFile instance of the appropriate type for the file at filepath inside the Dandiset rooted at dandiset_path. If dandiset_path is not set, it will default to filepath’s parent directory.

If bids_dataset_description is set, the file will be assumed to lie within the BIDS dataset with the given dataset_description.json file at its root, resulting in a BIDSAsset.

If filepath is a directory, it must be of a type represented by a LocalDirectoryAsset subclass; otherwise, an UnknownAssetError exception will be raised.

A regular file named dandiset.yaml will only be represented by a DandisetMetadataFile instance if it is at the root of the Dandiset.

A regular file that is not of a known type will be represented by a GenericAsset instance.

dandi.files.find_bids_dataset_description(dirpath: str | Path, dandiset_path: str | Path | None = None) → BIDSDatasetDescriptionAsset | None

Added in version 0.46.0.

Look for the topmost dataset_description.json file in the directory dirpath and each of its parents, stopping when a dandiset.yaml file is found or dandiset_path is reached.

dandi.files.find_dandi_files(*paths: str | Path, dandiset_path: str | Path | None = None, allow_all: bool = False, include_metadata: bool = False) → Iterator[DandiFile]

Yield all DANDI files at or under the paths in paths (which may be either files or directories). Files & directories whose names start with a period are ignored. Directories are only included in the return value if they are of a type represented by a LocalDirectoryAsset subclass, in which case they are not recursed into.

Parameters:

• dandiset_path – The path to the root of the Dandiset in which the paths are located. All paths in paths must be equal to or subpaths of dandiset_path. If None, then the Dandiset path for each asset found is implicitly set to the parent directory.

• allow_all – If true, unrecognized assets and the Dandiset’s dandiset.yaml file are returned as GenericAsset and DandisetMetadataFile instances, respectively. If false, they are not returned at all.

• include_metadata – If true, the Dandiset’s dandiset.yaml file is returned as a DandisetMetadataFile instance. If false, it is not returned at all (unless allow_all is true).