`dandi.files`#

New in version 0.36.0.

dandi.files defines functionality for working with local files & directories (as opposed to remote resources on a DANDI Archive server) that are of interest to DANDI. The classes for such files & directories all inherit from DandiFile, which has two immediate subclasses: DandisetMetadataFile, for representing dandiset.yaml files, and LocalAsset, for representing files that can be uploaded as assets to DANDI Archive.

class dandi.files.BIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset])[source]#

Bases: LocalFileAsset

New in version 0.46.0.

Base class for non-dataset_description.json assets in BIDS datasets

property bids_dataset_description: BIDSDatasetDescriptionAsset#: The dataset_description.json file for the containing dataset

bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset]#

A weak reference to the dataset_description.json file for the containing dataset.

Users are advised to use bids_dataset_description to access the dataset_description.json file instead.

property bids_path: str#: /-separated path to the asset from the root of the BIDS dataset

property bids_root: Path#: The directory on the filesystem in which the BIDS dataset is located

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset[source]#: Return the Dandi metadata for the asset

get_validation_bids_version() → str[source]#

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#: Attempt to validate the file and return a list of errors encountered

class dandi.files.BIDSDatasetDescriptionAsset(filepath: Path, dandiset_path: Path | None, path: str, dataset_files: list[BIDSAsset] = <factory>, _dataset_errors: list[ValidationResult] | None = None, _asset_errors: dict[str, list[ValidationResult]] | None = None, _asset_metadata: dict[str, BareAsset] | None = None, _bids_version: str | None = None)[source]#

Bases: LocalFileAsset

New in version 0.46.0.

The dataset_description.json file for a BIDS dataset, used to perform operations on the dataset as a whole

property bids_root: Path#: The directory on the filesystem in which the BIDS dataset is located

dataset_files: list[BIDSAsset]#: A list of all other assets in the dataset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#: Attempt to validate the file and return a list of errors encountered

class dandi.files.DandiFile(filepath: Path, dandiset_path: Path | None)[source]#

Bases: ABC

Abstract base class for local files & directories of interest to DANDI

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

abstract get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → CommonModel[source]#: Return the Dandi metadata for the file

abstract get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#: Attempt to validate the file and return a list of errors encountered

property modified: datetime#: The time at which the file was last modified

property size: int#: The size of the file

class dandi.files.DandisetMetadataFile(filepath: Path, dandiset_path: Path | None)[source]#

Bases: DandiFile

Representation of a dandiset.yaml file

as_readable() → LocalReadableFile[source]#: New in version 0.50.0.

Returns a Readable instance wrapping the local file

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → DandisetMeta[source]#: Return the Dandiset metadata inside the file

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#: Attempt to validate the file and return a list of errors encountered

class dandi.files.GenericAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]#

Bases: LocalFileAsset

Representation of a generic regular file, one that is not of any known type

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

class dandi.files.GenericBIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset])[source]#

Bases: BIDSAsset, GenericAsset

New in version 0.46.0.

An asset in a BIDS dataset that is not an NWB file, a Zarr directory, or a dataset_description.json file. Note that, unlike the non-BIDS classes, this includes video files.

bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset]#

A weak reference to the dataset_description.json file for the containing dataset.

Users are advised to use bids_dataset_description to access the dataset_description.json file instead.

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#: Attempt to validate the file and return a list of errors encountered

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

class dandi.files.LocalAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]#

Bases: DandiFile

Representation of a file or directory that can be uploaded to a DANDI Archive as an asset of a Dandiset

abstract get_digest() → Digest[source]#: Calculate a DANDI etag digest for the asset using the appropriate algorithm for its type

abstract get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset[source]#: Return the Dandi metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#: Attempt to validate the file and return a list of errors encountered

abstract iter_upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) → Iterator[dict][source]#

Upload the asset with the given metadata to the given Dandiset, returning a generator of status dicts.

Parameters:

dandiset (RemoteDandiset) – the Dandiset to which the asset will be uploaded
metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s path attribute if no such field is already present.
jobs (int) – Number of threads to use for uploading; defaults to 5
replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset

Returns:

A generator of dicts containing at least a "status" key. Upon successful upload, the last dict will have a status of "done" and an "asset" key containing the resulting RemoteAsset.

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) → RemoteAsset[source]#

Upload the file as an asset with the given metadata to the given Dandiset and return the resulting asset. Blocks until the upload is complete.

Parameters:

dandiset (RemoteDandiset) – the Dandiset to which the file will be uploaded
metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s path attribute if no such field is already present.
jobs (int) – Number of threads to use for uploading; defaults to 5
replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset

Return type:

RemoteAsset

class dandi.files.LocalDirectoryAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]#

Bases: LocalAsset, Generic[P]

Representation of a directory that can be uploaded to a DANDI Archive as a single asset of a Dandiset. It is generic in P, bound to dandi.misctypes.BasePath.

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

abstract property filetree: P#: The path object for the root of the hierarchy of files within the directory

iterfiles(include_dirs: bool = False) → Iterator[P][source]#: Yield all files within the directory

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

property size: int#: The total size of the files in the directory

class dandi.files.LocalFileAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]#

Bases: LocalAsset

Representation of a regular file that can be uploaded to a DANDI Archive as an asset of a Dandiset

as_readable() → LocalReadableFile[source]#: New in version 0.50.0.

Returns a Readable instance wrapping the local file

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

get_digest() → Digest[source]#: Calculate a dandi-etag digest for the asset

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset[source]#: Return the Dandi metadata for the asset

iter_upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) → Iterator[dict][source]#

Upload the file as an asset with the given metadata to the given Dandiset, returning a generator of status dicts.

Parameters:

dandiset (RemoteDandiset) – the Dandiset to which the file will be uploaded
metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s path attribute if no such field is already present.
jobs (int) – Number of threads to use for uploading; defaults to 5
replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset

Returns:

A generator of dicts containing at least a "status" key. Upon successful upload, the last dict will have a status of "done" and an "asset" key containing the resulting RemoteAsset.

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

class dandi.files.LocalZarrEntry(parts: tuple[str, ...], filepath: Path, zarr_basepath: Path)[source]#

Bases: BasePath

A file or directory within a ZarrAsset

exists() → bool[source]#: True iff the resource at the given path exists

filepath: Path#: The path to the actual file or directory on disk

get_digest() → Digest[source]#: Calculate the DANDI etag digest for the entry. If the entry is a directory, the algorithm will be the Dandi Zarr checksum algorithm; if it is a file, it will be MD5.

is_dir() → bool[source]#: True if the resource at the given path exists and is a directory

is_file() → bool[source]#: True if the resource at the given path exists and is a file

iterdir() → Iterator[LocalZarrEntry][source]#: Returns a generator of the paths under the instance, which must be a directory

property modified: datetime#: The time at which the entry was last modified

property parent: LocalZarrEntry#: The parent path of the object. The parent of the root of a path hierarchy is itself.

property size: int#: The size of the entry. For a directory, this is the total size of all entries within it.

zarr_basepath: Path#: The path to the root of the Zarr file tree

class dandi.files.NWBAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]#

Bases: LocalFileAsset

Representation of a local NWB file

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset[source]#: Return the Dandi metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#

Validate NWB asset

If schema_version was provided, we only validate basic metadata, and completely skip validation using nwbinspector.inspect_nwbfile

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

class dandi.files.NWBBIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset])[source]#

Bases: BIDSAsset, NWBAsset

New in version 0.46.0.

An NWB file in a BIDS dataset

bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset]#

A weak reference to the dataset_description.json file for the containing dataset.

Users are advised to use bids_dataset_description to access the dataset_description.json file instead.

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset[source]#: Return the Dandi metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#

Validate NWB asset

If schema_version was provided, we only validate basic metadata, and completely skip validation using nwbinspector.inspect_nwbfile

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

class dandi.files.VideoAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]#

Bases: LocalFileAsset

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

class dandi.files.ZarrAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]#

Bases: LocalDirectoryAsset[LocalZarrEntry]

Representation of a local Zarr directory

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

property filetree: LocalZarrEntry#: The LocalZarrEntry for the root of the hierarchy of files within the Zarr asset

get_digest() → Digest[source]#: Calculate a dandi-zarr-checksum digest for the asset

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset[source]#: Return the Dandi metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#: Attempt to validate the file and return a list of errors encountered

iter_upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) → Iterator[dict][source]#

Upload the Zarr directory as an asset with the given metadata to the given Dandiset, returning a generator of status dicts.

Parameters:

dandiset (RemoteDandiset) – the Dandiset to which the Zarr will be uploaded
metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s path attribute if no such field is already present.
jobs (int) – Number of threads to use for uploading; defaults to 5
replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset; if the old asset is a Zarr, the Zarr will be updated & reused for the new asset

Returns:

A generator of dicts containing at least a "status" key. Upon successful upload, the last dict will have a status of "done" and an "asset" key containing the resulting RemoteAsset.

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

stat() → ZarrStat[source]#: Return various details about the Zarr asset

class dandi.files.ZarrBIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset])[source]#

Bases: ZarrAsset, BIDSAsset

New in version 0.46.0.

A Zarr directory in a BIDS dataset

dandiset_path: Path | None#: The path to the root of the Dandiset, if there is one

filepath: Path#: The path to the actual file or directory on disk

get_metadata(digest: Digest | None = None, ignore_errors: bool = True) → BareAsset[source]#: Return the Dandi metadata for the asset

get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) → list[ValidationResult][source]#: Attempt to validate the file and return a list of errors encountered

path: str#: The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

class dandi.files.ZarrStat(size: int, digest: Digest, files: list[LocalZarrEntry])[source]#

Bases: object

Details about a Zarr asset

digest: Digest#: The Dandi Zarr checksum of the asset

files: list[LocalZarrEntry]#: A list of all files in the asset in unspecified order

size: int#: The total size of the asset

dandi.files.dandi_file(filepath: str | Path, dandiset_path: str | Path | None = None, bids_dataset_description: BIDSDatasetDescriptionAsset | None = None) → DandiFile[source]#

Return a DandiFile instance of the appropriate type for the file at filepath inside the Dandiset rooted at dandiset_path. If dandiset_path is not set, it will default to filepath’s parent directory.

If bids_dataset_description is set, the file will be assumed to lie within the BIDS dataset with the given dataset_description.json file at its root, resulting in a BIDSAsset.

If filepath is a directory, it must be of a type represented by a LocalDirectoryAsset subclass; otherwise, an UnknownAssetError exception will be raised.

A regular file named dandiset.yaml will only be represented by a DandisetMetadataFile instance if it is at the root of the Dandiset.

A regular file that is not of a known type will be represented by a GenericAsset instance.

dandi.files.find_bids_dataset_description(dirpath: str | Path, dandiset_path: str | Path | None = None) → BIDSDatasetDescriptionAsset | None[source]#: New in version 0.46.0.

Look for the topmost dataset_description.json file in the directory dirpath and each of its parents, stopping when a dandiset.yaml file is found or dandiset_path is reached.

dandi.files.find_dandi_files(*paths: str | Path, dandiset_path: str | Path | None = None, allow_all: bool = False, include_metadata: bool = False) → Iterator[DandiFile][source]#

Yield all DANDI files at or under the paths in paths (which may be either files or directories). Files & directories whose names start with a period are ignored. Directories are only included in the return value if they are of a type represented by a LocalDirectoryAsset subclass, in which case they are not recursed into.

Parameters:

dandiset_path – The path to the root of the Dandiset in which the paths are located. All paths in paths must be equal to or subpaths of dandiset_path. If None, then the Dandiset path for each asset found is implicitly set to the parent directory.
allow_all – If true, unrecognized assets and the Dandiset’s dandiset.yaml file are returned as GenericAsset and DandisetMetadataFile instances, respectively. If false, they are not returned at all.
include_metadata – If true, the Dandiset’s dandiset.yaml file is returned as a DandisetMetadataFile instance. If false, it is not returned at all (unless allow_all is true).

dandi.files#

`dandi.files`#