dandi.organize

Organize and structure NWB files according to DANDI conventions.

This module provides functionality for organizing neuroscience data files according to DANDI’s file organization schema. Features include: - Automatic path generation from metadata - BIDS-like subject/session organization - Metadata-driven file naming - Validation of organized paths - Support for videos and generic files

Module Attributes

MAX_METADATA_ERRORS_SHOWN

How many per-file metadata errors format_metadata_load_failures() spells out before deferring the rest to the log file

Functions

create_dataset_yml_template(filepath)
create_unique_filenames_from_metadata(metadata)	Create unique filenames given metadata
detect_link_type(srcfile, destdir)	Determine what type of links the filesystem will let us make from the file srcfile to the directory destdir.
filter_invalid_metadata_rows(metadata_rows)	Split into two lists - valid and invalid entries
format_metadata_load_failures(metadata_excs, ...)	Compose an actionable message about files whose metadata failed to load.
get_obj_id(object_id)	Given full object_id, get its shortened version
is_undefined(value)	Return True if None or an empty container
organize(paths[, dandiset_path, invalid, ...])
organize_external_files(metadata, ...)	Organizes the external_files into the new Dandiset folder structure.
populate_dataset_yml(filepath, metadata)
validate_organized_path(asset_path, ...)

Classes

CopyMode(value)
FileOperationMode(value)
OrganizeInvalid(value)

class dandi.organize.CopyMode(value)

COPY = 'copy'

HARDLINK = 'hardlink'

MOVE = 'move'

SYMLINK = 'symlink'

copy(old_path: str | Path, new_path: str | Path) → None

class dandi.organize.FileOperationMode(value)

AUTO = 'auto'

COPY = 'copy'

DRY = 'dry'

HARDLINK = 'hardlink'

MOVE = 'move'

SIMULATE = 'simulate'

SYMLINK = 'symlink'

as_copy_mode() → CopyMode

dandi.organize.MAX_METADATA_ERRORS_SHOWN = 5

How many per-file metadata errors format_metadata_load_failures() spells out before deferring the rest to the log file

class dandi.organize.OrganizeInvalid(value)

FAIL = 'fail'

WARN = 'warn'

dandi.organize.create_dataset_yml_template(filepath)

dandi.organize.create_unique_filenames_from_metadata(metadata: list[dict], required_fields: Sequence[str] | None = None) → list[dict]

Create unique filenames given metadata

Parameters:

• metadata (list of dict) – List of metadata records

• required_fields (sequence of str, optional) – Names from dandi_layout_fields to consider "required_if_not_empty"

Returns:

Adjusted metadata. A copy, which might have removed some metadata fields Do not rely on it being the same

Return type:

list of dict

dandi.organize.detect_link_type(srcfile: str | Path, destdir: str | Path) → FileOperationMode

Determine what type of links the filesystem will let us make from the file srcfile to the directory destdir. If symlinks are allowed, returns "symlink". Otherwise, if hard links are allowed, returns "hardlink". Otherwise, returns "copy".

dandi.organize.filter_invalid_metadata_rows(metadata_rows)

Split into two lists - valid and invalid entries

dandi.organize.format_metadata_load_failures(metadata_excs: Sequence[tuple[dict, tuple | None]], npaths: int) → str

Compose an actionable message about files whose metadata failed to load.

The message names each offending path together with the exception type and the exception’s own message, and it says how to get at the full tracebacks, rather than only stating that they exist at DEBUG level.

Parameters:

• metadata_excs (sequence of (dict, tuple or None)) – Pairs of a metadata record and, if loading it raised, a (exception class, exception message, traceback) tuple

• npaths (int) – Total number of paths that were considered

Returns:

A multi-line message suitable for logging at WARNING level

Return type:

str

dandi.organize.get_obj_id(object_id)

Given full object_id, get its shortened version

dandi.organize.is_undefined(value)

Return True if None or an empty container

dandi.organize.organize(paths: Sequence[str], dandiset_path: str | None = None, invalid: OrganizeInvalid = OrganizeInvalid.FAIL, files_mode: FileOperationMode = FileOperationMode.AUTO, devel_debug: bool = False, update_external_file_paths: bool = False, media_files_mode: CopyMode | None = None, required_fields: Sequence[str] | None = None, jobs: int | None = None) → None

dandi.organize.organize_external_files(metadata: list[dict], dandiset_path: str, copy_mode: CopyMode) → None

Organizes the external_files into the new Dandiset folder structure.

Parameters:

• metadata (list) – list of metadata dictionaries created during the call to pynwb_utils._get_pynwb_metadata

• dandiset_path (str) – full path of the main dandiset folder.

• copy_mode (CopyMode)

dandi.organize.populate_dataset_yml(filepath, metadata)

dandi.organize.validate_organized_path(asset_path: str, filepath: Path, dandiset_path: Path) → list[ValidationResult]

Parameters:

• asset_path (str) – The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)

• filepath (pathlib.Path) – The actual filesystem path of the asset (used to construct ValidationResult objects)

• dandiset_path (pathlib.Path) – The path to the root of the Dandiset (used to construct ValidationResult objects)