Add per-asset dataStandard, HED standard, and extensions to StandardsType by yarikoptic · Pull Request #371 · dandi/dandi-schema

yarikoptic · 2026-02-24T22:11:01Z

Summary

Add dataStandard: Optional[List[StandardsType]] field to BareAsset so dandi-cli can declare per-asset data standards (NWB, BIDS, HED, OME/NGFF) that flow through to AssetsSummary during aggregation
Add version: Optional[str] field to StandardsType to track standard versions (e.g. NWB 2.7.0, BIDS 1.9.0, HED 8.2.0)
Add extensions: Optional[List[StandardsType]] self-referencing field to StandardsType for standard extensions (NWB ndx-* extensions, HED library schemas)
Add hed_standard constant alongside existing nwb_standard, bids_standard, ome_ngff_standard
Collect per-asset dataStandard entries in aggregate_assets_summary(), with deprecated heuristic fallbacks for older clients (remove after 2026-12-01)
Register dataStandard in migrate() _FIELDS_INTRODUCED for downgrade pruning
Comprehensive CLAUDE.md with architecture docs and schema change checklist

Test plan

test_hed_standard_structure — verifies hed_standard has correct name, identifier, schemaKey
test_aggregate_per_asset_datastandard — HED declared per-asset flows to summary
test_aggregate_per_asset_datastandard_no_duplication — BIDS not duplicated when declared both per-asset and via heuristic
All 243 existing tests pass

PR in dandi-cli:

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions dandi-cli#1811

🤖 Generated with Claude Code

codecov · 2026-02-24T22:58:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.94%. Comparing base (bf5c98e) to head (702f464).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #371      +/-   ##
==========================================
+ Coverage   97.92%   97.94%   +0.02%     
==========================================
  Files          18       18              
  Lines        2405     2431      +26     
==========================================
+ Hits         2355     2381      +26     
  Misses         50       50

Flag	Coverage Δ
unittests	`97.94% <100.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

candleindark · 2026-03-03T19:07:02Z

+    dataStandard: Optional[List[StandardsType]] = Field(
+        None,
+        description="Data standard(s) applicable to this asset.",
+        json_schema_extra={"readOnly": True},


readOnly should set to true only in the case that the dataStandard field is assigned and managed solely by the server per https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-validation-00#rfc.section.9.4. Is it the case that the dataStandard field is intended to be assigned and managed solely by the server?

I would guess the answer is no -- that it is intended to be extracted from the source files by the DANDI CLI running client-side.

INTERESTING! We also have readOnly on above "TODO" attributes which I guess nobody ever assigned to any asset. Although we are indeed to populate it from client (or on server) but since we have other similar wasGeneratedBy also editable, I see no point in making this readOnly, and I guess it is really our own thing, hence

Suggested change

json_schema_extra={"readOnly": True},

json_schema_extra={"nskey": DANDI_NSKEY},

right?

Good point — this is populated client-side by dandi-cli, not server-managed, so readOnly is incorrect. Replaced with "nskey": DANDI_NSKEY in 702f464 RF: address PR review - replace readOnly with nskey, fix extensions field.

bendichter · 2026-03-03T19:46:42Z

@yarikoptic this looks great! It will be exciting to be able to query the archive for NWB version and extension usage for data going forward. I agree with @candleindark that this probably should not be readOnly. Other than that, looks good. It would be great to also see a draft of dandi-cli that extracts this information. I'd prefer to have a draft of that before merging this, in case we are able to catch any edge cases.

yarikoptic · 2026-03-03T19:50:46Z

It would be great to also see a draft of dandi-cli that extracts this information

It is there waiting for you

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions dandi-cli#1811

candleindark

Left some minor suggestions for change

candleindark · 2026-03-03T19:13:13Z

+    dataStandard: Optional[List[StandardsType]] = Field(
+        None,
+        description="Data standard(s) applicable to this asset.",
+        json_schema_extra={"readOnly": True},


Shouldn't we have an "nskey" key in the json_schema_extra dict? If dataStandard is not defined in a known ontology, we I think we should use ours, i.e.,

json_schema_extra={"nskey": DANDI_NSKEY}

Agreed — added "nskey": DANDI_NSKEY in 702f464 RF: address PR review - replace readOnly with nskey, fix extensions field.

it should still have readOnly as this is a metadata element that should not be editable using the UI or API. it is also filled in on the server side.

IIRC that readOnly only for web ui entry -- we are the ones to provide metadata records via API while uploading assets. So if we forbid that in API, nothing would fill that in AFAIK.

webUI does not ATM care about any asset level metadata entry, so all that readOnly is of no effect and somewhat inconsistent ATM anyways so I would not split hairs over it if someone pushes 'readOnly' there

i'm pretty positive asset summary is filled by the server validation.

sorry didn't realize this is asset level. i was thinking dandiset level. however, there are a set of fields that the server unsets when uploaded.

candleindark · 2026-03-03T19:58:17Z

+        description="Version of the standard used.",
+        json_schema_extra={"nskey": "schema"},
+    )
+    extensions: Optional[List["StandardsType"]] = Field(


Suggested change

extensions: Optional[List["StandardsType"]] = Field(

extensions: Optional[List[StandardsType]] = Field(

With from __future__ import annotations at the beginning of the file, you don't need the type to be wrapped in a string.

Right — removed the quotes since from __future__ import annotations handles it. Fixed in 702f464 RF: address PR review - replace readOnly with nskey, fix extensions field.

candleindark · 2026-03-03T20:07:28Z

+    extensions: Optional[List["StandardsType"]] = Field(
+        None,
+        description="Extensions to the standard used in this dataset "
+        "(e.g. NWB extensions like ndx-*, HED library schemas).",


Shouldn't we set "nskey" key in the json_schema_extra dict? If extensions is not defined in a known ontology, we I think we should use ours, i.e.,

json_schema_extra={"nskey": DANDI_NSKEY}

Agreed — added "nskey": DANDI_NSKEY to extensions in 702f464 RF: address PR review - replace readOnly with nskey, fix extensions field.

candleindark · 2026-03-03T20:22:25Z

    identifier="DOI:10.25504/FAIRsharing.9af712",
 ).model_dump(mode="json", exclude_none=True)

+hed_standard = StandardsType(


Defining hed_standard in lower case is consistent with the established pattern, as in ome_ngff_standard. However, all the standard variables, nwb_standard, bids_standard, ome_ngff_standard, and hed_standard should be specified in upper case laters since they function as constants.

I think that should be done in a separate PR though.

Agreed — keeping lowercase here for consistency with the existing pattern, and the rename to UPPER_CASE can be done in a separate PR as you suggest.

bendichter · 2026-03-03T20:27:59Z

I approve the dandi dli PR. Once we respond to @candleindark 's comments, this is good to go from my end

…s on StandardsType - Bump DANDI_SCHEMA_VERSION to 0.8.0, add 0.7.0 to ALLOWED_INPUT_SCHEMAS - Add `dataStandard: Optional[List[StandardsType]]` to BareAsset for per-asset standard declarations (NWB, BIDS, HED, OME/NGFF) - Add `version: Optional[str]` to StandardsType - Add `extensions: Optional[List["StandardsType"]]` (self-referencing) to StandardsType for NWB ndx-* extensions, HED library schemas, etc. - Add `hed_standard` constant (RRID:SCR_014074) - Collect per-asset dataStandard in aggregate_assets_summary(), with deprecated path/encoding heuristic fallbacks (remove after 2026-12-01) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…cklist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@candleindark

…ield Address @candleindark's review comments on PR #371: - BareAsset.dataStandard: replace readOnly=True with nskey=DANDI_NSKEY (field is populated client-side by dandi-cli, not server-managed) - StandardsType.extensions: add nskey=DANDI_NSKEY, fix description ("in this dataset" -> removed, since standard applies per-asset), and remove unnecessary string quotes on forward ref (from __future__ import annotations handles it) Co-Authored-By: Claude Code 2.1.104 / Claude Opus 4.6 <noreply@anthropic.com>

yarikoptic · 2026-04-13T18:49:12Z

@@ -880,6 +897,11 @@ class StandardsType(BaseType):
    identifier="DOI:10.25504/FAIRsharing.9af712",
 ).model_dump(mode="json", exclude_none=True)


@kabilar please add additional constructs for anything LINC related -- like those for bigtiff etc, and then here

https://github.com/dandi/dandi-cli/pull/1811/changes#diff-70a4bb759311189af6f4468d78254d863319fc4660ebe437060b6634c5a7b724R395

as to use them in dandi-cli

yarikoptic mentioned this pull request Feb 24, 2026

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions dandi/dandi-cli#1811

Open

5 tasks

yarikoptic commented Mar 3, 2026

View reviewed changes

Comment thread dandischema/metadata.py Outdated

yarikoptic requested review from bendichter, candleindark and satra March 3, 2026 16:31

yarikoptic force-pushed the enh-hed branch from be40b26 to 571978e Compare March 3, 2026 16:33

candleindark reviewed Mar 3, 2026

View reviewed changes

yarikoptic and others added 2 commits April 13, 2026 14:30

docs: comprehensive CLAUDE.md with architecture, conventions, and che…

721bd11

…cklist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

yarikoptic force-pushed the enh-hed branch from 571978e to 721bd11 Compare April 13, 2026 18:31

yarikoptic commented Apr 13, 2026

View reviewed changes

	json_schema_extra={"readOnly": True},
	json_schema_extra={"nskey": DANDI_NSKEY},

	extensions: Optional[List["StandardsType"]] = Field(
	extensions: Optional[List[StandardsType]] = Field(

		@@ -880,6 +897,11 @@ class StandardsType(BaseType):
		identifier="DOI:10.25504/FAIRsharing.9af712",
		).model_dump(mode="json", exclude_none=True)

Conversation

yarikoptic commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

codecov bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

yarikoptic commented Mar 3, 2026

Uh oh!

candleindark left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yarikoptic commented Feb 24, 2026 •

edited

Loading

codecov bot commented Feb 24, 2026 •

edited

Loading