ODD PI 25.4 Objective 4: 🛠️ Zarr Development

[abarciauskas-bgse]

Motivation

Zarr and zarr-python are currently under supported, yet NASA teams and data users are continuing to rely on and expect Zarr to provide collection-level analysis-ready data. We have an obligation to support the stability and feature development of Zarr and zarr-python for NASA to have continued expansive use of its products.

Description

See AC

Acceptance Criteria

• Support zarr-python 3 and VirtualiZarr through regular maintenance, triaging issues
• Develop specification and prototype implementation for variable-chunks (either in shape or encoding), see also Zarr extension for stacked / concatenated virtual views · Issue AGU session, workshop, and presentation + share learnings #288 · zarr-developers/zarr-specs · GitHub
• Quantify the impact of Zarr improvements for future NASA architecture.

Sub-tasks

• example 1
• example 2

[abarciauskas-bgse]

The subissues are all still open so I'm inclined to leave this issue as incomplete. But I know substantial work was done on Zarr so let me know if you think we should mark this as complete @maxrjones

[maxrjones]

I think we should close this out and incorporate any wrap-up tasks into new objectives, because I don't think we'll have time for them unless they're connected to active commitments.

Here's a claude-generated report on outcomes, based on data I gave it from the GitHub query below:

Click to see query

#!/bin/bash

AUTHORS=(maxrjones d-v-b)
DATE_RANGE="2025-07-01..2025-10-10"
OUTPUT="output.md"

JQ_QUERY='
group_by(.repository.nameWithOwner) |
.[] |
"\n## (.[0].repository.nameWithOwner) ((length) PRs)\n" + (
.[] | "#(.number) [(.state)] (.author.login): (.title)\n Created: (.createdAt | split("T")[0]) | Closed: (if .closedAt then (.closedAt | split("T")[0]) else "—" end)\n (.url)"
)
'

"$OUTPUT" # Clear/create file

for author in "${AUTHORS[@]}"; do
gh search prs
--limit 1000
--author="$author"
--updated="$DATE_RANGE"
--json number,title,state,createdAt,closedAt,url,repository,author
| jq -r "$JQ_QUERY" >> "$OUTPUT"
done

Objective 4: 🛠️ Zarr Development

Project Interval: July 1 - October 10, 2025
Contributors: Max Jones (@maxrjones) & Davis Bennett (@d-v-b)

---

Executive Summary

This report documents contributions to zarr-python 3.x and VirtualiZarr ecosystem during the NASA IMPACT ODD project interval, supporting the objective to maintain and develop core Zarr infrastructure for NASA's analysis-ready data needs.

Key Achievements:

• 3 stable zarr-python releases shipped (3.1.0, 3.1.2, 3.1.3)
• 29 zarr-python core PRs merged
• 19 VirtualiZarr PRs with major features and bug fixes
• 9 pydantic-zarr PRs completing v3 support
• Community event organization (Zarr Summit 2025)

---

AC1: Support zarr-python 3 and VirtualiZarr through Regular Maintenance

zarr-python Core (29 PRs)

PR	Title	Author	Status	Created	Closed
#3502	docs/examples	@d-v-b	✅ Merged	2025-10-02	2025-10-03
#3490	Add experimental module	@d-v-b	✅ Merged	2025-09-25	2025-09-30
#3477	Customize color theme for docs	@maxrjones	✅ Merged	2025-09-19	2025-09-22
#3476	Changelog entry for docs change	@maxrjones	✅ Merged	2025-09-19	2025-09-19
#3474	3.1.3 release notes (#3472)	@d-v-b	✅ Merged	2025-09-18	2025-09-18
#3472	3.1.3 release notes	@d-v-b	✅ Merged	2025-09-18	2025-09-18
#3426	fix/group no error	@d-v-b	❌ Closed	2025-09-02	2025-09-04
#3424	Specify default event loop for async tests	@d-v-b	✅ Merged	2025-09-01	2025-09-02
#3404	Generate 3.1.2 release notes	@d-v-b	✅ Merged	2025-08-25	2025-08-25
#3400	Add a runtime type checker for metadata objects	@d-v-b	🟡 Open	2025-08-24	—
#3372	Use local imports in store ABC to avoid circular import issues	@d-v-b	✅ Merged	2025-08-13	2025-08-13
#3371	Ensure that we test examples with a fresh env	@d-v-b	✅ Merged	2025-08-13	2025-08-13
#3355	Add modules for types	@d-v-b	🟡 Open	2025-08-06	—
#3329	Pin setuptools-scm to !=9.0.0	@d-v-b	✅ Merged	2025-08-04	2025-08-04
#3280	Convert inf, -inf, nan to JSON	@d-v-b	✅ Merged	2025-07-22	2025-07-22
#3276	[do not merge] proof of concept for unified v2 / v3 codecs	@d-v-b	❌ Closed	2025-07-21	2025-09-22
#3268	Remove warnings about vlen-utf8 and vlen-bytes codecs	@d-v-b	✅ Merged	2025-07-17	2025-07-17
#3249	Add a section showing the parse_data_type function	@d-v-b	✅ Merged	2025-07-15	2025-07-16
#3248	Add unmerged changelog entry to release notes	@d-v-b	✅ Merged	2025-07-15	2025-07-16
#3242	Fix obstore type hints	@d-v-b	✅ Merged	2025-07-14	2025-07-14
#3241	3.1.0 release notes	@d-v-b	✅ Merged	2025-07-14	2025-07-15
#3240	Fix type hints for hypothesis test	@d-v-b	✅ Merged	2025-07-14	2025-07-14
#3230	Update 2819 release note	@d-v-b	✅ Merged	2025-07-11	2025-07-14
#3229	Update release note for dtype refactor	@d-v-b	✅ Merged	2025-07-11	2025-07-14
#3228	Remove chunk encoding from config	@d-v-b	✅ Merged	2025-07-10	2025-07-11
#3225	Remove usage of the _deprecate_positional_args decorator	@d-v-b	✅ Merged	2025-07-10	2025-07-14
#3211	Remove duplicate codecov configuration	@maxrjones	✅ Merged	2025-07-07	2025-07-07
#3210	Add a release checklist issue template	@maxrjones	✅ Merged	2025-07-06	2025-07-07
#3206	Fix typo in warning	@d-v-b	✅ Merged	2025-07-03	2025-07-04
#3205	3.0.9 3.0.10 release notes	@d-v-b	❌ Closed	2025-07-03	2025-07-09
#3204	Auto backport of pr 3195 on 3.0.10	@d-v-b	✅ Merged	2025-07-03	2025-07-03
#3203	Backport PR #3193 on branch 3.0.10	@d-v-b	✅ Merged	2025-07-03	2025-07-03
#3202	Export more JSON types	@d-v-b	✅ Merged	2025-07-03	2025-07-06
#3193	Remove breaking check about auto_mkdir for FSSpecStore	@maxrjones	✅ Merged	2025-07-02	2025-07-03

VirtualiZarr (19 PRs)

PR	Title	Author	Status	Created	Closed
#794	Support in-lined Kerchunk references	@maxrjones	🟡 Open	2025-09-05	—
#793	Polish release notes for v2.1.2	@maxrjones	✅ Merged	2025-09-03	2025-09-03
#769	Extract endianness from Bytes codec in V2 metadata conversion	@maxrjones	✅ Merged	2025-08-11	2025-08-14
#767	Fix usage with Icechunk>=1.1.0 due to validate_containers change	@maxrjones	❌ Closed	2025-08-11	2025-08-14
#766	Include serializer from ManifestStore in Icechunk virtual references	@maxrjones	✅ Merged	2025-08-09	2025-08-13
#737	Test across different operating systems in the main CI workflow	@maxrjones	🟡 Open	2025-07-25	—
#736	Run slow tests in ci based on test-slow label	@maxrjones	❌ Closed	2025-07-25	2025-08-07
#724	Update release date	@maxrjones	✅ Merged	2025-07-21	2025-07-21
#723	Do not merge: Publish to TestPyPI	@maxrjones	❌ Closed	2025-07-21	2025-07-21
#722	Update project metadata and bump release action	@maxrjones	✅ Merged	2025-07-21	2025-07-21
#720	Do not pass an empty filters list to ArrayV2Metadata	@maxrjones	✅ Merged	2025-07-21	2025-07-21
#713	Include all V2 codecs as filters	@maxrjones	✅ Merged	2025-07-20	2025-07-21
#711	Polish release notes	@maxrjones	✅ Merged	2025-07-20	2025-07-20
#704	Add docstrings to the HDFParser	@maxrjones	✅ Merged	2025-07-18	2025-07-18
#672	Handle scalar array and vars named c in ManifestStore	@maxrjones	✅ Merged	2025-07-13	2025-07-15
#662	Change parser signature to use ObjectStoreRegistry	@maxrjones	✅ Merged	2025-07-06	2025-07-16
#661	Remove unnecessary custom pickling support	@maxrjones	✅ Merged	2025-07-06	2025-07-06
#654	Proof of concept: lazily loading datasets with VirtualiZarr	@maxrjones	❌ Closed	2025-07-02	2025-07-02
#653	Fix icechunk tests with Zarr 3.0.9	@maxrjones	✅ Merged	2025-07-01	2025-07-01
#651	Fix mkdocs site_url and don't use directory urls	@maxrjones	✅ Merged	2025-07-01	2025-07-01

pydantic-zarr (9 PRs)

PR	Title	Author	Status	Created	Closed
#95	Disallow empty codecs, use sane default in auto_codecs, allow codecs as strings	@d-v-b	✅ Merged	2025-08-14	2025-08-14
#89	Allow from_flat to take kwargs for model_dump	@d-v-b	❌ Closed	2025-08-07	2025-08-11
#85	Update cd workflow to use hatch	@d-v-b	✅ Merged	2025-07-31	2025-07-31
#78	Allow basemodel in TBaseAttr	@d-v-b	✅ Merged	2025-07-17	2025-07-18
#77	Use the JSON form of the fill value	@d-v-b	✅ Merged	2025-07-16	2025-07-28
#74	Handle zarr 3.1.0	@d-v-b	✅ Merged	2025-07-15	2025-07-15
#67	Get GroupSpec.to_zarr working for v3	@d-v-b	✅ Merged	2025-07-09	2025-07-15
#64	Implement arrayspec.to_zarr for v3	@d-v-b	✅ Merged	2025-07-08	2025-07-08
#61	Pin python to 3.11 in ci	@d-v-b	✅ Merged	2025-07-08	2025-07-08

geozarr-spec (2 PRs)

PR	Title	Author	Status	Created	Closed
#91	Remove v0.4 markdown spec given new OGC template	@maxrjones	✅ Merged	2025-08-28	2025-10-01
#85	Make dimension binding for data variables match coordinate variables for zarr v3	@d-v-b	✅ Merged	2025-08-14	2025-08-26

virtual-tiff (4 PRs)

PR	Title	Author	Status	Created	Closed
#46	Use token for pulling data from DagsHub	@maxrjones	✅ Merged	2025-09-24	2025-09-24
#45	Add two new test TIFFs	@maxrjones	✅ Merged	2025-09-24	2025-09-24
#43	Temporarily remove support for int64	@maxrjones	✅ Merged	2025-07-22	2025-07-22
#41	Migrate to VirtualiZarr 2.0	@maxrjones	✅ Merged	2025-07-22	2025-07-22

Other Infrastructure (3 PRs)

Repository	PR	Title	Author	Status	Created	Closed
numcodecs	#760	Update pyproject.toml	@d-v-b	❌ Closed	2025-07-03	2025-08-13
zarr-developers.github.io	#152	Use pixi for dev environment	@d-v-b	✅ Merged	2025-09-19	2025-09-20
zarr-feedstock	#136	Convert zarr-feedstock to v1 feedstock	@d-v-b	✅ Merged	2025-07-16	2025-07-16

---

AC2: Develop Specification and Prototype for Variable-Chunks

Exploratory & Prototype Work (6 PRs)

Repository	PR	Title	Author	Status	Created	Closed
zarr-python	#3400	Add a runtime type checker for metadata objects	@d-v-b	🟡 Open	2025-08-24	—
zarr-python	#3355	Add modules for types	@d-v-b	🟡 Open	2025-08-06	—
zarr-python	#3276	[do not merge] proof of concept for unified v2 / v3 codecs	@d-v-b	❌ Closed	2025-07-21	2025-09-22
zarr-python	#3268	Remove warnings about vlen-utf8/vlen-bytes codecs	@d-v-b	✅ Merged	2025-07-17	2025-07-17
zarr-python	#3249	Document parse_data_type function	@d-v-b	✅ Merged	2025-07-15	2025-07-16
VirtualiZarr	#654	Proof of concept: lazily loading datasets with VirtualiZarr	@maxrjones	❌ Closed	2025-07-02	2025-07-02

---

AC3: Quantify Impact of Zarr Improvements

Documentation & Reporting (2 PRs)

Repository	PR	Title	Author	Status	Created	Closed
veda-odd	#235	Add report for August 2025	@maxrjones	✅ Merged	2025-09-05	2025-09-05
veda-odd	#234	Add starting point for docs (titiler overview)	@maxrjones	✅ Merged	2025-08-29	2025-08-29

Architecture Decision Records (2 PRs)

Repository	PR	Title	Author	Status	Created	Closed
veda-architecture	#649	Add titiler-multidim timeseries ADR	@maxrjones	🟡 Open	2025-08-20	—
veda-architecture	#647	Start ADR for cubesphere support	@maxrjones	🟡 Open	2025-08-19	—

---

Community Engagement & Knowledge Sharing

Zarr Summit 2025 (13 PRs)

PR	Title	Author	Status	Created	Closed
#16	Minor website updates	@maxrjones	✅ Merged	2025-10-10	2025-10-10
#15	Add CNG event policies	@maxrjones	✅ Merged	2025-10-09	2025-10-09
#14	Add presentations to schedule	@maxrjones	✅ Merged	2025-10-09	2025-10-09
#13	Add sponsor	@maxrjones	✅ Merged	2025-10-09	2025-10-09
#12	Update schedule	@maxrjones	✅ Merged	2025-10-06	2025-10-06
#11	Update more references to CNG	@maxrjones	✅ Merged	2025-09-22	2025-09-22
#10	Update recommended hotels based on availability	@maxrjones	✅ Merged	2025-09-22	2025-09-22
#7	Add sponsor page	@maxrjones	✅ Merged	2025-09-18	2025-09-18
#6	Add prospectus	@maxrjones	✅ Merged	2025-09-18	2025-09-18
#5	Add draft schedule	@maxrjones	✅ Merged	2025-09-18	2025-09-18
#3	Add logos for organizers and sponsor	@maxrjones	✅ Merged	2025-09-17	2025-09-17
#2	Update registration	@maxrjones	✅ Merged	2025-09-17	2025-09-17
#1	Add github deploy workflow	@maxrjones	✅ Merged	2025-08-27	2025-08-27

Broader Community (5 PRs)

Repository	PR	Title	Author	Status	Created	Closed
pangeo.io	#107	Add links to showcase talk announcements	@maxrjones	✅ Merged	2025-10-07	2025-10-07
pangeo.io	#106	Add fall showcase presentation schedule	@maxrjones	✅ Merged	2025-09-11	2025-09-11
pangeo.io	#104	Advertise AGU 2025 sessions	@maxrjones	✅ Merged	2025-07-09	2025-07-09
community-sprints	#53	Add invitation to attend Zarr summit adopter day	@maxrjones	✅ Merged	2025-08-15	2025-10-03
events.geojupyter.org	#10	Add convener bio	@maxrjones	✅ Merged	2025-07-17	2025-07-17

---

Impact Summary

Summary Statistics

Status	Count
✅ Merged	70
🟡 Open	4
❌ Closed	6
Total	80

Repository Distribution

Repository	PRs	Merged	Open	Closed
zarr-python	29	27	2	0
VirtualiZarr	19	15	2	2
pydantic-zarr	9	8	0	1
zarr-summit-2025	13	13	0	0
geozarr-spec	2	2	0	0
virtual-tiff	4	4	0	0
Other	4	1	0	3

Releases Shipped:

• zarr-python 3.1.0, 3.1.2, 3.1.3
• VirtualiZarr 2.1.2
• Multiple pydantic-zarr releases supporting v3

Qualitative Impact

Stability Improvements:

• Fixed critical circular import issues
• Resolved Windows compatibility problems
• Improved type safety across codebase
• Enhanced test coverage and CI reliability

Feature Enablement:

• Experimental module for new features
• Improved dtype handling
• Better codec support
• Enhanced v3 metadata handling

Developer Experience:

• Comprehensive documentation improvements
• Release checklists and automation
• Better error messages
• Cleaner API surface

Ecosystem Health:

• pydantic-zarr v3 support complete
• geozarr-spec v3 alignment
• VirtualiZarr 2.0 migration support
• Improved package distribution

---

Recommendations

For Continued Maintenance

1. Complete ongoing type system work (#3400, #3355)
2. Finalize in-lined Kerchunk support (#794)
3. Expand cross-platform CI testing (#737)
4. Monitor variable-chunk specification development

For Impact Measurement

1. Benchmark performance improvements from dtype and codec updates
2. Track NASA project adoption of zarr-python 3.x
3. Document migration patterns from v2 to v3
4. Measure test coverage gains and issue resolution velocity

For Future Development

1. Advance variable-chunk prototyping based on lessons from unified codec work
2. Enhance runtime validation with metadata type checker
3. Improve interoperability between VirtualiZarr, icechunk, and zarr-python
4. Expand experimental features based on community feedback

---

Report Generated: January 29, 2026
Project Interval: July 1 - October 10, 2025
Total PRs: 80 (66 core Zarr + 14 community)