Sunset kerchunk and support all of virtualizarr parsers with a new top level API#1272
Conversation
…solidated API to virtualize() and removed consolidate_metadata()
|
|
chuckwondo
left a comment
There was a problem hiding this comment.
@betolink, this looks great. I have only a few minor suggestions, but I'm approving, as nothing is a show-stopper from my perspective.
|
|
||
| if access == "direct": | ||
| credentials_endpoint, region = get_granule_credentials_endpoint_and_region( | ||
| granules[0] |
There was a problem hiding this comment.
The list of granules could be empty, so I suggest handling this case with a pertinent ValueError, otherwise the user will get a less helpful IndexError here.
There was a problem hiding this comment.
Done, not sure if the Obstore registry instantiation is the best way to place this but for now... if the granules are not a empty list and they have data inks, in theory they could be virtualized. We can later use a custom "flag" if there is one in CMR about QA e.g. "vritualizable"
Co-authored-by: Chuck Daniels <cjdaniels4@gmail.com>
|
@chuckwondo since I updated the code with the granule validation for Obstore, I think I need your approval again =) |
danielfromearth
left a comment
There was a problem hiding this comment.
Awesome work, @betolink! This is such a good step in making the virtualization process more streamlined :)
…solidated API to virtualize() and removed consolidate_metadata()
Description
Addresses #1271 by consolidating
consolidate_metadata,open_virtual_mfdatasetsandopen_virtual_datasetinto a singlevirtualize()API. The user can control which parser earthaccess needs to use but there is a hierarchy: dmrpp first, then HDF then the rest.We may need another couple methods in case our users decide to use VirtualiZarr directly but don't want to deal with configuring Obstore/Icechunk. Not required for this PR IMO.
"Ready for review" checklist
Merge checklist
closes #1)CHANGELOG.mdupdatedREADME.mdupdatedpre-commit.ci autofixif pre-commit is failing)📚 Documentation preview 📚: https://earthaccess--1272.org.readthedocs.build/en/1272/