Load Zarr objects via recipe and catalog

[valeriupredoi]

Further on from #2584 and extending #2785 by exposing the load Zarr fnctionality to front-end (recipe), this is what I think we need to do:

• allow Zarr catalogs, eg this catalog as a valid DataSource member; possible hurdles:
  • assemble a list of available catalogs
  • have metadata attached to each catalog so it can be easily selected
• interface via recipe so that the Zarr catalog is loaded and searched over, rather than a local or intake-esgf object or store/catalog; possible hurdles:
  • create new recipe key-value that identifies the correct Zarr catalog that is needed for each dataset
  • or, create a generic marker that labels datasets to be loaded from Zarr objects, so that all available catalogs be searched for per specific (in Zarr) dataset
• allow for redundancy: search multiple catalogs of one or more datasets are not available from a certain catalog
• handle auth credentials if Zarr objects are not in public buckets - I would strongly not implement this, we may be falling in a deep security rabbit hole

[valeriupredoi]

I will be doing the work on this, but I also assigned @bouweandela since I'll need to consult with him on technical matters, alas, asking @ESMValGroup/technical-lead-development-team for any useful input/opinions 🍻

[bouweandela]

I would suggest using intake-esm for this, e.g. https://pangeo-data.github.io/pangeo-cmip6-cloud/accessing_data.html#loading-an-esm-collection

[bouweandela]

Another possibility is xcube, I have a first draft in #2917.