The problem
PR #561 introduced validation that raises a ValueError when a DataRequirement filters or groups on columns declared in columns_requiring_finalisation.
This prevents silent zero-result bugs when those columns contain pd.NA if lazy loading was being used.
The validation currently blocks filtering/grouping on columns_requiring_finalisation unconditionally for any DataCatalog, even when the loaded catalog is already fully finalised (e.g. produced by the "complete" CMIP6 parser where finalised=True and these columns are populated).
This makes those columns effectively unusable with DataCatalog even when they're safe to use.
Definition of "done"
The problem
PR #561 introduced validation that raises a
ValueErrorwhen aDataRequirementfilters or groups on columns declared incolumns_requiring_finalisation.This prevents silent zero-result bugs when those columns contain
pd.NAif lazy loading was being used.The validation currently blocks filtering/grouping on
columns_requiring_finalisationunconditionally for anyDataCatalog, even when the loaded catalog is already fully finalised (e.g. produced by the "complete" CMIP6 parser wherefinalised=Trueand these columns are populated).This makes those columns effectively unusable with
DataCatalogeven when they're safe to use.Definition of "done"