Skip to content

Bug fixing dreq_query#195

Open
tylov wants to merge 4 commits intoCMIP-Data-Request:mainfrom
tylov-fork:dreq_query_fixes
Open

Bug fixing dreq_query#195
tylov wants to merge 4 commits intoCMIP-Data-Request:mainfrom
tylov-fork:dreq_query_fixes

Conversation

@tylov
Copy link
Copy Markdown

@tylov tylov commented Nov 20, 2025

No description provided.

1. create_dreq_tables_for_request() has **kwarg argument, but it was not passed
   down to _get_base_dict(), so the args where not updated. Fixed.
2. create_dreq_tables_for_variables() did not have **kwargs, but also
   calls _get_base_dict() which should have used it. Fixed.
3. A number of dict keys are used internally with old names *after* the
   dict was updated to use new keys. Changed the keys to the new names.
out-commented .update() on dreq_tables in get_dimensions().
@tylov
Copy link
Copy Markdown
Author

tylov commented Nov 20, 2025

Fixes issues when using dreq_raw_raw_export.json as you have to specify consoliate=False and export='raw' in the parameters. Also fixes missing keys in get_dimension_sizes()
Should be squashed when merged.

@tylov
Copy link
Copy Markdown
Author

tylov commented Nov 21, 2025

As noted, there are only two main issues: missing **kwargs and missing keys in get_dimension_sizes()
Example code to demonstrate the fixes:

import json
from data_request_api.query import dreq_query

dreq_content_file="./CMIP7_DReq_Software/data_request_api/data_request_api/tests/test_datasets/dreq_raw_export.json"

with open(dreq_content_file) as f:
    content = json.load(f)

dreq_version = "v1.2.2.2"

request_tables = dreq_query.create_dreq_tables_for_request(content, dreq_version, consolidate=False, export='raw')
variable_tables = dreq_query.create_dreq_tables_for_variables(content, dreq_version, consolidate=False, export='raw')
dim_sizes = dreq_query.get_dimension_sizes(variable_tables)

I am new to this API, but from the docs I found, it says export_dreq_lists_json --all_opportunities v1.2.2.2 CMIP7_dreq_content.json should create a json file compatible with dreq_raw_export.json, but it does not.

@JamesAnstey
Copy link
Copy Markdown
Contributor

Thanks @tylov - we have been a little lax in keeping the raw export supported in the API because it represents the "backend" of Airtable databases, which is used to generate the official public DR releases (which are stored in the release export json file).

Can I ask why you're making use of the raw export json file? In doing so there's a risk of using DR information that's in an official release. E.g. the backend variables table contains all variables that were input to the DR consultation / thematic papers process, so there are a bunch that in the end no one requested and hence are not in official DR releases.

@tylov
Copy link
Copy Markdown
Author

tylov commented Nov 26, 2025

I used the raw export files just for some internal testing, however thanks for clarifying its purpose. If you plan to remove the support for raw input using these functions, please close the PR, otherwise it fixes some logical issues with them, as they could not use the raw export, although they were seemingly designed to do so initially.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants