CoDICE: Fixing attrs bugs for CAVA by tech3371 · Pull Request #2891 · IMAP-Science-Operations-Center/imap_processing

tech3371 · 2026-03-31T15:13:03Z

Change Summary

closes #2765

Overview

Fixes for three products that needed fixes for CAVA, L2 omni, sectored and lo-sw-species.

File changes

Updated CDF attr YAML and code to not carry l1b attrs which caused few issues. L2 needs to read from L2 attrs file.

Testing

tech3371 · 2026-03-31T15:16:25Z

@jtniehof pining you for awareness.

Copilot

Pull request overview

This PR updates CoDICE L2 metadata handling (primarily CDF variable attributes) to improve CAVA compliance and avoid incorrectly carrying L1B attributes into L2 products (per #2765).

Changes:

Rebuild several L2 coordinate variables as explicit xr.DataArrays with L2 CDF attributes (instead of reusing L1B coordinate DataArrays/attrs).
Add/propagate epoch_delta_plus / epoch_delta_minus coordinates for hi-omni and hi-sectored products and adjust related attribute handling.
Update multiple CoDICE CDF attribute YAMLs (formats, fill values, display types, scaling, and species placeholder replacement support).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`imap_processing/codice/codice_l2.py`	Reworks L2 coordinate construction/attrs; adds epoch delta coordinates; applies species attribute placeholder replacements.
`imap_processing/cdf/config/imap_codice_l2-lo-species_variable_attrs.yaml`	Updates lo-species coordinate and data variable attribute definitions (formats/scales/valid ranges).
`imap_processing/cdf/config/imap_codice_l2-hi-sectored_variable_attrs.yaml`	Updates hi-sectored coordinate/data attribute definitions, including species placeholder-based DEPEND/LABL_PTR wiring.
`imap_processing/cdf/config/imap_codice_l2-hi-omni_variable_attrs.yaml`	Updates hi-omni attribute definitions (formats, scales, display types, etc.).
`imap_processing/cdf/config/imap_codice_l1a_variable_attrs.yaml`	Minor attribute value/type tweaks and reordering/augmentation for DE attrs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

imap_processing/codice/codice_l2.py

imap_processing/cdf/config/imap_codice_l2-lo-species_variable_attrs.yaml

imap_processing/cdf/config/imap_codice_l2-hi-sectored_variable_attrs.yaml

subagonsouth

Nothing blocking a merge, but a couple questions and an idea to reduce duplicated code.

subagonsouth · 2026-03-31T15:20:36Z

imap_processing/cdf/config/imap_codice_l2-hi-omni_variable_attrs.yaml

  FIELDNAM: epoch delta minus
-  FILLVAL: -9223372036854775808
-  FORMAT: I18
+  FILLVAL: *real_fillval


I'm not sure that I understand this change. I would expect epoch_delta_minus/plus to be int64 to match the epoch datatype and contain integer nanoseconds.

I was getting this warning when running ISTP checks. I remember now. Epoch delta were double because we had to divide by something and so on. That's why deltas were double type.

epoch_delta_plus FILLVAL value of '9.223372036854776E18' is non-standard. The recommended value is '-1.0E31'. FORMAT 'I32' is invalid for a CDF_DOUBLE type variable

I think the fix there is to change the type of the variable, as Tim suggested, not the fill (and the format).

so if delta's data is float, convert to int? I haven't looked closely but it's ok to round up for delta if they are float type? If that is ok, I can totally revert and change data type of delta.

This is a mildly uninformed opinion since I haven't tracked the time handling through the l1 code.

Since the deltas are in ns, rounding shouldn't cause an issue in terms of scientific interpretation or anything like that. There might be some slight weirdness with certain tools if adjacent times get rounded differently, which might imply a small gap between successive measurements or overlap between them. I did a chunk check of an existing omni hi l2 and it looks like the fractional part of all the deltas is zero, so that feels like something not worth worrying about right now.

Perfect! Thank you for that information! It's scientific user experience that I don't enough and didn't want to assume :)

I will make changes Tim suggested!

I am going to revert these epoch updates and update l1a to convert to int in upcoming PR. Want to get this CAVA support for Leng Yeng.

subagonsouth · 2026-03-31T15:25:07Z

imap_processing/codice/codice_l2.py

+    epoch_attrs = l1b_dataset["epoch"].attrs.copy()
+    epoch_attrs["DELTA_MINUS_VAR"] = "epoch_delta_minus"
+    epoch_attrs["DELTA_PLUS_VAR"] = "epoch_delta_plus"
+    l1b_dataset["epoch"].attrs = epoch_attrs


Is the copy() needed here? Is it not possible to just do:

l1b_dataset["epoch"].attrs["DELTA_MINUS_VAR"] = "epoch_delta_minus" l1b_dataset["epoch"].attrs["DELTA_PLUS_VAR"] = "epoch_delta_plus"

I tried but something going on. Can I implement this change in the upcoming PR?

subagonsouth · 2026-03-31T15:26:31Z

imap_processing/codice/codice_l2.py


+    # TODO: remove this addition once SAMMI or CDF attr manager handles this
+    # Update epoch attrs to add epoch_delta_plus and epoch_delta_minus
+    epoch_attrs = l1b_dataset["epoch"].attrs.copy()


Same question about needing to copy and reassign the epoch attrs here.

subagonsouth · 2026-03-31T15:30:03Z

imap_processing/codice/codice_l2.py

+        "energy_h": xr.DataArray(
+            l1b_dataset["energy_h"].values,
+            dims=("energy_h",),
+            attrs=cdf_attrs.get_variable_attributes("energy_h", check_schema=False),
+        ),
        "energy_h_label": xr.DataArray(
            l1b_dataset["energy_h"].values.astype(str),
            dims=("energy_h",),
            attrs=cdf_attrs.get_variable_attributes(
                "energy_h_label", check_schema=False
            ),
        ),
-        "energy_he3": l1b_dataset["energy_he3"],
+        "energy_he3": xr.DataArray(
+            l1b_dataset["energy_he3"].values,
+            dims=("energy_he3",),
+            attrs=cdf_attrs.get_variable_attributes("energy_he3", check_schema=False),
+        ),


Lots of repeated code here. I wonder if it would be possible to do something like:

for coord in LIST_OF_COORDS: new_coords[coord] = xr.DataArray( l1b_dataset[coord].values, dims=(coord,), attrs=cdf_attrs.get_variable_attributes(coord, check_schema=False), )

I am going to make this change in upcoming PR. Really good suggestion!

vmartinez-cu · 2026-03-31T15:40:44Z

imap_processing/cdf/config/imap_codice_l2-hi-omni_variable_attrs.yaml

  VALIDMAX: 16777216.0
  VALIDMIN: 0.0
  DICT_KEY: SPASE>Particle>ParticleType:Ion,ParticleQuantity:NumberFlux,Qualifier:Differential
  CATDESC: c (Root 2 spacing)


Just a comment but not needed for this PR - It might be worth considering a more descriptive CATDESC for the particle variables. HIT L2 attrs have some examples of more descriptive strings for this field.

I hoping to get minimal changes needed for CAVA today to unblock scientist. But I plan to go back and do more thorough updates for CoDICE L2. I will keep this in mind!

I hoping to get minimal changes needed for CAVA today to unblock scientist. But I plan to go back and do more thorough updates for CoDICE L2. I will keep this in mind!

👍 Once the CoDICE team can look at the data in CAVA I think they can give much better feedback on the metadata...otherwise it's hard to tell the CATDESC from the FIELDNAM from the LABLAXIS.

imap_processing/cdf/config/imap_codice_l2-hi-omni_variable_attrs.yaml

imap_processing/codice/codice_l2.py

tech3371 requested a review from Copilot March 31, 2026 15:13

tech3371 self-assigned this Mar 31, 2026

Copilot started reviewing on behalf of tech3371 March 31, 2026 15:13 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

subagonsouth approved these changes Mar 31, 2026

View reviewed changes

vmartinez-cu reviewed Mar 31, 2026

View reviewed changes

imap_processing/cdf/config/imap_codice_l2-hi-omni_variable_attrs.yaml Show resolved Hide resolved

vmartinez-cu reviewed Mar 31, 2026

View reviewed changes

imap_processing/codice/codice_l2.py Show resolved Hide resolved

vmartinez-cu reviewed Mar 31, 2026

View reviewed changes

imap_processing/codice/codice_l2.py Show resolved Hide resolved

CoDICE: Fixing attrs bugs for CAVA

63bc277

tech3371 force-pushed the cod_l2_attrs branch from 9d2a267 to 63bc277 Compare March 31, 2026 17:08

test fixes

234466a

tech3371 merged commit 28fec45 into IMAP-Science-Operations-Center:dev Mar 31, 2026
14 checks passed

tech3371 mentioned this pull request Mar 31, 2026

L2 SPASE and ISTP requirements #2577

Open

10 tasks

Conversation

tech3371 commented Mar 31, 2026

Change Summary

Overview

File changes

Testing

Uh oh!

tech3371 commented Mar 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

subagonsouth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tech3371 Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tech3371 Mar 31, 2026 •

edited

Loading