-
Notifications
You must be signed in to change notification settings - Fork 5
GLOWS: limit reprocessing of L3e to N CR's #64
Description
Summary
Once a GLOWS L3d is processed, the L3e initializer then compares the new L3d to the previous L3d and finds the first CR which has changed. It then reprocesses all L3e from that point forward.
Update to only reprocess L3e for changed CRs and a configurable number of CRs later.
Rationale: a given CR of L3d data affects the L3e for that CR and maybe one or two more, as the solar wind propagates out. It does not have to reprocess L3e for all time. The knowledge of new L3c should be enough to decide which L3e to update.
This is essentially an efficiency feature: the current behavior is enough to ensure that the L3e are always up to date, but may result in excess reprocessing.
Estimate of 8 hours from #44.
Requirements
- Add a parameter to the GLOWS
pipeline-settings-l3bcdefile to specify the number of subsequent CRs to process with updated L3d (henceforth N, N integer >= 0). - Update L3e processor to identify "changed" CRs, either from changes in L3d or reprocessed L3c.
- Process all L3e from each changed CR and N subsequent CRs.
- Deduplicate this list first, e.g. if 2308 and 2310 changed, and N is 2, reprocess 2308-2312, don't reprocess 2310 twice.
- No need to check if the L3e already exist (other than for version number); just make if there are enough data to create for that CR.
Related tickets
This does not block and is not blocked by #60, but they may interact if we update the dependency system.
Successor to #44.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status