BUG - GLOWS L1B SPICE "insufficent ephemeris data" #2900
Description
Description of the issue
Some GLOWS jobs are failing due to insufficient ephemeris data. I am wondering if this is an issue related to Tim's comment on #2871 about timestamps?
This blocks development of GLOWS L3!
Steps to reproduce the issue
L1B histogram timestamps effected:
| 2026-03-25T00:00:00 | ||
|---|---|---|
| 2026-03-25T00:00:00 | ||
| 2026-03-25T00:00:00 | ||
| 2026-03-24T00:00:00 | ||
| 2026-01-27T00:00:00 | ||
| 2026-03-10T00:00:00 | ||
| 2026-03-16T00:00:00 | ||
| 2026-02-27T00:00:00 | ||
| 2026-01-28T00:00:00 | ||
| 2026-03-23T00:00:00 | ||
| 2026-03-24T00:00:00 | ||
| 2026-03-25T00:00:00 | ||
| 2026-02-28T00:00:00 | ||
| 2026-03-19T00:00:00 | ||
| 2026-03-14T00:00:00 | ||
| 2026-03-05T00:00:00 | ||
| 2026-03-06T00:00:00 | ||
| 2026-03-04T00:00:00 | ||
| 2026-03-22T00:00:00 | ||
| 2026-03-03T00:00:00 | ||
| 2026-03-21T00:00:00 | ||
| 2026-03-15T00:00:00 | ||
| 2026-03-02T00:00:00 | ||
| 2026-03-20T00:00:00 | ||
| 2026-01-29T00:00:00 | ||
| 2026-03-01T00:00:00 | ||
| 2026-01-27T00:00:00 | ||
| 2026-03-10T00:00:00 | ||
| 2026-03-11T00:00:00 | ||
| 2026-03-12T00:00:00 | ||
| 2026-03-18T00:00:00 | ||
| 2026-03-17T00:00:00 | ||
| 2026-03-13T00:00:00 | ||
| 2026-03-09T00:00:00 | ||
| 2026-03-08T00:00:00 | ||
| 2026-03-07T00:00:00 | ||
| 2026-02-15T00:00:00 | ||
| 2026-02-16T00:00:00 | ||
| 2026-02-25T00:00:00 | ||
| 2026-02-13T00:00:00 | ||
| 2026-02-17T00:00:00 | ||
| 2026-02-14T00:00:00 | ||
| 2026-02-24T00:00:00 | ||
| 2026-02-21T00:00:00 | ||
| 2026-02-23T00:00:00 | ||
| 2026-02-26T00:00:00 | ||
| 2026-02-22T00:00:00 | ||
| 2026-02-19T00:00:00 | ||
| 2026-02-18T00:00:00 | ||
| 2026-02-03T00:00:00 | ||
| 2026-02-20T00:00:00 | ||
| 2026-02-12T00:00:00 | ||
| 2026-02-06T00:00:00 | ||
| 2026-02-08T00:00:00 | ||
| 2026-02-07T00:00:00 | ||
| 2026-01-30T00:00:00 | ||
| 2026-02-11T00:00:00 | ||
| 2026-02-09T00:00:00 | ||
| 2026-02-10T00:00:00 | ||
| 2026-02-05T00:00:00 | ||
| 2026-02-04T00:00:00 | ||
| 2026-01-25T00:00:00 | ||
| 2026-02-01T00:00:00 | ||
| 2026-01-26T00:00:00 | ||
| 2026-01-23T00:00:00 | ||
| 2026-01-20T00:00:00 | ||
| 2026-01-21T00:00:00 | ||
| 2026-01-22T00:00:00 | ||
| 2026-01-31T00:00:00 | ||
| 2026-01-19T00:00:00 | ||
| 2026-01-18T00:00:00 | ||
| 2026-01-17T00:00:00 | ||
| 2025-12-31T00:00:00 | ||
| 2026-01-16T00:00:00 | ||
| 2026-01-01T00:00:00 | ||
| 2026-01-14T00:00:00 | ||
| 2025-12-25T00:00:00 | ||
| 2026-01-13T00:00:00 | ||
| 2025-12-22T00:00:00 | ||
| 2025-12-21T00:00:00 | ||
| 2025-12-28T00:00:00 | ||
| 2025-12-27T00:00:00 | ||
| 2025-12-20T00:00:00 | ||
| 2025-12-24T00:00:00 | ||
| 2025-12-30T00:00:00 | ||
| 2025-12-23T00:00:00 | ||
| 2025-12-06T00:00:00 | ||
| 2025-12-26T00:00:00 | ||
| 2025-12-07T00:00:00 | ||
| 2025-12-29T00:00:00 | ||
| 2025-12-05T00:00:00 | ||
| 2025-12-15T00:00:00 | ||
| 2025-12-09T00:00:00 | ||
| 2025-12-18T00:00:00 | ||
| 2025-12-10T00:00:00 | ||
| 2025-12-11T00:00:00 | ||
| 2025-12-14T00:00:00 | ||
| 2025-12-08T00:00:00 | ||
| 2025-12-13T00:00:00 | ||
| 2025-12-12T00:00:00 |
Expected vs Actual behavior
No response
Code Snippet (If applicable)
Processing job details
Processing log
Job properties
2026-04-01 01:36:28 - INFO:imap_processing.cli:Interpreting dependency argument as a JSON file: imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json
2026-04-01 01:36:28 - INFO:imap_data_access.io:Downloading file imap/dependency/glows/l1b/2026/03/imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json from https://api.imap-mission.com/api-key/download/imap/dependency/glows/l1b/2026/03/imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json to /app/data/imap/dependency/glows/l1b/2026/03/imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json
2026-04-01 01:36:29 - INFO:imap_data_access.io:File /app/data/imap/dependency/glows/l1b/2026/03/imap_glows_l1b_hist-cba4cd9b_20260325-repoint00196_v001.json downloaded successfully
2026-04-01 01:36:29 - INFO:imap_processing.cli:IMAP Processing Version: 1.0.25.post18.dev0+b0f1dc4
2026-04-01 01:36:29 - INFO:imap_processing.cli:Processing Glows level l1b
2026-04-01 01:36:29 - INFO:imap_processing.cli:Beginning preprocessing (download dependencies)
2026-04-01 01:36:29 - INFO:imap_data_access.io:Downloading file imap/spice/spin/imap_2026_083_2026_084_01.spin from https://api.imap-mission.com/api-key/download/imap/spice/spin/imap_2026_083_2026_084_01.spin to /app/data/imap/spice/spin/imap_2026_083_2026_084_01.spin
2026-04-01 01:36:29 - INFO:imap_data_access.io:File /app/data/imap/spice/spin/imap_2026_083_2026_084_01.spin downloaded successfully
2026-04-01 01:36:29 - INFO:imap_data_access.io:Downloading file imap/spice/spin/imap_2026_084_2026_085_01.spin from https://api.imap-mission.com/api-key/download/imap/spice/spin/imap_2026_084_2026_085_01.spin to /app/data/imap/spice/spin/imap_2026_084_2026_085_01.spin
2026-04-01 01:36:30 - INFO:imap_data_access.io:File /app/data/imap/spice/spin/imap_2026_084_2026_085_01.spin downloaded successfully
2026-04-01 01:36:30 - INFO:imap_data_access.io:Downloading file imap/spice/spin/imap_2026_085_2026_086_01.spin from https://api.imap-mission.com/api-key/download/imap/spice/spin/imap_2026_085_2026_086_01.spin to /app/data/imap/spice/spin/imap_2026_085_2026_086_01.spin
2026-04-01 01:36:31 - INFO:imap_data_access.io:File /app/data/imap/spice/spin/imap_2026_085_2026_086_01.spin downloaded successfully
2026-04-01 01:36:31 - INFO:imap_data_access.io:Downloading file imap/spice/lsk/naif0012.tls from https://api.imap-mission.com/api-key/download/imap/spice/lsk/naif0012.tls to /app/data/imap/spice/lsk/naif0012.tls
2026-04-01 01:36:32 - INFO:imap_data_access.io:File /app/data/imap/spice/lsk/naif0012.tls downloaded successfully
2026-04-01 01:36:32 - INFO:imap_data_access.io:Downloading file imap/spice/fk/imap_130.tf from https://api.imap-mission.com/api-key/download/imap/spice/fk/imap_130.tf to /app/data/imap/spice/fk/imap_130.tf
2026-04-01 01:36:32 - INFO:imap_data_access.io:File /app/data/imap/spice/fk/imap_130.tf downloaded successfully
2026-04-01 01:36:32 - INFO:imap_data_access.io:Downloading file imap/spice/fk/imap_science_120.tf from https://api.imap-mission.com/api-key/download/imap/spice/fk/imap_science_120.tf to /app/data/imap/spice/fk/imap_science_120.tf
2026-04-01 01:36:33 - INFO:imap_data_access.io:File /app/data/imap/spice/fk/imap_science_120.tf downloaded successfully
2026-04-01 01:36:33 - INFO:imap_data_access.io:Downloading file imap/spice/sclk/imap_sclk_0145.tsc from https://api.imap-mission.com/api-key/download/imap/spice/sclk/imap_sclk_0145.tsc to /app/data/imap/spice/sclk/imap_sclk_0145.tsc
2026-04-01 01:36:33 - INFO:imap_data_access.io:File /app/data/imap/spice/sclk/imap_sclk_0145.tsc downloaded successfully
2026-04-01 01:36:33 - INFO:imap_data_access.io:Downloading file imap/spice/spk/de440.bsp from https://api.imap-mission.com/api-key/download/imap/spice/spk/de440.bsp to /app/data/imap/spice/spk/de440.bsp
2026-04-01 01:36:35 - INFO:imap_data_access.io:File /app/data/imap/spice/spk/de440.bsp downloaded successfully
2026-04-01 01:36:35 - INFO:imap_data_access.io:Downloading file imap/spice/spk/imap_recon_20250925_20260325_v01.bsp from https://api.imap-mission.com/api-key/download/imap/spice/spk/imap_recon_20250925_20260325_v01.bsp to /app/data/imap/spice/spk/imap_recon_20250925_20260325_v01.bsp
2026-04-01 01:36:37 - INFO:imap_data_access.io:File /app/data/imap/spice/spk/imap_recon_20250925_20260325_v01.bsp downloaded successfully
2026-04-01 01:36:37 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_2025_358_2026_085_002.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_2025_358_2026_085_002.ah.bc to /app/data/imap/spice/ck/imap_2025_358_2026_085_002.ah.bc
2026-04-01 01:36:47 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_2025_358_2026_085_002.ah.bc downloaded successfully
2026-04-01 01:36:47 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_2026_084_2026_091_001.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_2026_084_2026_091_001.ah.bc to /app/data/imap/spice/ck/imap_2026_084_2026_091_001.ah.bc
2026-04-01 01:36:48 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_2026_084_2026_091_001.ah.bc downloaded successfully
2026-04-01 01:36:48 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_dps_2025_359_2026_084_001.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_dps_2025_359_2026_084_001.ah.bc to /app/data/imap/spice/ck/imap_dps_2025_359_2026_084_001.ah.bc
2026-04-01 01:36:49 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_dps_2025_359_2026_084_001.ah.bc downloaded successfully
2026-04-01 01:36:49 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_dps_2025_359_2026_087_003.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_dps_2025_359_2026_087_003.ah.bc to /app/data/imap/spice/ck/imap_dps_2025_359_2026_087_003.ah.bc
2026-04-01 01:36:49 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_dps_2025_359_2026_087_003.ah.bc downloaded successfully
2026-04-01 01:36:49 - INFO:imap_data_access.io:Downloading file imap/spice/ck/imap_dps_2026_085_2026_089_001.ah.bc from https://api.imap-mission.com/api-key/download/imap/spice/ck/imap_dps_2026_085_2026_089_001.ah.bc to /app/data/imap/spice/ck/imap_dps_2026_085_2026_089_001.ah.bc
2026-04-01 01:36:50 - INFO:imap_data_access.io:File /app/data/imap/spice/ck/imap_dps_2026_085_2026_089_001.ah.bc downloaded successfully
2026-04-01 01:36:50 - INFO:imap_data_access.io:Downloading file imap/glows/l1a/2026/03/imap_glows_l1a_hist_20260325-repoint00196_v001.cdf from https://api.imap-mission.com/api-key/download/imap/glows/l1a/2026/03/imap_glows_l1a_hist_20260325-repoint00196_v001.cdf to /app/data/imap/glows/l1a/2026/03/imap_glows_l1a_hist_20260325-repoint00196_v001.cdf
2026-04-01 01:36:50 - INFO:imap_data_access.io:File /app/data/imap/glows/l1a/2026/03/imap_glows_l1a_hist_20260325-repoint00196_v001.cdf downloaded successfully
2026-04-01 01:36:50 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-exclusions-by-instr-team_20251112_v001.dat from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-exclusions-by-instr-team_20251112_v001.dat to /app/data/imap/ancillary/glows/imap_glows_l1b-exclusions-by-instr-team_20251112_v001.dat
2026-04-01 01:36:51 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-exclusions-by-instr-team_20251112_v001.dat downloaded successfully
2026-04-01 01:36:51 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat to /app/data/imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat
2026-04-01 01:36:51 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat downloaded successfully
2026-04-01 01:36:51 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-map-of-uv-sources_20250923_v001.dat from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-map-of-uv-sources_20250923_v001.dat to /app/data/imap/ancillary/glows/imap_glows_l1b-map-of-uv-sources_20250923_v001.dat
2026-04-01 01:36:52 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-map-of-uv-sources_20250923_v001.dat downloaded successfully
2026-04-01 01:36:52 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_pipeline-settings_20251112_v001.json from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_pipeline-settings_20251112_v001.json to /app/data/imap/ancillary/glows/imap_glows_pipeline-settings_20251112_v001.json
2026-04-01 01:36:52 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_pipeline-settings_20251112_v001.json downloaded successfully
2026-04-01 01:36:52 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-suspected-transients_20251112_v001.dat from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-suspected-transients_20251112_v001.dat to /app/data/imap/ancillary/glows/imap_glows_l1b-suspected-transients_20251112_v001.dat
2026-04-01 01:36:53 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-suspected-transients_20251112_v001.dat downloaded successfully
2026-04-01 01:36:53 - INFO:imap_data_access.io:Downloading file imap/ancillary/glows/imap_glows_l1b-conversion-table-for-anc-data_20251112_v001.json from https://api.imap-mission.com/api-key/download/imap/ancillary/glows/imap_glows_l1b-conversion-table-for-anc-data_20251112_v001.json to /app/data/imap/ancillary/glows/imap_glows_l1b-conversion-table-for-anc-data_20251112_v001.json
2026-04-01 01:36:53 - INFO:imap_data_access.io:File /app/data/imap/ancillary/glows/imap_glows_l1b-conversion-table-for-anc-data_20251112_v001.json downloaded successfully
2026-04-01 01:36:53 - INFO:imap_processing.cli:Furnishing kernels: ['naif0012.tls', 'imap_130.tf', 'imap_science_120.tf', 'imap_sclk_0145.tsc', 'de440.bsp', 'imap_recon_20250925_20260325_v01.bsp', 'imap_2025_358_2026_085_002.ah.bc', 'imap_2026_084_2026_091_001.ah.bc', 'imap_dps_2025_359_2026_084_001.ah.bc', 'imap_dps_2025_359_2026_087_003.ah.bc', 'imap_dps_2026_085_2026_089_001.ah.bc']
2026-04-01 01:36:53 - INFO:imap_processing.spice.spin:Using the following spin-tables in processing: ['imap_2026_083_2026_084_01.spin', 'imap_2026_084_2026_085_01.spin', 'imap_2026_085_2026_086_01.spin']
2026-04-01 01:36:53 - INFO:imap_processing.cli:Beginning actual processing
/usr/local/lib/python3.12/site-packages/imap_processing/ancillary/ancillary_dataset_combiner.py:366: UserWarning: loadtxt: input contained no data: "/app/data/imap/ancillary/glows/imap_glows_l1b-map-of-excluded-regions_20251112_v001.dat"
data = np.loadtxt(filepath, comments="#")
/usr/local/lib/python3.12/site-packages/imap_processing/ancillary/ancillary_dataset_combiner.py:389: UserWarning: Input line 1 contained no data and will not be counted towards `max_rows=50000`. This differs from the behaviour in NumPy <=1.22 which counted lines rather than rows. If desired, the previous behaviour can be achieved by using `itertools.islice`.
Please see the 1.23 release notes for an example on how to do this. If you wish to ignore this warning, use `warnings.filterwarnings`. This warning is expected to be removed in the future and is given only once per `loadtxt` call.
data = np.loadtxt(filepath, comments="#", dtype=str)
Traceback (most recent call last):
File "/usr/local/bin/imap_cli", line 8, in <module>
Processing GLOWS l1b
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/site-packages/imap_processing/cli.py", line 1705, in main
instrument.process()
File "/usr/local/lib/python3.12/site-packages/imap_processing/cli.py", line 447, in process
products = self.do_processing(dependencies)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/imap_processing/cli.py", line 727, in do_processing
glows_l1b(
File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b.py", line 78, in glows_l1b
output_dataarrays = process_histogram(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b.py", line 304, in process_histogram
l1b_fields = xr.apply_ufunc(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/xarray/computation/apply_ufunc.py", line 1267, in apply_ufunc
return apply_dataarray_vfunc(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/xarray/computation/apply_ufunc.py", line 312, in apply_dataarray_vfunc
result_var = func(*data_vars)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/xarray/computation/apply_ufunc.py", line 820, in apply_variable_ufunc
result_data = func(*input_data)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/numpy/lib/_function_base_impl.py", line 2518, in __call__
return self._call_as_normal(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/numpy/lib/_function_base_impl.py", line 2511, in _call_as_normal
return self._vectorize_call(func=func, args=vargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/numpy/lib/_function_base_impl.py", line 2592, in _vectorize_call
res = self._vectorize_call_with_signature(func, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/numpy/lib/_function_base_impl.py", line 2630, in _vectorize_call_with_signature
results = func(*(arg[index] for arg in args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b.py", line 300, in create_histogram_l1b
return HistogramL1B( # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 20, in __init__
File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b_data.py", line 850, in __post_init__
self.update_spice_parameters()
File "/usr/local/lib/python3.12/site-packages/imap_processing/glows/l1b/glows_l1b_data.py", line 956, in update_spice_parameters
imap_state = geometry.imap_state(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/imap_processing/spice/geometry.py", line 141, in imap_state
state, _ = spiceypy.spkezr(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/spiceypy/spiceypy.py", line 148, in with_errcheck
res = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/spiceypy/spiceypy.py", line 12917, in spkezr
check_for_spice_error(None)
File "/usr/local/lib/python3.12/site-packages/spiceypy/spiceypy.py", line 132, in check_for_spice_error
raise dynamically_instantiate_spiceyerror(
spiceypy.utils.exceptions.SpiceSPKINSUFFDATA:
================================================================================
Toolkit version: CSPICE_N0067
SPICE(SPKINSUFFDATA) --
Insufficient ephemeris data has been loaded to compute the state of -43 (IMAP) relative to 10 (SUN) at the ephemeris epoch 2026 MAR 25 10:02:53.932.
spkezr_c --> SPKEZR --> SPKEZ --> SPKGEO
================================================================================Additional notes, affected areas, and suggested fixes
No response
Metadata
Metadata
Assignees
Type
Projects
Status