BUG - dependencies cannot be passed in as .json files

[vineetbansal]

Description of the issue

imap-cli -h indicates:

  --dependency DEPENDENCY
Dependency information in str format.Example:'[...]' A path to a JSON file containing this same information may also bepassed in. If dependency is a string ending in '.json', it will be interpreted as such a file path.

However, saving the dependency string as a .json file and passing it in results in a ValueError.

Steps to reproduce the issue

The following works fine:

$imap_cli --instrument swapi --data-level l1 --version v001 --start-date 20260330 --dependency '[{"type": "science", "files": ["imap_swapi_l0_raw_20260330_v001.pkts"]}, {"type": "spice", "files": ["naif0012.tls", "imap_sclk_0140.tsc"]}]'

Now save the same string in the file dep.json

$echo '[{"type": "science", "files": ["imap_swapi_l0_raw_20260330_v001.pkts"]}, {"type": "spice", "files": ["naif0012.tls", "imap_sclk_0140.tsc"]}]' > dep.json

and call the command against this file:

$imap_cli --instrument swapi --data-level l1 --version v001 --start-date 20260330 --dependency dep.json

2026-03-30 16:01:12 - INFO:imap_processing.cli:Interpreting dependency argument as a JSON file: dep.json
Traceback (most recent call last):
  File "/media/vineetb/delta/projects/imap/repos/imap_processing/.venv/bin/imap_cli", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/media/vineetb/delta/projects/imap/repos/imap_processing/imap_processing/cli.py", line 1686, in main
    args = _parse_args()
           ^^^^^^^^^^^^^
  File "/media/vineetb/delta/projects/imap/repos/imap_processing/imap_processing/cli.py", line 270, in _parse_args
    dependency_filepath = download(args.dependency)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/vineetb/delta/projects/imap/repos/imap_processing/.venv/lib/python3.12/site-packages/imap_data_access/io.py", line 95, in download
    path_obj = generate_imap_file_path(file_path.name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/vineetb/delta/projects/imap/repos/imap_processing/.venv/lib/python3.12/site-packages/imap_data_access/file_validation.py", line 42, in generate_imap_file_path
    raise ValueError(
ValueError: Invalid file type for dep.json. It does not matchany file formats.

Expected vs Actual behavior

imap_cli should behave identically in both cases (as per the documentation shown in usage).

Code Snippet (If applicable)

See above.

Additional notes, affected areas, and suggested fixes

A previous PR on this got closed without coming to a resolution.

The code is trying to download the file without declaring this intention in the documentation. Placing the .json at a pre-determined place is not just inconvenient, it also pollutes IMAP_DATA_DIR with custom dependencies for a user of imap_cli. The proper fix to satisfy both the users and developers of imap_cli seems to be:

diff --git a/imap_processing/cli.py b/imap_processing/cli.py
index bda67b62..726f7bf6 100644
--- a/imap_processing/cli.py
+++ b/imap_processing/cli.py
@@ -267,7 +267,7 @@ def _parse_args() -> argparse.Namespace:
         logger.info(
             f"Interpreting dependency argument as a JSON file: {args.dependency}"
         )
-        dependency_filepath = download(args.dependency)
+        dependency_filepath = download(args.dependency) if not Path(args.dependency).exists() else args.dependency
         with open(dependency_filepath) as f:
             args.dependency = f.read()

[maxinelasp]

Hi, the dependency file must match the file format and the folder structure as defined in our science file naming convention. For example, you would create a dependency file in the following location:

data/imap/dependency/swapi/l1/2026/03/imap_swapi_l1_sci_20260330_v001.json

with your dependency string. Then, you would call the CLI with --dependency imap_swapi_l1_sci_20260330_v001.json

This is not well documented anywhere yet, so maybe this ticket can be used to add documentation surrounding this issue @tech3371 ?

Hope this helps @vineetbansal !

[vineetbansal]

Thanks @maxinelasp - My initial concern was that if I start putting arbitrary files in IMAP_DATA_DIR, it would get polluted long term without me realizing it, since there's no checksumming of files during imap-data-access download.

I now realize that if I follow the procedure that the batch_starter code is using internally (i.e. generate a unique dependency_hash of the dependencies), then I'm guaranteed not to clash with any "official" SDC files. However, all these tweaks have to be done at a programmatic level on our end.

I do think it is a useful feature for users with a single line of code change (and @subagonsouth agrees), so if you think it's useful to just support it rather than changing your documentation, you might want to go that route. Otherwise looks like I do have a workaround.