feat!(ingest, workflows): add GCA downloads to mirror and enable use as segment grouping overrides in ingest

[anna-parker]

resolves #5905

Alternative to #5917 - have action download hydrated assemblies from NCBI virus, unzip and re-hydrate assemblies, then rezip assemblies with tar.zstd.

In #5917 the action downloaded the hydrated assemblies, unzipped and then rezipped with tar.zstd

BREAKING CHANGE

Users of ingest with the grouping_override key should rename it to grouping_override_url.

Numbers

The dehydrated files are 19M, unzipped 260M and 2.3G unzipped and dehydrated.
(I can even zip the dehydrated + zipped files once more with tar.zstd and get them down to 9.7M).
The unzipped and dehydrated file when rezipped with tar.zstd is 79M.

Download, re-hydration and zipping took 40min in the CLI action for influenza A, download from the mirror locally took seconds, unzipping also took under a minute. (Downloading from the mirror and rehydrating in ingest took 50min)

Testing using CCHF on the preview

I am now using these results for CCHF on the preview:

{
    "GCA_000854165": [
        "AY389361.2",
        "AF467768.2",
        "U88410.1"
    ],
    "GCF_000854165": [
        "NC_005301.3",
        "NC_005300.2",
        "NC_005302.1"
    ]
}

are now all together: https://calculate-gca-groups-2.loculus.org/seq/LOC_000N04F.1 and https://calculate-gca-groups-2.loculus.org/seq/LOC_000PX88.1

Screenshot

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by appropriate, automated tests.
• Any manual testing that has been done is documented (i.e. what exactly was tested?)

🚀 Preview: Add preview label to enable

[claude]

This PR may be related to: #5905 (Ingest: add gca accession downloads to the ingest pipeline)

[claude]

Claude finished @anna-parker's task in 2m 52s —— View job

---

Code Review

• Gather context and read changed files
• Review .github/workflows/datasets-assembly-mirror.yml
• Review ingest/scripts/calculate_assembly_groups.py
• Review ingest/Snakefile changes
• Review kubernetes/loculus/values.schema.json and values.yaml
• Post inline comments and final review summary

---

Good overall implementation. The approach of downloading, rehydrating, and mirroring assembly data as tar.zst archives is clean, and the Snakefile flow for conditional group override calculation is logically structured. A few issues to address:

Breaking change: The rename of grouping_override → grouping_override_url in the schema and Snakefile is a breaking change for any existing configs using the old key. Worth a note in the PR or a migration check.

Missing schema entry: grouping_ignore_list_url is referenced in the Snakefile (config.get("grouping_ignore_list_url", "")) but has no entry in values.schema.json. This leaves it undocumented and unvalidated for Helm users.

No tests: The new calculate_assembly_groups.py script has no automated tests. The PR checklist acknowledges this — it would be good to add at least a basic unit test for the happy path and the duplicate-detection logic.

See inline comments for specific issues.

[theosanderson]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0f71257fb4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[theosanderson]

Not code I know well but looks sensible to me!