VED-1017: Generate JSON ACK file - with E2E test changes

[JamesW1-NHS]

Summary

This one is not to be merged to master. It has unverified commits. Please merge the identical PR #1168 instead.

• Routine Change
• ❗ Breaking Change

Acceptance Criteria

A structured JSON ACK is generated when batch processing completes (EOF received / completion confirmed).

The ACK includes:

• Batch-level metadata (e.g. system, version, generated date, filename, provider)
• A summary section showing total, successful, and failed record counts
• A failures section listing only failed records with relevant error details
• No per-record success acknowledgements are included
• The ACK file is generated only once batch completion is confirmed (via EOF)

Conforms to the Design document: https://nhsd-confluence.digital.nhs.uk/pages/viewpage.action?pageId=1187373499&spaceKey=Vacc&title=Immunisation%2BFHIR%2BAPI%2B-%2BACK%2BFile%2BManagement#addcomment

Note: I have marked this as a breaking change as it requires changes to the E2E Automation Tests. These are ongoing. This currently builds successfully in master because the tests requiring modification have been temporarily patched.

Note to Testers

• Changes are to ack_backend only. No change to API or other lambdas.
• E2E Automation tests require modification (see above).

Reviews Required

• Dev
• Test
• Tech Author
• Product Owner

Review Checklist

ℹ️ This section is to be filled in by the reviewer.

• I have reviewed the changes in this PR and they fill all of the acceptance criteria of the ticket.
• If there were infrastructure, operational, or build changes, I have made sure there is sufficient evidence that the changes will work.
• If there were changes that are outside of the regular release processes e.g. account infrastructure to setup, manual setup for external API integrations, secrets to set, then I have checked that the developer has flagged this to the Tech Lead as release steps.
• I have checked that no Personal Identifiable Data (PID) is logged as part of the changes.

[github-actions]

This branch is working on a ticket in the NHS England VED JIRA Project. Here's a handy link to the ticket:

VED-1017

[dlzhry2nhs]

Nice work, has improvements for the overall test suite and tests the new JSON ack functionality nicely.

Adding the new test cases has brought the automation test suite up from 6 mins - 10 mins. Is it worth removing some of the repeated parametrized test cases from the smoke tests and leaving for functional only. Might also be worth dropping 1-2 batch cases so we only test key scenarios, as that can be quite slow. Ideally 5-8 mins for the pipeline tests...but could always look at it in another ticket.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
97.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[JamesW1-NHS]

Done in #1168