Support sandboxes in collections

[josephjclark]

Description

Sandboxes can't support projects that use collections because collection names are globally unique, and the v1 collections API has no way to disambiguate between a parent and its sandbox. This PR relaxes the uniqueness constraint to per-project and adds a v2 API that resolves collections by (project_id, name).

On the lifecycle side, empty collection records are now cloned into a sandbox when it's provisioned so that collections.get('my-collection') resolves correctly without copying production data. When a sandbox is merged back into its parent, collection names are synchronised (new collections in the sandbox are created empty in the parent, collections missing from the sandbox are removed from the parent). Collection data is never synced.

V1 continues to work for any project that doesn't have name collisions. When a v1 lookup finds the same name across multiple projects, it returns 409 Conflict pointing clients at v2. API version is selected via the x-api-version header (missing or "1" = v1, "2" = v2, anything else = 400 Bad Request). Header-based versioning was chosen in discussion with @stuartc and @josephjclark to keep a single collections base path.

Closes #3548

Validation steps

Setup: create a project with a collection

1. Create a new project called Cat Shelter Registry
2. Navigate to Project Settings > Collections and create a collection called cats
3. Create a workflow called "Fetch Cat Facts" with:
   • A cron trigger (any schedule is fine, we will run it manually)
   • A single job using @openfn/language-http@latest with this code:

     get('https://catfact.ninja/facts?limit=10');
     
     each('$.data.data[*]', fn(state) => {
       const fact = state.data;
       const id = fact.fact.substring(0, 20).replace(/\s/g, '-').toLowerCase();
       return collections.set('cats', `fact-${id}`, {
         fact: fact.fact,
         length: fact.length
       })(state);
     });

4. Run the workflow manually by clicking Run on the workflow page
5. Navigate to Project Settings > Collections. You should see: cats listed with a storage size greater than 0
6. Click the download button next to the cats collection. Open the downloaded JSON file. You should see 10 items, each with a fact and length field.

This is your parent project with production data. Keep it open.

1. Sandbox provision clones empty collections

1. From the Cat Shelter Registry project, navigate to Sandboxes
2. Click Create Sandbox and name it "sandbox-test"
3. Once created, click into the sandbox to open it
4. Inside the sandbox, navigate to Project Settings > Collections

Verify: The cats collection is listed. Click the download button. Open the JSON file -- it should be an empty array []. The sandbox has the collection, but none of the parent's 10 cat facts were copied.

2. Collection data is never merged back

1. Inside the sandbox from step 1, open the "Fetch Cat Facts" workflow
2. Edit the job to fetch breed data instead:

   get('https://catfact.ninja/breeds?limit=5');
   
   each('$.data.data[*]', fn(state) => {
     const breed = state.data;
     return collections.set('cats', `breed-${breed.breed}`, {
       breed: breed.breed,
       country: breed.country,
       coat: breed.coat
     })(state);
   });

3. Run the workflow manually
4. Navigate to Project Settings > Collections inside the sandbox. Click download on cats. Open the JSON -- you should see 5 breed items.
5. Now go back to Sandboxes and merge the sandbox into the parent
6. After the merge, open the parent project (Cat Shelter Registry)
7. Navigate to Project Settings > Collections. Click download on cats.

Verify: Open the JSON. You should still see the original 10 cat fact items (with fact and length fields). None of the sandbox's 5 breed items (with breed, country, coat fields) should be present. Collection data is never merged.

3. New collections in sandbox are created in parent on merge

1. Create a new sandbox from Cat Shelter Registry called "sandbox-breeds"
2. Open the sandbox and navigate to Project Settings > Collections
3. Create a new collection called cat-breeds
4. Open the workflow and edit the job:

   get('https://catfact.ninja/breeds?limit=10');
   
   each('$.data.data[*]', fn(state) => {
     const breed = state.data;
     return collections.set('cat-breeds', breed.breed, {
       country: breed.country,
       origin: breed.origin,
       coat: breed.coat
     })(state);
   });

5. Run the workflow. Go to Project Settings > Collections and download cat-breeds -- you should see 10 breed items.
6. Go back to Sandboxes and merge this sandbox into the parent
7. Open the parent project and navigate to Project Settings > Collections

Verify: You should now see two collections listed: cats and cat-breeds. Click download on cat-breeds and open the JSON. It should be an empty array []. The collection was created in the parent, but the sandbox's breed data was not copied over.

4. Collections deleted in sandbox are deleted from parent on merge

1. In the parent project, go to Project Settings > Collections and create a collection called temp-data
2. Create a new sandbox called "sandbox-cleanup"
3. Open the sandbox and navigate to Project Settings > Collections. You should see both cats and temp-data listed.
4. Delete the temp-data collection from the sandbox
5. Confirm only cats remains in the sandbox's collections list
6. Go back to Sandboxes and merge this sandbox into the parent
7. Open the parent project and navigate to Project Settings > Collections

Verify: temp-data is no longer listed. Only cats (and cat-breeds if you did step 3) remain. The collection and all its items were removed from the parent because it was deleted in the sandbox.

API test setup

Steps 5-7 use curl to test the API directly. First, get a token and project ID:

1. In Lightning, click your profile icon (top right) and go to Tokens
2. Click Generate New Token, give it a name, and copy the token value
3. In your terminal, save it for the curl commands:

   export TOKEN="paste-your-token-here"

4. To get the project ID, open the Cat Shelter Registry project in Lightning. The URL will look like http://localhost:4000/projects/abc123-def456-.../w. The UUID after /projects/ is your project ID:

   export PROJECT_ID="abc123-def456-..."

5. V1 conflict detection (API)

This requires an unmerged sandbox so that cats exists in two projects. If you merged all sandboxes in previous steps, create a new one first.

# V1 lookup -- the name "cats" exists in multiple projects
curl -s -H "Authorization: Bearer $TOKEN" \
     http://localhost:4000/collections/cats | jq .

Verify: The response is 409 Conflict:

{
  "error": "Multiple collections found with this name. Use API v2 with a project_id."
}

Now use v2 with the project ID to resolve it:

curl -s -H "Authorization: Bearer $TOKEN" \
     -H "x-api-version: 2" \
     http://localhost:4000/collections/$PROJECT_ID/cats | jq .

Verify: The response is 200 with the parent's cat fact items.

6. Version header validation (API)

curl -s -H "Authorization: Bearer $TOKEN" \
     -H "x-api-version: 99" \
     http://localhost:4000/collections/cats | jq .

Verify: The response is 400 Bad Request:

{
  "error": "Unsupported API version: \"99\". Supported versions: 1, 2."
}

7. V1 backwards compatibility

Delete all sandboxes first (so cats only exists in one project), or use a project with a uniquely named collection.

curl -s -H "Authorization: Bearer $TOKEN" \
     http://localhost:4000/collections/cats | jq .

Verify: The response is 200 with the collection items, exactly as it worked before this change. No x-api-version header is needed when the collection name is unambiguous.

Additional notes for the reviewer

AI Usage

• I have used Claude Code
• I have used another model
• I have not used AI

Pre-submission checklist

• I have performed an AI review of my code (we recommend using /review with Claude Code)
• I have implemented and tested all related authorization policies.
• I have updated the changelog.
• I have ticked a box in "AI usage" in this PR

[josephjclark]

Testing and QA notes:

• The existing worker & collections adaptor should continue to work against the new lightning server, no changes needed
• If you create two collections with the same id, and request that id through the legacy API, you should get an error
• After updating the adaptor and worker, this branch should return the correct collection (even for duplicate collection names)

[josephjclark]

@elias-ba one thing I didn't think to check is access control. A project should not be able to access the collections of another project.

How can we enforce this?

Using the CLI you can send any access token you want and so long as the project is within that access token's scope, you're good.

In the worker, we send a JWT which I believe is scoped only to access for that workflow.

That must be sufficient right? We just look at the scope of the access token?

[codecov]

Codecov Report

❌ Patch coverage is 98.98990% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 89.63%. Comparing base (46d36bc) to head (d1fa4e3).

Files with missing lines	Patch %	Lines
...ightning_web/controllers/collections_controller.ex	96.29%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4613      +/-   ##
==========================================
+ Coverage   89.56%   89.63%   +0.06%     
==========================================
  Files         444      445       +1     
  Lines       21505    21570      +65     
==========================================
+ Hits        19262    19335      +73     
+ Misses       2243     2235       -8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[elias-ba]

Good question @josephjclark , and yes, the access token scope is sufficient. The controller already enforces it.

Every action calls authorize/2 which hits Lightning.Policies.Collections.authorize/3. That policy has two clauses:

• For a user subject, it delegates to ProjectUsers.authorize(:access_project, user, ...) which checks membership against the collection's project
• For a run subject, it directly compares Runs.get_project_id_for_run(run) == collection.project_id

So a CLI user with a PAT can only touch collections in projects they're a member of, and a worker's run token can only touch collections in the same project as the run it was issued for. There's a test that specifically covers the worker case ("with a run token, cannot access a different project's collection" returns 401) and another for the user case ("returns 401 when user does not have access to the project").

One subtle thing worth noting: v2 makes this tighter by design. With v1, a request with a stale or leaked token could at least look up a collection by name globally; v2 requires you to know (and be authorized for) the project_id upfront.

[elias-ba]

Hey @josephjclark this is now looking good to me and it has all things we discussed this morning. Please do test / QA this extensively and let me know if you find anything weird

[github-actions]

Security Review

⚠️ Automated security review did not complete.

Claude hit the max-turns limit or encountered an error before posting findings.
A manual review of S0 (project-scoped data access), S1 (authorization policies),
and S2 (audit trail coverage) is recommended for this PR.

See the workflow run for details.

[midigofrank]

@elias-ba this looks great, I'm a little worried with the approach to remove the routes from the router. One thing that worries me is that the collections_router plug hijacks all actions and manually calls the controller, in my mind I'm expecting it handle the x-api-version and backward compatibility only. I don't like that we're losing visibility of the routes from the router.ex

I don't know the exact discussion you had with @stuartc , so I have added him as a reviewer as he has more context. Everything else looks perfect.