Remove support for prefix matching on bucket names

[Sleepful]

Issue: #400

Summary

The refactor cleanly removes the U+FFFF sentinel, COLLATE "C", and prefix matching from the Postgres compactor, replacing them with incremental bucket discovery and per-bucket exact-match compaction. EXPLAIN plans confirm optimal index usage with no COLLATE and no Filter lines.

Tradeoffs

1. Incremental discovery vs. single-pass: The new approach adds one small index-only query per 200 buckets. For extreme deployments (millions of buckets), this adds ~5000 round-trips, but each is sub-millisecond. The plan's analysis is sound — the tradeoff is heavily in favor of correctness and simplicity.

2. Breaking change for CLI users: Users passing bucket definition names (-b global) will now get a validation error instead of silently compacting matching buckets. This is intentional and documented. No existing tests used prefix matching, confirming this was an unused feature path.

Manual validation

CLI warning message

node service/lib/entry.js compact -c ../powersync.yaml -b global
info: Successfully registered Module Core.
error: Invalid bucket names: global. Pass full bucket names (e.g., "global[]"), not bucket definition names (e.g., "global").
error: Recipe `ps` failed on line 7 with exit code 1

Run compact tests

PG_STORAGE_TEST_URL="postgres://postgres:postgres@localhost:5432/powersync_storage_test" pnpm --filter @powersync/service-module-postgres-storage test -- --run

✓ test/src/connection-report-storage.test.ts (14 tests) 36ms

 Test Files  5 passed (5)
      Tests  74 passed (74)
   Start at  01:55:12
   Duration  29.56s (transform 747ms, setup 3.22s, import 3.49s, tests 22.46s, environment 0ms)

Run EXPLAIN

These represent the queries that are used in file

    PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres \
      -d powersync_storage_test -c "SET search_path TO powersync;" \
      -c "SELECT '=== DISCOVERY: incremental bucket enumeration ===' AS query;" \
      -c "EXPLAIN SELECT DISTINCT bucket_name FROM bucket_data WHERE group_id = 1 AND bucket_name > '' ORDER BY bucket_name ASC LIMIT 200;" \
      -c "SELECT '=== SINGLE BUCKET: no cursor ===' AS query;" \
      -c "EXPLAIN SELECT op, op_id, source_table, table_name, row_id, source_key, bucket_name FROM bucket_data WHERE group_id = 1 AND bucket_name = 'global[]' ORDER BY op_id DESC LIMIT 10000;" \
      -c "SELECT '=== SINGLE BUCKET: with cursor ===' AS query;" \
      -c "EXPLAIN SELECT op, op_id, source_table, table_name, row_id, source_key, bucket_name FROM bucket_data WHERE group_id = 1 AND bucket_name = 'global[]' AND op_id < 50000 ORDER BY op_id DESC LIMIT 10000;"

just explain query	Source	Lines
DISCOVERY: incremental bucket enumeration	PostgresCompactor.ts:90-98	compactAllBuckets() discovery query: SELECT DISTINCT bucket_name ... WHERE bucket_name > $lastBucket ORDER BY bucket_name ASC LIMIT 200
SINGLE BUCKET: no cursor	PostgresCompactor.ts:126-149	compactSingleBucket() on first iteration, when upperOpIdLimit = BIGINT_MAX (line 123) — effectively no cursor constraint
SINGLE BUCKET: with cursor	PostgresCompactor.ts:126-149	Same query on subsequent iterations, after upperOpIdLimit = lastBatchItem.op_id (line 158) makes the AND op_id < $upperOpIdLimit clause a real constraint

                       query
---------------------------------------------------
 === DISCOVERY: incremental bucket enumeration ===
(1 row)

                                             QUERY PLAN
-----------------------------------------------------------------------------------------------------
 Limit  (cost=0.15..8.17 rows=1 width=32)
   ->  Result  (cost=0.15..8.17 rows=1 width=32)
         ->  Unique  (cost=0.15..8.17 rows=1 width=32)
               ->  Index Only Scan using unique_id on bucket_data  (cost=0.15..8.17 rows=1 width=32)
                     Index Cond: ((group_id = 1) AND (bucket_name > ''::text))
(5 rows)

              query
----------------------------------
 === SINGLE BUCKET: no cursor ===
(1 row)

                                          QUERY PLAN
----------------------------------------------------------------------------------------------
 Limit  (cost=0.15..8.17 rows=1 width=200)
   ->  Index Scan Backward using unique_id on bucket_data  (cost=0.15..8.17 rows=1 width=200)
         Index Cond: ((group_id = 1) AND (bucket_name = 'global[]'::text))
(3 rows)

               query
------------------------------------
 === SINGLE BUCKET: with cursor ===
(1 row)

                                          QUERY PLAN
-----------------------------------------------------------------------------------------------
 Limit  (cost=0.15..8.17 rows=1 width=200)
   ->  Index Scan Backward using unique_id on bucket_data  (cost=0.15..8.17 rows=1 width=200)
         Index Cond: ((group_id = 1) AND (bucket_name = 'global[]'::text) AND (op_id < 50000))
(3 rows)

Skipped validation

I did not run the MongoDB tests, seems like those files were not touched. Let me know if I should run them, or if CI takes care of it. Basically this command: pnpm --filter @powersync/service-module-mongodb-storage test -- --run

[CLAassistant]

All committers have signed the CLA.

[changeset-bot]

🦋 Changeset detected

Latest commit: 4d3ff2c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-postgres-storage	Patch
@powersync/service-core	Patch
@powersync/service-schema	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-core	Patch
@powersync/service-module-mongodb-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

Thanks!

[Sleepful]

Pushed two fixes:

1. Updated test to use updateSyncRulesFromYaml() interface (aligned with upstream changes)
2. Ran prettier on the three affected files

Build and format checks pass locally. Ready for re-review.

[Sleepful]

Fixed the actual test failure: getBucketDataBatch was returning 0 items because the test used bare bucket names (global[]) but storage version 2 now uses versioned names (1#global[]). Updated the test to use bucketRequest() / bucketRequestMap() helpers. All 92 postgres-storage tests pass locally.

[Sleepful]

⚠️ Flaky CI test ⚠️

There was a CI failure in Postgres Test (12) (https://github.com/powersync-ja/powersync-service/actions/runs/22217913667/job/64265997954?pr=499) — looks like a flaky test. It's replicating changing primary key in wal_stream.test.ts, which this PR doesn't touch. Passes on all other PG versions (11, 13–18) and on main.

I re-ran the failed test without any changes and it passed ✅

[Sleepful]

rebased