Skip to content

Firebase cold-storage approach#3552

Open
isTravis wants to merge 15 commits intomainfrom
tr/firebase-reduce2
Open

Firebase cold-storage approach#3552
isTravis wants to merge 15 commits intomainfrom
tr/firebase-reduce2

Conversation

@isTravis
Copy link
Copy Markdown
Member

@isTravis isTravis commented Apr 3, 2026

This is an attempt to drastically reduce our firebase storage. Checkpoints are static and only loaded at page-load, there is no need for them to be storaged in firebase. Furthermore, we are storing checkpoints and changes for pubs that are essentially done in perpetuity in firebase. If we instead 'cold store' checkpoints in postgres, we can remove a large majority of content from firebase, reducing our costs.

Adds a DraftCheckpoints table in Postgres as the single source of truth for draft checkpoints, replacing Firebase's checkpoints/ nodes. The server now loads checkpoints from Postgres first and layers any new Firebase changes on top, falling through to the original Firebase-only path when no PG checkpoint exists. The client writes new checkpoints to Postgres via POST /api/draftCheckpoint instead of Firebase. This is a zero-downtime change: before bootstrapCheckpoints runs, every draft load silently falls back to Firebase.

A new coldStorage tool freezes inactive drafts (default 30+ days) by storing their full state in Postgres—including discussions and cumulative stepMaps for release discussion-anchor mapping—then wiping their Firebase path entirely. The existing cleanupFirebase cron is updated to use PG checkpoints as the prune threshold (capturing stepMaps before deleting Firebase changes). Cold storage runs weekly on Saturday, cleanup on Sunday. Deployment order: deploy → run migration (2026_04_02_addColdCheckpointIdToDrafts) → run bootstrapCheckpoints --execute → let crons run. All scripts are idempotent and safe to re-run.

No impact on user experience - all 'coldStored' drafts load just as they would previously. Performance likely will improve since we're pulling checkpoint data from on-machine pg rather than firebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant