-
Notifications
You must be signed in to change notification settings - Fork 0
feat: Option B1 staging→production Neon promotion pipeline #1405
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
316d9c1
2a79d6a
ccf2b2a
f690479
2ee5e44
5f036d8
d8747af
9e59e94
07cf56f
f3e9bcb
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,18 +2,15 @@ name: Database Migrations | |
|
|
||
| # Runs on every PR that touches migration files, and on every push to main. | ||
| # | ||
| # PR behaviour → validate + dry-run (no side effects); posts a comment with | ||
| # the list of pending migrations. | ||
| # main behaviour → validate + apply migrations per-backend (best-effort | ||
| # sequential). Each backend (D1, PostgreSQL) is atomic within | ||
| # itself; cross-backend rollback is not supported — a failure in | ||
| # one backend does not roll back an already-applied backend. | ||
| # PR behaviour → validate + dry-run (no side effects); posts a comment with | ||
| # the list of pending migrations. | ||
| # main behaviour → apply migrations to Neon staging branch (DIRECT_DATABASE_URL_STAGING). | ||
| # release behaviour → apply migrations to Neon production branch (DIRECT_DATABASE_URL) — Option B1 promotion. | ||
| # | ||
| # Backend support: | ||
| # Cloudflare D1 — always attempted when CLOUDFLARE_API_TOKEN is set. | ||
| # PostgreSQL — attempted when DIRECT_DATABASE_URL is set (e.g. PlanetScale | ||
| # / Neon / self-hosted). Requires DIRECT_DATABASE_URL to | ||
| # match what prisma/prisma.config.ts expects. | ||
| # Cloudflare D1 — always attempted when CLOUDFLARE_API_TOKEN is set (push to main only). | ||
| # PostgreSQL — staging: on push to main (DIRECT_DATABASE_URL_STAGING). | ||
| # production: on GitHub Release published (DIRECT_DATABASE_URL). | ||
|
|
||
| on: | ||
| push: | ||
|
|
@@ -30,12 +27,21 @@ on: | |
| - 'admin-migrations/**' | ||
| - 'prisma/migrations/**' | ||
| - 'prisma/schema.prisma' | ||
| release: | ||
| # Option B1: GitHub Release is the promotion gate — publishing a release | ||
| # triggers Prisma migrations against the Neon production branch. | ||
| types: [published] | ||
| workflow_dispatch: | ||
| inputs: | ||
| dry_run: | ||
| description: 'Perform a dry-run only (validate, no writes)' | ||
| type: boolean | ||
| default: false | ||
| target: | ||
| description: 'Migration target (staging | production | both)' | ||
| type: choice | ||
| options: [staging, production, both] | ||
| default: staging | ||
|
|
||
| env: | ||
| DENO_VERSION: '2.x' | ||
|
|
@@ -122,7 +128,7 @@ jobs: | |
| # Expose secret presence as env vars; secrets context is not | ||
| # available in step-level if: conditions. | ||
| HAS_CLOUDFLARE_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN != '' }} | ||
| HAS_DIRECT_DB_URL: ${{ secrets.DIRECT_DATABASE_URL != '' }} | ||
| HAS_DIRECT_DB_URL: ${{ secrets.DIRECT_DATABASE_URL_STAGING != '' || secrets.DIRECT_DATABASE_URL != '' }} | ||
| steps: | ||
| - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
|
|
||
|
|
@@ -171,7 +177,7 @@ jobs: | |
| if: env.HAS_DIRECT_DB_URL == 'true' | ||
| env: | ||
| # prisma/prisma.config.ts prefers DIRECT_DATABASE_URL over DATABASE_URL | ||
| DIRECT_DATABASE_URL: ${{ secrets.DIRECT_DATABASE_URL }} | ||
| DIRECT_DATABASE_URL: ${{ secrets.DIRECT_DATABASE_URL_STAGING }} | ||
| run: | | ||
| echo "=== Pending PostgreSQL migrations ===" | ||
| # Use Prisma to report pending migrations without applying them | ||
|
|
@@ -287,20 +293,18 @@ jobs: | |
| }); | ||
| } | ||
|
|
||
| # ─── 3. Apply migrations (main push only) ──────────────────────────────── | ||
| # ─── 3. Apply D1 migrations (main push only) ───────────────────────────── | ||
| migrate: | ||
| name: Apply Migrations | ||
| name: Apply D1 Migrations | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| needs: [validate] | ||
| if: >- | ||
| (github.event_name == 'push' && github.ref == 'refs/heads/main') || | ||
| (github.event_name == 'workflow_dispatch' && inputs.dry_run == false) | ||
| github.event_name == 'push' && github.ref == 'refs/heads/main' | ||
| env: | ||
| # Expose secret presence as env vars; secrets context is not | ||
| # available in step-level if: conditions. | ||
| HAS_CLOUDFLARE_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN != '' }} | ||
| HAS_DIRECT_DB_URL: ${{ secrets.DIRECT_DATABASE_URL != '' }} | ||
| steps: | ||
| - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
|
|
||
|
|
@@ -380,43 +384,66 @@ jobs: | |
| fi | ||
| done | ||
|
|
||
| # ── PostgreSQL migrations ──────────────────────────────────────── | ||
| # Each individual Prisma migration runs in its own transaction; | ||
| # requires DIRECT_DATABASE_URL (see prisma/prisma.config.ts). | ||
| - name: PostgreSQL — apply migrations | ||
| id: pg_migrate | ||
| if: env.HAS_DIRECT_DB_URL == 'true' | ||
| env: | ||
| # prisma/prisma.config.ts prefers DIRECT_DATABASE_URL over DATABASE_URL | ||
| DIRECT_DATABASE_URL: ${{ secrets.DIRECT_DATABASE_URL }} | ||
| run: | | ||
| echo "Applying PostgreSQL migrations..." | ||
| if OUTPUT=$(deno run -A npm:prisma migrate deploy 2>&1); then | ||
| echo "$OUTPUT" | ||
| echo "✅ PostgreSQL migrations applied" | ||
| echo "result=success" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "$OUTPUT" | ||
| echo "❌ PostgreSQL migration failed" | ||
| echo "result=failed" >> "$GITHUB_OUTPUT" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # ── Summary ────────────────────────────────────────────────────── | ||
| - name: Migration summary | ||
| - name: D1 migration summary | ||
| if: always() | ||
| run: | | ||
| echo "=== Migration Summary ===" | ||
| echo "=== D1 Migration Summary ===" | ||
| D1_MAIN="${{ steps.d1_main.outputs.result }}" | ||
| D1_ADMIN="${{ steps.d1_admin.outputs.result }}" | ||
| PG="${{ steps.pg_migrate.outputs.result }}" | ||
|
|
||
| echo "D1 main DB : ${D1_MAIN:-skipped}" | ||
| echo "D1 admin DB : ${D1_ADMIN:-skipped}" | ||
| echo "PostgreSQL : ${PG:-skipped}" | ||
|
|
||
| if [ "${D1_MAIN}" = "failed" ] || [ "${D1_ADMIN}" = "failed" ] || [ "${PG}" = "failed" ]; then | ||
| echo "❌ One or more migrations failed" | ||
| if [ "${D1_MAIN}" = "failed" ] || [ "${D1_ADMIN}" = "failed" ]; then | ||
| echo "❌ One or more D1 migrations failed" | ||
| exit 1 | ||
| fi | ||
| echo "✅ All migrations completed successfully" | ||
| echo "✅ D1 migrations completed successfully" | ||
|
|
||
| # ─── 4. Apply PostgreSQL migrations → Staging ──────────────────────────── | ||
| pg_migrate_staging: | ||
| name: Apply Prisma Migrations → Staging | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 15 | ||
| needs: [validate] | ||
| if: >- | ||
| (github.event_name == 'push' && github.ref == 'refs/heads/main') || | ||
| (github.event_name == 'workflow_dispatch' && (inputs.target == 'staging' || inputs.target == 'both')) | ||
| env: | ||
| DIRECT_DATABASE_URL: ${{ secrets.DIRECT_DATABASE_URL_STAGING }} | ||
| steps: | ||
| - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| - name: Setup Deno environment | ||
| uses: ./.github/actions/setup-deno-env | ||
| with: | ||
| deno-version: ${{ env.DENO_VERSION }} | ||
| - name: Apply Prisma migrations to staging | ||
| if: env.DIRECT_DATABASE_URL != '' | ||
| run: deno run -A npm:prisma migrate deploy | ||
|
|
||
| # ─── 5. Apply PostgreSQL migrations → Production ───────────────────────── | ||
| pg_migrate_production: | ||
| name: Apply Prisma Migrations → Production | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 15 | ||
| needs: [validate] | ||
| if: >- | ||
| github.event_name == 'release' || | ||
| (github.event_name == 'workflow_dispatch' && (inputs.target == 'production' || inputs.target == 'both')) | ||
| env: | ||
|
Comment on lines
+431
to
+434
|
||
| DIRECT_DATABASE_URL: ${{ secrets.DIRECT_DATABASE_URL }} | ||
| steps: | ||
| - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| - name: Setup Deno environment | ||
| uses: ./.github/actions/setup-deno-env | ||
| with: | ||
| deno-version: ${{ env.DENO_VERSION }} | ||
| - name: Ensure DIRECT_DATABASE_URL is set for production migrations | ||
| run: | | ||
| if [ -z "${{ env.DIRECT_DATABASE_URL }}" ]; then | ||
| echo "ERROR: DIRECT_DATABASE_URL secret is not configured; cannot run production Prisma migrations." | ||
| exit 1 | ||
| fi | ||
| - name: Apply Prisma migrations to production | ||
| run: deno run -A npm:prisma migrate deploy | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,12 +14,14 @@ | |
| # Phase 0 of the Neon PostgreSQL migration. | ||
| # | ||
| # Required repository secrets: | ||
| # NEON_API_KEY – Neon API token (Settings → API Keys in the Neon console) | ||
| # NEON_PROJECT_ID – Neon project ID (e.g. twilight-river-73901472) | ||
| # NEON_DATABASE_URL – Connection string for the *production* branch. | ||
| # Format: postgresql://ROLE:PASSWORD@HOST/DATABASE?sslmode=require | ||
| # The database name and role are derived from this URL at | ||
| # runtime — do NOT hardcode them in this file. | ||
| # NEON_API_KEY – Neon API token (Settings → API Keys in the Neon console) | ||
| # NEON_PROJECT_ID – Neon project ID (e.g. twilight-river-73901472) | ||
| # NEON_DATABASE_URL – Connection string for the *staging* Neon branch. | ||
| # This corresponds to the branch that acts as the parent for new PR branches, | ||
| # but the parent branch itself is configured explicitly in this workflow (e.g. `parent: staging`). | ||
| # Format: postgresql://ROLE:PASSWORD@HOST/DATABASE?sslmode=require | ||
| # The database name and role are derived from this URL at | ||
| # runtime — do NOT hardcode them in this file. | ||
| # | ||
| # NOTE — Why no Hyperdrive? | ||
| # Hyperdrive is a Cloudflare edge service. It is only accessible from code | ||
|
|
@@ -160,9 +162,9 @@ jobs: | |
| PYEOF | ||
|
|
||
| # ── 3. Create the Neon branch ────────────────────────────────────────── | ||
| # The action creates a new branch off `production` in the configured Neon | ||
| # The action creates a new branch off `staging` in the configured Neon | ||
| # project. The branch is a zero-cost, copy-on-write clone — it shares | ||
| # storage with production until writes diverge. | ||
| # storage with staging until writes diverge. | ||
| # If the branch already exists (re-run or force-push), the action returns | ||
| # the existing branch ID without error. | ||
| - name: Create Neon database branch | ||
|
|
@@ -171,9 +173,9 @@ jobs: | |
| with: | ||
| project_id: ${{ secrets.NEON_PROJECT_ID }} | ||
| branch_name: ${{ steps.context.outputs.branch_name }} | ||
| # Fork from the Default (production) branch so PR branches have | ||
| # production-parity schema and data. | ||
| parent: production | ||
| # Fork from the staging branch so PR branches have staging-parity | ||
| # schema and data (staging is the parent of all PR branches). | ||
| parent: staging | ||
| api_key: ${{ secrets.NEON_API_KEY }} | ||
| # Database name and role are derived from NEON_DATABASE_URL above. | ||
| username: ${{ steps.db-config.outputs.db_role }} | ||
|
|
@@ -222,7 +224,7 @@ jobs: | |
| # pending migrations to apply" — that is expected on every fresh branch | ||
| # and should NOT trigger a restore. | ||
| if echo "$STATUS" | grep -q "P3009"; then | ||
| echo "⚠️ P3009 detected — restoring branch to production tip for a clean slate." | ||
| echo "⚠️ P3009 detected — restoring branch to staging tip for a clean slate." | ||
|
|
||
| if [ -z "$BRANCH_ID" ]; then | ||
| echo "::error::BRANCH_ID is empty. Cannot restore branch. Check create-branch step outputs." | ||
|
Comment on lines
226
to
230
|
||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,88 @@ | ||||||||
| # Neon Database Branching Strategy | ||||||||
|
|
||||||||
| ## Branch Hierarchy | ||||||||
|
|
||||||||
| ```mermaid | ||||||||
| graph TD | ||||||||
| production["production<br/>(live Worker HYPERDRIVE target;<br/>migrations applied on GitHub Release)"] | ||||||||
| staging["staging<br/>(GitHub main branch;<br/>migrations applied on every merge to main)"] | ||||||||
| pr["pr-NNNN<br/>(per-PR isolation;<br/>forked from staging;<br/>deleted on PR close)"] | ||||||||
| dev["dev/<name><br/>(personal dev branches;<br/>forked from staging)"] | ||||||||
|
|
||||||||
| production --> staging | ||||||||
| staging --> pr | ||||||||
| staging --> dev | ||||||||
| ``` | ||||||||
|
|
||||||||
| ## Branch Descriptions | ||||||||
|
|
||||||||
| | Branch | Purpose | Migrations Applied By | | ||||||||
| |---|---|---| | ||||||||
| | `production` | Live Neon branch — Cloudflare Worker reads through Hyperdrive | GitHub Release (`on: release: types: [published]`) | | ||||||||
| | `staging` | Tracks `main`; source-of-truth for PR branches | Every merge to `main` (`on: push: branches: [main]`) | | ||||||||
| | `pr-NNNN` | Per-PR isolation; forked from `staging` at PR open | `neon-branch-create.yml` via Prisma migrate deploy | | ||||||||
| | `dev/*` | Personal dev branches; forked from `staging` | Manual | | ||||||||
|
|
||||||||
| ## Workflow Triggers | ||||||||
|
|
||||||||
| ### Push to `main` (`db-migrate.yml`) | ||||||||
|
|
||||||||
| On every merge to `main`: | ||||||||
|
|
||||||||
| 1. D1 edge-cache migrations applied (`migrate` job — Cloudflare D1 only) | ||||||||
| 2. Prisma migrations applied to **staging** Neon branch (`pg_migrate_staging` job) | ||||||||
|
|
||||||||
| ### GitHub Release published (`db-migrate.yml`) | ||||||||
|
|
||||||||
| On every published GitHub Release (tag `v*`): | ||||||||
|
|
||||||||
| 1. Prisma migrations applied to **production** Neon branch (`pg_migrate_production` job) | ||||||||
|
|
||||||||
| This is the **Option B1** promotion gate: staging migrations are promoted to production only when a release is cut. | ||||||||
|
|
||||||||
| ### Pull Request opened/updated (`neon-branch-create.yml`) | ||||||||
|
|
||||||||
| On every PR that touches `prisma/**`, schema, or DB middleware: | ||||||||
|
|
||||||||
| 1. A new Neon branch `pr-NNNN` is forked from **staging** (not production) | ||||||||
| 2. All pending Prisma migrations are applied to the new branch | ||||||||
| 3. A PR comment is posted with the connection string | ||||||||
|
|
||||||||
| ## Secrets Required | ||||||||
|
|
||||||||
| | Secret | Description | | ||||||||
| |---|---| | ||||||||
| | `NEON_API_KEY` | Neon API token (Settings → API Keys in the Neon console) | | ||||||||
| | `NEON_PROJECT_ID` | Neon project ID (e.g. `twilight-river-73901472`) | | ||||||||
| | `NEON_DATABASE_URL` | Connection string for the **staging** branch; used by `neon-branch-create.yml` to resolve the DB name/role and keep the DB+role pairing consistent for PR branches (the branch parent is configured separately as `staging`; not used for migrations directly) | | ||||||||
| | `DIRECT_DATABASE_URL_STAGING` | Direct connection string for the **staging** branch; used by Prisma `migrate deploy` on push to `main` | | ||||||||
| | `DIRECT_DATABASE_URL` | Direct connection string for the **production** branch; used by Prisma `migrate deploy` on GitHub Release | | ||||||||
|
|
||||||||
| ## Promotion Flow (Option B1) | ||||||||
|
|
||||||||
| ```mermaid | ||||||||
| sequenceDiagram | ||||||||
| participant Dev as Developer | ||||||||
| participant PR as Pull Request | ||||||||
| participant Main as main branch | ||||||||
| participant Staging as Neon staging | ||||||||
| participant PRDB as Neon pr-NNNN branch | ||||||||
| participant Prod as Neon production | ||||||||
| participant Worker as Cloudflare Worker | ||||||||
|
|
||||||||
| Dev->>PR: Open PR (forks pr-NNNN from staging) | ||||||||
| PR->>PRDB: Prisma migrations applied to pr-NNNN branch | ||||||||
| Dev->>Main: Merge PR to main | ||||||||
| Main->>Staging: Prisma migrations applied to staging | ||||||||
| Main->>Worker: D1 edge-cache migrations applied | ||||||||
| Dev->>Main: Publish GitHub Release (v*) | ||||||||
| Main->>Prod: Prisma migrations promoted to production | ||||||||
| Worker->>Prod: Reads live data via Hyperdrive | ||||||||
| ``` | ||||||||
|
|
||||||||
| ## Local Development | ||||||||
|
|
||||||||
| For local development, use a personal `dev/<name>` Neon branch forked from `staging`. | ||||||||
| Set `DIRECT_DATABASE_URL` in your `.dev.vars` to point at your dev branch. | ||||||||
|
jaypatrick marked this conversation as resolved.
|
||||||||
| Set `DIRECT_DATABASE_URL` in your `.dev.vars` to point at your dev branch. | |
| In your `.dev.vars`, set `CLOUDFLARE_HYPERDRIVE_LOCAL_CONNECTION_STRING_HYPERDRIVE` to the connection string for your dev branch (used by the local Worker/Hyperdrive). | |
| In your `.env.local`, set `DIRECT_DATABASE_URL` to the same dev-branch connection string for use by the Prisma CLI. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
workflow_dispatchwithinputs.dry_run: truewill still satisfy this job condition and can apply migrations to staging. Add aninputs.dry_run == falseguard (only for workflow_dispatch) so dry-run dispatches never runmigrate deployside effects.