Skip to content

Add S3 archive write-path foundation#43

Draft
pthmas wants to merge 1 commit intomainfrom
pierrick/s3-integration
Draft

Add S3 archive write-path foundation#43
pthmas wants to merge 1 commit intomainfrom
pierrick/s3-integration

Conversation

@pthmas
Copy link
Copy Markdown
Collaborator

@pthmas pthmas commented Mar 30, 2026

Summary

This PR adds the S3 archive write-path foundation for Atlas.

When archive is enabled, the indexer now writes a canonical per-block archive entry into PostgreSQL inside the same transaction as the normal indexed data, and a background uploader drains those entries to an S3-compatible object store as zstd-compressed JSON bundles.

This also adds archive state tracking, contiguous-head manifest generation, MinIO support for local debugging, and automated backend coverage for the archive flow.

Scope

Included in this PR:

  • clap/config support for archive runtime settings
  • archive_blocks and archive_state migrations
  • canonical block + receipts archive bundle generation
  • async S3 uploader with durable retry via PostgreSQL
  • contiguous archive manifest at {prefix}/v1/manifest.json
  • MinIO support in local docker-compose
  • automated unit + integration tests for archive success, retry, and manifest behavior

Explicitly not included in this PR:

  • atlas-server archive dump
  • atlas-server archive restore
  • any S3 read path
  • any cache changes
  • any API or frontend changes
  • DA or metadata archival
  • historical backfill of already-indexed blocks

This is intentionally part 1 only. A follow-up PR will add the operator-facing archive dump/restore commands on top of this foundation.

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 30, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f053967a-4dc0-4082-8065-bb33459a243d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch pierrick/s3-integration

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant