Dev to Main — March 2026

.flake8

@@ -0,0 +1,3 @@
+[flake8]
+ignore = E501,W291,W293,D401,D400,E402,E302,D200,D202,D205,W503,E203,D204,D403
+extend-exclude = .venv,.uv-cache

.github/workflows/benchmark.yml

@@ -0,0 +1,66 @@
+name: Annotation Benchmark
+
+on:
+  workflow_dispatch:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  benchmark:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    container: python:3.11.9
+
+    services:
+      postgres:
+        image: postgres:15
+        env:
+          POSTGRES_PASSWORD: postgres
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+
+    env:
+      POSTGRES_PASSWORD: postgres
+      POSTGRES_USER: postgres
+      POSTGRES_DB: postgres
+      POSTGRES_HOST: postgres
+      POSTGRES_PORT: 5432
+      GOOGLE_API_KEY: TEST
+      GOOGLE_CSE_ID: TEST
+      PROFILE_DIR: profiles
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Install uv and set the python version
+        uses: astral-sh/setup-uv@v5
+
+      - name: Install the project
+        run: uv sync --locked --group dev
+
+      - name: Create profiles directory
+        run: mkdir -p profiles
+
+      - name: Run benchmark tests
+        run: |
+          uv run pytest tests/automated/integration/benchmark \
+            -m "manual and benchmark" \
+            --benchmark-json=benchmark-results.json \
+            -v
+
+      - name: Post benchmark summary
+        run: uv run python scripts/post_benchmark_summary.py
+
+      - name: Upload benchmark results
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark-results-${{ github.sha }}
+          path: |
+            benchmark-results.json
+            profiles/
+          retention-days: 90

.github/workflows/python_checks.yml

@@ -18,5 +18,5 @@ jobs:
         uses: reviewdog/action-flake8@v3
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
-          flake8_args: --ignore E501,W291,W293,D401,D400,E402,E302,D200,D202,D205,W503,E203,D204,D403
+          flake8_args: .
           level: warning

.gitignore

@@ -10,3 +10,5 @@ runs/
 model.safetensors
 training_args.bin
 /temp/
+.devcontainer/.env.secrets
+.devcontainer/

.pre-commit-config.yaml

@@ -0,0 +1,11 @@
+repos:
+  - repo: https://github.com/pycqa/flake8
+    rev: 7.3.0
+    hooks:
+      - id: flake8
+        additional_dependencies:
+          - flake8-docstrings
+          - flake8-simplify
+          - flake8-unused-arguments
+          - flake8-annotations
+

CLAUDE.md

@@ -113,6 +113,10 @@ See `ENV.md` for the full reference. Key categories:
 
 ## Important Notes
 
+- Follow the repository's linting style guide when writing or editing Python:
+  - Run `flake8` with this exact ignore set in mind: `E501,W291,W293,D401,D400,E402,E302,D200,D202,D205,W503,E203,D204,D403`
+  - Keep import ordering and blank-line usage compliant with `flake8` defaults aside from the ignore list above
+  - Prefer concise docstrings where present and avoid introducing new docstring violations outside ignored codes
 - The app exposes its API docs at `/api` (not the default `/docs` — `/docs` redirects to `/api`).
 - CORS is configured for `localhost:8888`, `pdap.io`, and `pdap.dev`.
 - Two permission levels: `access_source_collector` (general) and `source_collector_final_review` (final review).

README.md

@@ -59,6 +59,16 @@ Thank you for your interest in contributing to this project! Please follow these
 
 ## Code Quality
 
-Docstrings and type hints are checked via a GitHub Action (`python_checks.yml`) using [pydocstyle](https://www.pydocstyle.org/en/stable/) and [mypy](https://mypy-lang.org/). These produce advisory PR comments and do *not* block merges.
+Linting runs via flake8 in GitHub Actions (`python_checks.yml`) and posts advisory review comments (non-blocking).
+
+To run the same lint checks before committing, install and use the `prek` pre-commit hook runner:
+
+```bash
+uv sync --group dev
+uv run prek install
+
+# run hooks on all files
+uv run prek run --all-files
+```
 
 Note: `python_checks.yml` only runs on pull requests from within the repo, not from forks.

alembic/versions/2026_02_15_1257-1fb2286a016c_add_internet_archive_to_batch_strategy_.py

@@ -0,0 +1,60 @@
+"""add internet_archive to batch_strategy enum
+
+Revision ID: 1fb2286a016c
+Revises: 1d3398f9cd8a
+Create Date: 2026-02-15 12:57:34.550327
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+
+from src.util.alembic_helpers import switch_enum_type
+
+# revision identifiers, used by Alembic.
+revision: str = '1fb2286a016c'
+down_revision: Union[str, None] = '759ce7d0772b'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    switch_enum_type(
+        table_name="batches",
+        column_name="strategy",
+        enum_name="batch_strategy",
+        new_enum_values=[
+            "example",
+            "ckan",
+            "muckrock_county_search",
+            "auto_googler",
+            "muckrock_all_search",
+            "muckrock_simple_search",
+            "common_crawler",
+            "manual",
+            "internet_archive",
+        ],
+    )
+
+
+def downgrade() -> None:
+    op.execute("""
+    DELETE FROM BATCHES
+        WHERE STRATEGY = 'internet_archive'
+    """)
+
+    switch_enum_type(
+        table_name="batches",
+        column_name="strategy",
+        enum_name="batch_strategy",
+        new_enum_values=[
+            "example",
+            "ckan",
+            "muckrock_county_search",
+            "auto_googler",
+            "muckrock_all_search",
+            "muckrock_simple_search",
+            "common_crawler",
+            "manual",
+        ],
+    )