feat: TTD Hardening Sprint S1 — Gates and Evidence Integrity #283
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,333 @@ | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # © James Ross Ω FLYING•ROBOTS <https://github.com/flyingrobots> | ||
| name: det-gates | ||
|
|
||
| on: | ||
| pull_request: | ||
| push: | ||
| branches: [main] | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: det-gates-${{ github.head_ref || github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| classify-changes: | ||
| name: classify-changes | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
| outputs: | ||
| run_full: ${{ steps.classify.outputs.run_full }} | ||
| run_reduced: ${{ steps.classify.outputs.run_reduced }} | ||
| run_none: ${{ steps.classify.outputs.run_none }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Detect changed files | ||
| id: changed | ||
| env: | ||
| BASE_REF: ${{ github.base_ref }} | ||
| EVENT_NAME: ${{ github.event_name }} | ||
| run: | | ||
| if [ "$EVENT_NAME" = "pull_request" ]; then | ||
| git fetch origin "$BASE_REF" --depth=1 | ||
| git diff --name-only "origin/$BASE_REF...HEAD" > changed.txt | ||
| else | ||
| git diff --name-only HEAD~1..HEAD > changed.txt || true | ||
| fi | ||
|
Comment on lines
+36
to
+42
Contributor
There was a problem hiding this comment. Push-to-main path only inspects the last commit — multi-commit merges or force pushes can slip through unclassified. Line 33: If branch protection rules prevent force pushes to 🤖 Prompt for AI Agents
Owner
Author
There was a problem hiding this comment. Acknowledged — The
Contributor
There was a problem hiding this comment.
Would you like me to open a GitHub issue to track the multi-commit merge edge case for future hardening? That way you have a backlog item if branch-protection rules ever change or if you want defense-in-depth for accidental force-pushes. ✏️ Learnings added
🧠 Learnings used |
||
| if [ "$EVENT_NAME" = "push" ] && [ ! -s changed.txt ]; then | ||
| echo "Warning: empty changelist on push, treating as full run" >&2 | ||
| echo "det-policy.yaml" > changed.txt | ||
| fi | ||
| echo "Changed files:" | ||
| cat changed.txt || true | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - name: Convert policy to JSON | ||
| run: | | ||
| yq -o=json det-policy.yaml > det-policy.json | ||
|
|
||
| - name: Classify path impact from det-policy.yaml | ||
| id: classify | ||
| run: | | ||
| node ./scripts/classify_changes.cjs det-policy.json changed.txt >> $GITHUB_OUTPUT | ||
|
|
||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| determinism-linux: | ||
| name: G1 determinism (linux) | ||
| needs: classify-changes | ||
| if: needs.classify-changes.outputs.run_full == 'true' | ||
| runs-on: ubuntu-latest | ||
|
Comment on lines
+59
to
+63
Contributor
There was a problem hiding this comment. 🧹 Nitpick | 🔵 Trivial No Rust/cargo caching across any of the six jobs — every PR pays full compilation cost from scratch. Six jobs ( Exception: ⚡ Example — add rust-cache to test jobs - uses: actions/checkout@v4
- name: Setup Rust
uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2 # Add this after toolchain setup
- name: Run parity tests (linux)
run: ...Also applies to: 80-84, 113-117, 147-151, 167-171, 185-189 🤖 Prompt for AI Agents
Owner
Author
There was a problem hiding this comment. Acknowledged — Adding
Contributor
There was a problem hiding this comment. Rate Limit Exceeded
|
||
| timeout-minutes: 15 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Setup Rust | ||
| uses: dtolnay/rust-toolchain@stable | ||
| - name: Run parity tests (linux) | ||
| run: | | ||
| cargo test -p echo-scene-port test_float_parity_with_js -- --nocapture 2>&1 | tee det-linux.log | ||
| grep -q " 0 passed" det-linux.log && echo "FATAL: zero tests matched filter" && exit 1 || true | ||
| - name: Run DIND suite (linux) | ||
| run: | | ||
| node scripts/dind-run-suite.mjs --mode run | tee dind-linux.log | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - name: Create digest table | ||
| env: | ||
| COMMIT_SHA: ${{ github.sha }} | ||
| RUN_ID: ${{ github.run_id }} | ||
| run: | | ||
| mkdir -p artifacts | ||
| echo "target,commit,run_id,digest" > artifacts/digest-table.csv | ||
| echo "linux,${COMMIT_SHA},${RUN_ID},$(sha256sum dind-report.json | cut -d' ' -f1)" >> artifacts/digest-table.csv | ||
| - name: Upload artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: det-linux-artifacts | ||
| path: | | ||
| det-linux.log | ||
| dind-linux.log | ||
| dind-report.json | ||
| artifacts/digest-table.csv | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| determinism-macos: | ||
| name: G1 determinism (macos) | ||
| needs: classify-changes | ||
| if: needs.classify-changes.outputs.run_full == 'true' | ||
| runs-on: macos-latest | ||
| timeout-minutes: 15 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Setup Rust | ||
| uses: dtolnay/rust-toolchain@stable | ||
| - name: Run parity tests (macos) | ||
| run: | | ||
| cargo test -p echo-scene-port test_float_parity_with_js -- --nocapture 2>&1 | tee det-macos.log | ||
| grep -q " 0 passed" det-macos.log && echo "FATAL: zero tests matched filter" && exit 1 || true | ||
| - name: Run DIND suite (macos) | ||
| run: | | ||
| node scripts/dind-run-suite.mjs --mode run | tee dind-macos.log | ||
| - name: Create digest table | ||
| env: | ||
| COMMIT_SHA: ${{ github.sha }} | ||
| RUN_ID: ${{ github.run_id }} | ||
| run: | | ||
| mkdir -p artifacts | ||
| echo "target,commit,run_id,digest" > artifacts/digest-table.csv | ||
| echo "macos,${COMMIT_SHA},${RUN_ID},$(shasum -a 256 dind-report.json | cut -d' ' -f1)" >> artifacts/digest-table.csv | ||
| - name: Upload artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: det-macos-artifacts | ||
| path: | | ||
| det-macos.log | ||
| dind-macos.log | ||
| dind-report.json | ||
| artifacts/digest-table.csv | ||
|
|
||
| static-inspection: | ||
| name: DET-001 Static Inspection | ||
| needs: classify-changes | ||
| if: needs.classify-changes.outputs.run_full == 'true' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 10 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Install ripgrep | ||
| run: command -v rg >/dev/null || (sudo apt-get update && sudo apt-get install -y ripgrep) | ||
| - name: Compute DETERMINISM_PATHS from policy | ||
| id: det_paths | ||
| run: | | ||
| PATHS=$(yq -o=json det-policy.yaml | jq -r ' | ||
| .crates | to_entries[] | | ||
| select(.value.class == "DET_CRITICAL") | | ||
| .value.paths[]' | | ||
| grep '^crates/' | sed 's|/\*\*$||' | sort -u | tr '\n' ' ') | ||
| echo "paths=$PATHS" >> "$GITHUB_OUTPUT" | ||
| - name: Run determinism check | ||
| id: det_check | ||
| env: | ||
| DETERMINISM_PATHS: ${{ steps.det_paths.outputs.paths }} | ||
| run: | | ||
| ./scripts/ban-nondeterminism.sh | tee static-inspection.log | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - name: Create report | ||
| if: always() | ||
| env: | ||
| DET_OUTCOME: ${{ steps.det_check.outcome }} | ||
| run: | | ||
| if [ "$DET_OUTCOME" = "success" ]; then | ||
| echo '{"claim_id": "DET-001", "status": "PASSED"}' > static-inspection.json | ||
| else | ||
| echo '{"claim_id": "DET-001", "status": "FAILED"}' > static-inspection.json | ||
| fi | ||
|
Comment on lines
+156
to
+165
Contributor
There was a problem hiding this comment. Evidence integrity gap: The Attack scenario:
Net result: A failed DET-001 check produces a VERIFIED evidence claim. The entire evidence chain becomes untrustworthy. 🔥 Proposed fix — read and validate artifact contentModify + const checkStaticInspection = () => {
+ try {
+ const report = JSON.parse(fs.readFileSync(
+ path.join(gatheredArtifactsDir, 'static-inspection', 'static-inspection.json'), 'utf8'));
+ return report.status === 'PASSED';
+ } catch {
+ return false;
+ }
+ };
+
const claims = [
{
id: 'DET-001',
- status: checkArtifact('static-inspection') ? 'VERIFIED' : 'UNVERIFIED',
+ status: checkStaticInspection() ? 'VERIFIED' : 'UNVERIFIED',
evidence: { workflow, run_id: runId, commit_sha: commitSha, artifact_name: 'static-inspection' }
},Also applies to: 302-307 🤖 Prompt for AI Agents |
||
| - name: Upload inspection artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: static-inspection | ||
| path: | | ||
| static-inspection.log | ||
| static-inspection.json | ||
|
|
||
| decoder-security: | ||
| name: G2 decoder security tests | ||
| needs: classify-changes | ||
| if: needs.classify-changes.outputs.run_full == 'true' || needs.classify-changes.outputs.run_reduced == 'true' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 10 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Setup Rust | ||
| uses: dtolnay/rust-toolchain@stable | ||
| - name: Run codec tests | ||
| run: | | ||
| cargo test -p echo-scene-codec --lib cbor::tests -- --nocapture 2>&1 | tee sec-tests.log | ||
| grep -q " 0 passed" sec-tests.log && echo "FATAL: zero tests matched filter" && exit 1 || true | ||
| - name: Upload security artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: sec-artifacts | ||
| path: | | ||
| sec-tests.log | ||
| docs/determinism/sec-claim-map.json | ||
|
|
||
| perf-regression: | ||
| name: G3 perf regression (criterion) | ||
| needs: classify-changes | ||
| # Runs for ALL non-NONCRITICAL changes (both DET_CRITICAL and DET_IMPORTANT). | ||
| # G3 is staging-optional per RELEASE_POLICY.md but always executes here. | ||
| if: needs.classify-changes.outputs.run_none != 'true' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Setup Rust | ||
| uses: dtolnay/rust-toolchain@stable | ||
| - name: Run benchmarks | ||
| run: | | ||
| cargo bench -p warp-benches --bench materialization_hotpath -- --output-format bencher | tee perf.log | ||
| - name: Upload perf artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: perf-artifacts | ||
| path: perf.log | ||
|
Comment on lines
198
to
218
Contributor
There was a problem hiding this comment. G3 "perf regression" doesn't detect regressions — there is no baseline comparison. The job runs Criterion benchmarks, captures output, and uploads it. That's a "benchmarks compile and run" gate. The name says "perf regression" but there is no historical baseline, no threshold, no comparison, and no failure criterion tied to performance. A 10x regression in At minimum, either:
If option 1 for now, document that regression detection is a future enhancement. 🤖 Prompt for AI Agents
Owner
Author
There was a problem hiding this comment. Acknowledged — G3 currently gates on benchmark compilation and execution, not regression detection. True regression detection requires baseline storage (e.g., criterion's baseline comparison or GitHub Action benchmark caching). This is a known limitation and a future enhancement.
Contributor
There was a problem hiding this comment. Rate Limit Exceeded
|
||
|
|
||
| build-repro: | ||
| name: G4 build reproducibility (wasm) | ||
| needs: classify-changes | ||
| if: needs.classify-changes.outputs.run_full == 'true' || needs.classify-changes.outputs.run_reduced == 'true' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| steps: | ||
| - name: Setup Rust (Global) | ||
| uses: dtolnay/rust-toolchain@stable | ||
| with: | ||
| targets: wasm32-unknown-unknown | ||
| - name: Checkout Build 1 | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| path: build1 | ||
| - name: Build 1 | ||
| run: | | ||
| cd build1 | ||
| rustup target add wasm32-unknown-unknown | ||
| cargo build --release --target wasm32-unknown-unknown -p ttd-browser | ||
| sha256sum target/wasm32-unknown-unknown/release/ttd_browser.wasm > ../hash1.txt | ||
| cp target/wasm32-unknown-unknown/release/ttd_browser.wasm ../build1.wasm | ||
| - name: Checkout Build 2 | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| path: build2 | ||
| - name: Build 2 | ||
| run: | | ||
| cd build2 | ||
| rustup target add wasm32-unknown-unknown | ||
| cargo build --release --target wasm32-unknown-unknown -p ttd-browser | ||
| sha256sum target/wasm32-unknown-unknown/release/ttd_browser.wasm > ../hash2.txt | ||
| cp target/wasm32-unknown-unknown/release/ttd_browser.wasm ../build2.wasm | ||
|
Comment on lines
+235
to
+252
Contributor
There was a problem hiding this comment. 🧹 Nitpick | 🔵 Trivial Redundant Lines 202 and 213 both run 🧹 Proposed cleanup - name: Build 1
run: |
cd build1
- rustup target add wasm32-unknown-unknown
cargo build --release --target wasm32-unknown-unknown -p ttd-browser
sha256sum target/wasm32-unknown-unknown/release/ttd_browser.wasm > ../hash1.txt
cp target/wasm32-unknown-unknown/release/ttd_browser.wasm ../build1.wasm
- name: Checkout Build 2
uses: actions/checkout@v4
with:
path: build2
- name: Build 2
run: |
cd build2
- rustup target add wasm32-unknown-unknown
cargo build --release --target wasm32-unknown-unknown -p ttd-browser🤖 Prompt for AI Agents
Owner
Author
There was a problem hiding this comment. Acknowledged — The
Contributor
There was a problem hiding this comment. Rate Limit Exceeded
|
||
| - name: Compare hashes | ||
| run: | | ||
| diff hash1.txt hash2.txt || (echo "Reproducibility failure: Hashes differ!" && exit 1) | ||
| echo "Hashes match: $(cat hash1.txt)" | ||
| - name: Upload build artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: build-repro-artifacts | ||
| path: | | ||
| hash1.txt | ||
| hash2.txt | ||
| build1.wasm | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| build2.wasm | ||
|
|
||
| validate-evidence: | ||
| name: Evidence schema / claim policy | ||
| needs: | ||
| - classify-changes | ||
| - determinism-linux | ||
| - determinism-macos | ||
| - static-inspection | ||
| - decoder-security | ||
| - perf-regression | ||
| - build-repro | ||
| if: always() && needs.classify-changes.result == 'success' && needs.classify-changes.outputs.run_none != 'true' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 10 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Download all artifacts | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| path: gathered-artifacts | ||
| - name: Verify artifact presence | ||
| env: | ||
| RUN_FULL: ${{ needs.classify-changes.outputs.run_full }} | ||
| run: | | ||
| ls -R gathered-artifacts | ||
| # Always required (run on both full and reduced) | ||
| [ -d gathered-artifacts/sec-artifacts ] || (echo "Missing sec-artifacts" && exit 1) | ||
| [ -d gathered-artifacts/build-repro-artifacts ] || (echo "Missing build-repro-artifacts" && exit 1) | ||
| [ -d gathered-artifacts/perf-artifacts ] || (echo "Missing perf-artifacts" && exit 1) | ||
| # Only required when run_full (these jobs are skipped for run_reduced) | ||
| if [ "$RUN_FULL" = "true" ]; then | ||
| [ -d gathered-artifacts/det-linux-artifacts ] || (echo "Missing det-linux-artifacts" && exit 1) | ||
| [ -d gathered-artifacts/det-macos-artifacts ] || (echo "Missing det-macos-artifacts" && exit 1) | ||
| [ -d gathered-artifacts/static-inspection ] || (echo "Missing static-inspection" && exit 1) | ||
| fi | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
Comment on lines
268
to
301
Contributor
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: cd /tmp && find . -name "validate_claims.cjs" 2>/dev/null | head -5Repository: flyingrobots/echo Length of output: 43 🏁 Script executed: git ls-files | grep -E "(validate_claims|det-gates)" | head -20Repository: flyingrobots/echo Length of output: 120 🏁 Script executed: cat scripts/validate_claims.cjsRepository: flyingrobots/echo Length of output: 2442 Content validation for FAILED evidence is missing — validate_claims.cjs only checks VERIFIED claims. The The script at for (const claim of data.claims) {
if (claim.status === 'VERIFIED') {
// validate required pointers...
}
}It exclusively validates claims marked Add an explicit content check in the 🤖 Prompt for AI Agents
Owner
Author
There was a problem hiding this comment. By design —
Contributor
There was a problem hiding this comment. Rate Limit Exceeded
|
||
| - name: Generate evidence pack | ||
| run: | | ||
| node scripts/generate_evidence.cjs gathered-artifacts | ||
| - name: Validate evidence pointers | ||
| run: | | ||
| node scripts/validate_claims.cjs gathered-artifacts/evidence.json | ||
| - name: Cross-check claim IDs against CLAIM_MAP | ||
| run: | | ||
| EVIDENCE_IDS=$(jq -r '.claims[].id' gathered-artifacts/evidence.json | sort) | ||
| CLAIM_MAP_IDS=$(yq -o=json docs/determinism/CLAIM_MAP.yaml | jq -r '.claims | keys[]' | sort) | ||
| EXTRA=$(comm -23 <(echo "$EVIDENCE_IDS") <(echo "$CLAIM_MAP_IDS")) | ||
| MISSING=$(comm -13 <(echo "$EVIDENCE_IDS") <(echo "$CLAIM_MAP_IDS")) | ||
| if [ -n "$EXTRA" ]; then | ||
| echo "ERROR: Claims in evidence.json but not in CLAIM_MAP.yaml:" && echo "$EXTRA" && exit 1 | ||
| fi | ||
| if [ -n "$MISSING" ]; then | ||
| echo "ERROR: Claims in CLAIM_MAP.yaml but not in evidence.json:" && echo "$MISSING" && exit 1 | ||
| fi | ||
| echo "All claim IDs synchronized" | ||
| - name: Verify sec-claim-map test IDs exist | ||
| run: | | ||
| MISSING="" | ||
| for tid in $(jq -r '.mappings[].test_id' docs/determinism/sec-claim-map.json); do | ||
| fn_name="${tid##*::}" | ||
| if ! grep -rq "fn ${fn_name}" crates/echo-scene-codec/src/; then | ||
| MISSING="$MISSING $tid" | ||
| fi | ||
| done | ||
| if [ -n "$MISSING" ]; then | ||
| echo "ERROR: sec-claim-map.json references non-existent tests:$MISSING" && exit 1 | ||
| fi | ||
| echo "All sec-claim-map test IDs verified" | ||
Uh oh!
There was an error while loading. Please reload this page.