Skip to content

feat: Rust state machine for investigation workflow (fn-17)#75

Draft
bordumb wants to merge 18 commits intomainfrom
fn-17
Draft

feat: Rust state machine for investigation workflow (fn-17)#75
bordumb wants to merge 18 commits intomainfrom
fn-17

Conversation

@bordumb
Copy link
Owner

@bordumb bordumb commented Jan 19, 2026

Summary

Implements a Rust-based investigation state machine with Python bindings, replacing Temporal workflow local variables with a deterministic, versioned state machine.

Key changes:

  • Rust core (core/): Event-sourced state machine with 8 phases, versioned JSON protocol (v1)
  • PyO3 bindings (core/bindings/python/): Panic-free FFI with custom exceptions
  • Python package (python-packages/investigator/): Runtime, security, envelope, and Temporal integration modules
  • Tests: 43 Rust tests + 74 Python integration tests + 7 E2E workflow tests

Architecture:

  • State machine runs inside Temporal activities (not workflow code) for side-effect isolation
  • brain_step activity: pure computation, no side effects
  • Signals/queries for HITL, continue_as_new at step threshold
  • Deny-by-default security with SQL pattern validation

Epic tasks completed: 16/16

Test plan

  • Rust tests: cd core && cargo test (43 tests)
  • Python integration tests: pytest python-packages/investigator/tests/ (74 tests)
  • E2E workflow tests: skippable when Temporal unavailable
  • Manual: just rust-dev then just test

🤖 Generated with Claude Code

bordumb and others added 15 commits January 19, 2026 01:50
- Add core/ Rust workspace with dataing_investigator crate
- Add PyO3 bindings crate with maturin configuration
- Set PROTOCOL_VERSION=1 for snapshot versioning
- Add clippy deny rules for panic-free code
- Include .flow/ epic and task tracking for fn-17

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Scope, CallKind, and CallMeta types to dataing_investigator crate:
- Scope: security context with user/tenant IDs and permissions
- CallKind: enum for LLM vs Tool calls
- CallMeta: metadata for pending external calls

Uses BTreeMap for deterministic serialization ordering and
serde(default) for forward compatibility.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add tagged enum types for state machine communication:
- Event: Start, CallResult, UserResponse, Cancel
- Intent: Idle, Call, RequestUser, Finish, Error

Uses serde tagged enum format for JSON wire protocol.
All variants tested with serialization roundtrip tests.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add state module with:
- Phase enum: Init, GatheringContext, GeneratingHypotheses,
  EvaluatingHypotheses, AwaitingUser, Synthesizing, Finished, Failed
- State struct: versioned snapshot with sequence/step counters
- generate_id(): unique ID generation with prefix
- BTreeMap for ordered evidence/call storage

Uses serde(default) on optional fields for forward compatibility.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement core state machine in machine.rs:
- Investigator::new(), restore(), snapshot() for lifecycle
- ingest() processes events and returns intents
- apply() handles Start, CallResult, UserResponse, Cancel events
- decide() emits appropriate Intent for each Phase

Key behaviors:
- Strict call_id validation: unexpected IDs trigger Failed phase
- Logical clock (step) incremented on each event
- Call metadata recorded in call_index for debugging
- Full workflow from Init -> GatheringContext -> GeneratingHypotheses
  -> EvaluatingHypotheses -> Synthesizing -> Finished

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add PyO3 bindings exposing the Rust state machine to Python:
- Investigator class with new(), restore(), snapshot(), ingest()
- current_phase() and current_step() helper methods
- JSON-based serialization for events and intents
- protocol_version() function

Maturin build verified with `maturin develop --uv`.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ions

Add complete panic-safe Python bindings:
- Custom exceptions: StateError, SerializationError, InvalidTransitionError
- catch_unwind at FFI boundary for panic safety
- Full docstrings for all methods
- is_terminal() helper method

Python API:
- Investigator(): new(), restore(), snapshot(), ingest()
- current_phase(), current_step(), is_terminal()
- Exception hierarchy for error handling

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Rust build commands to Justfile:
- rust-check: verify Rust toolchain
- rust-build: cargo build --release
- rust-dev: maturin develop --uv (install to venv)
- rust-test: cargo test
- rust-lint: cargo clippy
- rust-clean: clean target/

Update setup workflow:
- `just setup` now runs `just rust-dev` after uv sync
- `just clean` includes rust-clean

Note: dataing-investigator requires maturin build, cannot be a uv source.
Use `just rust-dev` after `uv sync` to install bindings.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create python-packages/investigator/ wrapping Rust bindings:
- pyproject.toml with hatchling build
- __init__.py re-exporting Investigator, exceptions, protocol_version
- Empty module files: envelope.py, security.py, runtime.py

Added to uv workspace and project dependencies.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Envelope module (fn-17.11):
- Envelope TypedDict for tracing context
- wrap/unwrap for JSON serialization
- create_trace() for new trace IDs
- create_child_envelope() for linked events

Security module (fn-17.12):
- SecurityViolation exception
- validate_tool_call() with deny-by-default
- Tool allowlist validation
- Table permission checks
- Forbidden SQL pattern detection (DROP, DELETE, etc.)
- create_scope() helper

Both modules exported from investigator package.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add runtime.py with:
- run_local() async function for executing investigations
- LocalInvestigator class for fine-grained control
- Security validation before every tool call
- Error handling for all intent types
- Snapshot/restore support
- Max steps limit to prevent infinite loops

Provides local testing without Temporal dependency.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- InvestigatorWorkflow uses Rust Investigator via brain_step activity
- Pure computation in activity, side effects in workflow
- Signal dedup via seen_signal_ids set
- continue_as_new at step threshold (100)
- HITL via user_response signal and get_status query
- Conditional temporal exports (requires temporalio optional)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- test_investigator.py: basics, events, serialization, errors, full cycle
- test_envelope.py: wrap/unwrap, trace IDs, child envelopes
- test_security.py: tool validation, forbidden SQL patterns, scopes
- test_runtime.py: LocalInvestigator and run_local function
- Added pytest and pytest-asyncio as dev dependencies
- 74 tests total, all passing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Full investigation lifecycle test
- Query status test
- Cancel signal test
- Deterministic replay test
- Brain step activity unit tests
- Signal deduplication test
- Tests skip if Temporal not available (SKIP_TEMPORAL_TESTS=1)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@vercel
Copy link

vercel bot commented Jan 19, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
dataing Ready Ready Preview, Comment Jan 19, 2026 8:29pm
dataing-app Ready Ready Preview, Comment Jan 19, 2026 8:29pm
dataing-docs Ready Ready Preview, Comment Jan 19, 2026 8:29pm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant