Skip to content

fix: classify update-step tool misses as agent_skipped#628

Merged
chubes4 merged 4 commits intomainfrom
fix/update-step-tool-miss-agent-skipped-625
Mar 5, 2026
Merged

fix: classify update-step tool misses as agent_skipped#628
chubes4 merged 4 commits intomainfrom
fix/update-step-tool-miss-agent-skipped-625

Conversation

@chubes4
Copy link
Member

@chubes4 chubes4 commented Mar 5, 2026

Summary

  • suppress generic error logging in ToolResultFinder when callers explicitly expect optional tool misses
  • update UpdateStep to emit a structured packet when the required handler tool is not called
  • treat that packet as a step-level agent_skipped signal in ExecuteStepAbility and complete the job with agent_skipped - handler_tool_not_called
  • add unit tests for ToolResultFinder missing-handler logging behavior

Why

At scale, update-step tool misses were being classified as empty_data_packet_returned failures, creating noisy failure metrics. This change gives a semantically correct terminal state for model/tool-use misses without masking true step failures.

Testing

  • homeboy test data-machine --skip-lint --path=\"/var/lib/datamachine/workspace/data-machine\" -- --filter \"PipelineBatchSchedulerTest|ToolResultFinderTest\"

@github-actions
Copy link

github-actions bot commented Mar 5, 2026

Homeboy Results — data-machine

Tooling versions

  • Homeboy CLI: homeboy 0.56.0
  • Extension: wordpress from https://github.com/Extra-Chill/homeboy-extensions
  • Extension revision: unknown
  • Action: Extra-Chill/homeboy-action@v1

ℹ️ PR test scope resolved to full for compatibility with installed Homeboy CLI

lint (changed files only)

  • PHPCS: LINT SUMMARY: 0 errors, 4 warnings
  • Fixable: 3 | Files with issues: 2 of 3
  • PHPStan: PHPSTAN SUMMARY: 133 errors at level 5

Homeboy Action v1 — homeboy 0.56.0

@chubes4
Copy link
Member Author

chubes4 commented Mar 5, 2026

Updated per review feedback:\n\n- Removed mapping for missing required update handler tool calls\n- UpdateStep now emits in packet metadata\n- ExecuteStepAbility now uses packet when failing job (fallback remains )\n\nThis keeps reserved for explicit skip tool usage () and classifies tool-miss protocol errors as failures with a precise reason string in status.

@chubes4
Copy link
Member Author

chubes4 commented Mar 5, 2026

Correction (previous comment had shell interpolation artifacts).

Updated per review feedback:

  • Removed agent_skipped mapping for missing required update handler tool calls
  • UpdateStep now emits failure_reason=required_handler_tool_not_called in packet metadata
  • ExecuteStepAbility now uses packet failure_reason when failing job (fallback remains empty_data_packet_returned)

This keeps agent_skipped reserved for explicit skip tool usage (skip_item) and classifies tool-miss protocol errors as failures with a precise reason string in status.

@chubes4
Copy link
Member Author

chubes4 commented Mar 5, 2026

Superseding PR body summary (can’t edit body via gh due projectCards API error in this repo context):

Final semantics in this branch

  • skip_item intent remains agent_skipped - <reason>
  • missing required update handler tool call is now failed - required_handler_tool_not_called
  • fallback remains failed - empty_data_packet_returned when no explicit failure reason is present

Code changes for that

  • UpdateStep: emits packet metadata failure_reason=required_handler_tool_not_called when no required handler tool call is found
  • ExecuteStepAbility: consumes packet failure_reason in failure path (datamachine_fail_job reason)
  • removed prior step-level agent_skipped mapping for this path

@chubes4
Copy link
Member Author

chubes4 commented Mar 5, 2026

Added follow-up commit for multi-handler determinism in UpdateStep.

What changed

  • UpdateStep now supports optional required_handler_slugs in flow step config
  • Validation enforces required_handler_slugs is a subset of handler_slugs
  • For multi-handler steps without required_handler_slugs, UpdateStep logs a warning and falls back to first handler (backward-compatible)
  • Missing required calls now include rich metadata:
    • configured_handler_slugs
    • required_handler_slugs
    • missing_required_handlers
  • Failure reason remains required_handler_tool_not_called

Why

This removes ambiguous “first handler by accident” behavior for multi-handler steps and makes required tool-call contract explicit.

@chubes4
Copy link
Member Author

chubes4 commented Mar 5, 2026

Added test coverage for the contract discussion:

  • new UpdateStepTest asserts missing required handler call emits failure_reason=required_handler_tool_not_called
  • new UpdateStepTest asserts required_handler_slugs can target a non-first handler in multi-handler configs

This gives immediate CI guardrails for the semantics we agreed on, independent of Homeboy audit capabilities.

Also opened Homeboy enhancement issue for architecture/layer ownership checks:
Extra-Chill/homeboy#482

@chubes4 chubes4 force-pushed the fix/update-step-tool-miss-agent-skipped-625 branch from a6350e3 to b1e4df1 Compare March 5, 2026 14:55
@chubes4 chubes4 merged commit 8e32910 into main Mar 5, 2026
2 checks passed
@chubes4 chubes4 deleted the fix/update-step-tool-miss-agent-skipped-625 branch March 5, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant