Summary
The gesture telemetry + overlay work is in good shape, but there are still a few follow-up improvements worth doing after the feature lands. These are not blockers for the current PR, but they would reduce maintenance cost and improve confidence in the recording pipeline.
Proposed follow-ups
- Extract shared interaction recording plumbing from
src/daemon/handlers/interaction.ts
Today several branches (press direct/ref/selector, fill ref/selector, related interaction flows) repeat the same shape:
- capture timing
- dispatch platform interaction
- build result payload / visualization payload
- record session action
- append gesture telemetry
- return response
A focused helper for that flow would reduce duplication in a hot-path handler and make future gesture/telemetry changes less error-prone.
- Add end-to-end artifact validation for recordings on both platforms
The current unit coverage around record stop is strong, but it still mostly validates orchestrated mocks. We should add at least one integration-level check per platform that asserts:
- the output video is playable
- the telemetry sidecar is written
- soft overlay failures still return the video artifact with
overlayWarning
This should target the real record start -> interactions -> record stop path rather than just internal helpers.
- Tighten packaging/runtime guarantees for Swift recording helper scripts
The overlay/trim scripts now live under ios-runner/AgentDeviceRunner/RecordingScripts, which is the right source location. The remaining follow-up is to decide whether distribution should:
- bundle them explicitly, or
- enforce/publish a clear runtime requirement that those scripts must be present next to the repo source tree.
The current lazy resolution and error messages are good enough for development, but packaging expectations are still implicit.
Benefits
- Lower maintenance cost in
interaction.ts
- Fewer regressions when adding or changing gesture telemetry fields
- Better confidence that released recordings are actually usable artifacts, not just unit-tested code paths
- Clearer distribution/runtime contract for the Swift post-processing helpers
Non-goals
This follow-up should not reopen the core architecture. The telemetry-first design, shared reference-frame helper, soft overlay failures, and current platform split are the intended direction.
Summary
The gesture telemetry + overlay work is in good shape, but there are still a few follow-up improvements worth doing after the feature lands. These are not blockers for the current PR, but they would reduce maintenance cost and improve confidence in the recording pipeline.
Proposed follow-ups
src/daemon/handlers/interaction.tsToday several branches (
pressdirect/ref/selector,fillref/selector, related interaction flows) repeat the same shape:A focused helper for that flow would reduce duplication in a hot-path handler and make future gesture/telemetry changes less error-prone.
The current unit coverage around
record stopis strong, but it still mostly validates orchestrated mocks. We should add at least one integration-level check per platform that asserts:overlayWarningThis should target the real
record start -> interactions -> record stoppath rather than just internal helpers.The overlay/trim scripts now live under
ios-runner/AgentDeviceRunner/RecordingScripts, which is the right source location. The remaining follow-up is to decide whether distribution should:The current lazy resolution and error messages are good enough for development, but packaging expectations are still implicit.
Benefits
interaction.tsNon-goals
This follow-up should not reopen the core architecture. The telemetry-first design, shared reference-frame helper, soft overlay failures, and current platform split are the intended direction.