Skip to content

Follow up recording telemetry cleanup and validation #247

@thymikee

Description

@thymikee

Summary

The gesture telemetry + overlay work is in good shape, but there are still a few follow-up improvements worth doing after the feature lands. These are not blockers for the current PR, but they would reduce maintenance cost and improve confidence in the recording pipeline.

Proposed follow-ups

  1. Extract shared interaction recording plumbing from src/daemon/handlers/interaction.ts

Today several branches (press direct/ref/selector, fill ref/selector, related interaction flows) repeat the same shape:

  • capture timing
  • dispatch platform interaction
  • build result payload / visualization payload
  • record session action
  • append gesture telemetry
  • return response

A focused helper for that flow would reduce duplication in a hot-path handler and make future gesture/telemetry changes less error-prone.

  1. Add end-to-end artifact validation for recordings on both platforms

The current unit coverage around record stop is strong, but it still mostly validates orchestrated mocks. We should add at least one integration-level check per platform that asserts:

  • the output video is playable
  • the telemetry sidecar is written
  • soft overlay failures still return the video artifact with overlayWarning

This should target the real record start -> interactions -> record stop path rather than just internal helpers.

  1. Tighten packaging/runtime guarantees for Swift recording helper scripts

The overlay/trim scripts now live under ios-runner/AgentDeviceRunner/RecordingScripts, which is the right source location. The remaining follow-up is to decide whether distribution should:

  • bundle them explicitly, or
  • enforce/publish a clear runtime requirement that those scripts must be present next to the repo source tree.

The current lazy resolution and error messages are good enough for development, but packaging expectations are still implicit.

Benefits

  • Lower maintenance cost in interaction.ts
  • Fewer regressions when adding or changing gesture telemetry fields
  • Better confidence that released recordings are actually usable artifacts, not just unit-tested code paths
  • Clearer distribution/runtime contract for the Swift post-processing helpers

Non-goals

This follow-up should not reopen the core architecture. The telemetry-first design, shared reference-frame helper, soft overlay failures, and current platform split are the intended direction.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions