Draft
Conversation
Add IVS (Interactive Video Service) as an alternative transport to WebRTC. Users select transport via `transport: "ivs"` in connect options — both paths return the same RealTimeClient interface. New files: transport-manager.ts (shared interface), ivs-connection.ts, ivs-manager.ts. Updated client.ts for transport selection, methods.ts to accept the shared interface, and WebRTCManager to implement it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add transport dropdown (WebRTC/IVS) to the test page. When IVS is selected, the IVS Web Broadcast SDK is loaded from CDN. The chosen transport is passed to client.realtime.connect(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bouncer's IVS handler waits for `ivs_joined` as the first client message after sending `ivs_stage_ready`. The message pump (which handles set_image/prompt) only starts after `ivs_joined` is received. Sending set_image before the stage handshake caused the bouncer to read it instead of `ivs_joined`, rejecting with "Expected ivs_joined message". Fix: reorder IVS connection phases so stage setup completes first, then send initial image/prompt once the bouncer's message pump is running. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The IVS SDK's StageStrategy callbacks take (participant) not (stage, participant). This caused TypeError: Cannot read properties of undefined (reading 'isLocal'). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
IVS SDK v1.14.0 does not pass participant to shouldPublishParticipant or shouldSubscribeToParticipant. Use argument-free callbacks instead: - publish: always true (only called for local publish-eligible participants) - subscribe: always AUDIO_VIDEO (subscribe to all remote streams) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SDK uses pnpm — package-lock.json was generated by mistake. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add transport-aware subscribe flow so viewers can watch IVS sessions without consuming inference server resources (SFU handles fan-out). - Add optional transport field to subscribe token encoding - Add subscribeIVS path: fetches viewer token from bouncer, creates subscribe-only IVS stage - Export getIVSBroadcastClient and IVSBroadcastModule for reuse Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The realtime client's baseUrl is a WebSocket URL (wss://), but the IVS subscribe endpoint is an HTTP GET. Convert the protocol before calling fetch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The subscribe stage was subscribing to ALL non-local participants, including the client's own camera feed. Now uses client_publish_participant_id from bouncer's ivs_stage_ready message to skip the client's publish participant and only receive the server's processed output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use server_publish_participant_id from the subscribe-ivs response to only subscribe to the server's inference output stream, preventing the viewer from accidentally receiving the client's camera input. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add requestRTCStats to IVS type declarations, store remote/local stage streams in IVSConnection, and expose them via getter methods proxied through IVSManager. Streams are cleared on cleanup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move delta-tracking state and parse logic into a reusable StatsParser class so it can be shared between WebRTCStatsCollector and the upcoming IVSStatsCollector. No external API change. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Polls IVS stage streams via requestRTCStats() at 1s intervals and feeds merged RTCStatsReport into StatsParser, mirroring the WebRTCStatsCollector pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire IVSStatsCollector into client.ts alongside WebRTC stats collection. Both transports now get stats events, video stall detection, and telemetry reporting. Add transport tag to telemetry reports for backend filtering. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Required for IVSManager's public getRemoteStreams/getLocalStreams methods to be type-checkable by downstream consumers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new latency tracking approaches for real-time streams: - Composite RTT: stitches client/server STUN RTTs + pipeline latency - Pixel Marker: embeds seq number in output frame pixels for true E2E measurement Both work across WebRTC and IVS transports, gated by client connect options. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate CompositeLatencyTracker + PixelLatencyProbe setup/teardown into a single pluggable LatencyDiagnostics class, reducing ~35 lines of inline wiring in client.ts to a simple instantiate/wire/stop pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
IVS publish stage was only sending video tracks and WHIP publish was missing the audio output track, causing audio to be silently dropped — a regression from existing WebRTC behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… start Reject remoteStreamPromise when IVS subscribe stage disconnects during setup, preventing the connect() flow from hanging forever. Store and clear the latency diagnostics delayed-start timer on disconnect to avoid restarting diagnostics after cleanup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With high pipeline delays (17-22s, e.g. packet loss + high FPS), probes were TTL-expired before the stamped frame reached the client, causing the measurement to silently drop and report stale low values instead of the actual delay. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Client burns binary seq marker into input frame pixels via canvas processing → server reads and re-stamps → client reads output and computes true round-trip latency through the video path. Changes: - pixel-latency-stamper.ts: canvas-based input frame stamper that wraps camera MediaStreamTrack (draw camera → stamp marker → captureStream) - pixel-latency.ts: E2E mode (stamp input + read output), stats tracking (sent/received/lost/corrupted/outOfOrder), periodic e2e_latency_report - latency-diagnostics.ts: createStamper() wraps localStream, deferred probe creation to start() for stamper availability - client.ts: create stamper before manager.connect() to substitute the published stream when pixelMarker is enabled - types.ts: add E2ELatencyReportMessage, update OutgoingMessage unions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The canvas stamper was created but not started until 1s after manager.connect(). IVS received a 300x150 canvas track producing zero frames, causing input_video_fps: 0.0 on the server. - Initialize canvas dimensions from track settings in constructor - Start draw loop in createStamper() before returning the stream - Remove redundant stamper.start() from LatencyDiagnostics.start() Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…p bias) Add pixelLatencyEvent and pixelLatencyReport events that surface corrupted, lost, and out-of-order stamps — not just successful ones. - Add PixelLatencyEvent discriminated union (ok, ok_reordered, corrupted, lost) - Add PixelLatencyReport periodic aggregate with pending count - Refactor PixelLatencyProbe constructor to options object (fixes broken onStats wiring) - Emit events for all outcomes in readFrame() and cleanUpOldProbes() - Existing pixelLatency event unchanged for backwards compat Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Read nackCount from inbound-rtp stats and compute per-sample delta. NACK rate > 0 with packet loss = 0 means the proxy cache is recovering losses before they count as lost. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Send delta (not cumulative) lost/corrupted/outOfOrder in e2e_latency_report to prevent Datadog double-counting 2. Wrap seq counter at 16 bits to match stamp encoding, preventing false losses after ~36h sessions 3. Start stats collectors when latencyTracking is configured, not only when telemetryEnabled — fixes composite latency RTT=0 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@aws/ivs-web-broadcast doesn't exist on npm. The correct package is amazon-ivs-web-broadcast. Fixes CI frozen-lockfile failure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…idatePairs Report interval now fires in E2E mode even without sendMessage, so the onReport callback receives periodic updates for local display. Re-add selectedCandidatePairs to WebRTCStats connection type and parser for ICE candidate display in rt-tester. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ing enabled Server (PR #749) gates EstimatedE2ELatencyReporter behind the latency_diagnostics query param. Without this, the server never sends latency_report messages and composite latency stops working. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rve source FPS The canvas + rAF + captureStream() pipeline ran at requestAnimationFrame rate (~120fps in headless Chrome), overwhelming the server regardless of the source camera's actual frame rate. Replace with MediaStreamTrackProcessor/Generator (Chrome 94+): - 1-in-1-out frame processing preserves source FPS naturally - 99% of frames pass through unchanged (zero copy, no quality loss) - Only stamped frames (~every 2s) go through OffscreenCanvas - Canvas approach kept as fallback for non-Chrome environments Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows callers to pass arbitrary query parameters (e.g. enable_recording) to the bouncer/upstream WS URL without SDK changes for each new param. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Zod 4 requires both key and value schemas for z.record(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stamp the 24-pixel marker across 4 bottom rows instead of 1. On read, per-bit majority voting across rows that pass sync check recovers from codec corruption (tolerates up to 2 corrupted rows). Backward compatible with single-row stamps. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous conflict resolution lost PixelLatencyEvent, PixelLatencyReport, PixelLatencyStatus types, onEvent/onReport callbacks, and delta reporting. Restores upstream's full API while keeping our multi-row MARKER_ROWS majority voting and always-fire report interval improvements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…pant Align SDK with updated bouncer that creates 2 tokens (client + server) with PUBLISH+SUBSCRIBE instead of 4 single-capability tokens. The SDK now joins the IVS stage once as 1 participant with a merged pub+sub strategy, adds jitterBuffer minDelay:0 and disables simulcast. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove WebSocket message handler on stage-ready timeout to prevent listener leak. Call stage.leave() on all reject paths during stage setup to ensure event handlers are torn down on failure. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add `probeIntervalMs` option to `latencyTracking` connect options. When set to 0, every input frame is stamped (via new auto-stamp path in PixelLatencyStamper), giving continuous delivery rate instead of near-binary 2-second windows. Cleanup timer decoupled to fixed 5s interval to avoid per-stamp overhead in every-frame mode. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When stage.join() fails or the stage disconnects during setup, close the bouncer control WebSocket right away so the bouncer releases the upstream inference server slot. Previously the WS stayed open until the manager's retry cleanup, leaving the server at capacity for subsequent attempts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ions
Adds jitterBufferMinDelay option (number | string) that flows through
client.connect() → IVSManager → IVSConnection → subscribeConfiguration.
Accepts numeric values or IVS presets ("low", "medium", "high", "default").
Defaults to 0 (previous hardcoded behavior) when not specified.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…into temp-merge-ivs
…g fields Add inbound video fields (firCount, pliCount, framesReceived, inter-frame delay, jitter buffer target/min delay, decode/jitter/processing time) and outbound video fields (nackCount, firCount, pliCount, retransmission stats, target bitrate, packet send delay, encode time). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…r fields Align SDK with updated backend metrics_report payload containing pipeline stats, FPS, queue depths, event loop latency, and server-side WebRTC metrics (rtt, jitter, loss, bitrate). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…load Expose all server-reported metrics (pipeline latency, FPS, queue depths, event loop latency, WebRTC stats) via a new serverMetrics event so benchmark tooling can capture them alongside client-side stats. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Status
🚧 Work in progress — do not merge
🤖 Generated with Claude Code