port over Python fallback fixes

[chenghao-mou]

Description

Fallback to VAD when there is an error from the barge in stream. Python PR: livekit/agents#5142

Changes Made

• Added LIVEKIT_DEV_MODE env so that bargein is disabled by default for self-hosted production env
• Fallback to VAD when barge in hits an unretryable error

Pre-Review Checklist

• Build passes: All builds (lint, typecheck, tests) pass locally
• AI-generated code reviewed: Removed unnecessary comments and ensured code quality
• Changes explained: All changes are properly documented and justified above
• Scope appropriate: All changes relate to the PR title, or explanations provided for why they're included
• Video demo: A small video demo showing changes works as expected and did not break any existing functionality using Agent Playground (if applicable)

Testing

• Automated tests added/updated (if applicable)
• All tests pass
• Make sure both restaurant_agent.ts and realtime_agent.ts work properly (for major changes)

Additional Notes

---

Note to reviewers: Please ensure the pre-review checklist is completed before starting your review.

[changeset-bot]

⚠️ No Changeset found

Latest commit: baeb74a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 5 additional findings.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 8 additional findings in Devin Review.

[devin-ai-integration]

🟡 Held user transcripts silently dropped when falling back from adaptive to VAD interruption

When disableInterruptionDetection() is called from fallbackToVadInterruption(), the transcriptBuffer in AudioRecognition is never flushed. STT events held during overlap speech (agent speaking + user speaking) are permanently lost.

The issue arises because disableInterruptionDetection() sets this.isInterruptionEnabled = false without first flushing the held transcripts. After this, all code paths that could flush or clear the buffer are blocked:

• flushHeldTranscripts() returns early at agents/src/voice/audio_recognition.ts:325 (!this.isInterruptionEnabled)
• shouldHoldSttEvent() returns false at agents/src/voice/audio_recognition.ts:400, so the START_OF_SPEECH branch at line 408-411 that clears the buffer is never reached
• onEndOfAgentSpeech() returns early at agents/src/voice/audio_recognition.ts:267 without flushing

This means if a user was speaking over the agent when the interruption detector timed out, those words are silently dropped during the transition to VAD-based interruption.

Prompt for agents

In agents/src/voice/audio_recognition.ts, the disableInterruptionDetection() method (lines 252-259) should flush the transcriptBuffer before setting isInterruptionEnabled = false. Before setting isInterruptionEnabled = false on line 253, add code to process all held events:

1. Save the current transcriptBuffer to a local variable
2. Clear transcriptBuffer and ignoreUserTranscriptUntil
3. Then set isInterruptionEnabled = false and proceed with the rest of the cleanup
4. After cleanup, iterate through the saved events and call this.onSTTEvent(ev) for each one (since isInterruptionEnabled is now false, they'll be processed normally without being re-held)

The key insight is that flushHeldTranscripts() cannot be used as-is because it checks isInterruptionEnabled. The events must be re-processed through onSTTEvent after disabling interruption so they flow through normal STT processing.

---

Was this helpful? React with 👍 or 👎 to provide feedback.