Short utterances in Latin-script languages transcribed as Cyrillic [Parakeet TDT v3]

[tajchert]

Summary

Parakeet TDT v3 transcribes short Polish (and likely other Latin-script Slavic) utterances in Cyrillic. For example, "Wpisz google kropka com" becomes "Впиш гугл к ком.". Longer sentences transcribe correctly. English is unaffected.

Root Cause

The v3 JointDecision CoreML model performs argmax internally and outputs a single token_id (Int32). The shared 8,192-token vocabulary contains 1,161 Cyrillic tokens (~14%) alongside 6,987 Latin tokens. For short audio, the model doesn't accumulate enough language context in the decoder LSTM, and phonetically similar Slavic languages cause it to pick Cyrillic tokens over Latin ones.

Because argmax happens inside the CoreML model, there is no point in the Swift pipeline where token probabilities are available for filtering or masking by language.

AsrManager.transcribe() ignores language

The public API:

public func transcribe(_ samples: [Float], source: AudioSource = .file) async throws -> ASRResult

has no language parameter. Callers (like TypeWhisper's Parakeet plugin) know the user's language but cannot pass it through.

Reproduction

1. Use AsrModels.downloadAndLoad(version: .v3) with AsrManager(config: .default)
2. Record a short (1-3 second) Polish phrase, e.g., "Wpisz google kropka com"
3. Transcribe — output will often be Cyrillic instead of Latin Polish
4. Try a longer Polish sentence (10+ seconds) — output will be correct Latin Polish

[tajchert]

I used https://github.com/TypeWhisper/typewhisper-mac and despite having a selected "Polish" language in the app settings, got output in Cyrillic for short sentences in Polish. I started digging and check it uses FluidAudio - but language parameter is never passed to it and the model speech detection pipeline is not aware of this selection, and there is not way to do so (or I don't see, but also it is not a simple one line fix).

[SeoFood]

We reproduced this downstream in TypeWhisper.

On our side, language is passed correctly all the way to the Parakeet plugin. The issue is after that: the current AsrManager.transcribe(...) API does not expose a usable language-constrained path for v3.

From what we can see, the v3 decoder path gets token_id, token_prob, and duration, but not full token logits. So adding a language parameter alone would not be enough to constrain decoding in practice.

We also verified this at runtime with the same short Polish clip using:

• language=pl
• language=ru
• no language hint

All three produced the same result, so the language hint currently has no observable effect.

Happy to help with a PR, but first wanted to ask: do you see the real fix here as requiring updated model exports / logits exposure, rather than just an API-level language parameter?

[Alex-Wengg]

@tajchert @SeoFood Could you share a Polish audio sample (WAV) that triggers Cyrillic confusions

[SeoFood]

I do not speak Polish either. For the downstream check, I used synthetic audio on macOS rather than a human recording.

Specifically, I generated a short Polish clip with the built-in Zosia voice and then converted it to WAV:

say -v Zosia -o /tmp/tw217-short.aiff 'Wpisz google kropka com'
afconvert -f WAVE -d LEI16 /tmp/tw217-short.aiff /tmp/tw217-short.wav

Then I transcribed the exact same file three ways with Parakeet active:

typewhisper transcribe /tmp/tw217-short.wav --language pl --json
typewhisper transcribe /tmp/tw217-short.wav --language ru --json
typewhisper transcribe /tmp/tw217-short.wav --json

All three produced the same result on our side:

Wpisz google.com

So this synthetic sample reproduces the core issue, the language hint has no observable effect.

What it does not reliably reproduce is the exact Cyrillic output from the original report. For that, I think we probably need a short real human Polish recording, not TTS.

If useful, I can still share the synthetic WAV I used, but I want to be clear that it demonstrates the ignored-language behavior more than the specific Cyrillic-confusion symptom.

[tajchert]

Hi, happy to help if possible, here it is: https://drive.google.com/file/d/1rMSD4l-Q_uk5YVt-K5E3aPb6UNa7sJvR/view?usp=drive_link

EDIT:
I recorded few more samples but failed to get one that transcribed to Cyrillic yet (yesterday it was 100% repeatable for me using same mic and phrase).
Drive folder

audio_sample_pl.m4a , output: Fpisch Google CrockerCon.
audio_sample_pl2.m4a , output: Speech google club dot com
audio_sample_pl3.m4a , output: Speech Google Club.com.
audio_sample_pl4.m4a , output: Fp, Google Krapka com.
audio_sample_pl5.m4a , output: Fp, Google Clark Com.
audio_sample_pl6.m4a , output: Fp, Google KrafkaCom.

Expected output: wpisz Google kropka com or wpisz Google.com

Added one more sample Cały spichlerz jest ze spiżu (file audio_sample_pl_complex.m4a ), got output: Tao is Piclash as the speech.

Model: Parakeet TDT v3, lanugage in Typewhisper: PL