Skip to content

fix: use large penalty for byte fallback in SentencePiece Viterbi

f6da4ab
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
Merged

fix: byte fallback penalty in SentencePiece Viterbi tokenizer #4

fix: use large penalty for byte fallback in SentencePiece Viterbi
f6da4ab
Select commit
Loading
Failed to load commit list.

Annotations

1 warning
test
succeeded Mar 26, 2026 in 26s