fix: implement Viterbi SentencePiece encoding (replaces greedy)#3

dndungu merged 1 commit intomainzerfoo/ztoken:mainfrom

fix/viterbi-sentencepiecezerfoo/ztoken:fix/viterbi-sentencepieceCopy head branch name to clipboard