This repo contains a set of scripts and MRI patches to build a pure zero-dependency tokenizer that is based on MRI's grammar.
Compiled binary is attached to release called latest. Only x86-64 Linux and macOS are supported.
This repository was archived by the owner on Nov 5, 2024. It is now read-only.