TralaIndo-32K Tokenizer Inference 🇮🇩

Tokenizer demo for Tralalabs/TralaIndo-32K-Tokenizer-HF.

This Space encodes text into tokens and token IDs, then decodes it back. If the Hugging Face fast tokenizer files are not usable, it automatically falls back to the original SentencePiece model.