How it works

Tokenization efficiency describes how compactly a tokenizer represents the same text. For an identical input, fewer tokens use less of a model's context window and may reduce token-based input charges, but they do not by themselves prove better model quality or lower end-to-end cost.

Each model family can use a different vocabulary, normalization process, pre-tokenizer, and merge strategy. Training-corpus coverage, writing system, spacing, morphology, emoji, and mixed scripts can therefore change how text is divided.

No. TokEvals labels exact local tokenizer output and exact provider results separately from estimates. Some provider APIs expose only a structured-message count, so TokEvals can estimate the raw content count but cannot truthfully show unavailable token IDs or boundaries.

Local tokenizer rows run in the browser from pinned public artifacts. Rows labeled Provider API send the entered text through the TokEvals server to that provider so the provider can count or tokenize it.

No. The ranking compares token count for the current text only. Model quality, output length, latency, pricing, caching, energy use, and task success are separate measurements that should be evaluated independently.