Simon Willison releases ttok 0.4: the token counter gets --list-models

Simon Willison has released ttok 0.4, the first update to his command-line token counter in roughly two years. The release makes three changes: it fixes a warning from the Click library, updates CI, and adds a --list-models command that prints the list of available models. Under the hood it still uses tiktoken, OpenAI's open-source tokenization library. The release is small, but it brings back into working order one of the simplest tools for managing the context window and request budgets.

What changed in 0.4

The update is mostly technical. The Click warning appeared every time the tool ran and cluttered the output. It is now fixed. The author also tidied up CI, the automated build and test pipeline for the project. After two years of inactivity, this matters: the tool once again builds and tests in a current environment instead of depending on settings that went out of date long ago.

The only new feature in the release is the --list-models flag. It shows which models ttok can count tokens for. Previously, you had to check a model name against the documentation or the tiktoken code. Now you can get the list straight from the terminal.

How to run it

ttok works with uvx, which runs Python tools through uv in a temporary, isolated environment. You do not need to install the package beforehand. Counting tokens in any text looks like this:

``` cat file.txt | uvx ttok ```

The text comes in on standard input, and the output is the token count. This format fits well into Unix pipelines: you can pipe the output of another command, a log, a repository dump, or document search results into ttok.

The release at a glance

• Version: ttok 0.4, published on GitHub.
• Author: Simon Willison.
• Foundation: OpenAI's open-source tiktoken library.
• New command: --list-models, a list of available models.
• Fixes: the Click warning and the CI update.
• Run without installing: via uvx.
• Previous update: about two years ago.

Where it comes in handy

Tokens determine two practical things: how much text fits into a model's context window, and how much a request will cost. Estimating by eye from the number of characters or words often misses. This is especially noticeable with code, JSON, tables, and non-English text, where there are usually more tokens than you would expect. Counting locally with a real tokenizer gives an exact figure without calling an API.

The typical use cases are simple. Before sending a large file to a model, check whether it fits in the window. Estimate the cost of batch-processing a set of documents. Compare how much shorter a prompt got after cleanup. Build a length check into a script or a pre-commit hook so that an overly long context does not go out in a request.

The limitation follows from the tool's foundation. tiktoken describes OpenAI's tokenizers. For models from other developers, which have their own token vocabularies, the result will be approximate, and it is better to get exact figures from the provider. For a rough sense of scale, this is usually enough.

If you only need ttok occasionally, you do not have to install it: uvx and one line in the terminal are enough. If you use it regularly, the --list-models command saves you from hunting for the right model name in the documentation.