Count Tokens in Python Before You Send

By the end of this guide you will have a small Python module that tells you exactly how large a request is before the request leaves your laptop — a plain string, a full chat message list, or a 40-page document you are about to paste into a prompt. It takes about fifteen minutes to set up and it costs nothing to run, because the measurement happens locally. Once it is in place you can refuse, trim or split anything that is too big, instead of finding out from an error message or a surprise line on your bill.

This is one guide in Managing AI API Costs and Tokens. Counting is the first habit in that series: every other cost technique — pricing a call, caching a reply, capping a month — depends on knowing the size of what you are about to send.

Prerequisites

You need Python 3.10 or newer and an activated virtual environment. If you have not made one yet, Create a Python Virtual Environment for AI walks through it. Check your version:

python --version

Install the counting library plus the OpenAI SDK, which you need only for the final step:

pip install "tiktoken>=0.7" "openai>=1.40" "python-dotenv>=1.0"
pip freeze > requirements.txt

tiktoken is OpenAI's own tokeniser — the same code that turns text into tokens on their side, published as a Python package you run yourself. Counting needs no API key. The last step does make one real call, so put your key in a .env file (a plain text file of NAME=value lines that never gets committed):

OPENAI_API_KEY=sk-your-key-here

Then add that file to .gitignore before you write another line:

echo ".env" >> .gitignore

A key in a public repository gets scraped and spent by strangers, often within the hour. One line of .gitignore prevents it.

Step 1 — Get the encoding that matches your model

A token is a chunk of text — usually a short word, part of a longer word, or a punctuation mark — and an encoding is the specific vocabulary that decides where those chunks begin and end. Different model families use different vocabularies, so the same sentence can come to a different number of tokens depending on which model you send it to. Getting the encoding right is the whole game; everything after this is arithmetic.

Never guess the encoding name. Ask tiktoken to look it up from the model name, and keep a fallback for model names your installed version has not heard of yet:

import tiktoken


def encoding_for(model: str):
    """Return the tokeniser tiktoken uses for a model, with a safe fallback."""
    try:
        return tiktoken.encoding_for_model(model)
    except KeyError:
        # Model not in this version's lookup table yet.
        return tiktoken.get_encoding("o200k_base")


for name in ("gpt-4o-mini", "gpt-4o", "text-embedding-3-small"):
    print(f"{name:24} -> {encoding_for(name).name}")

Run it and you will see the newer chat models resolve to o200k_base while the embedding model resolves to cl100k_base. That difference matters: counting an embedding input with a chat encoding gives you a number that is close but not correct.

The first call downloads the vocabulary file for that encoding and caches it on disk. Every call after that is offline and takes microseconds.

Step 2 — Count a plain string

With an encoding in hand, counting is two lines. encode turns text into a list of integers, one per token, and len counts them. Decoding a single id back to text shows you exactly where the tokeniser drew its boundaries, which is the fastest way to build intuition:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o-mini")

text = "Counting tokens before you send keeps the bill predictable."
ids = enc.encode(text)

print("token ids:", ids[:8])
print("pieces:   ", [enc.decode([i]) for i in ids[:8]])
print("tokens:   ", len(ids))
print("words:    ", len(text.split()))

Look at the pieces line. Most of them arrive as a leading space plus a whole word, because the vocabulary was built from real text and common words earned their own entry. Longer or rarer words split into two or three fragments.

Why a token is not a word

For ordinary English prose, a token works out at roughly three-quarters of a word on average. That ratio is not a rule the tokeniser follows; it is a by-product of how the vocabulary was built. The vocabulary was trained by repeatedly merging the most frequent pairs of characters in a large body of mostly English text, so the sequences that appeared constantly — the, ing, people — ended up as single entries, while everything rarer has to be assembled from smaller pieces.

That explains the two cases where the ratio falls apart.

Code and structured data tokenise poorly because they are dense in characters that rarely form frequent pairs: brackets, underscores, colons, repeated indentation. A variable name like customer_lifetime_value is one word to you and several tokens to the model. A JSON payload can easily cost twice what the same information costs as a sentence.

Text outside the Latin alphabet tokenises worse still. Characters that appeared rarely in training are stored as raw bytes, so a single character can consume two or three tokens on its own. The newer o200k_base vocabulary is noticeably better here than the older cl100k_base, but the gap with English never closes completely.

The practical consequence: never budget by word count. If your input is code, tables, or a language other than English, measure it.

Step 3 — Count a whole messages list

Real requests are not bare strings. You send a list of messages, each a dictionary with a role and content, and the API wraps every one of them in invisible formatting before the model sees it. Those wrappers cost tokens too. If you only count the content values you will under-report every request, and the gap grows with the number of messages in a conversation.

The convention for the chat models is three tokens of overhead per message, one extra if a message carries an optional name field, and three more at the end because the reply is primed with the assistant role. Save this as tokens.py — the next step imports from it:

"""tokens.py — count the tokens in a chat request without sending it."""
import tiktoken

TOKENS_PER_MESSAGE = 3   # each message is wrapped in role/content framing
TOKENS_PER_NAME = 1      # optional "name" field costs one more
TOKENS_FOR_REPLY = 3     # the reply is primed with the assistant role


def encoding_for(model: str):
    try:
        return tiktoken.encoding_for_model(model)
    except KeyError:
        return tiktoken.get_encoding("o200k_base")


def count_message_tokens(messages: list[dict], model: str = "gpt-4o-mini") -> int:
    """Return the input token count for a list of chat messages."""
    enc = encoding_for(model)
    total = 0
    for message in messages:
        total += TOKENS_PER_MESSAGE
        for key, value in message.items():
            total += len(enc.encode(value))
            if key == "name":
                total += TOKENS_PER_NAME
    return total + TOKENS_FOR_REPLY


if __name__ == "__main__":
    demo = [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Summarise these notes in three bullets."},
    ]
    naive = sum(len(encoding_for("gpt-4o-mini").encode(m["content"])) for m in demo)
    print("content only:", naive)
    print("with framing:", count_message_tokens(demo))

Run python tokens.py. The two numbers differ by around a dozen tokens on this tiny example — trivial on its own, but a chat that carries thirty turns of history pays that overhead thirty times, and it is exactly the sort of drift that makes a request bump into the model's limit when your own arithmetic said it would fit. If you have already hit that wall, Fix the Context-Length-Exceeded Error in Python explains the error itself; counting first is how you stop meeting it.

Note that the loop counts role as well as content, because the role string is real text in the request. Also note what it does not count: tool and function definitions, image inputs, and any extra parameters. For a text-only chat the number is close enough to budget against; the moment you attach tools, treat it as a floor rather than a total.

Step 4 — Guard a long input before you spend anything

Now put the count to work. The pattern is always the same three moves: build the messages, measure them, and only then decide. When the measurement is over your ceiling you either refuse loudly or trim deliberately — but you never send blind.

Pick a ceiling well below the model's real context limit. The limit covers your input and the reply, so leaving headroom is what stops a long answer from being cut off mid-sentence:

import os

import tiktoken
from dotenv import load_dotenv
from openai import OpenAI

from tokens import count_message_tokens

load_dotenv()  # reads .env, which is listed in .gitignore
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

MODEL = "gpt-4o-mini"
MAX_INPUT_TOKENS = 8000        # your ceiling, not the model's
enc = tiktoken.encoding_for_model(MODEL)


def build(document: str, question: str) -> list[dict]:
    return [
        {"role": "system", "content": "Answer only from the document."},
        {"role": "user", "content": f"{document}\n\nQuestion: {question}"},
    ]


def ask_about(document: str, question: str, allow_trim: bool = False) -> str:
    messages = build(document, question)
    used = count_message_tokens(messages, MODEL)

    if used > MAX_INPUT_TOKENS:
        if not allow_trim:
            raise ValueError(f"{used} tokens exceeds the {MAX_INPUT_TOKENS} limit")
        overflow = used - MAX_INPUT_TOKENS
        ids = enc.encode(document)
        document = enc.decode(ids[: max(0, len(ids) - overflow - 20)])
        messages = build(document, question)
        used = count_message_tokens(messages, MODEL)
        print(f"trimmed to {used} tokens")

    print(f"sending {used} input tokens")
    response = client.chat.completions.create(model=MODEL, messages=messages)
    print("billed input tokens:", response.usage.prompt_tokens)
    return response.choices[0].message.content


if __name__ == "__main__":
    notes = "The team agreed to ship on Friday. " * 400
    print(ask_about(notes, "What was decided?", allow_trim=True))

Two details are worth copying. The trim happens on token ids, not characters, so the result lands exactly on the budget instead of near it — slicing a string by character length is guesswork that can still overshoot. And the script prints response.usage.prompt_tokens next to your own figure, so you can see how close your local estimate was on a real call. For a text-only request like this one they should be within a handful of tokens.

Trimming is the blunt option and it throws information away. When the document genuinely matters, split it instead and process the parts — Split Long Text into Chunks for AI APIs shows how to cut on sentence boundaries so each piece still reads sensibly.

Encoding quick reference

Use this to sanity-check what encoding_for_model gives you. You should still call the lookup rather than hard-coding a name, because the mapping changes as models are added.

Anthropic's Claude models use a different tokeniser that tiktoken cannot reproduce. Counts from o200k_base are a rough sighting shot for Claude, nothing more; use the token-counting endpoint in Anthropic's own SDK when you need a real figure. The OpenAI vs Anthropic API for Beginners comparison covers the other differences between the two SDKs.

Troubleshooting

ModuleNotFoundError: No module named 'tiktoken' — the package is installed somewhere your interpreter is not looking, almost always because the virtual environment is not active. Activate it (source .venv/bin/activate, or .venv\Scripts\activate on Windows) and install again. Fix ModuleNotFoundError: No Module Named openai has the full diagnosis; it is the same fault with a different package name.

KeyError raised from encoding_for_model — you passed a model name the installed version of tiktoken does not know. Either upgrade the package (pip install -U tiktoken) or rely on the try/except KeyError fallback from step 1, which is why that wrapper exists.

ConnectionError on the very first encode call — tiktoken fetches the vocabulary file once and caches it. Behind a proxy or on an offline machine the fetch fails. Run the script once with internet access to warm the cache, or set the TIKTOKEN_CACHE_DIR environment variable to a folder you have pre-populated on another machine.

Your count is lower than usage.prompt_tokens — you are sending something the formula does not model, usually tool definitions or an image. Compare the two numbers on a text-only call to confirm your maths is right, then add a fixed allowance for the extras and keep your ceiling conservative.

When to use this vs. alternatives

• Counting locally vs. reading usage from the response. The usage object is authoritative but arrives after you have been billed. Use tiktoken to decide whether to send, and usage to reconcile what you actually spent — they answer different questions and belong in the same script.
• Counting locally vs. dividing characters by four. The divide-by-four shortcut needs no dependency and is fine for a rough log line. It quietly under-counts code and badly under-counts non-Latin text, so it is unsafe as the guard that decides whether a request goes out.
• Counting vs. capping max_tokens. Capping limits only the reply. A 60,000-token input still bills in full even if the answer is two words. Count the input; cap the output; do both.

Counting is the measurement layer everything else sits on. Once your script knows the size of a request, turn that number into money with Estimate OpenAI API Costs with Python, stop paying twice for identical prompts using Cache AI Responses in Python to Cut Costs, and put a hard ceiling on the month with Set a Monthly AI API Spending Limit. Back to Managing AI API Costs and Tokens.

Related guides

• Estimate OpenAI API Costs with Python — turn the token counts from this page into a currency figure before you run a batch.
• Fix the Context-Length-Exceeded Error in Python — what to do when a request was already too big and the API said so.
• Split Long Text into Chunks for AI APIs — divide an oversized document instead of trimming the end off it.
• Cache AI Responses in Python to Cut Costs — the cheapest tokens are the ones you never send twice.
• Understanding LLM APIs — how requests, responses and usage reporting fit together.