Tokens are the hidden unit that controls everything about how LLMs read your prompts and generate responses — including what you pay. This lesson builds a complete, practical understanding of tokenization from first principles, with hands-on Python examples and real cost estimation techniques you can apply immediately.
You're building a chatbot for customer support. You've crafted a careful system prompt, written a detailed user message, and wired everything up to the OpenAI API. Then the bill arrives at the end of the month and it's three times what you expected. Or worse — your carefully written prompt gets truncated halfway through a critical instruction, and the model starts hallucinating because it never received the full context. Both of these problems trace back to the same root cause: you didn't have a clear mental model of how LLMs actually read and process text.
Tokens are the fundamental unit of how every large language model (LLM) — whether it's GPT-4, Claude, Gemini, or an open-source model like Llama — perceives, processes, and generates language. They're not words. They're not characters. They're something in between, and the distinction matters enormously once you start building real applications. Understanding tokens means understanding why "ChatGPT" costs differently than "chat gpt", why certain languages eat through your context window faster than others, and why a response that looks short to you might actually be expensive.
By the end of this lesson, you'll have a working mental model of tokenization that you can immediately apply to your prompt engineering, cost estimation, and system design decisions.
What you'll learn:
No prior machine learning knowledge is required. You should be comfortable reading simple Python code and have a basic idea of what an LLM is — a model that takes text in and generates text out. That's all you need to start.
Think about how you read a sentence. You don't process it letter by letter, and you don't wait until you've read the entire paragraph before understanding begins. You chunk language into meaningful units — words, prefixes, suffixes — almost automatically.
LLMs do something similar, but more systematic. Before any text reaches the model's neural network, it goes through a tokenizer: a preprocessing step that breaks raw text into smaller chunks called tokens. Each token is then converted into a number (its token ID), and that sequence of numbers is what the model actually works with. The model never sees the word "customer" — it sees something like [43534].
A token is typically somewhere between a single character and a short word. In practice:
"the" is usually 1 token"uncharacteristically" might be 4–5 tokens" the" and "the" can be different tokens)The most important thing to internalize right now: tokens are not words. Treating them as equivalent is one of the most common and costly mistakes developers make.
Modern LLMs almost universally use a technique called Byte-Pair Encoding (BPE) — or a close variant of it — to build their tokenizer. Let's walk through how it works from first principles.
Imagine you're building a vocabulary for a tokenizer. You start with a huge corpus of text — billions of web pages, books, and code. Initially, your vocabulary consists of every individual character that appears: a, b, c, ... z, digits, punctuation, unicode characters, spaces, and so on.
Then you run a compression algorithm:
e and r appearing together as er)erEach merge creates a new token that represents a chunk of text that appears together very frequently. After 50,000 or so merges, you have a vocabulary of roughly 50,000–100,000 tokens. Common words like " the", " and", " is" end up as single tokens because they appear together constantly. Rare words get broken into their component sub-word pieces.
This is why common words cost less (in tokens) than rare words. And it's why this approach is brilliant: you never hit an "unknown word" problem. Any text — even words the model has never seen — can be encoded as a sequence of sub-word tokens.
Key insight
The tokenizer is trained separately from the model itself, and it's fixed at training time. When you use GPT-4, you're using a specific, frozen tokenizer. Different model families (GPT, Claude, Gemini) use different tokenizers, so the same text may tokenize differently across providers.
The best way to build intuition is to watch tokenization happen. OpenAI provides a free tool called the Tokenizer at platform.openai.com/tokenizer. Navigate there and you'll see a text box where you can paste any text and watch it get highlighted in alternating colors — each color represents a single token.
But let's also do this programmatically, because you'll need this skill when building applications. The tiktoken library is OpenAI's open-source tokenizer:
pip install tiktoken
import tiktoken
# Load the tokenizer used by GPT-4 and GPT-3.5-turbo
enc = tiktoken.get_encoding("cl100k_base")
text = "The quarterly revenue report shows a 12.4% increase in EMEA markets."
tokens = enc.encode(text)
print(f"Token IDs: {tokens}")
print(f"Token count: {len(tokens)}")
# Decode each token individually to see the chunks
for token_id in tokens:
chunk = enc.decode([token_id])
print(repr(chunk))
Running this gives you output like:
Token IDs: [791, 38177, 13254, 3307, 5039, 264, 220, 717, 13, 19, 4, 5376, 304, 36085, 16717, 13]
Token count: 16
'The'
' quarterly'
' revenue'
' report'
' shows'
' a'
' '
'12'
'.'
'4'
'%'
' increase'
' in'
' EM'
'EA'
' markets'
'.'
Notice a few things immediately:
" quarterly" is one token (space included), but "EMEA" splits into "EM" and "EA" — because EMEA is less common in the training data than quarterly12.4 becomes four tokens: 12, ., 4, %Now that you have the basics, let's look at where things get unintuitive. These are exactly the scenarios that cause unexpected behavior and cost overruns in production.
Numbers tokenize poorly. Each digit often becomes its own token:
text_a = "The transaction ID is 8472916304857."
text_b = "The transaction ID is eight billion four hundred seventy-two million."
tokens_a = enc.encode(text_a)
tokens_b = enc.encode(text_b)
print(f"Numeric version: {len(tokens_a)} tokens")
print(f"Written-out version: {len(tokens_b)} tokens")
You might expect the numeric version to be shorter. Often it's comparable or longer, because each digit in a long number gets its own token. This matters if you're feeding large datasets of numeric IDs, financial data, or timestamps into a prompt.
This is one of the most impactful and least-discussed aspects of tokenization. Because BPE tokenizers are trained predominantly on English text, English words have very efficient representations. Other languages — particularly those with non-Latin scripts — tokenize much less efficiently:
english = "The customer submitted a support request."
spanish = "El cliente envió una solicitud de soporte."
japanese = "顧客がサポートリクエストを送信しました。"
arabic = "أرسل العميل طلب دعم."
for label, text in [("English", english), ("Spanish", spanish),
("Japanese", japanese), ("Arabic", arabic)]:
count = len(enc.encode(text))
print(f"{label}: {count} tokens — '{text}'")
You'll typically see English using the fewest tokens, Spanish moderately more, and Japanese or Arabic potentially using 3–5x as many tokens per equivalent meaning. If you're building a multilingual product, this directly affects both your costs and your effective context window capacity.
Code tokenizes relatively well because programming keywords, operators, and common patterns appear frequently in training data. But variable names, string literals, and comments vary widely:
code = """
def calculate_churn_rate(active_users, churned_users):
if active_users == 0:
return 0.0
return round((churned_users / active_users) * 100, 2)
"""
print(f"Code snippet: {len(enc.encode(code))} tokens")
Warning
Long, descriptive variable names and verbose comments add tokens. In extremely token-constrained situations (like few-shot examples in a tight context window), compressed variable names save space — though readability suffers. It's a real trade-off.
Structured data, JSON, and heavily indented code cost more than you'd think:
json_compact = '{"user_id":1042,"status":"active","plan":"enterprise"}'
json_pretty = '''{
"user_id": 1042,
"status": "active",
"plan": "enterprise"
}'''
print(f"Compact JSON: {len(enc.encode(json_compact))} tokens")
print(f"Pretty JSON: {len(enc.encode(json_pretty))} tokens")
The pretty-printed version typically uses 20–40% more tokens just from whitespace. When you're feeding large JSON payloads into a prompt — API responses, database records, configuration — always strip unnecessary whitespace first.
Every LLM has a context window: the maximum number of tokens it can "see" at once, including both input and output. Think of it like working memory. The model can only reason about what fits inside the window. Everything outside it doesn't exist.
Common context window sizes as of 2024:
Here's the critical thing people miss: the context window is shared between your input and your output. When you call an API, you set a parameter called max_tokens (or max_output_tokens), which is the maximum length of the response. That response budget comes out of the same pool as your input.
So if you have a 16,000-token context window, and your system prompt + user message is 12,000 tokens, you only have 4,000 tokens left for the model's response. If you set max_tokens=8000 hoping for a long answer, the model will be cut off — or depending on the API, you'll get an error.
Here's how to track this in practice:
import tiktoken
def count_tokens_for_chat(messages: list[dict], model: str = "gpt-4o") -> int:
"""
Estimate token count for a list of messages in OpenAI chat format.
Includes overhead tokens for message formatting.
"""
enc = tiktoken.encoding_for_model(model)
total = 0
for message in messages:
# Each message has ~4 tokens of overhead for role/formatting
total += 4
for key, value in message.items():
total += len(enc.encode(str(value)))
total += 2 # reply priming tokens
return total
messages = [
{"role": "system", "content": "You are a data analyst. Answer concisely with specific numbers."},
{"role": "user", "content": "Summarize the key trends in our Q3 sales data for the APAC region, focusing on month-over-month changes."}
]
estimated_tokens = count_tokens_for_chat(messages)
print(f"Estimated input tokens: {estimated_tokens}")
print(f"Tokens remaining for output (in 16k window): {16385 - estimated_tokens}")
Tip
Always count your input tokens before sending an API request in production code. Running into a context length error mid-pipeline is expensive and disruptive. Build a check that raises a clear warning if you're within, say, 10% of the limit.
LLM APIs charge by the token — almost universally. But the pricing is not symmetric: input tokens and output tokens have different prices, and output tokens are typically more expensive.
Why? Because generating each output token requires a full forward pass through the model. Reading input tokens is processed more efficiently in parallel.
A typical pricing structure (approximate, always check current rates):
| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|---|---|---|
| GPT-4o | $5.00 | $15.00 |
| GPT-4o-mini | $0.15 | $0.60 |
| Claude 3.5 Sonnet | $3.00 | $15.00 |
Let's build a simple cost estimator:
def estimate_cost(
input_tokens: int,
output_tokens: int,
input_price_per_million: float,
output_price_per_million: float
) -> float:
"""Returns estimated cost in USD."""
input_cost = (input_tokens / 1_000_000) * input_price_per_million
output_cost = (output_tokens / 1_000_000) * output_price_per_million
return input_cost + output_cost
# Example: 500 API calls, each with ~2000 input tokens and ~500 output tokens
calls = 500
input_per_call = 2000
output_per_call = 500
total_input = calls * input_per_call
total_output = calls * output_per_call
cost = estimate_cost(
input_tokens=total_input,
output_tokens=total_output,
input_price_per_million=5.00, # GPT-4o input
output_price_per_million=15.00 # GPT-4o output
)
print(f"Total input tokens: {total_input:,}")
print(f"Total output tokens: {total_output:,}")
print(f"Estimated cost: ${cost:.4f}")
This kind of estimation is something you should be doing before you deploy any LLM-powered feature. A feature that feels fast and cheap in testing can become surprisingly expensive at 100,000 calls per day.
Warning
Don't confuse "I set max_tokens=500" with "I'll always spend 500 output tokens." The model stops generating when it reaches a natural stopping point OR hits the limit, whichever comes first. Billing is based on actual tokens generated, not the maximum you allowed.
Work through this exercise to solidify your understanding. You'll need Python and tiktoken installed.
The Scenario: You're building a document summarization pipeline. Your application takes customer contract PDFs (converted to plain text), sends them to an LLM with a system prompt, and gets back a structured summary. You want to understand your token economics before going to production.
Step 1: Set up your tokenizer
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
system_prompt = """You are a legal document analyst. Extract the following from the contract:
1. Parties involved (names and roles)
2. Contract duration (start and end dates)
3. Key financial terms (payment amounts, schedules, penalties)
4. Termination conditions
5. Governing law
Return your analysis as structured JSON."""
sample_contract = """
SERVICE AGREEMENT
This Service Agreement ("Agreement") is entered into as of January 15, 2024,
between Meridian Analytics LLC ("Service Provider"), a Delaware limited liability
company, and Thornfield Capital Group ("Client"), a New York corporation.
TERM: This Agreement shall commence on February 1, 2024, and continue for a
period of twenty-four (24) months, terminating on January 31, 2026, unless
earlier terminated in accordance with Section 8.
COMPENSATION: Client agrees to pay Service Provider a monthly retainer of
$18,500, due on the first business day of each calendar month. Late payments
shall accrue interest at 1.5% per month. A setup fee of $5,000 is due upon
execution of this Agreement.
TERMINATION: Either party may terminate this Agreement with sixty (60) days
written notice. Client may terminate immediately for cause if Service Provider
fails to cure a material breach within thirty (30) days of written notice.
GOVERNING LAW: This Agreement shall be governed by the laws of the State of
New York, without regard to conflict of law principles.
"""
Step 2: Analyze the token counts
system_tokens = len(enc.encode(system_prompt))
contract_tokens = len(enc.encode(sample_contract))
total_input = system_tokens + contract_tokens + 4 # overhead
print(f"System prompt: {system_tokens} tokens")
print(f"Contract text: {contract_tokens} tokens")
print(f"Estimated total input: {total_input} tokens")
Step 3: Estimate costs for production scale
Assume your expected output is ~400 tokens per summary (structured JSON with the five fields). You process 200 contracts per day.
daily_contracts = 200
output_per_summary = 400
daily_input = daily_contracts * total_input
daily_output = daily_contracts * output_per_summary
# Calculate for GPT-4o
daily_cost_gpt4o = estimate_cost(daily_input, daily_output, 5.00, 15.00)
# Calculate for GPT-4o-mini
daily_cost_mini = estimate_cost(daily_input, daily_output, 0.15, 0.60)
print(f"\nDaily token usage:")
print(f" Input: {daily_input:,} tokens")
print(f" Output: {daily_output:,} tokens")
print(f"\nDaily cost estimate:")
print(f" GPT-4o: ${daily_cost_gpt4o:.4f}")
print(f" GPT-4o-mini: ${daily_cost_mini:.4f}")
print(f"\nMonthly estimate:")
print(f" GPT-4o: ${daily_cost_gpt4o * 30:.2f}")
print(f" GPT-4o-mini: ${daily_cost_mini * 30:.2f}")
Step 4: Experiment with compression
Try removing whitespace from the contract text and re-measuring:
import re
compressed_contract = re.sub(r'\n\s*\n', '\n', sample_contract.strip())
compressed_tokens = len(enc.encode(compressed_contract))
print(f"\nOriginal contract: {contract_tokens} tokens")
print(f"Compressed contract: {compressed_tokens} tokens")
print(f"Savings: {contract_tokens - compressed_tokens} tokens per document")
print(f"Monthly savings at 200/day (GPT-4o): ${((contract_tokens - compressed_tokens) * 200 * 30 / 1_000_000) * 5:.2f}")
This exercise connects token mechanics directly to real engineering decisions: model selection, prompt compression, and cost forecasting.
Mistake 1: Assuming 1 word = 1 token
The rule of thumb you'll see online is "roughly 750 words = 1,000 tokens" for English. That's a reasonable starting estimate, but for non-English text, numbers, code, or JSON, it breaks down completely. Always measure with tiktoken or the equivalent for your model family.
Mistake 2: Forgetting that context includes conversation history In multi-turn chat applications, every previous message in the conversation is re-sent to the model on each turn. A conversation that's been going for 20 exchanges might be passing 8,000+ tokens before the latest user message even arrives. If you're not truncating or summarizing old messages, costs compound rapidly.
Mistake 3: Counting only your text, not the formatting Chat APIs wrap messages in formatting overhead. OpenAI's API adds tokens for role labels, message boundaries, and reply priming. It's usually 3–10 extra tokens per message — small in isolation, significant at scale.
Mistake 4: Confusing max_tokens with response length
max_tokens sets a ceiling, not a target. The model generates until it naturally completes or hits the ceiling. If you set max_tokens=2000 for summaries that are naturally 200 tokens long, you're not wasting money on unused tokens — but you're also not getting the output you might expect if the model stops short.
Mistake 5: Not accounting for different tokenizers across providers
A prompt that's 1,200 tokens in cl100k_base (OpenAI) might be 1,400 tokens in Claude's tokenizer. If you're comparing costs across providers or planning to switch models, retokenize with the correct tokenizer for each model. Claude provides a Python SDK with tokenization utilities; Gemini provides similar tooling through the Google AI SDK.
Tip
When in doubt, add a 10–15% buffer to your token estimates. Real production data is always messier than your test samples.
You now have a working mental model of one of the most important — and most overlooked — concepts in LLM application development.
Here's what we covered:
tiktoken or equivalent tools before making API calls.Understanding tokens isn't just trivia — it's the foundation of responsible LLM system design. Every advanced topic in prompt engineering, retrieval-augmented generation, and multi-agent orchestration will require you to think in tokens.
Where to go next: