Understanding Tokens: How LLMs Tokenize Text and Why It Affects Your Inputs, Outputs, and Costs

You're building a chatbot for customer support. You've crafted a careful system prompt, written a detailed user message, and wired everything up to the OpenAI API. Then the bill arrives at the end of the month and it's three times what you expected. Or worse — your carefully written prompt gets truncated halfway through a critical instruction, and the model starts hallucinating because it never received the full context. Both of these problems trace back to the same root cause: you didn't have a clear mental model of how LLMs actually read and process text.

Tokens are the fundamental unit of how every large language model (LLM) — whether it's GPT-4, Claude, Gemini, or an open-source model like Llama — perceives, processes, and generates language. They're not words. They're not characters. They're something in between, and the distinction matters enormously once you start building real applications. Understanding tokens means understanding why "ChatGPT" costs differently than "chat gpt", why certain languages eat through your context window faster than others, and why a response that looks short to you might actually be expensive.

By the end of this lesson, you'll have a working mental model of tokenization that you can immediately apply to your prompt engineering, cost estimation, and system design decisions.

What you'll learn:

• What tokens are and how text is broken into them using Byte-Pair Encoding (BPE)
• Why token boundaries don't align neatly with words or characters
• How tokenization varies across languages, code, and special characters
• How to calculate and estimate token usage for inputs and outputs
• How context windows, pricing, and model limits are all token-denominated

Prerequisites

No prior machine learning knowledge is required. You should be comfortable reading simple Python code and have a basic idea of what an LLM is — a model that takes text in and generates text out. That's all you need to start.

What Is a Token, Exactly?

Think about how you read a sentence. You don't process it letter by letter, and you don't wait until you've read the entire paragraph before understanding begins. You chunk language into meaningful units — words, prefixes, suffixes — almost automatically.

LLMs do something similar, but more systematic. Before any text reaches the model's neural network, it goes through a tokenizer: a preprocessing step that breaks raw text into smaller chunks called tokens. Each token is then converted into a number (its token ID), and that sequence of numbers is what the model actually works with. The model never sees the word "customer" — it sees something like [43534].

A token is typically somewhere between a single character and a short word. In practice:

• A common English word like "the" is usually 1 token
• A longer or rarer word like "uncharacteristically" might be 4–5 tokens
• A space before a word is often included in the token itself (so " the" and "the" can be different tokens)
• Punctuation marks are often their own tokens
• Numbers can be tokenized digit by digit or as groups

The most important thing to internalize right now: tokens are not words. Treating them as equivalent is one of the most common and costly mistakes developers make.

How Tokenization Actually Works: Byte-Pair Encoding

Modern LLMs almost universally use a technique called Byte-Pair Encoding (BPE) — or a close variant of it — to build their tokenizer. Let's walk through how it works from first principles.

Imagine you're building a vocabulary for a tokenizer. You start with a huge corpus of text — billions of web pages, books, and code. Initially, your vocabulary consists of every individual character that appears: a, b, c, ... z, digits, punctuation, unicode characters, spaces, and so on.

Then you run a compression algorithm:

1. Count every adjacent pair of tokens in your corpus
2. Find the most frequent pair (say, e and r appearing together as er)
3. Merge them into a single new token: er
4. Repeat this process thousands of times

Each merge creates a new token that represents a chunk of text that appears together very frequently. After 50,000 or so merges, you have a vocabulary of roughly 50,000–100,000 tokens. Common words like " the", " and", " is" end up as single tokens because they appear together constantly. Rare words get broken into their component sub-word pieces.

This is why common words cost less (in tokens) than rare words. And it's why this approach is brilliant: you never hit an "unknown word" problem. Any text — even words the model has never seen — can be encoded as a sequence of sub-word tokens.

Key insight: The tokenizer is trained separately from the model itself, and it's fixed at training time. When you use GPT-4, you're using a specific, frozen tokenizer. Different model families (GPT, Claude, Gemini) use different tokenizers, so the same text may tokenize differently across providers.

Seeing Tokenization in Action

The best way to build intuition is to watch tokenization happen. OpenAI provides a free tool called the Tokenizer at platform.openai.com/tokenizer. Navigate there and you'll see a text box where you can paste any text and watch it get highlighted in alternating colors — each color represents a single token.

But let's also do this programmatically, because you'll need this skill when building applications. The tiktoken library is OpenAI's open-source tokenizer:

pip install tiktoken

import tiktoken

# Load the tokenizer used by GPT-4 and GPT-3.5-turbo
enc = tiktoken.get_encoding("cl100k_base")

text = "The quarterly revenue report shows a 12.4% increase in EMEA markets."

tokens = enc.encode(text)
print(f"Token IDs: {tokens}")
print(f"Token count: {len(tokens)}")

# Decode each token individually to see the chunks
for token_id in tokens:
    chunk = enc.decode([token_id])
    print(repr(chunk))

Running this gives you output like:

Token IDs: [791, 38177, 13254, 3307, 5039, 264, 220, 717, 13, 19, 4, 5376, 304, 36085, 16717, 13]
Token count: 16

'The'
' quarterly'
' revenue'
' report'
' shows'
' a'
' '
'12'
'.'
'4'
'%'
' increase'
' in'
' EM'
'EA'
' markets'
'.'

Notice a few things immediately:

• " quarterly" is one token (space included), but "EMEA" splits into "EM" and "EA" — because EMEA is less common in the training data than quarterly
• The number 12.4 becomes four tokens: 12, ., 4, %
• The space before most words is included in the word's token — this is important to understand when you're thinking about token boundaries

The Surprising Cases That Trip People Up

Now that you have the basics, let's look at where things get unintuitive. These are exactly the scenarios that cause unexpected behavior and cost overruns in production.

Numbers and Math Are Expensive

Numbers tokenize poorly. Each digit often becomes its own token:

text_a = "The transaction ID is 8472916304857."
text_b = "The transaction ID is eight billion four hundred seventy-two million."

tokens_a = enc.encode(text_a)
tokens_b = enc.encode(text_b)

print(f"Numeric version: {len(tokens_a)} tokens")
print(f"Written-out version: {len(tokens_b)} tokens")

You might expect the numeric version to be shorter. Often it's comparable or longer, because each digit in a long number gets its own token. This matters if you're feeding large datasets of numeric IDs, financial data, or timestamps into a prompt.

Non-English Text Uses More Tokens

This is one of the most impactful and least-discussed aspects of tokenization. Because BPE tokenizers are trained predominantly on English text, English words have very efficient representations. Other languages — particularly those with non-Latin scripts — tokenize much less efficiently:

english = "The customer submitted a support request."
spanish = "El cliente envió una solicitud de soporte."
japanese = "顧客がサポートリクエストを送信しました。"
arabic = "أرسل العميل طلب دعم."

for label, text in [("English", english), ("Spanish", spanish), 
                     ("Japanese", japanese), ("Arabic", arabic)]:
    count = len(enc.encode(text))
    print(f"{label}: {count} tokens — '{text}'")

You'll typically see English using the fewest tokens, Spanish moderately more, and Japanese or Arabic potentially using 3–5x as many tokens per equivalent meaning. If you're building a multilingual product, this directly affects both your costs and your effective context window capacity.

Code: Surprisingly Efficient (But Not Always)

Code tokenizes relatively well because programming keywords, operators, and common patterns appear frequently in training data. But variable names, string literals, and comments vary widely:

code = """
def calculate_churn_rate(active_users, churned_users):
    if active_users == 0:
        return 0.0
    return round((churned_users / active_users) * 100, 2)
"""

print(f"Code snippet: {len(enc.encode(code))} tokens")

Warning: Long, descriptive variable names and verbose comments add tokens. In extremely token-constrained situations (like few-shot examples in a tight context window), compressed variable names save space — though readability suffers. It's a real trade-off.

Whitespace and Formatting Add Up

Structured data, JSON, and heavily indented code cost more than you'd think:

json_compact = '{"user_id":1042,"status":"active","plan":"enterprise"}'
json_pretty = '''{
    "user_id": 1042,
    "status": "active",
    "plan": "enterprise"
}'''

print(f"Compact JSON: {len(enc.encode(json_compact))} tokens")
print(f"Pretty JSON: {len(enc.encode(json_pretty))} tokens")

The pretty-printed version typically uses 20–40% more tokens just from whitespace. When you're feeding large JSON payloads into a prompt — API responses, database records, configuration — always strip unnecessary whitespace first.

Context Windows: Your Token Budget

Every LLM has a context window: the maximum number of tokens it can "see" at once, including both input and output. Think of it like working memory. The model can only reason about what fits inside the window. Everything outside it doesn't exist.

Common context window sizes as of 2024:

• GPT-3.5-turbo: 16,385 tokens
• GPT-4o: 128,000 tokens
• Claude 3.5 Sonnet: 200,000 tokens
• Llama 3.1 (70B): 128,000 tokens

Here's the critical thing people miss: the context window is shared between your input and your output. When you call an API, you set a parameter called max_tokens (or max_output_tokens), which is the maximum length of the response. That response budget comes out of the same pool as your input.

So if you have a 16,000-token context window, and your system prompt + user message is 12,000 tokens, you only have 4,000 tokens left for the model's response. If you set max_tokens=8000 hoping for a long answer, the model will be cut off — or depending on the API, you'll get an error.

Here's how to track this in practice:

import tiktoken

def count_tokens_for_chat(messages: list[dict], model: str = "gpt-4o") -> int:
    """
    Estimate token count for a list of messages in OpenAI chat format.
    Includes overhead tokens for message formatting.
    """
    enc = tiktoken.encoding_for_model(model)
    total = 0
    
    for message in messages:
        # Each message has ~4 tokens of overhead for role/formatting
        total += 4
        for key, value in message.items():
            total += len(enc.encode(str(value)))
    
    total += 2  # reply priming tokens
    return total

messages = [
    {"role": "system", "content": "You are a data analyst. Answer concisely with specific numbers."},
    {"role": "user", "content": "Summarize the key trends in our Q3 sales data for the APAC region, focusing on month-over-month changes."}
]

estimated_tokens = count_tokens_for_chat(messages)
print(f"Estimated input tokens: {estimated_tokens}")
print(f"Tokens remaining for output (in 16k window): {16385 - estimated_tokens}")

Tip: Always count your input tokens before sending an API request in production code. Running into a context length error mid-pipeline is expensive and disruptive. Build a check that raises a clear warning if you're within, say, 10% of the limit.

How Token Pricing Works

LLM APIs charge by the token — almost universally. But the pricing is not symmetric: input tokens and output tokens have different prices, and output tokens are typically more expensive.

Why? Because generating each output token requires a full forward pass through the model. Reading input tokens is processed more efficiently in parallel.

A typical pricing structure (approximate, always check current rates):

Model	Input (per 1M tokens)	Output (per 1M tokens)
GPT-4o	$5.00	$15.00
GPT-4o-mini	$0.15	$0.60
Claude 3.5 Sonnet	$3.00	$15.00

Let's build a simple cost estimator:

def estimate_cost(
    input_tokens: int,
    output_tokens: int,
    input_price_per_million: float,
    output_price_per_million: float
) -> float:
    """Returns estimated cost in USD."""
    input_cost = (input_tokens / 1_000_000) * input_price_per_million
    output_cost = (output_tokens / 1_000_000) * output_price_per_million
    return input_cost + output_cost

# Example: 500 API calls, each with ~2000 input tokens and ~500 output tokens
calls = 500
input_per_call = 2000
output_per_call = 500

total_input = calls * input_per_call
total_output = calls * output_per_call

cost = estimate_cost(
    input_tokens=total_input,
    output_tokens=total_output,
    input_price_per_million=5.00,    # GPT-4o input
    output_price_per_million=15.00   # GPT-4o output
)

print(f"Total input tokens: {total_input:,}")
print(f"Total output tokens: {total_output:,}")
print(f"Estimated cost: ${cost:.4f}")

This kind of estimation is something you should be doing before you deploy any LLM-powered feature. A feature that feels fast and cheap in testing can become surprisingly expensive at 100,000 calls per day.

Warning: Don't confuse "I set max_tokens=500" with "I'll always spend 500 output tokens." The model stops generating when it reaches a natural stopping point OR hits the limit, whichever comes first. Billing is based on actual tokens generated, not the maximum you allowed.

Hands-On Exercise

Work through this exercise to solidify your understanding. You'll need Python and tiktoken installed.

The Scenario: You're building a document summarization pipeline. Your application takes customer contract PDFs (converted to plain text), sends them to an LLM with a system prompt, and gets back a structured summary. You want to understand your token economics before going to production.

Step 1: Set up your tokenizer

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

system_prompt = """You are a legal document analyst. Extract the following from the contract:
1. Parties involved (names and roles)
2. Contract duration (start and end dates)
3. Key financial terms (payment amounts, schedules, penalties)
4. Termination conditions
5. Governing law

Return your analysis as structured JSON."""

sample_contract = """
SERVICE AGREEMENT

This Service Agreement ("Agreement") is entered into as of January 15, 2024, 
between Meridian Analytics LLC ("Service Provider"), a Delaware limited liability 
company, and Thornfield Capital Group ("Client"), a New York corporation.

TERM: This Agreement shall commence on February 1, 2024, and continue for a 
period of twenty-four (24) months, terminating on January 31, 2026, unless 
earlier terminated in accordance with Section 8.

COMPENSATION: Client agrees to pay Service Provider a monthly retainer of 
$18,500, due on the first business day of each calendar month. Late payments 
shall accrue interest at 1.5% per month. A setup fee of $5,000 is due upon 
execution of this Agreement.

TERMINATION: Either party may terminate this Agreement with sixty (60) days 
written notice. Client may terminate immediately for cause if Service Provider 
fails to cure a material breach within thirty (30) days of written notice.

GOVERNING LAW: This Agreement shall be governed by the laws of the State of 
New York, without regard to conflict of law principles.
"""

Step 2: Analyze the token counts

system_tokens = len(enc.encode(system_prompt))
contract_tokens = len(enc.encode(sample_contract))
total_input = system_tokens + contract_tokens + 4  # overhead

print(f"System prompt: {system_tokens} tokens")
print(f"Contract text: {contract_tokens} tokens")
print(f"Estimated total input: {total_input} tokens")

Step 3: Estimate costs for production scale

Assume your expected output is ~400 tokens per summary (structured JSON with the five fields). You process 200 contracts per day.

daily_contracts = 200
output_per_summary = 400

daily_input = daily_contracts * total_input
daily_output = daily_contracts * output_per_summary

# Calculate for GPT-4o
daily_cost_gpt4o = estimate_cost(daily_input, daily_output, 5.00, 15.00)

# Calculate for GPT-4o-mini
daily_cost_mini = estimate_cost(daily_input, daily_output, 0.15, 0.60)

print(f"\nDaily token usage:")
print(f"  Input: {daily_input:,} tokens")
print(f"  Output: {daily_output:,} tokens")
print(f"\nDaily cost estimate:")
print(f"  GPT-4o: ${daily_cost_gpt4o:.4f}")
print(f"  GPT-4o-mini: ${daily_cost_mini:.4f}")
print(f"\nMonthly estimate:")
print(f"  GPT-4o: ${daily_cost_gpt4o * 30:.2f}")
print(f"  GPT-4o-mini: ${daily_cost_mini * 30:.2f}")

Step 4: Experiment with compression

Try removing whitespace from the contract text and re-measuring:

import re
compressed_contract = re.sub(r'\n\s*\n', '\n', sample_contract.strip())
compressed_tokens = len(enc.encode(compressed_contract))

print(f"\nOriginal contract: {contract_tokens} tokens")
print(f"Compressed contract: {compressed_tokens} tokens")
print(f"Savings: {contract_tokens - compressed_tokens} tokens per document")
print(f"Monthly savings at 200/day (GPT-4o): ${((contract_tokens - compressed_tokens) * 200 * 30 / 1_000_000) * 5:.2f}")

This exercise connects token mechanics directly to real engineering decisions: model selection, prompt compression, and cost forecasting.

Common Mistakes & Troubleshooting

Mistake 1: Assuming 1 word = 1 token The rule of thumb you'll see online is "roughly 750 words = 1,000 tokens" for English. That's a reasonable starting estimate, but for non-English text, numbers, code, or JSON, it breaks down completely. Always measure with tiktoken or the equivalent for your model family.

Mistake 2: Forgetting that context includes conversation history In multi-turn chat applications, every previous message in the conversation is re-sent to the model on each turn. A conversation that's been going for 20 exchanges might be passing 8,000+ tokens before the latest user message even arrives. If you're not truncating or summarizing old messages, costs compound rapidly.

Mistake 3: Counting only your text, not the formatting Chat APIs wrap messages in formatting overhead. OpenAI's API adds tokens for role labels, message boundaries, and reply priming. It's usually 3–10 extra tokens per message — small in isolation, significant at scale.

Mistake 4: Confusing max_tokens with response length max_tokens sets a ceiling, not a target. The model generates until it naturally completes or hits the ceiling. If you set max_tokens=2000 for summaries that are naturally 200 tokens long, you're not wasting money on unused tokens — but you're also not getting the output you might expect if the model stops short.

Mistake 5: Not accounting for different tokenizers across providers A prompt that's 1,200 tokens in cl100k_base (OpenAI) might be 1,400 tokens in Claude's tokenizer. If you're comparing costs across providers or planning to switch models, retokenize with the correct tokenizer for each model. Claude provides a Python SDK with tokenization utilities; Gemini provides similar tooling through the Google AI SDK.

Tip: When in doubt, add a 10–15% buffer to your token estimates. Real production data is always messier than your test samples.

Summary & Next Steps

You now have a working mental model of one of the most important — and most overlooked — concepts in LLM application development.

Here's what we covered:

• Tokens are sub-word chunks, not words or characters. They're created through Byte-Pair Encoding by merging frequently co-occurring character sequences.
• Token counts vary dramatically based on language, content type, and formatting. English text is efficient; non-English scripts, long numbers, and heavily formatted JSON are not.
• Context windows are token-denominated and shared between input and output. Filling your context with input leaves less room for output.
• Pricing is per-token, asymmetric — output tokens typically cost 3–4x more than input tokens. Calculate your expected token economics before you deploy.
• You can and should measure token usage programmatically using tiktoken or equivalent tools before making API calls.

Understanding tokens isn't just trivia — it's the foundation of responsible LLM system design. Every advanced topic in prompt engineering, retrieval-augmented generation, and multi-agent orchestration will require you to think in tokens.

Where to go next:

• Prompt Engineering Fundamentals — Now that you understand tokens, learn how to write prompts that use them efficiently
• Managing Context Windows in Long-Document Applications — Deep dive into chunking, summarization, and retrieval strategies for documents that exceed context limits
• Choosing the Right Model for Your Use Case — Token economics are one major factor; learn how to evaluate capability vs. cost trade-offs across model families
• Building a Token-Aware Rate Limiter — Practical implementation of pre-flight token counting and cost guardrails for production APIs