Under the Hood: Tokens & Context
You've used AI — now understand how it processes text. Learn about tokens, context windows, and why AI 'forgets' things.
What Are Tokens, Really?
When you type a message to an AI, it doesn’t read words like you do. It breaks your text into tokens — small chunks of text that the model processes.
Here’s how tokenisation typically works:
| Text | Approximate Tokens |
|---|---|
| ”Hello” | 1 token |
| ”artificial intelligence” | 2 tokens |
| ”I love programming in Python” | 5 tokens |
| ”antidisestablishmentarianism” | 5-6 tokens |
Rule of thumb: 1 token ≈ ¾ of a word in English. So 1,000 tokens ≈ 750 words.
Different languages tokenise differently. English is efficient (fewer tokens per word), while languages like Japanese or Arabic may use more tokens per word.
The Context Window
The context window is the AI’s “working memory” — the maximum amount of text it can consider at once.
| Model | Context Window | Roughly Equals |
|---|---|---|
| GPT-4o mini | 128K tokens | ~96,000 words |
| Claude Sonnet | 200K tokens | ~150,000 words |
| Gemini 1.5 | 1M tokens | ~750,000 words |
This includes everything: your messages, the AI’s responses, system instructions, and any documents you’ve uploaded.
Why AI “Forgets”
In a long conversation, the AI doesn’t truly “remember” anything. Each response is generated from what’s currently in the context window. Once the conversation exceeds that limit, the oldest parts get dropped.
This is why:
- Long conversations can feel disjointed
- AI might “forget” something you said 20 messages ago
- Starting a new chat sometimes gives better results
Temperature Revisited
Now you know about tokens, here’s a deeper look at temperature:
- Temperature 0: Always picks the most probable next token. Very consistent but can feel robotic.
- Temperature 0.7: Good balance of creativity and coherence (most chatbots default here).
- Temperature 1.0+: More random token selection. Creative but can produce nonsense.
When you ask AI to “be more creative,” you’re essentially asking it to consider less-probable token choices.
Why This Matters for You
Understanding tokens helps you:
- Write better prompts — keep them concise but complete
- Manage long conversations — start fresh when things get off track
- Understand pricing — API access is charged per token
- Upload documents smartly — know the limits of what AI can process at once
Try it: Paste a long document into ChatGPT and ask a question about the beginning. Then ask about the end. If the document is within the context window, it should handle both well.
Quick Quiz
Test what you just learned. Pick the best answer for each question.
Q1 Roughly how many tokens is the word 'hamburger'?
Q2 What happens when your conversation exceeds the context window?
Q3 Why do AI companies charge by the token?
Q4 A context window of 128K tokens means the AI can handle roughly...