Tokens and Context Window Explained: Why AI Forgets Long Chats

By Ravindra Bagale, trainer · LinkedIn · Last updated: 19 October 2026

Generative AI course · Chapter 12 · Estimated reading time: 15 minutes

1. Suppose the AI forgot our rule

Suppose we are Neha, an operations executive in Nagpur. In the morning we start a chat to prepare a vendor report. Our first message sets 3 rules:

For this whole chat: use ₹ with Indian commas, keep every answer under 150 words, and never name any vendor, use Vendor A, Vendor B.

Then we work all day in the same chat. We paste 6 long email threads, 3 tables and a policy document. Around 5 pm, the answers are suddenly 400 words long, and one of them names a real vendor.

The AI did not become careless. Our rules, written at 10 am, have been pushed out of the part of the chat the model can still see. That part is the context window.

2. Tokens in 2 minutes

You met tokens in the last chapter. A quick reminder:

• A token is a small piece of text: often a whole common word, sometimes part of a word, a number or a symbol.
• OpenAI's rule of thumb for English: 1 token is about 4 characters, or about ¾ of a word, so 100 tokens is about 75 words.
• Indian-language text and numbers usually take more tokens than plain English.
• Each company counts a little differently, so treat counts as estimates.

3. What is a context window?

The context window is the total amount of text, in tokens, that the model can look at in one go: your messages, its replies, uploaded files, hidden instructions, and the answer it is writing.

Think of a whiteboard in a meeting room. Everything the AI needs must be written on the board. When the board is full, the oldest notes are rubbed off, or squeezed into a short summary, to make room.

The model does not remember anything outside the board. It has no memory of its own between messages; the app sends the board to it again every time you press Enter.

What goes on the board:

1. Hidden system instructions from the app (you do not see these).
2. Your custom instructions, memory, and project instructions, if you use them.
3. The chat so far: your messages and the AI's replies.
4. Files you uploaded, or the parts of them the app selected.
5. Web search results, if search was used.
6. The answer the AI is writing now.

So the space for your text is smaller than the full window. OpenAI says this clearly on its pricing page: the part available for your input is smaller than the total window.

4. How big is the window? (checked October 2026)

"K" means thousand: 27K is 27,000 tokens. Using the rule of thumb, 27,000 tokens is roughly 20,000 English words, but remember that hidden instructions and the reply share the same space.

Plans change often. Before you pay for a plan for long documents, check the official pricing or help page.

5. Why long chats drift and forget

When the chat grows past the window, apps do one of these (each app is different and does not always tell you):

• Drop the oldest messages, so early rules disappear.
• Summarise older parts, so details get lost.
• Search inside long files and send only the parts that look relevant to your question.

Even inside the window, a model pays less attention to some parts of a very long text. Important rules buried in the middle of a long chat are easier to miss than rules at the start or at the end of your latest message.

The signs that your chat is too long:

1. The AI ignores a rule you set earlier (length, format, names).
2. It asks for information you already gave.
3. It mixes up two documents or two people.
4. Answers get slower, or the app shows a message like "this chat is too long".

6. Worked example 1: paste a long PDF

Suppose Neha uploads a 40-page company travel policy and asks:

What is the daily food allowance for a trip to Mumbai, and what proof do I need?

A 40-page policy can easily be 20,000 words. With the rule of thumb (100 tokens ≈ 75 words), that is roughly 27,000 tokens: more than the whole free ChatGPT window, and close to the full Gemini free window, before the chat has even started.

What usually happens:

• The app reads only some pages, or searches inside the file for parts that match "food allowance".
• If the allowance is in a table on page 31, and the search picks a different page, the answer can be wrong or incomplete.

The better way: ask for the page and the exact line.

From the attached policy only:
1. Quote the exact line about the daily food allowance for metro cities, with the page number.
2. Quote the exact line about the proof needed for food claims, with the page number.
If you cannot find a line, say "Not found" instead of guessing.

Expected output (yours will be similar, not the same):

1. Page 31, Table 4: "Metro cities (Mumbai, Delhi, Bengaluru, Chennai, Kolkata, Hyderabad):
   daily food allowance ₹[amount] per day."
2. Not found. I could not find a line about proof for food claims in the parts I could read.
   Please check the "Claims and receipts" section, or upload that section separately.

Now Neha can open page 31 and check, and she knows exactly which part to upload next. Ask for quotes and page numbers whenever you work with long files. If the quote is not in the file, the answer was invented.

Split big files. If a document is long, upload only the chapter you need, or split the PDF into parts (most free PDF tools can split pages) and ask about one part at a time.

7. Worked example 2: the "summarise, then continue" trick

When a chat becomes long, do not keep going. Make the AI write a short handover note, then start a fresh chat with it.

Step 1: in the old chat, paste:

We will continue in a new chat. Write a handover summary of this chat in under 200 words:
1. Goal of the work
2. Rules I set (format, length, names)
3. Decisions made so far
4. Open questions and the next step
Use bullet points. Do not add anything new.

Expected output (yours will be similar, not the same):

Goal: a monthly vendor performance report for October.

Rules:
- Use ₹ with Indian commas.
- Every answer under 150 words.
- Never name vendors; use Vendor A, Vendor B, Vendor C.

Decisions:
- Vendor A: on-time delivery 92%, keep.
- Vendor B: 3 late deliveries; send a warning email.
- Report format: table + 3 bullet points of action.

Open questions:
- Vendor C invoice of ₹1,25,000 is not matched with the purchase order.
Next step: draft the warning email to Vendor B.

Step 2: check the summary. Correct anything wrong. This is your chance to fix the AI's memory.

Step 3: open a new chat and paste:

Here is the summary of my earlier work. Follow all the rules in it.
[paste the corrected summary]
Now: draft the warning email to Vendor B.

The new chat starts with a clean whiteboard that holds only what matters, with your rules at the top. Answers become short and focused again.

8. Worked example 3: count tokens yourself

Way 1: the free Tokenizer page (no coding)

1. Open platform.openai.com/tokenizer in your browser.
2. Paste any text.
3. Read the number of tokens and characters; each token is shown in a different colour.

Way 2: a short Python script

If you have Python on your computer, install OpenAI's free tiktoken library:

pip install tiktoken

Save this as count_tokens.py:

import sys
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def report(label, text):
    tokens = enc.encode(text)
    words = len(text.split())
    print(f"{label}: {len(text)} characters, {words} words, {len(tokens)} tokens")

report("Short message", "Please send me the invoice by Friday.")
report("Email", "Dear Sir, I am writing to request two days of leave on 26 and 27 October "
                "for a family function in Nashik. I will finish the monthly report before I leave "
                "and Priya has agreed to handle urgent calls. Regards, Neha")

if len(sys.argv) > 1:
    text = open(sys.argv[1], encoding="utf-8").read()
    report(sys.argv[1], text)

Run it with any text file of your own (we used a made-up notes.txt of team notes):

python count_tokens.py notes.txt

Output (this is the real output we got on 19 October 2026 with tiktoken 0.14.0):

Short message: 37 characters, 7 words, 8 tokens
Email: 212 characters, 41 words, 50 tokens
notes.txt: 15300 characters, 2820 words, 3420 tokens

Check the rule of thumb: 2,820 words ÷ 0.75 = 3,760, and the real count is 3,420. Close enough for planning. Also, 15,300 characters ÷ 4 = 3,825. The rule is an estimate, and it is a little high for simple English.

o200k_base is the encoding used by recent OpenAI models. Gemini and Claude use their own tokenizers, so their counts will be a little different. For planning, the rule of thumb is enough.

9. Will it fit? A quick check

Before you paste something long, do this 3-step check:

1. Count words (Word and Google Docs show a word count).
2. Estimate tokens: words ÷ 0.75. For Hindi or Marathi text, roughly double it.
3. Compare with your plan's window, and leave space for the chat and the answer.

10. 6 habits for long work with AI

1. One task, one chat. Start a new chat for a new task.
2. Rules at the top, and repeat them. Put key rules in your latest message too: "Remember: under 150 words."
3. Use projects for fixed rules. Project instructions are added to every chat in the project.
4. Handover summary before a chat gets too long.
5. Send only what is needed: one chapter, not the whole book; the relevant table, not the whole spreadsheet.
6. Ask for quotes and page numbers when the AI reads files.

11. Compare the 4 tools

12. Beginner mistakes and quick fixes

13. Try at home

Recap

Open your longest AI chat now, ask for a handover summary, and continue in a new chat. Notice how much sharper the next answer is.

Previous chapter: What Is an LLM? Large Language Models Explained Simply

Next chapter: Why AI Hallucinates and How to Fact-Check Any AI Answer

Ravindra Bagale, trainer: linkedin.com/in/ravindra-bagale. The people, vendors, policy and notes on this page are made up for learning. Context window sizes were checked on the official ChatGPT pricing page and Google Gemini help pages on 19 October 2026 and can change. The Python output was produced with tiktoken 0.14.0 on 19 October 2026. AI answers change every time, so your output will be similar, not the same.