Everyday storyThe STD booth meter
Before mobile phones, a long-distance call meant a trip to the STD booth. A little meter ticked up as you talked, and the bill depended on how long you spoke, not on how important the call was. A one-minute call to say "reached safely" cost the same per minute as a one-minute call about a wedding.
Now imagine a strange rule: every time you want to say something new, you must first repeat the whole conversation so far, from the very start. Your first sentence costs a few paise. Your tenth costs far more, because you have to recite the nine before it first, and it takes longer before the person on the other end hears the new part. Keep talking for an hour and most of the meter is spent on repetition.
AI apps work a lot like that booth with that rule. The meter counts small pieces of text rather than minutes, it runs in both directions (what the model reads and what it writes), and the model on the other end really does need the whole conversation repeated every time, because it forgets everything the moment it hangs up.
In a real productAI chat apps and AI features inside everyday apps
AI services sell access by the amount of text the model reads and writes, and the apps built on them pass that cost on through monthly plans, free-tier message limits or "credits". You may have noticed a very long chat start to feel sluggish, or an app suggest you "start a new chat for better results". You may have seen a cheaper plan that only lets you use a "lite" model, or a daily limit that resets at midnight. That's the meter at work.
It also shapes things you don't see. A food app's help bot keeps its replies short. A shopping app writes product summaries once a night and stores them, instead of asking the model every time someone opens the page. A banking app's assistant answers "what's my balance" without calling a big model at all. Each of those is a team watching the meter, because at lakhs of users a day, a few extra paise per message adds up to lakhs of rupees a month.
The ideaHow it works
What a token is
An doesn't read words or letters. It reads : pieces of words. Before your message reaches the model, a chops it up using a fixed vocabulary of tens of thousands of pieces. Common English words like "the" or "plan" are a single token. Longer or rarer words split: "unbelievable" might become "un", "believ" and "able". Spaces and punctuation ride along inside tokens too.
Some rules of thumb for English:
- about 3 words make 4 tokens, so a 300-word email is roughly 400 tokens;
- about 4 characters make one token;
- a page of a book is roughly 500 to 800 tokens.
Numbers, code, web links and odd spellings split into many more pieces. A 16-digit order number can take six or eight tokens where a word takes one.
Indian languages are where this bites. Many tokenisers were built mostly from English text, so Hindi, Tamil or Bengali words often break into several small pieces each. The same 30-word message can be twice as many tokens or more, which means it costs more, fills the model's memory faster and takes longer to answer. Newer tokenisers with bigger vocabularies have narrowed the gap a lot, but not closed it. Hinglish typed in English letters is its own puzzle: the words are spelled many different ways, so they split unpredictably.
Pictures and audio are counted in tokens too. A cuts a photo into small patches and turns each into a token, so one image can cost as much as a few hundred words of text. The lesson is the same everywhere: count with the model's own tokeniser, not by guessing from words, because two models can count the same text quite differently.
Two meters on every call
Every call to a model runs two meters:
- Input tokens: everything the model reads.
- Output tokens: everything it writes. These are usually priced higher, often several times higher.
Why is writing dearer? Reading happens in one big parallel pass. Writing happens one token at a time: the model works out a token, adds it to the text, then works out the next one, and the serving it is tied up the whole time. A long answer keeps an expensive machine busy for longer than a long question does.
There's a hidden third kind. A writes out its working before the final answer: it plans, checks, and sometimes tries a few approaches. Those are billed as output even when the app never shows them. A two-line answer can quietly carry 2,000 tokens of thinking behind it.
Why the model reads everything again
What does the model read on each call? Everything inside its : the app's (its instructions), the chat so far, any documents the app has added, and finally your question. The model keeps no memory between calls. To "remember" what you said five messages ago, the app has to send it again.
That's why long chats get slower and pricier. Message 6 carries messages 1 to 5 along with it.
- 1
YouChat app
Message 1: plan a 3-day Goa trip
- 2
Chat appAI model
Rules + message 1 (about 400 tokens in)
- 3
AI modelYou
A day-by-day plan (about 500 tokens out)
- 4
YouChat app
Message 6: swap day 2 for a beach day
- 5
Chat appAI model
Rules + all 6 messages + replies (about 4,000 tokens in)
- 6
AI modelYou
Slower first word, bigger bill, same short request
Let's put numbers on it. Say the rules are 500 tokens, each question is about 30 tokens, and each question plus its reply adds about 330 tokens to the history.
- Turn 1 sends about 530 input tokens.
- Turn 10 sends 500 + (9 × 330) + 30, about 3,500 tokens.
- Turn 20 sends about 6,800 tokens.
Now add them up. Across a 10-turn chat the model reads about 20,000 input tokens in total, while it writes only about 3,000. Across 20 turns, the total input is about 73,000. Twice as many turns, nearly four times the input. That's the STD booth rule in action: the cost of a chat grows much faster than its length, and the old history, not the new question, is most of the bill.
There's also a ceiling. Every model has a maximum context window. Once a chat hits it, something must be dropped or squeezed, or the call fails. Long before that ceiling, quality can slip, because the model has to pick the one useful line out of pages of old material, a problem called .
How fast it feels
Speed has two parts too, matching the two meters.
First the model reads the whole prompt in one parallel pass, called . Nothing appears on screen until it's done, so the grows with a longer input. A short prompt might take a fifth of a second; a 50,000-token document might take several seconds before the first word.
Then the model writes, one token at a time, called . Its pace is measured in , say about 50 for a big model on a busy server. A 500-token answer takes about 10 seconds to finish; a 1,500-token essay takes about 30.
- 1
Chat appAI model
A prompt of about 2,000 tokens
- 2
AI modelYour screen
Reads all 2,000 in one go, so the first word shows after about half a second
- 3
AI modelYour screen
Then writes about 50 tokens a second, one at a time
- 4
AI modelYour screen
The last of 500 tokens lands at about 10.5 seconds
hides much of this: the app shows words as they arrive, so you start reading after half a second instead of staring at a spinner for ten. But it doesn't make the answer finish sooner. Only a shorter answer, a faster model or a shorter prompt does that.
Counting the bill
Here is a sum. Take 1 lakh users sending 10 messages a day, each call about 1,300 tokens in and 200 out.
- Calls a day: 1 lakh × 10 = 10 lakh.
- Input tokens: 10 lakh × 1,300 = 130 crore a day.
- Output tokens: 10 lakh × 200 = 20 crore a day.
With example prices of ₹25 per million input tokens and ₹100 per million output tokens (made-up round numbers, not any real price list), that's about ₹32,500 for input plus ₹20,000 for output: roughly ₹52,500 a day, or about ₹16 lakh a month.
Notice two things. First, input is 87% of the tokens and over 60% of the cost, even though each output token is four times the price. In chat apps, history and instructions usually dominate. Second, divide by users: about ₹16 per user per month. If the plan sells for ₹99, that's fine. If it's free, someone is paying ₹16 lakh a month. This per-user figure is the feature's , and it decides whether growth makes or loses money.
The levers
Every lever comes from the sums above: send fewer tokens, write fewer tokens, or pay less for each one.
- Shorter instructions. Cut repeated rules and long examples; every token in the system prompt is paid on every single call.
- . Many providers charge much less for an opening that's identical to a recent call. Keep the fixed rules at the very start and anything that changes at the end.
- Trim old history. Keep the last few exchanges word for word and replace older ones with a short summary, called .
- An and a request for short answers. Output is the dearest token.
- . Send easy jobs to a that costs a tenth or less.
- Cheaper timing. Work nobody is waiting for, like overnight summaries, can go through a , usually at a big discount.
- Don't ask twice. A reuses an earlier answer when a new question means the same thing.
The big idea: you pay for every token the model reads and writes on every call, and the model rereads the whole conversation each time, so keep what you send lean and the answers as short as they can usefully be.
For experienced engineers
Most current tokenisers are byte-level BPE or SentencePiece-style unigram models with vocabularies from about 32K to over 200K entries; larger vocabularies compress multilingual text and code better at the cost of a bigger embedding table. Token counts are model-specific: re-baseline every cost model when you switch model families, because a different tokeniser can move your bill 10–30% with no change in list price. Use the provider's counting endpoint or the response's usage field; never estimate one vendor's model with another's tokeniser. Output is dearer because decode is memory-bandwidth-bound and serial (each step streams the weights and the whole KV cache), while prefill is compute-bound and parallel; that's also why TTFT tracks prompt length and inter-token latency tracks model size, batch size and context length. Reasoning models bill thinking as output and are effort-sensitive, so set the effort or thinking budget per route and budget from measured usage, not visible length. Watch pricing structure as well as rates: some providers reprice the whole request above a long-context threshold, and prompt caching often carries a write premium plus a short TTL, so cache only prefixes that are reused within it. For agents, input dominates (ratios of hundreds to one are common), so cache hit rate drives cost; measure cost per completed task, not per call, and put hard step and token ceilings inside the loop.
Step by stepOne help chat, costed to the paisa
Riya's biryani is late, and she opens the food app's help chat. Let's follow her conversation through the meter, using the same example prices: ₹25 per million input tokens and ₹100 per million output tokens.
-
Riya types "My biryani is 40 minutes late, where is it?" That's about 25 tokens. Before calling the model, the help bot looks up her order and the rider's position in the app's own systems. That costs nothing on the token meter.
-
The bot builds the prompt. The system prompt (tone, refund rules, what the bot may and may not promise) is about 1,200 tokens. The order details are about 300. Add Riya's 25: about 1,525 input tokens.
-
The model replies with an apology and a new arrival time: about 120 output tokens. Cost: 1,525 × ₹25 per million is about ₹0.038, and 120 × ₹100 per million is about ₹0.012. Total: about 5 paise. Time: about 0.4 seconds to the first word, then 120 tokens at 50 a second, so the reply finishes in under 3 seconds.
- 1
RiyaHelp bot
My biryani is 40 minutes late
- 2
Help botOrder system
Fetch order A17 and the rider position
- 3
Order systemHelp bot
Rider 15 minutes away, stuck in traffic
- 4
Help botAI model
Rules + order details + message, about 1,525 tokens in
- 5
AI modelHelp bot
A reply of about 120 tokens, costing about 5 paise
- 6
Help botRiya
Apology and a new arrival time, in about 3 seconds
-
Riya asks for a refund (20 tokens). The prompt now carries everything before: 1,525 + 120 + 20 = about 1,665 in, and the reply is about 150 out.
-
Two more short turns (she confirms, the bot confirms the refund) bring inputs of about 1,830 and 1,920, with replies of 80 and 60.
-
Add it up. Input: about 6,940 tokens across four turns, of which the same 1,200-token system prompt makes up 4,800. Output: about 410 tokens. Cost: about ₹0.17 for input plus ₹0.04 for output, roughly 22 paise for the whole chat.
-
Scale it. At 2 lakh help chats a day, that's about ₹44,000 a day, or about ₹13 lakh a month.
-
Now pull the levers.
- The 1,200-token rules are identical in every call, so put them first and use prompt caching. If cached input costs, say, a tenth of the normal rate, those 4,800 tokens drop from about 12 paise to about 1.2 paise, and the chat falls to about 11 paise.
- "Where is my order" is an easy question. Route it to a small model at, say, a tenth of the price, and keep the big model for angry customers and tricky refunds. If 70% of chats are easy, the average falls to roughly 4 to 5 paise.
- Cap replies at 150 tokens. The bot was never meant to write essays.
The monthly bill drops from about ₹13 lakh to under ₹3 lakh, and Riya still gets her answer in under 3 seconds. Nobody touched the quality of the hard cases; the savings came from not paying big-model prices to repeat the same rules on easy questions.
What happens when…Situations
Real situations this idea runs into, and how well-built systems handle each one.
What happens when someone pastes a 300-page PDF
A student uploads a 300-page textbook and asks, "What does chapter 7 say about photosynthesis?" At about 600 tokens a page, that's 1.8 lakh tokens. It may not fit in the context window at all. If it does, each question costs about ₹4.50 in input alone at our example price, the first word takes many seconds, and every follow-up question sends all 300 pages again.
Well-built apps don't send the whole book. They split it into small passages once, and for each question they fetch only the few passages that match, a pattern called . Five relevant passages of about 500 tokens each is 2,500 tokens instead of 1.8 lakh: roughly seventy times cheaper per question, and faster too.
What happens when a chat goes on for 200 messages
A user plans a whole wedding in one chat over a week. By message 200, each new call carries tens of thousands of tokens of history: menus that were rejected, venues that were dropped, a guest list from three versions ago. The cost per message has grown many times over, the replies come slower, and the model starts mixing up old and new decisions.
Keep the recent exchanges word for word and fold the older ones into a short running summary of what's been decided. The summary is refreshed every so often. Users notice nothing, except that answers stay quick and stop bringing back the rejected caterer.
What happens when most of your users write in Hindi
An app launches in Hindi and Marathi and finds its token bill per message is about twice the English estimate. Nothing is broken: the tokeniser simply splits those scripts into more pieces. Replies are also slower, because there are more tokens to write, and long chats hit the context ceiling sooner.
Measure with real messages in each language, using the model's own tokeniser, before setting prices or limits. Prefer models whose vocabularies handle Indian scripts well, and test them, since this varies a lot. Price plans per message or per month rather than per token, so users in one language don't pay double for the same help.
What happens when a reasoning model thinks hard about everything
A team switches its whole app to a reasoning model because it scored better on hard questions. The next bill is several times bigger, even though the visible answers are no longer. For "what time does the store close?" the model was writing a thousand tokens of careful thought, all billed as output.
Set a per kind of request: none or low for simple lookups and chit-chat, higher only for multi-step problems. Track output tokens from the provider's usage figures, not from the length of the answer on screen.
What happens when every request goes to the biggest model
An app sends "hi", "thanks" and "what's your refund policy?" to the same large, slow, expensive model it uses for complicated complaints. Most of its traffic is easy, so most of its bill pays for power it doesn't need, and easy answers are slower than they should be.
Put a router in front. A quick check (often a tiny model or simple rules) sorts each message and sends easy ones to a small model and hard ones to the large one. A goes further: try the small model first and move up only if its answer fails a check.
What happens when a bug or an attacker loops the meter
A bug makes an AI agent retry the same failing step all night, each try sending 20,000 tokens. Or someone scripts thousands of maximum-length requests against a free tier. By morning the bill is lakhs of rupees. This is called : the service stays up, but the money runs out.
Put limits at several levels: a per user per day, a maximum size per request, a cap on steps and tokens inside any agent loop, and a spending alert per feature that pages someone within the hour, not at the end of the month.
Watch outCommon mistakes
Estimating the bill from the first message
A team measures one question and one answer, multiplies by expected traffic and signs off the budget. In real use, people have 8-turn conversations, and each turn resends everything before it. The real bill comes in at four or five times the plan, and the feature gets paused in its second month.
Model a whole typical conversation, turn by turn, including the system prompt, history and any documents. Count with the model's own tokeniser, then check the estimate against the provider's actual usage figures in the first week of real traffic.
Putting things that change at the top of the prompt
A developer starts every prompt with "Today is Tuesday, 14:32. User: Riya Sharma." It seems tidy. But prompt caching only works when the opening is exactly the same as a recent call, so that one changing line at the top means the 1,200 tokens of rules after it are never cached, and every call pays full price.
Put the fixed parts first (instructions, tool descriptions, policies), in the same order every time, and the parts that change (time, user details, retrieved passages, the question) at the end.
Letting answers run as long as the model likes
With no output limit and no instruction about length, the model writes three paragraphs where one line would do, or returns a long, wordy format when a short list was needed. Output is the dearest token, and longer answers also take longer to finish.
Ask for the shortest useful answer, use a compact format, and set an output limit that fits the job, say 150 tokens for a support reply. Check that the limit doesn't cut off answers people actually need.
Watching only the monthly invoice
The bill arrives as one big number. Nobody knows that a single "summarise my inbox" feature, used by 3% of users, is eating half of it, or that one customer's integration retries every failed call ten times.
Record input and output tokens for every call, tagged by feature, user and model, a practice called . Look at cost per conversation and per user every day, with alerts when a feature jumps.
Interview angleIn an interview
Token cost comes up in almost every AI design question, often as "estimate what this will cost" or "your bill doubled, what do you do?" Interviewers want to see you do the token sum out loud and know which parts of the prompt drive it.
A strong answer covers:
- a per-call token budget: system prompt, history, retrieved documents, question, and expected output;
- how history makes each turn bigger, and how you'd cap it (a window plus a summary);
- the back-of-the-envelope sum: calls a day × tokens per call × an assumed price, then cost per user;
- speed as two numbers: time to first token (prompt length) and tokens per second (answer length);
- the levers, in order of impact: routing, caching a stable prefix, trimming context, output limits, batching offline work;
- guardrails: per-user limits, per-feature budgets and tracking cost per request.
Follow-up: "Your bill doubled but traffic is flat. What do you check?" Tokens per call by feature, to see what grew: longer prompts, a broken cache (something changing at the top), runaway retries or loops, a switch to a reasoning model, or a new tokeniser after a model upgrade.
Follow-up: "Why is output priced higher than input?" Input is read in one parallel pass, while output is written one token at a time, keeping the GPU busy far longer per token.
Follow-up: "How would you cut cost five times without hurting quality?" Route easy traffic to a small model, cache the fixed prefix, trim history to a summary plus recent turns, and cap output, then confirm with an evaluation set that answers didn't get worse.
Try it in the simulator
Goal: Add GPU servers until answers stop timing out, and watch the monthly cost rise.
60 chats a second arrive, but each GPU server works on only 8 answers at a time, about 0.4 s each: at best about 20 answers a second per server. Two servers manage about 40, so a third of the chats time out. Add GPU servers behind the load balancer, connecting each one to the model store like the others, and run again. Three barely cope; four run comfortably. Watch the monthly cost climb with every server you add: this is what every token costs the company that runs the model.
Hint
In the simulator the GPU servers are ordinary servers set to be slow and limited, just like real ones running a big model. Divide what each can hold at once by how long one answer takes to see why so few answers a second fit.
Quick checkCheck your understanding
Words in this lessonThe jargon, in plain English
Every technical word used above. In the lesson, tap any word with an ⓘ to see its meaning right there.
- Back-of-the-envelope estimate
- A quick, rounded calculation, like 1 crore requests a day ≈ 100 a second, used to decide which parts of a design matter. Close enough beats exact.
- Batch API
- A way to send non-urgent model requests in bulk to be answered within hours, usually at a lower price than instant answers.
- Code-mixing
- Using two languages in one sentence, like 'timer lagao, and play some music'.
- Compaction
- Replacing the older part of a long conversation or agent run with a short summary, so the work can carry on within the context window.
- Context rot
- Answer quality slipping as a context fills with old, irrelevant or conflicting material, even before it reaches the window's limit.
- Context window
- Everything an AI model reads on one call: the app's instructions, the chat so far, any documents and the question. It has a maximum size, counted in tokens.
- Cost attribution
- Tagging every model call with the feature, team or customer it served, so spending can be traced to its source.
- Decode
- The second phase of answering, where the model writes its reply one token at a time, each step waiting for the one before. Its speed sets how fast the words stream out.
- Denial of wallet
- An attack or bug that runs up huge bills by making an AI system do expensive work again and again, such as sending endless very long requests.
- GPU
- A chip with thousands of small cores that does many calculations at once; AI models run on them, and they're expensive.
- LLM
- Large language model: an AI model trained on huge amounts of text that writes answers one word piece at a time.
- Model cascade
- Trying a cheap model first and passing the request to a stronger, dearer one only when the first is unsure or fails a check.
- Model routing
- Choosing which AI model handles each request, for example sending easy questions to a small, cheap model and hard ones to a stronger one.
- Multimodal model
- An AI model that can take in or produce more than text, such as pictures, voice or video.
- Output limit
- The most tokens an app lets an AI model write in one answer. When the limit is reached, the answer stops, even in the middle of a sentence.
- Prefill
- The first phase of answering, where the model reads the whole prompt in one go, in parallel. Longer prompts make prefill take longer, which delays the first word of the reply.
- Prompt caching
- A model provider remembering its work on the opening part of a prompt, so later requests that start with exactly the same text are cheaper and faster.
- RAG
- Retrieval-augmented generation: first finding relevant passages in your own documents, then having an AI answer only from them.
- Reasoning model
- A model trained to work through a problem in hidden steps before it answers. Much better at multi-step problems, but slower and dearer per answer.
- Semantic cache
- A cache that reuses an earlier AI answer when a new question means the same thing, skipping the expensive model.
- Small language model
- A language model small enough to run cheaply or on a phone or laptop, usually trained or distilled for narrower jobs.
- Streaming
- Sending something in small pieces as soon as each piece is ready, instead of waiting until the whole thing is finished.
- System prompt
- The fixed opening instructions an app gives an AI model on every call: its role, rules and tone.
- Thinking budget
- A cap on how much a reasoning model may think for one request, trading answer quality against time and cost. Some providers call it reasoning effort.
- Thinking tokens
- The working-out a reasoning model writes before its final answer. They are usually charged for even when you can't see them.
- Time to first token
- How long an AI model takes to send back the first piece of its answer. It decides how quickly a reply seems to start, even when the full answer takes longer.
- Token (AI)
- A small piece of text, roughly a word or part of one, that an AI model reads and writes one at a time.
- Tokeniser
- The part of an AI system that cuts text into tokens using a fixed vocabulary. Common English words become one token, while numbers, rare words and many Indian-language words split into several.
- Tokens per second
- How fast a model writes once it has started answering. Together with time to first token, it describes how quick a reply feels.
- Unit economics
- The cost and the revenue for one unit of a business, such as one conversation or one paying user, which shows whether growth makes or loses money.
- Usage limit
- How much of a service one person or plan may use in a set time, such as 40 messages or a number of AI tokens a day. It resets at a known time.
Enjoyed this? Keep going for free
This lesson is a free preview. Sign up free to save your progress and earn XP, and to unlock all 232 lessons, the simulator challenges and mock interviews.
Sign up free