A short question can produce a surprisingly complicated AI bill. The words visible in a chat box are only part of what a model may need to process. Behind them could sit system instructions, earlier messages, uploaded documents, search results and rules for any connected tools.
Most commercial AI services meter this activity using tokens. Understanding what goes in, what comes out and what can be reused helps explain why two apparently similar AI features may have very different costs.
What is a token?
A token is a small piece of information that an AI model processes. In written English, it might be a whole word, part of a word, punctuation or even a space attached to another fragment. The exact split depends on the model and language.
OpenAI's introductory guidance suggests that one token is roughly four characters, or around three quarters of an English word. That is a useful estimate, not an invoice calculator. Product names, unusual spellings, numbers and languages other than English can produce different results.
When someone enters a prompt, the text is converted into tokens. The model processes those tokens and generates its response as another sequence of tokens. This gives us the two basic parts of usage: tokens in and tokens out.
Tokens in are more than the visible question
If a user asks, "Can I return this item?", the application may send far more than those six words. It might also include:
- Instructions that define the assistant's role and boundaries
- Earlier messages from the conversation
- Relevant sections from policies, product data or uploaded documents
- Descriptions of tools the assistant is allowed to use
- Information returned by searches, databases or connected systems
All of this becomes context: the information available to the model for that request. A larger context can improve an answer when the material is relevant. Feeding in everything available, however, can increase cost, slow processing and sometimes make it harder for the model to focus on the useful details.
Conversation history deserves particular attention. A long-running assistant may resend much of the previous conversation each time a new message arrives. Cached material can be cheaper to process, but it still occupies space in the model's context window, as Anthropic's context-window guidance makes clear.
Tokens out are created one step at a time
AI text is not normally written as a complete paragraph in one movement. The model predicts a suitable next token, adds it to the sequence, then repeats the process. That is why answers appear progressively and why a long response takes more generation work than a short one.
Output can also be priced differently from input. OpenAI's API pricing, for example, lists separate rates for input, cached input and output, with the exact figures varying by model and processing option. Other providers use their own combinations of input charges, output charges, cache charges and tool fees.
Some models also use additional reasoning or "thinking" tokens before returning the visible answer. The treatment varies by provider and model, so estimates should be based on the actual service rather than a simple words-in, words-out calculation.
Where KV caching fits
Generating each new token involves looking back at the tokens already processed. Recalculating all of that work from scratch at every step would be wasteful. A key-value cache, usually shortened to KV cache, stores internal calculations for earlier tokens so they can be reused while the response is being produced. NVIDIA's technical explanation describes this as reusing already computed keys and values for future tokens.
For a non-technical analogy, imagine someone taking useful notes while reading a long briefing. When they reach the final page, they can consult the notes rather than rereading every previous page before writing each sentence.
Prompt caching is related but easier to see from a product and billing perspective. If many requests begin with the same instructions, examples or reference material, a provider may be able to reuse work from that shared opening. OpenAI's prompt-caching guidance explains that cache hits depend on matching prompt prefixes, which is why stable material is usually placed first and changing customer-specific information later.
Caching is useful, but it is not a magic free lane. Providers differ in how long cached content is kept, what qualifies for a match and how reads, writes or storage are charged. It also does not remove the cost of new input or generated output.
How token costs quietly multiply
The rough shape of an AI bill is easy to state:
number of requests × input used × output generated × model rate
Real systems add more branches to the tree. One customer action might trigger a classification request, a database search, a second model call to draft an answer and a final call to check it. Web search, image analysis, audio, document storage and other tools may have separate charges.
Retries also matter. An inexpensive request that fails often can cost more than a better-designed request using a slightly more capable model. Cost should therefore be measured against successful outcomes, not simply against the cheapest price per million tokens.
Using AI shrewdly
Shrewd token use is not about stripping every prompt down until the answer becomes unreliable. It is about paying for information and processing that genuinely help the task.
- Choose the model for the job. Routine classification or summarisation may not need the most capable and expensive option. Test smaller models against real examples rather than assuming bigger is always better.
- Retrieve what is relevant. A support assistant may need three paragraphs from a handbook, not the entire handbook on every request.
- Manage conversation history. Older exchanges can sometimes be summarised, removed or stored outside the active context, provided important decisions and facts are preserved.
- Keep reusable instructions stable. Consistent prompt openings can improve the chance of benefiting from prompt caching.
- Ask for the output the process needs. A two-sentence customer reply does not require a 1,000-word internal essay.
- Count the whole workflow. Include retries, background calls, tool usage, cache storage and human review when estimating the real cost.
- Monitor cost by feature and outcome. A monthly total is less useful than knowing which feature, customer journey or automated process is consuming the tokens.
AI costs can look tiny at the level of one request and substantial at the scale of thousands of customers or repeated internal workflows. The sensible question is not simply, "How do we use fewer tokens?" It is, "Which tokens are helping us produce a reliable business result, and which are merely along for the ride?"
