Cost per token is the price, in money or compute, of processing one input token or generating one output token with a language model. In AI infrastructure and artificial intelligence economics, it is the basic unit of inference cost: providers bill by it, and teams that run their own models derive it from hardware cost divided by token throughput.
Because a single request may consume thousands of tokens, small differences in cost per token compound into large differences in operating budgets. The metric is usually quoted per million tokens and is almost always split into input and output prices, since the two phases of inference load the hardware very differently.
How cost per token is calculated
For a hosted API, cost per token is simply the published list price, typically stated per million input tokens and per million output tokens. For self-hosted inference, it must be derived: the hourly cost of the serving hardware is divided by the number of tokens served per hour at realistic utilization.
A simplified self-hosted formula is cost per token = hourly infrastructure cost / (tokens per second x 3600 x utilization). The infrastructure cost should include accelerators, host servers, networking, power, cooling and engineering overhead, not only the GPU rental rate. The table shows the main inputs and how each one moves the result.
| Input | Typical unit | Effect on cost per token |
|---|---|---|
| Accelerator price | Currency per GPU hour | Higher price raises cost proportionally |
| Aggregate throughput | Tokens per second per node | Higher throughput lowers cost |
| Utilization | Percent of hours under load | Idle capacity raises effective cost |
| Model size | Billions of parameters | Larger models need more memory and compute |
| Context length | Tokens per request | Long contexts consume memory and reduce batch size |
The most frequently underestimated input is utilization. A cluster that is busy only 30 percent of the time effectively costs more than three times as much per token as the same cluster at full load, which is why low-traffic workloads are often cheaper on shared APIs or serverless inference.
Why input and output tokens are priced differently
Output tokens usually cost several times more than input tokens, commonly between three and five times on commercial price lists. The reason lies in the two phases of transformer inference: the prefill phase and the decode phase.
| Property | Prefill phase (input) | Decode phase (output) |
|---|---|---|
| Processing pattern | All prompt tokens in parallel | One token at a time per sequence |
| Hardware bottleneck | Compute bound | Memory bandwidth bound |
| GPU efficiency | High arithmetic utilization | Low utilization per sequence |
| Latency metric | Time to first token | Time per output token |
| Relative price | Lower | Higher |
During decoding, every new token requires reading the model weights and the KV cache from memory, so the accelerator spends much of its time waiting for data. Techniques such as continuous batching, speculative decoding and PagedAttention raise the number of sequences decoded together and therefore lower the effective output price.
Factors and techniques that change cost per token
Cost per token is not a fixed property of a model; it depends heavily on serving configuration. The same model can differ in cost by an order of magnitude between a naive deployment and an optimized one. The following table lists the main levers.
| Technique | Mechanism | Typical trade-off |
|---|---|---|
| Quantization | Weights stored in 8 or 4 bits | Possible small accuracy loss |
| Prompt caching | Reused prefixes skip prefill | Only helps repeated prompts |
| Batch inference | Asynchronous jobs at full load | Hours of latency instead of seconds |
| Model routing | Simple requests sent to small models | Routing errors on hard inputs |
| Model distillation | Smaller model trained on large model outputs | Narrower capability |
| Semantic caching | Stored answers for similar queries | Risk of stale or mismatched answers |
Providers reflect these levers in pricing tiers: cached input tokens are often billed at a steep discount, and batch APIs commonly cost around half of the interactive price. Reasoning models add another dimension, because their hidden reasoning tokens are billed as output even though the user never sees them.
Cost per token compared with related cost metrics
Cost per token is a unit price, and it can mislead when used alone. Decision makers should pair it with metrics that express cost per useful outcome.
| Metric | What it expresses | When it is more useful |
|---|---|---|
| Cost per token | Price of one processed token | Comparing providers for one model |
| Cost per request | Tokens per request times price | Budgeting a specific feature |
| Cost per task | All calls needed to finish a job | Evaluating agents and pipelines |
| Tokens per second | Serving throughput | Capacity planning |
| Model FLOPs utilization | Share of peak compute achieved | Diagnosing hardware efficiency |
A cheaper model with a lower cost per token can still be more expensive per task if it needs longer prompts, more retries or several agent steps to reach the same quality.
Common mistakes when estimating cost per token
Most budgeting errors come from counting tokens incorrectly or from ignoring overhead. The following mistakes recur in practice.
- Assuming tokens equal words: English text averages roughly 1.3 tokens per word, and code or non-Latin scripts often need more.
- Ignoring tokenizer differences: the same text produces different token counts on different model families.
- Forgetting system prompts and context: retrieved documents and conversation history are billed on every call.
- Overlooking reasoning tokens: hidden reasoning can multiply output volume.
- Using peak throughput: benchmarks at full batch size overstate the throughput real traffic achieves.
- Comparing list prices only: quality, latency targets and rate limits change the real cost.
A reliable estimate therefore measures actual token logs from a pilot, separates input, cached input and output volumes, and applies the real utilization expected in production rather than benchmark figures.
The simple explanation
A language model reads and writes text in small pieces called tokens, each about three quarters of an English word. Cost per token is what each of those pieces costs, much like a phone plan that charges by the minute.
Reading your question is cheaper than writing the answer, because writing happens one piece at a time. The longer the prompts and answers, the bigger the bill, so careful teams keep prompts short and reuse what they can.
How it is used in practice
An online retailer plans a support assistant that will handle about 50,000 conversations per day. A pilot shows each conversation averages 3,000 input tokens, mostly a fixed system prompt and retrieved policy text, and 400 output tokens. That is roughly 150 million input and 20 million output tokens daily, and at the chosen list prices the output share alone is substantial because output costs four times more per token.
The infrastructure team enables prompt caching for the fixed system prompt, routes simple order status questions to a smaller model, and moves overnight summary reports to a batch API. Measured over the next month, the blended cost per conversation falls by more than half, while customer satisfaction scores stay unchanged.