Cost per token

On this page 7

Cost per token is the price, in money or compute, of processing one input token or generating one output token with a language model. In AI infrastructure and artificial intelligence economics, it is the basic unit of inference cost: providers bill by it, and teams that run their own models derive it from hardware cost divided by token throughput.

Because a single request may consume thousands of tokens, small differences in cost per token compound into large differences in operating budgets. The metric is usually quoted per million tokens and is almost always split into input and output prices, since the two phases of inference load the hardware very differently.

How cost per token is calculated

For a hosted API, cost per token is simply the published list price, typically stated per million input tokens and per million output tokens. For self-hosted inference, it must be derived: the hourly cost of the serving hardware is divided by the number of tokens served per hour at realistic utilization.

A simplified self-hosted formula is cost per token = hourly infrastructure cost / (tokens per second x 3600 x utilization). The infrastructure cost should include accelerators, host servers, networking, power, cooling and engineering overhead, not only the GPU rental rate. The table shows the main inputs and how each one moves the result.

InputTypical unitEffect on cost per token
Accelerator priceCurrency per GPU hourHigher price raises cost proportionally
Aggregate throughputTokens per second per nodeHigher throughput lowers cost
UtilizationPercent of hours under loadIdle capacity raises effective cost
Model sizeBillions of parametersLarger models need more memory and compute
Context lengthTokens per requestLong contexts consume memory and reduce batch size
Inputs that determine self-hosted cost per token

The most frequently underestimated input is utilization. A cluster that is busy only 30 percent of the time effectively costs more than three times as much per token as the same cluster at full load, which is why low-traffic workloads are often cheaper on shared APIs or serverless inference.

Why input and output tokens are priced differently

Output tokens usually cost several times more than input tokens, commonly between three and five times on commercial price lists. The reason lies in the two phases of transformer inference: the prefill phase and the decode phase.

PropertyPrefill phase (input)Decode phase (output)
Processing patternAll prompt tokens in parallelOne token at a time per sequence
Hardware bottleneckCompute boundMemory bandwidth bound
GPU efficiencyHigh arithmetic utilizationLow utilization per sequence
Latency metricTime to first tokenTime per output token
Relative priceLowerHigher
Prefill and decode phases compared

During decoding, every new token requires reading the model weights and the KV cache from memory, so the accelerator spends much of its time waiting for data. Techniques such as continuous batching, speculative decoding and PagedAttention raise the number of sequences decoded together and therefore lower the effective output price.

Factors and techniques that change cost per token

Cost per token is not a fixed property of a model; it depends heavily on serving configuration. The same model can differ in cost by an order of magnitude between a naive deployment and an optimized one. The following table lists the main levers.

TechniqueMechanismTypical trade-off
QuantizationWeights stored in 8 or 4 bitsPossible small accuracy loss
Prompt cachingReused prefixes skip prefillOnly helps repeated prompts
Batch inferenceAsynchronous jobs at full loadHours of latency instead of seconds
Model routingSimple requests sent to small modelsRouting errors on hard inputs
Model distillationSmaller model trained on large model outputsNarrower capability
Semantic cachingStored answers for similar queriesRisk of stale or mismatched answers
Optimization levers and their typical impact

Providers reflect these levers in pricing tiers: cached input tokens are often billed at a steep discount, and batch APIs commonly cost around half of the interactive price. Reasoning models add another dimension, because their hidden reasoning tokens are billed as output even though the user never sees them.

Cost per token is a unit price, and it can mislead when used alone. Decision makers should pair it with metrics that express cost per useful outcome.

MetricWhat it expressesWhen it is more useful
Cost per tokenPrice of one processed tokenComparing providers for one model
Cost per requestTokens per request times priceBudgeting a specific feature
Cost per taskAll calls needed to finish a jobEvaluating agents and pipelines
Tokens per secondServing throughputCapacity planning
Model FLOPs utilizationShare of peak compute achievedDiagnosing hardware efficiency
Related metrics for AI inference economics

A cheaper model with a lower cost per token can still be more expensive per task if it needs longer prompts, more retries or several agent steps to reach the same quality.

Common mistakes when estimating cost per token

Most budgeting errors come from counting tokens incorrectly or from ignoring overhead. The following mistakes recur in practice.

  • Assuming tokens equal words: English text averages roughly 1.3 tokens per word, and code or non-Latin scripts often need more.
  • Ignoring tokenizer differences: the same text produces different token counts on different model families.
  • Forgetting system prompts and context: retrieved documents and conversation history are billed on every call.
  • Overlooking reasoning tokens: hidden reasoning can multiply output volume.
  • Using peak throughput: benchmarks at full batch size overstate the throughput real traffic achieves.
  • Comparing list prices only: quality, latency targets and rate limits change the real cost.

A reliable estimate therefore measures actual token logs from a pilot, separates input, cached input and output volumes, and applies the real utilization expected in production rather than benchmark figures.

The simple explanation

A language model reads and writes text in small pieces called tokens, each about three quarters of an English word. Cost per token is what each of those pieces costs, much like a phone plan that charges by the minute.

Reading your question is cheaper than writing the answer, because writing happens one piece at a time. The longer the prompts and answers, the bigger the bill, so careful teams keep prompts short and reuse what they can.

How it is used in practice

An online retailer plans a support assistant that will handle about 50,000 conversations per day. A pilot shows each conversation averages 3,000 input tokens, mostly a fixed system prompt and retrieved policy text, and 400 output tokens. That is roughly 150 million input and 20 million output tokens daily, and at the chosen list prices the output share alone is substantial because output costs four times more per token.

The infrastructure team enables prompt caching for the fixed system prompt, routes simple order status questions to a smaller model, and moves overnight summary reports to a batch API. Measured over the next month, the blended cost per conversation falls by more than half, while customer satisfaction scores stay unchanged.