Batch inference is the practice of running a trained model over a large, predefined collection of inputs as one job and storing the predictions for later use, rather than answering individual requests as they arrive. In AI infrastructure, it is also called offline inference, and it is the standard way to score customer databases, embed document archives, label datasets or process millions of prompts with a language model.
Because nobody waits for an individual answer, batch inference optimizes for throughput and cost per prediction instead of latency. Jobs can run on cheaper or preemptible hardware, use very large batch sizes and finish within a window measured in minutes to hours, which often makes them several times cheaper than equivalent real-time serving.
How a batch inference job runs
A batch inference job reads input records from storage, runs the model on them in groups and writes the outputs back to storage. The job is usually triggered by a scheduler, a data pipeline or an API call that submits a file of requests, and it ends when every record has a result or has been marked as failed.
- Input staging: records are read from a data warehouse, object storage or a JSON Lines file and split into shards.
- Preprocessing: each shard is tokenized, resized or normalized exactly as during training.
- Model execution: workers load the model once and process inputs in large batches to keep accelerators fully utilized.
- Postprocessing and write-back: predictions are decoded, joined with record identifiers and written to a table or file.
- Validation and retry: failed shards are retried, and summary statistics are checked before results are published.
The key engineering property is idempotence: each shard can be re-run without producing duplicates, so a job can survive worker failures, spot instance reclamation or a partial outage without restarting from the beginning.
Batch inference compared with online serving
Batch inference and online inference use the same model but optimize opposite metrics. Online serving through an inference endpoint must answer each request within a latency target, whereas a batch job only has to finish before a deadline.
| Property | Batch inference | Online inference |
|---|---|---|
| Primary goal | Maximum throughput per dollar | Low latency per request |
| Trigger | Schedule, pipeline event or submitted file | Individual user or service request |
| Typical latency | Minutes to about 24 hours per job | Milliseconds to a few seconds |
| Batch size | Large and fixed, limited by memory | Small and dynamic, formed by the server |
| Hardware | Spot or preemptible capacity, shared clusters | Reserved capacity sized for peaks |
| Freshness of results | As old as the last job run | Computed from current input |
| Failure handling | Retry shards later | Must fail fast or fall back |
Techniques such as dynamic batching and continuous batching blur this line inside an online server, but they still serve live requests. True batch inference differs because the full workload is known in advance, which allows sorting, packing and scheduling that are impossible for unpredictable traffic.
Throughput levers and cost drivers
The cost of batch inference is determined mainly by how fully the accelerators are used for the duration of the job. Every idle second on a GPU is paid for but produces nothing, so most optimization targets utilization.
| Parameter | Typical setting | Effect |
|---|---|---|
| Batch size | Largest size that fits in memory | Raises throughput until compute or memory bandwidth saturates |
| Sequence length sorting | Group inputs of similar length | Reduces padding waste in language models |
| Numeric precision | FP16, BF16, INT8 or FP8 | Lower precision increases throughput, with a quality check needed |
| Shard size | Minutes of work per shard | Balances retry cost against scheduling overhead |
| Data loading parallelism | Several reader threads per worker | Prevents the GPU from waiting on input |
| Hardware class | Spot, preemptible or off-peak capacity | Often cuts compute cost substantially |
Several hosted language model APIs offer a batch tier with a discount of about half the standard per-token price in exchange for a completion window of up to 24 hours. The discount reflects the provider’s freedom to schedule the work into idle capacity, and the exact terms vary by provider and change over time.
When to choose batch inference
Batch inference is the right choice when predictions can be computed before they are needed and the input set is known at job time. The decision table below summarizes common situations.
| Situation | Recommended mode | Reason |
|---|---|---|
| Nightly churn scores for all customers | Batch inference | Inputs are fixed and results are read the next day |
| Embedding a document archive for retrieval | Batch inference | Millions of items, no user waiting |
| Evaluating a new model on a test set | Batch inference | Reproducible, cheap, easy to compare |
| Chat assistant answering a user | Online inference | A person waits for each reply |
| Fraud check during card payment | Online inference | Depends on the current transaction |
| Product recommendations on a homepage | Hybrid | Precompute candidates in batch, rerank online |
The hybrid pattern is common in production: an expensive model precomputes scores or embeddings in batch, and a lightweight online step combines them with fresh signals at request time.
Risks and common mistakes
The main weakness of batch inference is stale predictions: results reflect the data and model at job time, so a customer who changed behavior this morning still carries yesterday’s score. Teams should state the acceptable staleness explicitly and schedule jobs accordingly.
- Training and serving skew: preprocessing in the batch job differs from training code, silently degrading accuracy.
- Silent partial failure: some shards fail and the output table is published incomplete; row counts must be reconciled.
- Unversioned outputs: predictions are overwritten without recording which model version produced them, which breaks auditing and rollback.
- Underused hardware: slow data loading leaves GPUs idle, making the job far more expensive than planned.
- Missing monitoring: no checks on prediction distributions, so model drift goes unnoticed between runs.
Tying each output to a model registry version and an input snapshot makes batch results reproducible and simplifies investigations when downstream metrics change.
A plain-language summary
Batch inference means asking a model many questions at once, in advance, and saving the answers. It is like a bakery that bakes all the bread before the shop opens instead of baking each loaf when a customer walks in.
Because nobody is standing at the counter waiting, the work can be done at night, on cheaper machines and in large trays. The result is much lower cost per answer, with one catch: the answers are only as fresh as the last baking run.
An example from practice
An online retailer wants a short, consistent description and a set of category tags for two million product listings written by different sellers. Calling a language model live for every listing view would be slow and expensive, and the descriptions rarely change.
The data team exports the listings to a JSON Lines file each night, submits it to a batch tier of a language model API and runs an embedding model in batch on a GPU cluster using spot capacity. Results are written to a table keyed by listing ID and model version. The full catalog is processed within a few hours, the per-token cost is roughly half the interactive price, and the storefront reads precomputed tags with no added page latency. Listings edited during the day are picked up in the next nightly run, which the business accepted as a reasonable delay.