Batch inference

Also known as: offline inference

On this page 7

Batch inference is the practice of running a trained model over a large, predefined collection of inputs as one job and storing the predictions for later use, rather than answering individual requests as they arrive. In AI infrastructure, it is also called offline inference, and it is the standard way to score customer databases, embed document archives, label datasets or process millions of prompts with a language model.

Because nobody waits for an individual answer, batch inference optimizes for throughput and cost per prediction instead of latency. Jobs can run on cheaper or preemptible hardware, use very large batch sizes and finish within a window measured in minutes to hours, which often makes them several times cheaper than equivalent real-time serving.

How a batch inference job runs

A batch inference job reads input records from storage, runs the model on them in groups and writes the outputs back to storage. The job is usually triggered by a scheduler, a data pipeline or an API call that submits a file of requests, and it ends when every record has a result or has been marked as failed.

  1. Input staging: records are read from a data warehouse, object storage or a JSON Lines file and split into shards.
  2. Preprocessing: each shard is tokenized, resized or normalized exactly as during training.
  3. Model execution: workers load the model once and process inputs in large batches to keep accelerators fully utilized.
  4. Postprocessing and write-back: predictions are decoded, joined with record identifiers and written to a table or file.
  5. Validation and retry: failed shards are retried, and summary statistics are checked before results are published.

The key engineering property is idempotence: each shard can be re-run without producing duplicates, so a job can survive worker failures, spot instance reclamation or a partial outage without restarting from the beginning.

Batch inference compared with online serving

Batch inference and online inference use the same model but optimize opposite metrics. Online serving through an inference endpoint must answer each request within a latency target, whereas a batch job only has to finish before a deadline.

PropertyBatch inferenceOnline inference
Primary goalMaximum throughput per dollarLow latency per request
TriggerSchedule, pipeline event or submitted fileIndividual user or service request
Typical latencyMinutes to about 24 hours per jobMilliseconds to a few seconds
Batch sizeLarge and fixed, limited by memorySmall and dynamic, formed by the server
HardwareSpot or preemptible capacity, shared clustersReserved capacity sized for peaks
Freshness of resultsAs old as the last job runComputed from current input
Failure handlingRetry shards laterMust fail fast or fall back
Batch inference versus online inference

Techniques such as dynamic batching and continuous batching blur this line inside an online server, but they still serve live requests. True batch inference differs because the full workload is known in advance, which allows sorting, packing and scheduling that are impossible for unpredictable traffic.

Throughput levers and cost drivers

The cost of batch inference is determined mainly by how fully the accelerators are used for the duration of the job. Every idle second on a GPU is paid for but produces nothing, so most optimization targets utilization.

ParameterTypical settingEffect
Batch sizeLargest size that fits in memoryRaises throughput until compute or memory bandwidth saturates
Sequence length sortingGroup inputs of similar lengthReduces padding waste in language models
Numeric precisionFP16, BF16, INT8 or FP8Lower precision increases throughput, with a quality check needed
Shard sizeMinutes of work per shardBalances retry cost against scheduling overhead
Data loading parallelismSeveral reader threads per workerPrevents the GPU from waiting on input
Hardware classSpot, preemptible or off-peak capacityOften cuts compute cost substantially
Parameters that shape batch inference efficiency

Several hosted language model APIs offer a batch tier with a discount of about half the standard per-token price in exchange for a completion window of up to 24 hours. The discount reflects the provider’s freedom to schedule the work into idle capacity, and the exact terms vary by provider and change over time.

When to choose batch inference

Batch inference is the right choice when predictions can be computed before they are needed and the input set is known at job time. The decision table below summarizes common situations.

SituationRecommended modeReason
Nightly churn scores for all customersBatch inferenceInputs are fixed and results are read the next day
Embedding a document archive for retrievalBatch inferenceMillions of items, no user waiting
Evaluating a new model on a test setBatch inferenceReproducible, cheap, easy to compare
Chat assistant answering a userOnline inferenceA person waits for each reply
Fraud check during card paymentOnline inferenceDepends on the current transaction
Product recommendations on a homepageHybridPrecompute candidates in batch, rerank online
Choosing between batch and online inference

The hybrid pattern is common in production: an expensive model precomputes scores or embeddings in batch, and a lightweight online step combines them with fresh signals at request time.

Risks and common mistakes

The main weakness of batch inference is stale predictions: results reflect the data and model at job time, so a customer who changed behavior this morning still carries yesterday’s score. Teams should state the acceptable staleness explicitly and schedule jobs accordingly.

  • Training and serving skew: preprocessing in the batch job differs from training code, silently degrading accuracy.
  • Silent partial failure: some shards fail and the output table is published incomplete; row counts must be reconciled.
  • Unversioned outputs: predictions are overwritten without recording which model version produced them, which breaks auditing and rollback.
  • Underused hardware: slow data loading leaves GPUs idle, making the job far more expensive than planned.
  • Missing monitoring: no checks on prediction distributions, so model drift goes unnoticed between runs.

Tying each output to a model registry version and an input snapshot makes batch results reproducible and simplifies investigations when downstream metrics change.

A plain-language summary

Batch inference means asking a model many questions at once, in advance, and saving the answers. It is like a bakery that bakes all the bread before the shop opens instead of baking each loaf when a customer walks in.

Because nobody is standing at the counter waiting, the work can be done at night, on cheaper machines and in large trays. The result is much lower cost per answer, with one catch: the answers are only as fresh as the last baking run.

An example from practice

An online retailer wants a short, consistent description and a set of category tags for two million product listings written by different sellers. Calling a language model live for every listing view would be slow and expensive, and the descriptions rarely change.

The data team exports the listings to a JSON Lines file each night, submits it to a batch tier of a language model API and runs an embedding model in batch on a GPU cluster using spot capacity. Results are written to a table keyed by listing ID and model version. The full catalog is processed within a few hours, the per-token cost is roughly half the interactive price, and the storefront reads precomputed tags with no added page latency. Listings edited during the day are picked up in the next nightly run, which the business accepted as a reasonable delay.