ML pipeline

Also known as: Machine learning pipeline

On this page 7

An ML pipeline, or machine learning pipeline, is an automated and versioned sequence of steps that moves data through ingestion, validation, feature engineering, training, evaluation and deployment, so that a model can be rebuilt reproducibly whenever data or code changes. In AI infrastructure, the pipeline is the unit that turns experimentation into a dependable production process.

Each step is a self-contained component with defined inputs, outputs and dependencies, usually arranged as a directed acyclic graph and executed by an orchestrator. The pipeline records which data, code and parameters produced each model, which is the foundation of auditability and MLOps practice in artificial intelligence.

How an ML pipeline is structured

An ML pipeline is structured as a chain of stages connected by artifacts: each stage reads versioned inputs, performs one job and writes outputs that the next stage consumes. Because stages communicate only through stored artifacts, any stage can be cached, rerun or replaced without repeating the whole workflow.

The orchestrator resolves the dependency graph, schedules containers or jobs on compute resources, retries failed steps and logs metadata. Typical triggers are a schedule, the arrival of new data, a code commit or an alert from model monitoring that signals drift.

StagePurposeTypical output
Data ingestionCollect raw data from sourcesVersioned dataset snapshot
Data validationCheck schema, ranges and distributionsValidation report, pass or fail
Feature engineeringTransform data into model inputsFeature tables or feature store entries
TrainingFit model parametersModel weights and training logs
EvaluationCompare against metrics and baselinesMetrics report and approval decision
Registration and deploymentPublish an approved modelModel registry entry, serving endpoint
Core stages of a training pipeline

Types of ML pipelines

The term covers several distinct pipelines that share tooling but serve different moments in the model lifecycle. Confusing them is a common source of design errors, especially between the training pipeline and the inference path.

VariantRuns whenMain concern
Data pipelineContinuously or on scheduleFreshness and data quality
Training pipelineNew data, code change or driftReproducibility and compute cost
Batch inference pipelineHourly, nightly or on demandThroughput and completeness
Online inference pathPer requestLatency and availability
Fine-tuning pipelineNew domain data for a foundation modelGPU cost and evaluation quality
Pipeline variants and when they run

For large language models, the pipeline often adds stages for data deduplication, filtering and tokenization, distributed training across a GPU cluster, and safety or quality evaluations before registration. These runs can occupy hundreds of accelerators for days, so checkpointing and resumability become first-class requirements.

Mature teams also distinguish the pipeline definition from a pipeline run. The definition is versioned code describing the graph, while each run is an immutable record of inputs, parameters, outputs and timings. This separation allows an auditor to answer which exact run produced a model that is serving traffic today, and it lets engineers compare two runs side by side when quality changes unexpectedly.

ML pipeline compared with neighboring concepts

An ML pipeline is a concrete, executable workflow, whereas MLOps is the broader discipline of operating models in production. The pipeline sits between components such as the feature store, experiment tracking and the model registry, and it hands its output to model serving.

ConceptScopeRelation to the pipeline
MLOpsPractices, roles and toolingPipelines are its main automation unit
ETL pipelineData movement and transformationFeeds the ML pipeline, ends before training
CI/CD pipelineBuilding and releasing softwareTests and deploys pipeline code itself
Experiment trackingRecords of runs and metricsStores metadata the pipeline emits
Model registryCatalog of approved model versionsReceives the pipeline’s final artifact
How the ML pipeline relates to adjacent infrastructure

A key distinction from ordinary software CI/CD is that ML pipelines must version data and models in addition to code. Two runs of identical code on different data can produce different models, so lineage records are essential.

Designing a reliable ML pipeline

A reliable ML pipeline is deterministic where possible, observable everywhere and gated at critical points. Practitioners pin library versions in container images, fix random seeds, snapshot datasets and store every artifact with a content hash.

  • Use the same feature code for training and serving to prevent training-serving skew.
  • Add automated quality gates: block deployment when metrics fall below the current production model.
  • Cache expensive steps so that a change in evaluation does not retrain the model.
  • Keep a human approval step for high-risk models, especially those that affect people.
  • Emit lineage metadata linking each model to its data snapshot, code commit and parameters.

Orchestration options range from general workflow engines such as Apache Airflow to ML-specific frameworks such as Kubeflow Pipelines, TensorFlow Extended or managed cloud services. The choice depends on team skills, existing infrastructure and the need for GPU scheduling more than on feature lists.

Common failure modes

Most pipeline failures are silent rather than crashing: the run completes, but the model is worse. The table summarizes symptoms that experienced teams watch for.

SymptomLikely causeFix
Offline metrics high, production poorTraining-serving skewShare feature logic, log serving inputs
Unrealistically strong evaluationData leakage across splitsSplit by time or entity, audit features
Results cannot be reproducedUnversioned data or dependenciesSnapshot data, pin environments
Sudden quality drop after retrainingUpstream schema changeStrict data validation stage
Runaway compute billsNo caching or failed retries loopingStep caching, retry limits, budgets
Symptoms, causes and fixes

A further risk is pipeline sprawl: dozens of slightly different copies maintained by separate teams. Shared, parameterized templates reduce maintenance and keep governance rules consistent.

Put simply

An ML pipeline is like a production line for AI models. Raw data goes in at one end. Stations along the line check it, clean it, teach the model and test it. A finished model comes out at the other end.

The main benefit is repeatability. When new data arrives, the whole line can run again on its own. It produces an updated model and keeps a record of exactly how it was made.

In practice

A regional grocery chain forecasts daily demand for about 20,000 products across 150 stores. Analysts originally retrained the forecasting model by hand in notebooks every few weeks, and a forgotten preprocessing step once caused two weeks of overstocked produce.

The data team rebuilt the process as an ML pipeline that runs every night: it ingests sales and promotion data, validates the schema and value ranges, computes features, retrains a gradient-boosted model, and compares its error with the current production model on the last 28 days. Only a better model is promoted to the registry and the batch inference job. After three months, forecast error fell by roughly 8 percent, and a supplier data format change was caught by the validation stage before it reached any store order.