An ML pipeline, or machine learning pipeline, is an automated and versioned sequence of steps that moves data through ingestion, validation, feature engineering, training, evaluation and deployment, so that a model can be rebuilt reproducibly whenever data or code changes. In AI infrastructure, the pipeline is the unit that turns experimentation into a dependable production process.
Each step is a self-contained component with defined inputs, outputs and dependencies, usually arranged as a directed acyclic graph and executed by an orchestrator. The pipeline records which data, code and parameters produced each model, which is the foundation of auditability and MLOps practice in artificial intelligence.
How an ML pipeline is structured
An ML pipeline is structured as a chain of stages connected by artifacts: each stage reads versioned inputs, performs one job and writes outputs that the next stage consumes. Because stages communicate only through stored artifacts, any stage can be cached, rerun or replaced without repeating the whole workflow.
The orchestrator resolves the dependency graph, schedules containers or jobs on compute resources, retries failed steps and logs metadata. Typical triggers are a schedule, the arrival of new data, a code commit or an alert from model monitoring that signals drift.
| Stage | Purpose | Typical output |
|---|---|---|
| Data ingestion | Collect raw data from sources | Versioned dataset snapshot |
| Data validation | Check schema, ranges and distributions | Validation report, pass or fail |
| Feature engineering | Transform data into model inputs | Feature tables or feature store entries |
| Training | Fit model parameters | Model weights and training logs |
| Evaluation | Compare against metrics and baselines | Metrics report and approval decision |
| Registration and deployment | Publish an approved model | Model registry entry, serving endpoint |
Types of ML pipelines
The term covers several distinct pipelines that share tooling but serve different moments in the model lifecycle. Confusing them is a common source of design errors, especially between the training pipeline and the inference path.
| Variant | Runs when | Main concern |
|---|---|---|
| Data pipeline | Continuously or on schedule | Freshness and data quality |
| Training pipeline | New data, code change or drift | Reproducibility and compute cost |
| Batch inference pipeline | Hourly, nightly or on demand | Throughput and completeness |
| Online inference path | Per request | Latency and availability |
| Fine-tuning pipeline | New domain data for a foundation model | GPU cost and evaluation quality |
For large language models, the pipeline often adds stages for data deduplication, filtering and tokenization, distributed training across a GPU cluster, and safety or quality evaluations before registration. These runs can occupy hundreds of accelerators for days, so checkpointing and resumability become first-class requirements.
Mature teams also distinguish the pipeline definition from a pipeline run. The definition is versioned code describing the graph, while each run is an immutable record of inputs, parameters, outputs and timings. This separation allows an auditor to answer which exact run produced a model that is serving traffic today, and it lets engineers compare two runs side by side when quality changes unexpectedly.
ML pipeline compared with neighboring concepts
An ML pipeline is a concrete, executable workflow, whereas MLOps is the broader discipline of operating models in production. The pipeline sits between components such as the feature store, experiment tracking and the model registry, and it hands its output to model serving.
| Concept | Scope | Relation to the pipeline |
|---|---|---|
| MLOps | Practices, roles and tooling | Pipelines are its main automation unit |
| ETL pipeline | Data movement and transformation | Feeds the ML pipeline, ends before training |
| CI/CD pipeline | Building and releasing software | Tests and deploys pipeline code itself |
| Experiment tracking | Records of runs and metrics | Stores metadata the pipeline emits |
| Model registry | Catalog of approved model versions | Receives the pipeline’s final artifact |
A key distinction from ordinary software CI/CD is that ML pipelines must version data and models in addition to code. Two runs of identical code on different data can produce different models, so lineage records are essential.
Designing a reliable ML pipeline
A reliable ML pipeline is deterministic where possible, observable everywhere and gated at critical points. Practitioners pin library versions in container images, fix random seeds, snapshot datasets and store every artifact with a content hash.
- Use the same feature code for training and serving to prevent training-serving skew.
- Add automated quality gates: block deployment when metrics fall below the current production model.
- Cache expensive steps so that a change in evaluation does not retrain the model.
- Keep a human approval step for high-risk models, especially those that affect people.
- Emit lineage metadata linking each model to its data snapshot, code commit and parameters.
Orchestration options range from general workflow engines such as Apache Airflow to ML-specific frameworks such as Kubeflow Pipelines, TensorFlow Extended or managed cloud services. The choice depends on team skills, existing infrastructure and the need for GPU scheduling more than on feature lists.
Common failure modes
Most pipeline failures are silent rather than crashing: the run completes, but the model is worse. The table summarizes symptoms that experienced teams watch for.
| Symptom | Likely cause | Fix |
|---|---|---|
| Offline metrics high, production poor | Training-serving skew | Share feature logic, log serving inputs |
| Unrealistically strong evaluation | Data leakage across splits | Split by time or entity, audit features |
| Results cannot be reproduced | Unversioned data or dependencies | Snapshot data, pin environments |
| Sudden quality drop after retraining | Upstream schema change | Strict data validation stage |
| Runaway compute bills | No caching or failed retries looping | Step caching, retry limits, budgets |
A further risk is pipeline sprawl: dozens of slightly different copies maintained by separate teams. Shared, parameterized templates reduce maintenance and keep governance rules consistent.
Put simply
An ML pipeline is like a production line for AI models. Raw data goes in at one end. Stations along the line check it, clean it, teach the model and test it. A finished model comes out at the other end.
The main benefit is repeatability. When new data arrives, the whole line can run again on its own. It produces an updated model and keeps a record of exactly how it was made.
In practice
A regional grocery chain forecasts daily demand for about 20,000 products across 150 stores. Analysts originally retrained the forecasting model by hand in notebooks every few weeks, and a forgotten preprocessing step once caused two weeks of overstocked produce.
The data team rebuilt the process as an ML pipeline that runs every night: it ingests sales and promotion data, validates the schema and value ranges, computes features, retrains a gradient-boosted model, and compares its error with the current production model on the last 28 days. Only a better model is promoted to the registry and the batch inference job. After three months, forecast error fell by roughly 8 percent, and a supplier data format change was caught by the validation stage before it reached any store order.