MLflow

On this page 8

MLflow is an open-source platform for managing the machine learning lifecycle: it records experiments, packages trained models in a standard format, keeps a versioned registry of those models and, in recent releases, traces and evaluates generative AI applications. Among AI frameworks & tools in artificial intelligence, it does not train models itself; it sits beside training libraries such as PyTorch, scikit-learn or XGBoost and captures what they produce.

Databricks released MLflow in 2018, and the project later moved to the Linux Foundation. It is written mainly in Python, offers REST, Java and R interfaces, and can run on a laptop, on a self-hosted server or as a managed service. Its central promise is reproducibility: any model in production can be traced back to the code version, parameters, data references and metrics that created it.

How MLflow organizes the machine learning lifecycle

MLflow organizes work into loosely coupled components that can be adopted one at a time. A team may start with experiment tracking alone and add the registry or serving later, because each component communicates through the same tracking server and storage layer.

The table below lists the main components and the problem each one solves.

ComponentWhat it stores or doesTypical user
MLflow TrackingRuns with parameters, metrics, tags and artifactsData scientist
MLflow ModelsA standard model directory with an MLmodel descriptorML engineer
Model RegistryNamed models, numbered versions, aliases and tagsML platform team
MLflow ProjectsReproducible runs defined by an MLproject fileResearcher
EvaluationMetrics and judges applied to models or promptsQuality reviewer
TracingSpans of LLM calls, tools and retrieval stepsGenAI developer
Core components of MLflow

Behind every component sit two stores. The backend store, usually a SQL database such as PostgreSQL, MySQL or SQLite, holds metadata like run identifiers and metric values. The artifact store, usually object storage such as Amazon S3, Azure Blob Storage or Google Cloud Storage, holds large files such as model weights, plots and datasets.

Experiment tracking with runs and autologging

Experiment tracking in MLflow records each training attempt as a run inside a named experiment. Code calls functions such as mlflow.log_param, mlflow.log_metric and mlflow.log_artifact, and the tracking UI then lets users sort, filter and compare runs side by side.

Autologging removes most of that manual code. A single call to mlflow.autolog() patches supported libraries, including scikit-learn, XGBoost, LightGBM, Keras and PyTorch Lightning, so that hyperparameters, training curves and the fitted model are logged automatically. Autologging is convenient, but it can log more artifacts than needed, which inflates storage costs on large hyperparameter sweeps.

The following table shows what a single run typically captures and why each item matters.

Recorded itemExampleWhy it matters
Parameterslearning_rate = 0.001Explain why results differ
Metricsvalidation AUC per epochRank and compare runs
ArtifactsModel files, confusion matrixReuse outputs without retraining
Tagsgit commit hash, ownerSearch and audit runs
Dataset inputsDataset name, digest, sourceDetect training data changes
Information recorded in a tracking run

The MLflow model format and the Model Registry

An MLflow model is a directory with an MLmodel file that lists one or more flavors, meaning the ways the model can be loaded. A PyTorch model may carry a native pytorch flavor and the generic python_function (pyfunc) flavor, which exposes a uniform predict() method to any downstream tool.

Each saved model can include a model signature that defines input and output schemas, an input example, and an environment specification such as a requirements.txt or conda file. These details let a serving system reject malformed requests and rebuild the exact dependency set used during training.

The Model Registry adds governance on top of the format. A registered model has numbered versions, and teams point aliases such as champion or challenger at specific versions. Older releases used fixed stages (Staging, Production, Archived); these stages are now deprecated in favor of aliases and tags, which support more flexible promotion workflows.

MLflow for generative AI and LLM applications

Recent MLflow releases, especially the MLflow 3 line, extend the platform from classic models to LLM applications and agents. The emphasis shifts from logging weights to recording prompts, traces and quality judgments, because many generative systems call hosted models rather than train their own.

  • Tracing records each step of a request as spans, with inputs, outputs and latency, and is compatible with OpenTelemetry conventions.
  • Automatic tracing integrations cover libraries such as LangChain, LlamaIndex, DSPy and several model provider SDKs.
  • A prompt registry stores versioned prompt templates that applications can load by name and alias.
  • Evaluation supports LLM-as-a-judge scorers for criteria such as relevance, groundedness and safety, alongside human feedback.

These features place MLflow in direct overlap with LangSmith and Langfuse for LLM observability, although the depth of each integration differs and changes quickly between releases.

MLflow is a lifecycle and governance tool, so its closest neighbors are tracking and observability products rather than training frameworks. The comparison below summarizes the main differences.

ToolMain focusHosting modelModel registry
MLflowFull lifecycle, classic ML and GenAISelf-hosted or managedYes, built in
Weights & BiasesExperiment tracking and visualizationMainly hosted serviceYes, artifact-based
TensorBoardTraining curves and graphsLocal viewerNo
LangSmithLLM tracing and evaluationMainly hosted servicePrompt hub only
LangfuseOpen-source LLM observabilitySelf-hosted or hostedPrompt management only
MLflow compared with neighboring tools

The decisive advantages of MLflow are its permissive Apache 2.0 license, its vendor-neutral model format and its breadth. The usual counterarguments are a less polished visualization layer than Weights & Biases and the operational work required to run it securely at scale.

Deployment choices, limitations and common mistakes

Running MLflow well depends mostly on storage and access control decisions. The open-source server historically shipped with minimal authentication, so production installations usually add a reverse proxy, single sign-on or the optional basic authentication module, or they use a managed offering.

The table below lists frequent problems, their causes and practical fixes.

SymptomLikely causeFix
Runs vanish after a restartLocal file store on ephemeral diskUse a SQL backend store
Slow tracking UIMillions of metric rows in SQLiteMove to PostgreSQL or MySQL
Model fails in servingMissing or wrong dependency pinsLog a pinned environment
Bad input reaches the modelNo model signature loggedLog a signature and input example
Exploding storage billsAutologging every sweep artifactDisable unneeded artifact logging
Common MLflow problems and fixes

A frequent misconception is that MLflow is an orchestration or training system. It does not schedule jobs or allocate GPUs; teams pair it with tools such as Airflow, Kubeflow or Ray for that work. Built-in serving through mlflow models serve is suitable for testing and moderate traffic, while high-throughput inference usually moves to a dedicated serving platform that loads the MLflow model.

In plain words

MLflow is like a lab notebook and a warehouse for machine learning. Every time a team trains a model, it writes down the settings it used, the scores it achieved and the files it produced, so nobody has to remember them later.

When a model is good enough, MLflow stores it on a labeled shelf with a version number. Anyone can then see which version is in use, how it was made and how to load it again. For chatbots and other AI assistants, it also records each step the assistant took to answer a question.

A real-world example

A data science team at a mid-sized insurer builds a model that predicts which claims need manual review. Over three weeks, four people run about 300 experiments with gradient boosting, varying tree depth, learning rate and feature sets. With autologging enabled, every run lands in one shared MLflow experiment backed by PostgreSQL and object storage.

The team filters runs by validation AUC, picks the best one at 0.87, and registers it as version 12 of the claims-triage model with the alias challenger. After a two-week shadow test against the current champion, an engineer moves the champion alias to version 12. The serving job loads the model by alias, so no code change is needed.

Six months later, an auditor asks how the model was built. The team opens version 12 and shows the exact parameters, data digest and code commit, answering the question in minutes instead of days.