Windows ML

On this page 8

Windows ML (Windows Machine Learning) is the on-device inference runtime built into Windows that lets desktop applications load trained models in the ONNX format and run them locally on the CPU, GPU or neural processing unit. Among AI frameworks & tools in artificial intelligence, it occupies the deployment end of the pipeline: models are trained elsewhere, for example in PyTorch or TensorFlow, then exported to ONNX and executed by Windows ML on the end user’s machine.

The runtime exists so that developers do not need to ship and maintain a separate inference engine for every hardware vendor. Windows ML abstracts the silicon, selects or accepts an execution target, and handles memory binding, tensor conversion and scheduling, which keeps the application code largely identical across very different PCs.

How Windows ML executes an ONNX model

Windows ML executes a model by loading an ONNX graph, binding inputs and outputs to tensors, and dispatching each operator to a hardware backend. The original API, introduced with Windows 10 version 1809 in 2018, exposed the WinRT namespace Windows.AI.MachineLearning with three central objects: LearningModel, LearningModelSession and LearningModelBinding.

The typical flow is short. The application loads the model file once, creates a session on a chosen device, binds input features such as an image or a float tensor, and calls Evaluate. The session caches the compiled graph and device resources, so the first call is markedly slower than later calls; this warm-up cost matters for interactive software.

The following table summarizes the core objects of the classic programming model and their responsibilities.

ObjectRoleLifetime
LearningModelHolds the parsed ONNX graph and its metadataLoaded once per model
LearningModelDeviceSelects CPU, default GPU or a specific adapterChosen at session creation
LearningModelSessionCompiles the graph for the device and runs itReused across many evaluations
LearningModelBindingMaps named inputs and outputs to tensors or imagesCreated per evaluation or reused
Core objects in the classic Windows ML API

In 2025 Microsoft redesigned the runtime. The current Windows ML is built on ONNX Runtime, ships through the Windows App SDK, and obtains vendor-specific execution providers (EPs) from Windows itself, which downloads and updates them for the hardware present in the device.

Classic and current Windows ML generations

The two generations share a name and the ONNX format but differ in architecture and API surface. Developers who read older samples should know which generation a sample targets, because the classic API relied on DirectML for GPU acceleration, while the current one delegates to ONNX Runtime and its providers.

AspectClassic Windows MLCurrent Windows ML
IntroducedWindows 10 version 1809, 2018Announced 2025, generally available late 2025
EngineInbox WinRT runtimeONNX Runtime
AcceleratorsCPU and GPU via DirectMLCPU, GPU and NPU via execution providers
DistributionPart of Windows or a NuGet packageWindows App SDK with system-managed providers
Main API styleWinRT objects such as LearningModelSessionONNX Runtime sessions plus provider catalog
Operator coverageFixed by the shipped opset versionFollows the ONNX Runtime release
Comparison of the two Windows ML generations

The practical consequence is that NPU support belongs to the current generation. Neural processing units in Copilot+ PCs from Qualcomm, Intel and AMD require vendor-specific compilers, and Windows ML hides them behind a common provider registration step instead of asking each application to bundle them.

Execution providers and hardware targets

An execution provider is the plug-in that translates ONNX operators into calls for one class of hardware. Windows ML can register the providers available on the device, after which ONNX Runtime partitions the graph: supported subgraphs run on the accelerator, and unsupported operators fall back to the CPU.

The table below lists typical targets; real performance depends on the model, numeric precision and drivers.

TargetStrengthTypical precisionCommon use
CPUUniversal availability, full operator coverageFP32, INT8Small models and fallback
GPUHigh parallel throughputFP16, FP32Image, video and larger networks
NPULow power for sustained workloadsINT8, INT4, FP16Background and always-on features
Typical hardware targets for Windows ML inference

Applications can let the runtime apply a device selection policy, such as preferring lowest power or highest performance, or they can pin a specific provider. Explicit selection gives predictable behavior during testing, while policy-based selection adapts better to the wide range of Windows hardware.

Windows ML is best understood as a Windows-specific distribution layer over ONNX inference, not as a training framework. It competes with, and partly builds on, neighboring tools in the same space.

ToolScopeRelationship to Windows ML
ONNX RuntimeCross-platform inference engineEngine underneath the current generation
DirectMLDirectX 12 machine learning libraryGPU backend of the classic generation
TensorRTNVIDIA GPU optimizer and runtimeCan serve as an NVIDIA execution provider
Core MLApple on-device runtimeEquivalent role on Apple platforms
LiteRTGoogle on-device runtimeEquivalent role on Android and embedded devices
How Windows ML relates to neighboring inference tools

The decisive difference from a plain ONNX Runtime installation is provider management by the operating system: instead of each application packaging vendor providers, Windows supplies shared, updated providers to all applications.

Preparing and deploying models for Windows ML

Deployment starts with a clean ONNX export. Teams usually export from PyTorch with a fixed or explicitly declared dynamic shape, verify the graph, and then apply quantization or graph optimization to fit the target accelerator.

  1. Export the trained model to ONNX with a supported opset version.
  2. Validate outputs against the original framework on a reference dataset.
  3. Quantize to INT8 or FP16 where the target provider benefits, and re-check accuracy.
  4. Compile or cache the model for NPU targets to avoid long first-run delays.
  5. Ship the model with the application and register execution providers at startup.

The optional AI Toolkit for Visual Studio Code helps convert, quantize and profile models, shortening the trial-and-error cycle of finding a variant that runs fully on the accelerator.

Limitations and common mistakes

Most problems with Windows ML come from operator coverage, precision and first-run latency, not from the API itself. The table pairs frequent symptoms with their usual causes.

SymptomLikely causeRemedy
Model fails to loadOpset newer than the runtime supportsRe-export with an older opset
Accelerator shows little speedupMany operators fall back to the CPUSimplify the graph or replace operators
Accuracy drops after deploymentAggressive INT8 or INT4 quantizationUse calibration data or mixed precision
First inference takes secondsGraph compilation on session creationCache compiled models and warm up early
Results differ across PCsDifferent providers and numeric pathsSet tolerances and test on each class
Frequent Windows ML problems and remedies

Windows ML is inference only: it does not train or fine-tune models. It is also limited to Windows, so cross-platform products typically call ONNX Runtime directly on other systems and use Windows ML only where its provider management pays off.

A non-technical view

Windows ML is the part of Windows that lets an app use an artificial intelligence model directly on your own computer, without sending data to the internet. A company trains the model elsewhere and packs it into a common file format. Windows ML then reads that file and runs it.

Think of it as a universal adapter. Computers contain very different chips, and each chip speaks its own language. Windows ML translates the model for whatever chip is present, so the same app works on many machines and uses the fastest or most power-efficient part available.

A concrete example

A small software company builds a photo management application for Windows that needs to remove backgrounds and tag images offline. Its data scientists train a segmentation network and a classifier in PyTorch, export both to ONNX with opset 17, and quantize the classifier to INT8 after checking that top-1 accuracy falls by less than one percentage point.

At startup, the application uses Windows ML to register the available execution providers and asks for a low-power policy for background tagging and a performance policy for interactive background removal. On a laptop with an NPU, tagging of a 10,000-photo library runs on the NPU while the user keeps working; on an older desktop, the same code runs on the GPU or CPU.

The result is one codebase and one installer for all customers. The team caches compiled models after the first launch, which reduces the initial delay on later starts, and no customer photo ever leaves the device.