LiteRT is Google’s on-device inference runtime for machine learning models, the direct successor of TensorFlow Lite (TFLite), renamed in 2024. It executes compact models stored in the .tflite FlatBuffers format on Android and iOS phones, embedded Linux boards, desktops and the web, using hardware accelerators such as GPUs and NPUs when they are available.
Among AI frameworks & tools, LiteRT occupies the edge deployment layer: models are trained elsewhere, typically in TensorFlow, Keras, JAX or PyTorch, then converted, optimized and shipped inside an application. The rename reflects that the runtime is no longer tied to TensorFlow and now serves models from several training frameworks.
How LiteRT runs models on device
LiteRT works in two stages: conversion and inference. A converter takes a trained model, simplifies its computation graph, fuses operations and serializes it into a single .tflite file. On the device, the runtime loads that file through memory mapping, allocates tensors once and executes the graph operator by operator.
The FlatBuffers format matters because it can be read without unpacking or parsing, which keeps startup fast and memory overhead low. Execution goes through a set of built-in operator kernels optimized for mobile CPUs, and any part of the graph can be handed to a hardware delegate that runs it on a faster processor.
The table below lists the main parts of the LiteRT pipeline and what each contributes.
| Component | Function | Where it runs |
|---|---|---|
| Converter | Turns trained models into .tflite files | Development machine |
| Model file | Stores graph, weights and metadata | Bundled with the app |
| Interpreter or compiled model API | Loads the model and executes inference | Target device |
| Delegates | Offload operations to accelerators | GPU, NPU or DSP |
| Metadata and support libraries | Describe inputs and preprocess data | App code |
Newer releases add a compiled model API that selects accelerators automatically and supports ahead-of-time compilation for NPUs, while the classic interpreter API remains available for existing applications.
Hardware acceleration and delegates
A delegate is a plug-in that takes over supported parts of the graph and runs them on specialized hardware. The default CPU path uses the XNNPACK library, which is highly optimized for ARM and x86 processors. When an operation is not supported by a delegate, LiteRT falls back to the CPU, which can split the graph and add data transfer costs.
Choosing the right accelerator depends on the device, the model’s operators and its numeric precision, as the following comparison shows.
| Accelerator | Best suited for | Main caveat |
|---|---|---|
| CPU with XNNPACK | Universal default, small models | Slower for large convolutional models |
| GPU delegate | Float models, vision workloads | Initialization time and partial operator coverage |
| NPU acceleration | Quantized models, sustained workloads | Vendor-specific support and compilation |
| NNAPI delegate | Older Android acceleration | Deprecated in recent Android versions |
| Core ML delegate | Apple Neural Engine on iOS | Limited operator set |
| Edge TPU | Coral embedded devices | Requires full integer quantization |
Measured speedups vary widely. A well-quantized vision model can run several times faster on an NPU than on the CPU, while a model full of unsupported operators may run slower on an accelerator because of graph partitioning. Profiling on real target devices is the only reliable guide.
Model optimization and quantization
Quantization is the central optimization technique in LiteRT. It stores weights, and often activations, in lower precision such as 8-bit integers instead of 32-bit floats, which shrinks the model and speeds up arithmetic on mobile hardware.
The options below differ in effort, size reduction and accuracy risk.
| Method | Size reduction | Accuracy impact | Effort |
|---|---|---|---|
| Dynamic range quantization | About 4x | Usually small | No calibration data |
| Full integer quantization | About 4x | Small to moderate | Representative dataset needed |
| Float16 quantization | About 2x | Minimal | No calibration data |
| Quantization-aware training | About 4x | Lowest loss for int8 | Retraining required |
Other techniques include pruning, weight clustering and operator fusion. For large generative models, Google also provides a dedicated LiteRT-LM layer that handles tokenization, key-value caching and streaming output for on-device language models.
LiteRT compared with other edge runtimes
LiteRT competes with several edge inference runtimes, and the choice usually follows the training framework, target platform and team skills. The comparison below covers the most common neighbors.
| Runtime | Model format | Strongest platforms | Typical origin framework |
|---|---|---|---|
| LiteRT | .tflite | Android, embedded Linux, web | TensorFlow, Keras, JAX, PyTorch |
| ExecuTorch | .pte | Mobile and embedded | PyTorch |
| Core ML | .mlpackage | Apple devices | Any, via conversion tools |
| ONNX Runtime | .onnx | Cross-platform, Windows | Any ONNX exporter |
| TensorFlow Lite Micro | C arrays of .tflite | Microcontrollers | TensorFlow, Keras |
LiteRT’s main advantages are mature Android integration, distribution through Google Play services and a large ecosystem of pretrained models, including those used by MediaPipe solutions. Its main weakness has been PyTorch conversion friction, which Google addresses with a dedicated PyTorch converter.
Common pitfalls when deploying LiteRT models
Most LiteRT problems appear at conversion time or on real hardware, not during training. The following practices prevent the most frequent failures.
- Unsupported operators: check operator compatibility before committing to an architecture, because custom layers may require Select TensorFlow ops that enlarge the binary.
- Mismatched preprocessing: normalization, color order and input size must match training exactly, or accuracy drops silently.
- Unrepresentative calibration data: full integer quantization with poor sample data causes large accuracy losses.
- Testing only on emulators: delegate behavior and speed differ across chipsets, so benchmark on physical devices.
- Ignoring thermal throttling: sustained inference can slow down after minutes of continuous use.
In simple terms
LiteRT is a tool that lets artificial intelligence run directly on your phone or another small device, without sending data to a distant server. Developers first teach a model on powerful computers, then shrink it and package it so it fits on the device.
It is a bit like compressing a large photo so it loads fast on a phone while still looking good. The result is AI that works offline, responds quickly and keeps personal data on the device.
A practical example
A team building a plant disease app for farmers needs it to work in fields with no mobile signal. They train an image classifier in Keras that recognizes 38 leaf conditions, reaching good accuracy on a validation set. The original float model is about 16 MB and takes roughly 120 milliseconds per image on a mid-range phone CPU.
The team converts it to LiteRT with full integer quantization, calibrating with 500 representative field photos. The model shrinks to about 4 MB, and accuracy drops by less than one percentage point. With NPU acceleration enabled on supported phones, inference falls to around 20 milliseconds, and the CPU path remains as a fallback for older devices.
Farmers can now photograph a leaf and receive a diagnosis instantly, even offline. The developers keep a benchmark suite on five physical phone models to catch performance regressions before each release.