MediaPipe

On this page 8

MediaPipe is an open-source framework from Google for building and running on-device machine learning pipelines that process live and streaming media such as camera video, still images, audio and text. Among AI frameworks & tools in artificial intelligence, it occupies a specific niche: it is not a library for training large models, but a runtime and toolkit for perception tasks such as hand tracking, pose estimation, face landmarks, object detection and segmentation on phones, browsers, desktops and embedded devices.

The project has two layers. The low-level MediaPipe Framework describes processing as a graph of calculators connected by data streams, while the higher-level MediaPipe Solutions offer ready-made tasks with pretrained models and simple APIs for Android, iOS, the web and Python. Its main value lies in low latency, privacy and offline operation, because inference happens locally rather than on a server.

How the MediaPipe graph processes media

A MediaPipe pipeline is a directed graph in which each node is a calculator, a self-contained C++ component that consumes inputs, performs one operation and emits outputs. Typical calculators decode frames, resize and normalize images, run a neural network, convert raw tensors into landmarks or boxes, and draw overlays.

Data moves through the graph as packets on streams. Every packet carries a timestamp, and the scheduler uses these timestamps to synchronize inputs that arrive at different rates, for example video frames and sensor readings. Values that stay constant during a run, such as a model file or configuration, travel as side packets. The graph itself is declared in a text configuration file, so engineers can rearrange processing steps without rewriting application logic.

Two design choices make real-time operation practical. First, flow control lets a graph drop or throttle frames when inference cannot keep pace, which prevents queues from growing and latency from accumulating. Second, MediaPipe often runs an expensive detector only occasionally and uses a cheaper tracker or landmark model on following frames, reusing the previous result as a region of interest.

The following table summarizes the core building blocks of the framework layer.

ConceptRole in the pipelinePractical consequence
CalculatorPerforms one processing stepReusable and testable in isolation
StreamCarries timestamped packets between nodesKeeps audio, video and sensor data aligned
Side packetSupplies constant inputs such as modelsLoaded once per graph run
Graph configDeclares nodes and connections in textPipeline changes without code changes
Executor and schedulerRuns calculators in parallel threadsBetter use of multicore CPUs and GPUs
Core concepts of the MediaPipe Framework

MediaPipe Solutions and the Tasks API

MediaPipe Solutions package complete pipelines with pretrained models so that developers can add perception features in a few lines of code. Since 2023, the recommended interface is the MediaPipe Tasks API, which replaced the older legacy solutions such as Face Mesh, Hands and Holistic with a consistent set of task classes. The suite also includes MediaPipe Model Maker for customizing models and MediaPipe Studio for testing tasks in a browser.

Models are distributed in the TensorFlow Lite format, now part of LiteRT, often wrapped in a .task bundle that combines the network with metadata and label files. The table below lists frequently used tasks and their typical outputs.

TaskInputTypical output
Hand landmarkerImage or video frame21 3D landmarks per hand and handedness
Pose landmarkerImage or video frame33 body landmarks and optional mask
Face landmarkerImage or video frame478 face landmarks and blendshape scores
Gesture recognizerImage or video frameGesture category with hand landmarks
Object detectorImage or video frameBounding boxes, labels and scores
Image segmenterImage or video framePer-pixel category or confidence masks
Audio and text classifiersAudio clip or text stringCategory labels with scores
LLM Inference APIText promptGenerated text from a compact language model
Common MediaPipe tasks and their outputs

Each task can run in image, video or live-stream mode. Image mode processes independent pictures, video mode assumes sequential frames with timestamps, and live-stream mode delivers results asynchronously through a callback, which suits camera previews where the interface must never block.

Platforms and hardware acceleration

MediaPipe is cross-platform by design: the same model and graph can serve an Android app, an iOS app, a web page and a Python script. Performance depends heavily on the hardware delegate chosen for inference. On CPUs, MediaPipe relies on the XNNPACK library for optimized operations, while GPU paths use OpenGL ES, Metal or WebGL depending on the platform.

PlatformMain interfaceAcceleration options
AndroidJava and Kotlin Tasks librariesCPU with XNNPACK, GPU delegate
iOSSwift and Objective-C Tasks librariesCPU, GPU through Metal
WebJavaScript packages with WebAssemblyWebAssembly CPU, WebGL or WebGPU paths
Pythonpip package for desktop and serversMainly CPU, limited GPU support
C++Full framework and custom graphsAll delegates, custom calculators
MediaPipe support by platform

Real throughput varies widely. A pose landmarker can exceed 30 frames per second on a recent smartphone with GPU acceleration, yet drop well below that on an older device or in a browser without GPU access. Model variant matters as much as hardware: many tasks ship in lite, full and heavy versions that trade accuracy for speed.

MediaPipe is often confused with general deep learning frameworks, but it sits downstream of training. TensorFlow and PyTorch are used to build and train networks, while MediaPipe packages trained networks into end-to-end media pipelines with preprocessing, postprocessing and synchronization included.

ToolPrimary purposeRelation to MediaPipe
MediaPipeReal-time on-device perception pipelinesComplete pipeline with pre- and postprocessing
LiteRTLightweight on-device inference runtimeRuns the models inside MediaPipe tasks
TensorFlow and PyTorchModel design and trainingProduce models that can be converted for MediaPipe
ONNX RuntimeCross-framework inference engineRuns single models, without media graph features
Core MLApple on-device inferencePlatform-specific alternative for iOS and macOS
OpenCVClassical computer vision libraryOften combined with MediaPipe for capture and drawing
MediaPipe and neighboring tools

The practical rule is simple: choose MediaPipe when the problem matches one of its supported perception tasks and must run locally in real time; choose a general inference runtime when the model is custom and the surrounding pipeline is already written.

Customizing models with Model Maker

MediaPipe Model Maker applies transfer learning to adapt a pretrained model to new categories with a modest dataset, often a few hundred labeled examples per class. It retrains only the final layers, which keeps training fast and allows it to run on an ordinary laptop or in a hosted notebook.

Supported customizations include image classification, object detection, gesture recognition and text classification. The result is exported as a model file that drops directly into the matching task. Model Maker does not support every task, and deep architectural changes still require full training in TensorFlow followed by conversion.

Limitations and common mistakes with MediaPipe

MediaPipe models are trained for typical conditions, so accuracy degrades with unusual lighting, motion blur, occlusion or camera angles far from the training data. Landmark depth values are relative estimates, not metric measurements, which is a frequent source of errors in applications that need real distances.

SymptomLikely causeRecommended fix
Jittery landmarks between framesImage mode used on videoUse video or live-stream mode with timestamps
Low frame rate on phonesCPU path or heavy model variantEnable the GPU delegate or a lite model
Missed detections at distanceSubject too small in frameCrop the region of interest or raise resolution
Code fails after upgradeLegacy solution APIs removedMigrate to the Tasks API
Biased accuracy across usersTraining data imbalanceTest on representative groups before release
Frequent problems and their fixes

Teams should also remember that on-device does not mean unregulated. Face and body landmarks can qualify as biometric data under privacy laws, so consent, retention and purpose limitation still apply even when no image leaves the device.

Understanding it simply

MediaPipe is a toolkit that lets a phone, laptop or web page see and understand a camera feed by itself. It can find your hands, your face or your body pose in each video frame and report where the important points are, such as fingertips or elbows.

Think of it as an assembly line for video. Each station does one small job: one prepares the picture, one runs the trained model, one turns the result into usable points. Because everything happens on the device, it is fast, works offline and keeps images private.

Applied in practice

A physiotherapy clinic wants patients to do knee rehabilitation exercises at home with feedback. A small development team builds a web app that uses the MediaPipe pose landmarker in live-stream mode through the JavaScript package. The app reads the hip, knee and ankle landmarks, calculates the knee flexion angle for every frame and counts repetitions when the angle passes from about 170 to below 90 degrees.

On a mid-range laptop with WebGL acceleration, the lite model runs at roughly 25 to 30 frames per second, which is enough for smooth feedback. Video never leaves the browser; only the repetition count and peak angles are sent to the therapist. After testing with different body types and lighting, the clinic adopts the app, and therapists receive objective range-of-motion data between visits instead of relying on patient memory.