Running AI models on phones, PCs and embedded devices instead of in the cloud, and how models are made small enough.
55 terms
4-bit quantization stores neural network weights in 4-bit integers, shrinking models about fourfold so they fit on phones, PCs and embedded devices.
AIoT, the artificial intelligence of things, combines connected sensors and devices with machine learning so that data is analyzed where it is collected.
Always-on sensing is the continuous, low-power monitoring of sensor data on a device so that small models detect events without waking the main processor.
AWQ is a post-training method that compresses language model weights to about 4 bits by protecting the channels that activations show to be most important.
A binarized neural network (BNN) constrains weights and activations to +1 or -1, so edge devices can run inference with cheap bitwise operations.
Cloud offloading in Edge AI moves heavy model inference from a phone or embedded device to remote servers when local compute, memory or energy fall short.
Cross-device federated learning trains one shared model across millions of phones and edge devices while raw training data never leaves each device.
Dynamic quantization stores model weights as 8-bit integers and quantizes activations at runtime, shrinking AI models for faster CPU inference on edge devices.
An early exit network is a neural network with intermediate classifiers that stop inference once a prediction is confident, saving compute on edge devices.
Edge inference runs a trained AI model directly on a phone, PC, camera or embedded device, producing predictions locally instead of in a cloud data center.
An edge inference runtime is the software engine that loads a trained model on a phone, PC or embedded device and executes it locally and efficiently.
Edge MLOps is the practice of building, deploying, monitoring and updating machine learning models that run on phones, gateways and embedded devices.
Embedded vision is the practice of running image capture and computer vision models directly on cameras, boards and devices instead of in the cloud.
Energy-efficient inference runs trained AI models with the least energy per prediction, a core requirement for battery-powered Edge AI devices.
Federated analytics computes aggregate statistics across many devices by sending queries to the data, so raw user records never leave phones or sensors.
Federated averaging (FedAvg) trains one shared model across many devices by averaging locally updated weights instead of collecting raw user data.
GPTQ is a one-shot post-training method that compresses language model weights to 3 or 4 bits using second-order error correction.
A hardware delegate is the runtime component in edge AI that hands parts of a neural network graph to a GPU, NPU or DSP for faster,…
Hardware-aware neural architecture search finds neural network designs that meet accuracy goals within the latency, memory and energy limits of a target device.
Hybrid inference runs part of the AI inference workload on a local device and part in the cloud, choosing per request where each model step should…
In-browser inference runs a trained AI model inside the web browser on the user's device, using WebAssembly, WebGPU or WebNN instead of a cloud server.
INT8 quantization stores neural network weights and activations as 8-bit integers, shrinking models and speeding up inference on edge devices.
Keyword spotting (KWS) is an always-on edge AI task in which a small on-device model listens for a short spoken phrase such as a wake word.
A local LLM is a large language model that runs entirely on a user's own PC, phone or server, keeping prompts and data off cloud infrastructure.
The lottery ticket hypothesis states that dense neural networks contain small subnetworks that, trained alone from their initial weights, match full accuracy.
Low-rank factorization compresses neural network weight matrices into products of smaller matrices, cutting parameters and compute for Edge AI deployment.
Magnitude pruning removes neural network weights with the smallest absolute values to shrink models for Edge AI with little loss in accuracy.
Mixed-precision quantization assigns different bit widths to different layers of a neural network so a compressed model keeps its accuracy on edge hardware.
MLPerf Tiny is the MLCommons benchmark suite that measures accuracy, latency and energy of machine learning inference on microcontroller-class devices.
MobileNet is a family of compact convolutional neural networks built from depthwise separable convolutions for fast image recognition on edge devices.
Model conversion translates a trained neural network into a format and operator set that an edge inference runtime can load and execute on a device.
Model footprint is the total memory, storage, compute and energy a machine learning model needs on a target device, a key constraint in Edge AI.
Model offloading keeps part of a neural network in CPU memory or storage and moves it to the accelerator only when needed, so large models fit…
Model sparsity is the share of zero-valued weights or activations in a neural network, used in edge AI to cut memory, compute and energy on devices.
Neural network pruning removes redundant weights, channels or layers from a trained model so it runs faster on edge devices with little accuracy loss.
On-device personalization adapts an AI model to one user directly on the phone or PC, so behavior improves while personal data stays local.
On-device RAG runs retrieval-augmented generation entirely on a phone or PC, grounding a local language model in private data without any cloud calls.
On-device training updates a machine learning model directly on a phone or embedded device, using local data without sending it to the cloud.
Operator fusion merges adjacent neural network operations into one kernel, cutting memory traffic and launch overhead for faster, leaner Edge AI inference.
An OTA model update delivers a new or retrained AI model to deployed edge devices over a network, without physical access or a full firmware reflash.
Per-channel quantization gives each output channel of a weight tensor its own scale, keeping compressed Edge AI models accurate at low bit widths.
Post-training quantization (PTQ) converts a trained model's weights and activations to low-precision integers without retraining, shrinking it for edge devices.
Quantization calibration runs sample data through a model to measure activation ranges and set the scales used when converting it to low-precision integers.
Quantization-aware training (QAT) simulates low-precision arithmetic during training so a neural network keeps its accuracy after conversion to INT8 or lower.
Secure aggregation is a cryptographic protocol that lets a server learn only the sum of device model updates in federated learning, never any single update.
SmoothQuant is a training-free quantization method that shifts activation outliers into weights so large language models can run with 8-bit math.
Split computing divides a neural network between a device and a server, running early layers locally and sending a compact tensor for the rest.
Split learning trains one neural network across a device and a server, so raw data stays local while only cut-layer activations and gradients travel.
SqueezeNet is a compact convolutional neural network that reaches AlexNet-level image accuracy with about 50 times fewer parameters for edge devices.
Structured pruning removes whole channels, filters, heads or layers from a neural network so the smaller model runs faster on edge AI hardware.