Artificial intelligence

Edge AI

Running AI models on phones, PCs and embedded devices instead of in the cloud, and how models are made small enough.

Browse A–Z

Terms

55 terms

  • 4-bit quantization

    4-bit quantization stores neural network weights in 4-bit integers, shrinking models about fourfold so they fit on phones, PCs and embedded devices.

  • AIoT

    AIoT, the artificial intelligence of things, combines connected sensors and devices with machine learning so that data is analyzed where it is collected.

  • Always-on sensing

    Always-on sensing is the continuous, low-power monitoring of sensor data on a device so that small models detect events without waking the main processor.

  • AWQ

    AWQ is a post-training method that compresses language model weights to about 4 bits by protecting the channels that activations show to be most important.

  • Binarized neural network BNN

    A binarized neural network (BNN) constrains weights and activations to +1 or -1, so edge devices can run inference with cheap bitwise operations.

  • Cloud offloading

    Cloud offloading in Edge AI moves heavy model inference from a phone or embedded device to remote servers when local compute, memory or energy fall short.

  • Cross-device federated learning

    Cross-device federated learning trains one shared model across millions of phones and edge devices while raw training data never leaves each device.

  • Dynamic quantization

    Dynamic quantization stores model weights as 8-bit integers and quantizes activations at runtime, shrinking AI models for faster CPU inference on edge devices.

  • Early exit network

    An early exit network is a neural network with intermediate classifiers that stop inference once a prediction is confident, saving compute on edge devices.

  • Edge inference

    Edge inference runs a trained AI model directly on a phone, PC, camera or embedded device, producing predictions locally instead of in a cloud data center.

  • Edge inference runtime

    An edge inference runtime is the software engine that loads a trained model on a phone, PC or embedded device and executes it locally and efficiently.

  • Edge MLOps

    Edge MLOps is the practice of building, deploying, monitoring and updating machine learning models that run on phones, gateways and embedded devices.

  • Embedded vision

    Embedded vision is the practice of running image capture and computer vision models directly on cameras, boards and devices instead of in the cloud.

  • Energy-efficient inference

    Energy-efficient inference runs trained AI models with the least energy per prediction, a core requirement for battery-powered Edge AI devices.

  • Federated analytics

    Federated analytics computes aggregate statistics across many devices by sending queries to the data, so raw user records never leave phones or sensors.

  • Federated averaging FedAvg

    Federated averaging (FedAvg) trains one shared model across many devices by averaging locally updated weights instead of collecting raw user data.

  • GPTQ

    GPTQ is a one-shot post-training method that compresses language model weights to 3 or 4 bits using second-order error correction.

  • Hardware delegate

    A hardware delegate is the runtime component in edge AI that hands parts of a neural network graph to a GPU, NPU or DSP for faster,…

  • Hardware-aware neural architecture search

    Hardware-aware neural architecture search finds neural network designs that meet accuracy goals within the latency, memory and energy limits of a target device.

  • Hybrid inference

    Hybrid inference runs part of the AI inference workload on a local device and part in the cloud, choosing per request where each model step should…

  • In-browser inference

    In-browser inference runs a trained AI model inside the web browser on the user's device, using WebAssembly, WebGPU or WebNN instead of a cloud server.

  • INT8 quantization

    INT8 quantization stores neural network weights and activations as 8-bit integers, shrinking models and speeding up inference on edge devices.

  • Keyword spotting KWS

    Keyword spotting (KWS) is an always-on edge AI task in which a small on-device model listens for a short spoken phrase such as a wake word.

  • Local LLM

    A local LLM is a large language model that runs entirely on a user's own PC, phone or server, keeping prompts and data off cloud infrastructure.

  • Lottery ticket hypothesis

    The lottery ticket hypothesis states that dense neural networks contain small subnetworks that, trained alone from their initial weights, match full accuracy.

  • Low-rank factorization

    Low-rank factorization compresses neural network weight matrices into products of smaller matrices, cutting parameters and compute for Edge AI deployment.

  • Magnitude pruning

    Magnitude pruning removes neural network weights with the smallest absolute values to shrink models for Edge AI with little loss in accuracy.

  • Mixed-precision quantization

    Mixed-precision quantization assigns different bit widths to different layers of a neural network so a compressed model keeps its accuracy on edge hardware.

  • MLPerf Tiny

    MLPerf Tiny is the MLCommons benchmark suite that measures accuracy, latency and energy of machine learning inference on microcontroller-class devices.

  • MobileNet

    MobileNet is a family of compact convolutional neural networks built from depthwise separable convolutions for fast image recognition on edge devices.

  • Model conversion

    Model conversion translates a trained neural network into a format and operator set that an edge inference runtime can load and execute on a device.

  • Model footprint

    Model footprint is the total memory, storage, compute and energy a machine learning model needs on a target device, a key constraint in Edge AI.

  • Model offloading

    Model offloading keeps part of a neural network in CPU memory or storage and moves it to the accelerator only when needed, so large models fit…

  • Model sparsity

    Model sparsity is the share of zero-valued weights or activations in a neural network, used in edge AI to cut memory, compute and energy on devices.

  • Neural network pruning

    Neural network pruning removes redundant weights, channels or layers from a trained model so it runs faster on edge devices with little accuracy loss.

  • On-device personalization

    On-device personalization adapts an AI model to one user directly on the phone or PC, so behavior improves while personal data stays local.

  • On-device RAG

    On-device RAG runs retrieval-augmented generation entirely on a phone or PC, grounding a local language model in private data without any cloud calls.

  • On-device training

    On-device training updates a machine learning model directly on a phone or embedded device, using local data without sending it to the cloud.

  • Operator fusion

    Operator fusion merges adjacent neural network operations into one kernel, cutting memory traffic and launch overhead for faster, leaner Edge AI inference.

  • OTA model update

    An OTA model update delivers a new or retrained AI model to deployed edge devices over a network, without physical access or a full firmware reflash.

  • Per-channel quantization

    Per-channel quantization gives each output channel of a weight tensor its own scale, keeping compressed Edge AI models accurate at low bit widths.

  • Post-training quantization PTQ

    Post-training quantization (PTQ) converts a trained model's weights and activations to low-precision integers without retraining, shrinking it for edge devices.

  • Quantization calibration

    Quantization calibration runs sample data through a model to measure activation ranges and set the scales used when converting it to low-precision integers.

  • Quantization-aware training QAT

    Quantization-aware training (QAT) simulates low-precision arithmetic during training so a neural network keeps its accuracy after conversion to INT8 or lower.

  • Secure aggregation

    Secure aggregation is a cryptographic protocol that lets a server learn only the sum of device model updates in federated learning, never any single update.

  • SmoothQuant

    SmoothQuant is a training-free quantization method that shifts activation outliers into weights so large language models can run with 8-bit math.

  • Split computing

    Split computing divides a neural network between a device and a server, running early layers locally and sending a compact tensor for the rest.

  • Split learning

    Split learning trains one neural network across a device and a server, so raw data stays local while only cut-layer activations and gradients travel.

  • SqueezeNet

    SqueezeNet is a compact convolutional neural network that reaches AlexNet-level image accuracy with about 50 times fewer parameters for edge devices.

  • Structured pruning

    Structured pruning removes whole channels, filters, heads or layers from a neural network so the smaller model runs faster on edge AI hardware.