On-device RAG

Also known as: local RAG

On this page 7

On-device RAG (also called local RAG) is a form of retrieval-augmented generation in which the embedding model, the vector index, the retriever and the language model all run on the user’s own hardware, so documents are searched and answers are generated without sending data to a cloud service. It is one of the most practical patterns in Edge AI, because it lets a small local model answer questions about personal files, messages or manuals that it never saw during training.

The approach trades the scale of data-center models for privacy, offline availability and predictable latency. Its design is dominated by tight budgets for memory, storage, battery and thermal headroom, which shape every choice from the embedding dimension to the context length.

How on-device RAG retrieves and generates locally

On-device RAG works as a two-phase pipeline: an indexing phase that turns local content into searchable vectors, and a query phase that retrieves relevant passages and feeds them to a local model. Both phases run on the device’s CPU, GPU or neural processing unit (NPU).

During indexing, documents are split into chunks of typically 200 to 500 tokens, each chunk is converted into a dense vector by a compact embedding model, and the vectors are stored in an on-device index, often inside SQLite or a dedicated embedded vector library. Indexing usually runs in the background while charging, because embedding thousands of chunks is the most energy-intensive step.

At query time the question is embedded with the same model, the index returns the top-k nearest chunks (commonly 3 to 8), and the chunks are inserted into the prompt of a quantized local language model. The model then writes an answer grounded in those passages, ideally citing which file each statement came from.

The table below lists the components and the constraints that typically apply on consumer hardware.

ComponentTypical on-device choiceMain constraint
ChunkerFixed-size or sentence-aware splittingChunk count drives index size
Embedding model20M to 150M parameters, 256 to 768 dimensionsIndexing time and battery
Vector indexFlat search, HNSW or IVF with quantized vectorsRAM and storage footprint
RerankerOptional small cross-encoderAdded query latency
Generator1B to 8B parameter model at 4-bit precisionMemory and tokens per second
Components of an on-device RAG stack

Resource budgets that shape the design

Every on-device RAG system is sized against fixed hardware budgets rather than elastic cloud capacity. The largest consumer of memory is the generator: a 3B parameter model at 4-bit quantization needs roughly 1.5 to 2 GB of RAM for weights, plus a KV cache that grows with context length.

Retrieved chunks directly consume context, so a prompt with eight chunks of 400 tokens adds about 3,200 tokens of prefill work. On a mobile NPU this can mean one to several seconds before the first token appears, which is why retrieval precision matters more than recall on small devices.

The following ranges are typical for current phones and laptops and vary with chip generation, runtime and model architecture.

ResourcePhone rangeLaptop range
Generator weights at 4-bit0.5 to 2 GB2 to 5 GB
Vector index for 50,000 chunks20 to 150 MB20 to 150 MB
Time to first token0.5 to 4 s0.2 to 2 s
Generation speed10 to 30 tokens/s20 to 80 tokens/s
Usable context window2,000 to 8,000 tokens4,000 to 32,000 tokens
Typical resource ranges for local RAG

Index size can be cut by vector quantization, such as int8 or binary embeddings, which shrinks storage by four to thirty-two times with a modest loss in retrieval accuracy. Matryoshka-style embeddings that can be truncated to fewer dimensions offer a similar trade-off.

On-device RAG differs from cloud RAG mainly in where data and computation live, and from fine-tuning in how knowledge enters the model. It also overlaps with neighboring Edge AI patterns such as hybrid inference and on-device personalization.

ApproachWhere knowledge livesKey strengthKey weakness
On-device RAGLocal index read at query timePrivate, offline, easy to updateLimited model quality and context
Cloud RAGServer-side index and modelLarge models, large corporaData leaves the device
Hybrid inferenceLocal retrieval, remote generationBetter answers when onlinePartial privacy exposure
On-device personalizationAdapted model weightsNo retrieval latencyHard to update or delete facts
Local LLM without retrievalTraining data onlySimplest stackNo access to personal content
How local retrieval compares with alternatives

A key advantage over weight-based adaptation is deletability: removing a document from the index removes it from future answers immediately, which supports data-protection obligations far more cleanly than retraining.

Designing a reliable local retrieval pipeline

A reliable on-device RAG pipeline is built by tuning retrieval first and choosing the smallest generator that answers well with good context. Weak retrieval cannot be rescued by a small model, while a strong retriever lets a 3B model perform close to much larger ones on document questions.

  • Use hybrid search that combines BM25 keyword scoring with vector similarity, because names, codes and numbers are often missed by embeddings alone.
  • Attach metadata filters such as date, app or folder to narrow the candidate set before vector search.
  • Index incrementally with change detection instead of rebuilding the whole index after each edit.
  • Store the index in encrypted, sandboxed storage tied to the operating system’s data protection.
  • Return source references with each answer so users can verify the passage.

The decision table below matches common situations with a recommended configuration.

SituationRecommended choiceReason
Small corpus under 10,000 chunksExact flat searchFast enough and no index tuning
Large mail or file archiveHNSW with int8 vectorsSublinear search within RAM limits
Many exact identifiersHybrid BM25 and vector searchEmbeddings blur codes and names
Low-end phone1B generator, top-3 chunksKeeps prefill under a few seconds
Strict offline requirementBundle all models with the appNo download or network dependency
Configuration choices by situation

Limitations and common failure modes

The main limitation of on-device RAG is that small generators reason poorly over many passages, so multi-document synthesis, long comparisons and arithmetic across sources degrade quickly. Retrieval also cannot fix a question whose answer is spread across dozens of files.

  • Stale index: files change but embeddings are not refreshed, so answers cite outdated content.
  • Embedding mismatch: upgrading the embedding model without re-indexing makes old and new vectors incompatible.
  • Context overflow: too many chunks push the prompt beyond the window and silently truncate evidence.
  • Thermal throttling: sustained indexing or long answers slow the chip and drain the battery.
  • Prompt injection from local files: a malicious document can contain instructions that the model obeys.

Privacy is strong but not automatic. Logs, crash reports and analytics can still leak query text, and indexes stored without encryption are readable by other software with file access, so the whole data path must be audited.

The idea in brief

On-device RAG gives a small AI model on your phone or laptop a way to look things up in your own files before it answers. Instead of relying only on what it memorized, it first searches your notes, emails or documents, picks the most relevant pieces and then writes a reply based on them.

Think of a student taking an open-book exam in a quiet room. The book is your data, the student is the model, and nobody outside the room sees either. The result is helpful answers about your own information that never leave your device, even without an internet connection.

A practical scenario

A field service technician maintains industrial pumps in plants where mobile coverage is weak and customer documents may not be uploaded to external servers. The company installs a tablet app that indexes about 1,200 pages of service manuals, wiring diagrams and past repair reports, producing roughly 9,000 chunks and a vector index of about 30 MB.

On site, the technician asks why a specific pump model reports a pressure fault after a seal change. The app runs hybrid search, retrieves four passages from the relevant manual and two earlier repair notes, and a 3B model quantized to 4 bits produces an answer in about five seconds, listing the likely causes and the torque values from the manual.

The outcome is faster diagnosis with no network dependency and no confidential documents leaving the tablet. When a manual is revised, only the changed pages are re-embedded overnight while the device charges.