On-device RAG (also called local RAG) is a form of retrieval-augmented generation in which the embedding model, the vector index, the retriever and the language model all run on the user’s own hardware, so documents are searched and answers are generated without sending data to a cloud service. It is one of the most practical patterns in Edge AI, because it lets a small local model answer questions about personal files, messages or manuals that it never saw during training.
The approach trades the scale of data-center models for privacy, offline availability and predictable latency. Its design is dominated by tight budgets for memory, storage, battery and thermal headroom, which shape every choice from the embedding dimension to the context length.
How on-device RAG retrieves and generates locally
On-device RAG works as a two-phase pipeline: an indexing phase that turns local content into searchable vectors, and a query phase that retrieves relevant passages and feeds them to a local model. Both phases run on the device’s CPU, GPU or neural processing unit (NPU).
During indexing, documents are split into chunks of typically 200 to 500 tokens, each chunk is converted into a dense vector by a compact embedding model, and the vectors are stored in an on-device index, often inside SQLite or a dedicated embedded vector library. Indexing usually runs in the background while charging, because embedding thousands of chunks is the most energy-intensive step.
At query time the question is embedded with the same model, the index returns the top-k nearest chunks (commonly 3 to 8), and the chunks are inserted into the prompt of a quantized local language model. The model then writes an answer grounded in those passages, ideally citing which file each statement came from.
The table below lists the components and the constraints that typically apply on consumer hardware.
| Component | Typical on-device choice | Main constraint |
|---|---|---|
| Chunker | Fixed-size or sentence-aware splitting | Chunk count drives index size |
| Embedding model | 20M to 150M parameters, 256 to 768 dimensions | Indexing time and battery |
| Vector index | Flat search, HNSW or IVF with quantized vectors | RAM and storage footprint |
| Reranker | Optional small cross-encoder | Added query latency |
| Generator | 1B to 8B parameter model at 4-bit precision | Memory and tokens per second |
Resource budgets that shape the design
Every on-device RAG system is sized against fixed hardware budgets rather than elastic cloud capacity. The largest consumer of memory is the generator: a 3B parameter model at 4-bit quantization needs roughly 1.5 to 2 GB of RAM for weights, plus a KV cache that grows with context length.
Retrieved chunks directly consume context, so a prompt with eight chunks of 400 tokens adds about 3,200 tokens of prefill work. On a mobile NPU this can mean one to several seconds before the first token appears, which is why retrieval precision matters more than recall on small devices.
The following ranges are typical for current phones and laptops and vary with chip generation, runtime and model architecture.
| Resource | Phone range | Laptop range |
|---|---|---|
| Generator weights at 4-bit | 0.5 to 2 GB | 2 to 5 GB |
| Vector index for 50,000 chunks | 20 to 150 MB | 20 to 150 MB |
| Time to first token | 0.5 to 4 s | 0.2 to 2 s |
| Generation speed | 10 to 30 tokens/s | 20 to 80 tokens/s |
| Usable context window | 2,000 to 8,000 tokens | 4,000 to 32,000 tokens |
Index size can be cut by vector quantization, such as int8 or binary embeddings, which shrinks storage by four to thirty-two times with a modest loss in retrieval accuracy. Matryoshka-style embeddings that can be truncated to fewer dimensions offer a similar trade-off.
On-device RAG compared with related approaches
On-device RAG differs from cloud RAG mainly in where data and computation live, and from fine-tuning in how knowledge enters the model. It also overlaps with neighboring Edge AI patterns such as hybrid inference and on-device personalization.
| Approach | Where knowledge lives | Key strength | Key weakness |
|---|---|---|---|
| On-device RAG | Local index read at query time | Private, offline, easy to update | Limited model quality and context |
| Cloud RAG | Server-side index and model | Large models, large corpora | Data leaves the device |
| Hybrid inference | Local retrieval, remote generation | Better answers when online | Partial privacy exposure |
| On-device personalization | Adapted model weights | No retrieval latency | Hard to update or delete facts |
| Local LLM without retrieval | Training data only | Simplest stack | No access to personal content |
A key advantage over weight-based adaptation is deletability: removing a document from the index removes it from future answers immediately, which supports data-protection obligations far more cleanly than retraining.
Designing a reliable local retrieval pipeline
A reliable on-device RAG pipeline is built by tuning retrieval first and choosing the smallest generator that answers well with good context. Weak retrieval cannot be rescued by a small model, while a strong retriever lets a 3B model perform close to much larger ones on document questions.
- Use hybrid search that combines BM25 keyword scoring with vector similarity, because names, codes and numbers are often missed by embeddings alone.
- Attach metadata filters such as date, app or folder to narrow the candidate set before vector search.
- Index incrementally with change detection instead of rebuilding the whole index after each edit.
- Store the index in encrypted, sandboxed storage tied to the operating system’s data protection.
- Return source references with each answer so users can verify the passage.
The decision table below matches common situations with a recommended configuration.
| Situation | Recommended choice | Reason |
|---|---|---|
| Small corpus under 10,000 chunks | Exact flat search | Fast enough and no index tuning |
| Large mail or file archive | HNSW with int8 vectors | Sublinear search within RAM limits |
| Many exact identifiers | Hybrid BM25 and vector search | Embeddings blur codes and names |
| Low-end phone | 1B generator, top-3 chunks | Keeps prefill under a few seconds |
| Strict offline requirement | Bundle all models with the app | No download or network dependency |
Limitations and common failure modes
The main limitation of on-device RAG is that small generators reason poorly over many passages, so multi-document synthesis, long comparisons and arithmetic across sources degrade quickly. Retrieval also cannot fix a question whose answer is spread across dozens of files.
- Stale index: files change but embeddings are not refreshed, so answers cite outdated content.
- Embedding mismatch: upgrading the embedding model without re-indexing makes old and new vectors incompatible.
- Context overflow: too many chunks push the prompt beyond the window and silently truncate evidence.
- Thermal throttling: sustained indexing or long answers slow the chip and drain the battery.
- Prompt injection from local files: a malicious document can contain instructions that the model obeys.
Privacy is strong but not automatic. Logs, crash reports and analytics can still leak query text, and indexes stored without encryption are readable by other software with file access, so the whole data path must be audited.
The idea in brief
On-device RAG gives a small AI model on your phone or laptop a way to look things up in your own files before it answers. Instead of relying only on what it memorized, it first searches your notes, emails or documents, picks the most relevant pieces and then writes a reply based on them.
Think of a student taking an open-book exam in a quiet room. The book is your data, the student is the model, and nobody outside the room sees either. The result is helpful answers about your own information that never leave your device, even without an internet connection.
A practical scenario
A field service technician maintains industrial pumps in plants where mobile coverage is weak and customer documents may not be uploaded to external servers. The company installs a tablet app that indexes about 1,200 pages of service manuals, wiring diagrams and past repair reports, producing roughly 9,000 chunks and a vector index of about 30 MB.
On site, the technician asks why a specific pump model reports a pressure fault after a seal change. The app runs hybrid search, retrieves four passages from the relevant manual and two earlier repair notes, and a 3B model quantized to 4 bits produces an answer in about five seconds, listing the likely causes and the torque values from the manual.
The outcome is faster diagnosis with no network dependency and no confidential documents leaving the tablet. When a manual is revised, only the changed pages are re-embedded overnight while the device charges.