Binary embeddings are vector representations in which each dimension is reduced to a single bit, usually by keeping only the sign of the original floating-point value. In Retrieval & knowledge work within artificial intelligence, they serve as a compact, fast first-stage index for semantic search, trading some ranking precision for a 32-fold reduction in memory compared with 32-bit float vectors.
The technique is also called binary quantization of embeddings. It does not require a new model: an existing embedding model produces dense vectors, and a simple thresholding step converts them into bit strings compared with Hamming distance, which modern processors compute with a handful of instructions.
How binary quantization turns float vectors into bits
Binary quantization maps every component of a dense vector to 1 when it is above a threshold and to 0 otherwise. The default threshold is zero, so the bit records the sign of the value; some pipelines instead use the per-dimension median computed on a calibration sample, which balances the number of ones and zeros and preserves more information.
The resulting bits are packed into bytes, so a 1,024-dimensional vector occupies 128 bytes instead of 4,096 bytes in float32. Similarity is then measured with Hamming distance, the count of differing bits, obtained through an XOR operation followed by a population count (popcount). Because these operations run natively on CPUs and GPUs, scanning millions of binary codes is far cheaper than computing cosine similarity on floats.
The following table shows how the same vector shrinks under common storage precisions.
| Precision | Bits per dimension | Bytes per vector | Relative size |
|---|---|---|---|
| Float32 | 32 | 4,096 | 100% |
| Float16 | 16 | 2,048 | 50% |
| Int8 (scalar quantization) | 8 | 1,024 | 25% |
| Binary | 1 | 128 | about 3% |
Why sign bits preserve semantic similarity
Binary codes work because high-dimensional embeddings spread meaning across many coordinates. The sign pattern captures the orientation of the vector, and orientation is exactly what cosine similarity measures. This relationship is related to random hyperplane hashing, a locality-sensitive hashing method in which the probability that two bits agree grows with the cosine similarity of the original vectors.
The approximation is not perfect. Values close to zero flip easily, so small differences between near neighbors are lost, and documents with nearly identical scores may swap positions. In practice, binary search alone typically retains roughly 85 to 95 percent of float retrieval quality on standard benchmarks, depending on the model, its dimensionality and whether it was trained with quantization in mind.
The factors below explain most of the variation in quality loss.
| Factor | Effect on quality | Practical note |
|---|---|---|
| Embedding dimensionality | More dimensions retain more information | Vectors under 384 dimensions lose noticeably more |
| Quantization-aware training | Reduces the gap to float search | Some recent models are tuned for binary output |
| Threshold choice | Median thresholds balance bit usage | Requires a calibration sample |
| Data domain | Narrow domains crowd vectors together | Fine distinctions suffer most |
Binary embeddings compared with other compression methods
Binary embeddings are the most aggressive form of scalar compression, but they are not the only way to shrink a vector index. Int8 scalar quantization keeps more precision at a quarter of the size, product quantization learns codebooks for subvectors, and Matryoshka embeddings shorten vectors by truncating dimensions rather than reducing bits. These methods are complementary and are often combined.
| Method | What is reduced | Typical compression | Main trade-off |
|---|---|---|---|
| Binary quantization | Bits per dimension | 32x | Largest precision loss |
| Int8 quantization | Bits per dimension | 4x | Needs value range calibration |
| Product quantization | Subvectors to code IDs | 16x to 64x | Requires training codebooks |
| Matryoshka truncation | Number of dimensions | 2x to 8x | Model must support it |
A frequent combination is Matryoshka truncation followed by binarization, which can reduce a vector to a few dozen bytes. Graph-based indexes such as HNSW and disk-oriented designs such as DiskANN can store binary codes as well, although exhaustive Hamming scans are already fast enough for many collections of up to tens of millions of items.
Rescoring and the two-stage retrieval pattern
Production systems rarely rank final results by Hamming distance alone. The standard design is oversample, then rescore: the binary index returns several times more candidates than needed, and those candidates are reordered with a more precise similarity score. This step recovers most of the lost quality at a small cost.
- Encode the query with the same embedding model used for the documents.
- Binarize the query and retrieve the top candidates by Hamming distance, often 4 to 10 times the final result count.
- Rescore candidates with the float query against binary or int8 document vectors, or with full float vectors loaded from disk.
- Optionally pass the shortlist to a cross-encoder reranker for the final order.
Asymmetric rescoring, in which the query stays in float precision while documents remain binary, is popular because it requires no extra document storage. With rescoring, reported recall frequently reaches 96 percent or more of the float baseline, which is why binary indexes are now a mainstream option in vector databases and RAG pipelines.
When to choose binary embeddings
Binary embeddings fit workloads where memory and latency dominate cost and a modest ranking error can be corrected downstream. They are less suitable when the collection is small, because the savings are negligible, or when every position in the ranking matters.
| Situation | Recommended choice | Reason |
|---|---|---|
| Hundreds of millions of vectors | Binary index with rescoring | Keeps the index in RAM |
| Fewer than one million vectors | Float32 or int8 | Savings do not justify quality loss |
| Strict ranking precision required | Int8 plus reranker | Binary codes blur near ties |
| Edge or mobile deployment | Binary or Matryoshka binary | Tight memory budgets |
| Low-dimensional model under 384 | Avoid binarization | Too little information per bit |
Limitations and common mistakes
The most common error is evaluating without rescoring and concluding that binary search is unusable. Other pitfalls come from mismatched pipelines and unrealistic expectations about what one bit can encode.
- Binarizing the query differently from documents, for example with another threshold, which silently corrupts distances.
- Assuming every model binarizes well; quality depends strongly on how the model distributes values around zero.
- Skipping a domain-specific evaluation, because public benchmark retention may not hold for legal, medical or code corpora.
- Discarding the float vectors entirely when later rescoring or reindexing may need them.
Understanding it simply
Imagine describing every document with a long list of yes-or-no answers instead of precise measurements. Each answer says only whether a feature is present or not. The list becomes tiny, and comparing two lists is just counting where the answers differ.
That is what binary embeddings do for search. They keep the rough direction of meaning while throwing away fine detail, so a computer can sift through enormous collections quickly. The key idea is a fast rough pass followed by a careful check of the best candidates, which keeps results accurate.
Applied in practice
A research platform indexes about 250 million scientific paragraphs with a 1,024-dimensional embedding model. Stored as float32, the vectors would need roughly one terabyte of memory, which would require a large and expensive server cluster. After binary quantization, the index shrinks to about 32 gigabytes and fits on a single machine.
For each query, the system retrieves the top 400 candidates by Hamming distance, rescores them against int8 vectors kept on fast disk storage, and passes the best 40 to a reranker. Internal evaluation shows retrieval quality within about two percent of the float baseline, while median search latency falls from hundreds of milliseconds to under 50 milliseconds, making the retrieval step of the RAG assistant affordable at scale.