Apache Avro is an open-source, row-oriented binary serialization format in which every piece of data is written and read according to a schema defined in JSON. Among data formats, Avro is known for compact encoding, fast serialization and well-defined rules for schema evolution, which let producers and consumers of data change independently without breaking each other.
Avro is used mainly in data pipelines and event streaming, for example as the message format of Apache Kafka topics together with a schema registry, and as a storage format for raw records in data lakes. It also defines a container file format and a simple remote procedure call protocol, although the RPC part is used far less today.
How Avro encodes data with a schema
Avro separates the structure of data from the data itself. The schema, written in JSON, declares a record name, its fields, their types and optional defaults. The binary encoding then contains only the values, in field order, with no field names and no type tags.
Integers and longs use variable-length zig-zag encoding, so small numbers take one byte; strings and bytes are written as a length followed by the content; a union is encoded as the index of the chosen branch followed by the value. Because nothing in the bytes identifies fields, a reader always needs the writer’s schema to decode them.
| Type category | Types | Typical use |
|---|---|---|
| Primitive | null, boolean, int, long, float, double, bytes, string | Basic field values |
| Record | Named set of typed fields | Entities and events |
| Enum | Fixed list of symbols | Status codes and categories |
| Array and map | Repeated values, string-keyed values | Lists and attributes |
| Union | One of several types, often null and another | Optional fields |
| Fixed | Byte sequence of fixed length | Hashes and identifiers |
| Logical types | decimal, date, timestamp-millis, uuid and others | Semantic meaning on top of primitives |
The object container file stores the writer schema once in the file header, followed by blocks of records separated by a random sync marker. Blocks can be compressed with codecs such as deflate, Snappy or zstandard, and the sync markers make files splittable for parallel processing.
Schema evolution and compatibility rules
Schema evolution is the main reason teams choose Avro. During decoding, Avro performs schema resolution between the writer schema, used when the data was produced, and the reader schema, which the consuming application expects. Fields are matched by name, not by position.
Fields present in the writer schema but missing in the reader schema are skipped. Fields present only in the reader schema take their declared default value; if there is no default, resolution fails. Certain type promotions are allowed, for example int to long or float to double.
| Change | Backward compatible | Forward compatible | Condition |
|---|---|---|---|
| Add a field | Yes | Yes | The new field has a default |
| Remove a field | Yes | Yes | The removed field had a default |
| Rename a field | Only with an alias | No | Aliases map the old name |
| Promote int to long | Yes | No | Reader uses the wider type |
| Change string to int | No | No | Incompatible types |
| Add an enum symbol | Yes | Only with an enum default | Old readers need a fallback |
In streaming systems, a schema registry stores every schema version and checks compatibility before a producer may register a new one. Each message then carries only a small schema identifier instead of the full schema, keeping messages compact.
Avro compared with other data formats
Avro competes with both text formats and other binary formats, and the right choice depends on access pattern and evolution needs. Avro is a row format optimized for writing and reading whole records, not for scanning a few columns.
| Format | Layout | Schema handling | Best fit |
|---|---|---|---|
| Apache Avro | Binary, row-oriented | Required, resolved at read time | Event streams, record ingestion |
| Protocol Buffers | Binary, row-oriented | Numbered fields, code generation | Service APIs and RPC |
| JSON | Text, row-oriented | Optional, external | Human-readable interchange |
| Parquet | Binary, columnar | Embedded in file footer | Analytical scans |
| ORC | Binary, columnar | Embedded in file footer | Analytical scans in Hive-style systems |
| Apache Arrow | In-memory, columnar | Embedded | Fast in-memory exchange |
Compared with Protocol Buffers, Avro needs no generated code and handles dynamically discovered schemas well, but it depends on the writer schema being available. Compared with Parquet, Avro writes records faster and suits append-heavy pipelines, while Parquet reads selected columns far more efficiently. A common architecture lands data in Avro and later compacts it into Parquet for analysis.
Practical use and design guidelines
Avro works best when schemas are treated as managed contracts between teams. A few design habits make evolution painless over years of changes.
- Give every optional field a default, usually by declaring a union with null first and a null default.
- Never reuse a removed field name for a different meaning.
- Use logical types for decimals and timestamps instead of plain strings.
- Choose a compatibility mode in the registry, such as backward or full, and enforce it in CI.
- Keep namespaces stable, because the full record name is part of schema identity.
Typical file sizes for Avro containers in data lakes range from tens to hundreds of megabytes; very small files waste metadata and slow down processing engines.
Limitations and common pitfalls
Avro’s compactness comes from dependence on the schema, and most problems arise when that dependence is overlooked. Raw Avro bytes without the writer schema are practically unreadable.
- Lost writer schema: messages stored without a schema identifier cannot be decoded later.
- Missing defaults: a field added without a default breaks old data for new readers.
- Column queries on Avro: analytical scans read every field and run slower than on columnar formats.
- Union ambiguity: in the JSON encoding, union values need type wrappers, which surprises many users.
Stripped to the essentials
Avro is a way of packing data tightly before sending or storing it. Instead of writing the label next to every value, as a spreadsheet with repeated headings would, it writes only the values and keeps the labels in a separate description called a schema.
Anyone who has the schema can unpack the data. When the description changes, for example a new field is added, Avro knows how to read old and new data side by side, so systems do not break when their data grows over time.
Where it is applied
A ride-hailing platform publishes trip events to Kafka from dozens of services: trip requested, driver assigned, trip completed. Each event type has an Avro schema registered in a schema registry with backward compatibility enforced. A message carries a few bytes of schema identifier and a compact binary body, typically several times smaller than the equivalent JSON.
When the pricing team adds a surge multiplier field with a default of 1.0, consumers that have not been updated keep working, and new consumers read old events with the default value. Events are archived hourly as compressed Avro container files in object storage and compacted daily into Parquet tables for analysts, so the same data serves both real-time processing and reporting.