Parquet, formally Apache Parquet, is an open-source columnar file format for storing large analytical datasets, in which the values of each column are stored together, compressed and accompanied by statistics. Among data formats, it is the de facto standard for data lakes and analytical pipelines, supported by Spark, Trino, DuckDB, pandas, Polars and most cloud warehouses.
Unlike row-oriented text formats such as CSV or JSON, a Parquet file is binary, typed and self-describing: it carries its own schema and metadata. Query engines can read only the columns and row ranges they need, which typically makes analytical scans far faster and storage much smaller than with text formats.
How a Parquet file is organized
A Parquet file is split horizontally into row groups and vertically into column chunks. Each row group holds a block of rows, and within it each column is stored as a separate chunk made of pages, the smallest unit of encoding and compression. A footer at the end of the file holds the schema, the location of every chunk and per-chunk statistics.
| Unit | Contains | Typical size |
|---|---|---|
| Row group | A horizontal slice of rows | Often 128 MB or a fixed row count, set by the writer |
| Column chunk | One column’s data within a row group | Varies with column width |
| Page | Encoded and compressed values | About 1 MB by default |
| Footer | Schema, offsets and statistics | Kilobytes to a few megabytes |
Because readers start with the footer, an engine can plan the read without scanning the data. It fetches only the column chunks a query references, which is called column projection, and skips whole row groups whose statistics show they cannot match a filter, known as predicate pushdown.
Row group size is a trade-off. Larger row groups compress better and reduce metadata, while smaller row groups allow finer skipping and more parallel readers. Optional page indexes and Bloom filters in newer writers refine skipping further, letting readers avoid individual pages or rule out values that min and max statistics cannot exclude.
Encoding, compression and types
Storing one column at a time puts similar values next to each other, which makes them highly compressible. Parquet first applies a lightweight encoding, then an optional general-purpose compression codec to each page.
| Technique | Kind | Works best for |
|---|---|---|
| Dictionary encoding | Encoding | Columns with few distinct values, such as country codes |
| Run-length encoding | Encoding | Long runs of repeated values and definition levels |
| Delta encoding | Encoding | Sorted integers and timestamps |
| Snappy | Compression codec | Fast reads with moderate compression |
| Zstandard | Compression codec | Better ratio at similar read speed, tunable levels |
| Gzip | Compression codec | High ratio with slower writes |
Parquet has a small set of physical types, such as 32-bit and 64-bit integers, doubles and byte arrays, plus logical types that give them meaning, such as decimal, date, timestamp and string. Nested structures such as lists and maps are stored using repetition and definition levels, a technique adopted from the Dremel paper, which lets nested data remain columnar.
Parquet compared with other data formats
Parquet is optimized for write once, read many times analytical workloads. Other formats in the same family serve different purposes, and the right choice depends on access pattern rather than fashion.
| Format | Layout | Best fit |
|---|---|---|
| Parquet | Columnar, binary, compressed on disk | Analytical storage and data lakes |
| CSV | Row-based plain text | Simple exchange and spreadsheets |
| JSON and NDJSON | Row-based text with nesting | APIs, logs and event streams |
| Apache Arrow | Columnar, in memory | Fast processing and exchange between tools |
| Avro | Row-based binary with schema | Streaming and record-at-a-time writes |
| ORC | Columnar binary on disk | Hive-centered analytical stacks |
Parquet and Apache Arrow complement each other: Parquet is the compact on-disk format, and Arrow is the in-memory format that engines decode it into for vectorized processing. Converting between them is efficient because both are columnar.
Using Parquet effectively
Parquet performance depends heavily on how files are written. The same data can be fast or slow to query depending on file size, sort order and partitioning.
- Aim for large files, typically hundreds of megabytes each, because thousands of tiny files multiply metadata and request overhead.
- Sort or cluster data by frequently filtered columns so row group statistics become selective.
- Partition directories by a low-cardinality column such as date, not by a high-cardinality identifier.
- Choose Zstandard or Snappy according to whether storage or write speed matters more.
- Keep schemas consistent and evolve them by adding optional columns rather than changing types.
Parquet files are immutable; updating a single row means rewriting a file. Table formats such as Apache Iceberg, Delta Lake and Apache Hudi add a transaction log on top of Parquet files to provide updates, deletes, snapshots and schema evolution.
Limitations and common mistakes
Parquet is a poor fit for frequent small writes, single-row lookups and human inspection. The table lists typical problems teams meet.
| Problem | Cause | Remedy |
|---|---|---|
| Small file problem | Streaming jobs writing tiny files | Compact files on a schedule |
| No row group skipping | Unsorted data spreads values evenly | Sort by common filter columns before writing |
| Schema conflicts | Same column written with different types | Enforce schemas or use a table format |
| Timestamp shifts | Mixed time zone and unit conventions | Standardize on UTC with explicit units |
| Slow point lookups | Columnar layout scans whole pages | Use a database or key-value store for lookups |
Compatibility can also vary. Older readers may not support newer logical types or encodings, so producers shared across many tools often stick to widely supported options.
The short version
Parquet stores a table column by column instead of row by row. Picture a filing cabinet where every drawer holds just one kind of information, all dates in one drawer and all prices in another, each tightly packed.
When someone asks for the average price, only the price drawer needs to be opened. That is why analysis on Parquet files is fast and the files are small, even though changing a single record is more work than in a spreadsheet.
Where it makes a difference
A mobile app company stores its event logs as compressed CSV files, about 2 TB per month, in cloud object storage. Analysts mostly query five of the sixty columns, filtered by date and event type, and each monthly report scans all the data and takes close to an hour.
The data team rewrites the logs into Parquet, partitioned by date, sorted by event type and compressed with Zstandard. Storage falls to a fraction of the original size, and the query engine now reads only the five needed columns and skips row groups for irrelevant event types. The same monthly report runs in a few minutes, and the cost of scanned data drops accordingly.