ORC, short for Optimized Row Columnar, is an open-source columnar file format for storing large tables, designed to make analytical reads fast and storage compact. It organizes rows into large stripes, stores each column of a stripe separately with type-specific encoding and compression, and embeds indexes and statistics that let readers skip data they do not need. Among data formats for big data processing, ORC is the closest relative of Parquet.
ORC originated in the Apache Hive project in 2013 as a replacement for earlier Hive file formats and later became the independent Apache ORC project. It remains the native format for transactional Hive tables and is supported by engines such as Spark, Trino, Presto and Impala.
How an ORC file is organized
An ORC file is divided into stripes, each holding a horizontal slice of rows stored column by column, followed by a file footer and a small postscript. A reader first opens the postscript at the end of the file, which describes compression and the footer length, then reads the footer to find stripe locations, the schema and file-level statistics.
Inside each stripe, data for every column is split into streams: a presence stream that marks null values, a data stream with the encoded values, and for some types a length or dictionary stream. Each stripe also has an index section with statistics for groups of rows, called row groups, which by default contain 10,000 rows.
The following table summarizes the main structural elements and what each one contributes.
| Element | Contents | Purpose |
|---|---|---|
| Postscript | Compression kind, footer length, version | Entry point for readers |
| File footer | Schema, stripe list, file statistics | Planning and file-level skipping |
| Stripe | Column streams for a slice of rows | Unit of parallel reading |
| Stripe index | Row group statistics and positions | Skipping inside a stripe |
| Stripe footer | Stream locations and encodings | Locating column data |
| Bloom filters | Optional per-column membership filters | Skipping on equality predicates |
Encoding, compression and statistics
ORC reduces file size by combining type-aware encoding with a general-purpose compression codec. Integers are written with run-length encoding that detects repeated values, fixed deltas and small bit widths, while strings use dictionary encoding when the number of distinct values is low relative to the row count and direct encoding otherwise.
After encoding, streams are compressed in chunks, typically of 256 KB, with a codec chosen at write time. Each column also carries minimum, maximum, count and null statistics at the file, stripe and row group levels, and numeric columns can include sums. These statistics power predicate pushdown, which lets a query with a filter such as a date range skip entire stripes or row groups.
| Codec | Relative size | Typical use |
|---|---|---|
| ZLIB | Small | Long-term storage, the traditional default |
| ZSTD | Small, often smaller than ZLIB | Good balance of ratio and speed |
| SNAPPY | Medium | Fast reads and writes |
| LZ4 | Medium | Very fast decompression |
| NONE | Largest | Debugging or pre-compressed data |
ORC compared with other data formats
ORC competes most directly with Parquet, and both serve the same purpose: efficient columnar storage for analytics. The practical differences lie in ecosystem support and in details of indexing and nested data handling, not in fundamental capability.
| Format | Layout | Typical strength |
|---|---|---|
| ORC | Columnar stripes with row group indexes | Hive ACID tables, fine-grained skipping |
| Parquet | Columnar row groups and pages | Broadest ecosystem support |
| Apache Avro | Row-based binary records | Streaming and schema evolution in pipelines |
| CSV | Row-based text | Simple exchange, no types or statistics |
| Apache Arrow | Columnar in-memory format | Zero-copy processing between tools |
Formats such as Apache Iceberg and Delta Lake are not alternatives to ORC but table formats that manage collections of data files and their metadata. Iceberg can use ORC files as its underlying storage, adding snapshots, schema evolution and partition management on top.
Tuning ORC files
ORC performance depends heavily on stripe size, row ordering and file size. Default stripe sizes in current writers are typically 64 MB, although some deployments use larger values, and the right setting balances read parallelism against memory needed while writing.
| Setting | Typical value | Effect |
|---|---|---|
| Stripe size | 64 to 256 MB | Larger stripes improve compression and scan speed |
| Row index stride | 10,000 rows | Granularity of skipping inside stripes |
| Compression block size | 256 KB | Trade-off between ratio and random access |
| Bloom filter columns | High-cardinality lookup keys | Faster equality filters, larger files |
| Dictionary threshold | Fraction of distinct values | Chooses dictionary or direct encoding |
Sorting data on frequently filtered columns before writing is often the single most effective optimization, because it makes minimum and maximum statistics narrow and lets predicate pushdown skip most of the file. Without sorting, every stripe may span the full value range and nothing can be skipped.
Limitations and pitfalls
ORC is optimized for large, append-oriented analytical tables, so it performs poorly when used as a general-purpose store. The most common issues come from file sizing and from expectations that do not match a columnar layout.
- Small files problem: thousands of tiny ORC files carry footer and open overhead for each file and erase the benefits of large stripes.
- Single-row access: reading one record requires decompressing whole chunks of several columns, so ORC is unsuitable for transactional lookups.
- In-place updates: files are immutable; updates and deletes need delta files and compaction, as in Hive ACID tables.
- Ecosystem gaps: some tools and libraries support Parquet more completely than ORC, which can influence the format choice.
Regular compaction into larger files and writing through engines that respect stripe settings resolve most of these problems in practice.
Stripped to the essentials
ORC is a way of saving very large tables so that computers can read only the parts they need. Instead of writing one row after another, it groups values by column, like a filing cabinet with a separate drawer for names, dates and prices.
Each drawer also has a label saying the smallest and largest value inside. If someone asks only for last week’s sales, the reader checks the labels and skips drawers that cannot contain the answer, which saves a great deal of time and disk space.
Where it is applied
A telecommunications operator stores call detail records in a Hive-based data lake, adding about two billion rows per day. Each record has around 40 columns, but typical reports use only five or six of them and filter by date and cell tower.
The engineering team writes daily partitions as ORC files of roughly 256 MB with ZSTD compression, sorted by tower identifier, and adds Bloom filters on the subscriber identifier for fraud investigations. The data occupies a fraction of the space of the previous text files.
Queries that once scanned the full day now read only the required columns and skip most stripes thanks to the sorted statistics. Investigators looking up a single subscriber benefit from the Bloom filters, which eliminate most stripes before any data is decompressed.