Delta Lake is an open-source table format and storage layer that adds ACID transactions, version history and schema enforcement to collections of Parquet files kept in a data lake. Among data formats, it does not define a new way to encode individual records; instead, it defines how many data files plus a transaction log together form one reliable, queryable table.
Delta Lake was originally developed at Databricks and released as open source in 2019, and it is now hosted by the Linux Foundation. It is a central building block of the lakehouse architecture, which aims to combine the low cost and openness of object storage with the reliability and management features traditionally found in data warehouses.
How the Delta Lake transaction log works
The defining component of Delta Lake is the _delta_log directory, an ordered log that records every change made to the table. Each committed transaction writes a new, sequentially numbered JSON commit file that lists the actions it performed, such as adding data files, removing data files or changing table metadata.
Readers reconstruct the current table state by replaying these actions. To keep this fast, Delta Lake periodically writes Parquet checkpoint files that summarize the whole state, by default every 10 commits, so a reader loads the latest checkpoint and only the few commits after it. The table below lists the main pieces of a Delta table on storage.
| Component | Format | Purpose |
|---|---|---|
| Data files | Parquet | Store the actual rows in columnar form |
| Commit files | JSON, one per version | Record add, remove and metadata actions |
| Checkpoint files | Parquet | Snapshot of table state for faster reads |
| Last checkpoint pointer | Small JSON file | Tells readers where the newest checkpoint is |
| Deletion vectors | Compact bitmap files | Mark deleted rows without rewriting files |
Concurrent writers use optimistic concurrency control. Each writer prepares its changes, then attempts to create the next commit file atomically. If another writer committed first, the late writer checks for logical conflicts and either retries or fails, which guarantees serializable or snapshot isolation depending on configuration.
Key features and table operations
Delta Lake turns a folder of immutable files into a table that supports database-style operations. Because Parquet files cannot be edited in place, updates and deletes are implemented by writing new files and logically removing old ones in a single commit.
| Feature | What it does | Practical benefit |
|---|---|---|
| ACID transactions | Commits all file changes atomically | No partial writes visible to readers |
| Time travel | Queries earlier versions by number or timestamp | Audits, reproducible reports, rollback |
| Schema enforcement | Rejects writes that do not match the schema | Prevents silent data corruption |
| Schema evolution | Adds columns when explicitly allowed | Accommodates changing sources |
| MERGE | Combines inserts, updates and deletes | Efficient upserts and change data capture |
| Change data feed | Exposes row-level changes between versions | Incremental downstream processing |
| OPTIMIZE and clustering | Compacts small files and co-locates related data | Faster scans and better data skipping |
Delta Lake also collects column-level statistics such as minimum and maximum values for each data file. Query engines use them for data skipping, reading only files that can contain matching rows, which often reduces scan volume dramatically on selective filters.
Delta Lake compared with Parquet and Apache Iceberg
Delta Lake is frequently confused with Parquet, but the two operate at different layers. Parquet is a file format for storing columnar data in one file, while Delta Lake is a table format that coordinates many Parquet files through a log. Apache Iceberg is the closest competitor, solving the same problem with a different metadata design.
| Aspect | Plain Parquet | Delta Lake | Apache Iceberg |
|---|---|---|---|
| Layer | File format | Table format | Table format |
| Transactions | None across files | ACID via transaction log | ACID via metadata snapshots |
| Metadata structure | Footer inside each file | Sequential JSON log with checkpoints | Metadata files, manifest lists, manifests |
| Partition handling | Directory conventions | Explicit partitions or liquid clustering | Hidden partitioning with transforms |
| Time travel | Not supported | Supported by version or timestamp | Supported by snapshot |
| Strongest ecosystem | Universal | Apache Spark and Databricks | Multi-engine, broad vendor support |
The practical gap between Delta Lake and Iceberg has narrowed. Both now offer row-level deletes, schema evolution and broad engine support, and Delta Lake provides UniForm, which writes Iceberg-compatible metadata alongside the Delta log so that Iceberg readers can query the same files. The choice is often driven by the existing platform rather than by core capabilities.
When to use Delta Lake
Delta Lake is most valuable when many processes write to the same data or when data must be corrected after it lands. For write-once archives that are rarely queried, plain Parquet may be simpler and sufficient.
| Situation | Recommendation | Reason |
|---|---|---|
| Streaming and batch jobs write one table | Use Delta Lake | Transactions keep readers consistent |
| Frequent upserts from operational systems | Use Delta Lake with MERGE | Avoids rewriting whole partitions manually |
| Regulatory audits require history | Use Delta Lake with tuned retention | Time travel reproduces past states |
| Static export shared with external tools | Plain Parquet may suffice | No coordination overhead needed |
| Several engines share tables equally | Compare Delta Lake with UniForm and Iceberg | Engine support decides interoperability |
A common design is the medallion architecture, in which raw data lands in a bronze layer, is cleaned into a silver layer and is aggregated into a gold layer, with each layer stored as Delta tables. This structure makes data lineage easier to follow and lets teams reprocess a layer from the one below it.
Limitations and operational pitfalls
Delta Lake removes many data lake problems, but it introduces maintenance duties that teams sometimes overlook. Most production issues stem from file layout and retention settings rather than from the format itself.
- Small file accumulation. Frequent small writes, especially from streaming, create thousands of tiny files; regular OPTIMIZE runs or auto-compaction are needed.
- VACUUM and time travel conflict. VACUUM deletes unreferenced files older than the retention threshold, 7 days by default, after which older versions can no longer be queried.
- Over-partitioning. Partitioning by a high-cardinality column produces many tiny partitions; liquid clustering or coarser partitions usually perform better.
- Protocol versions. Enabling newer table features raises the reader and writer protocol version, which can make the table unreadable by older engines.
- Not a database server. Delta Lake has no built-in query service or indexes in the database sense; performance depends on the engine and on data layout.
Storage costs can also grow unexpectedly, because every update retains old files until vacuumed. Teams should set retention periods deliberately, balancing audit requirements against storage spending.
The short version
Imagine a large binder of pages where many people add, replace and remove pages at the same time. Without rules, someone could read the binder halfway through a change and get a confusing mix of old and new pages.
Delta Lake adds a numbered logbook to the front of the binder. Every change is written in the logbook in one step, so readers always see a complete, consistent version, and they can even look up what the binder contained last week. The pages themselves are ordinary Parquet files; the logbook is what makes them behave like one reliable table.
Where it makes a difference
An online marketplace ingests about 40 million order events per day from a streaming platform into its data lake, while analysts run revenue dashboards on the same data every 15 minutes. With plain Parquet folders, dashboards occasionally showed partial batches, and correcting late refunds meant rewriting entire daily partitions overnight.
After converting the order data to Delta Lake, the streaming job appends events in transactional commits, and a MERGE job applies refunds and status changes every hour. Dashboards now always read a complete, consistent table version. When a faulty deployment wrote incorrect currency values, engineers used time travel to compare versions and restored the table to the last correct version within minutes, avoiding a full reload of three days of data. A nightly OPTIMIZE job keeps file counts manageable, and VACUUM retention is set to 30 days to match the finance team’s audit window.