Delta Lake

On this page 7

Delta Lake is an open-source table format and storage layer that adds ACID transactions, version history and schema enforcement to collections of Parquet files kept in a data lake. Among data formats, it does not define a new way to encode individual records; instead, it defines how many data files plus a transaction log together form one reliable, queryable table.

Delta Lake was originally developed at Databricks and released as open source in 2019, and it is now hosted by the Linux Foundation. It is a central building block of the lakehouse architecture, which aims to combine the low cost and openness of object storage with the reliability and management features traditionally found in data warehouses.

How the Delta Lake transaction log works

The defining component of Delta Lake is the _delta_log directory, an ordered log that records every change made to the table. Each committed transaction writes a new, sequentially numbered JSON commit file that lists the actions it performed, such as adding data files, removing data files or changing table metadata.

Readers reconstruct the current table state by replaying these actions. To keep this fast, Delta Lake periodically writes Parquet checkpoint files that summarize the whole state, by default every 10 commits, so a reader loads the latest checkpoint and only the few commits after it. The table below lists the main pieces of a Delta table on storage.

ComponentFormatPurpose
Data filesParquetStore the actual rows in columnar form
Commit filesJSON, one per versionRecord add, remove and metadata actions
Checkpoint filesParquetSnapshot of table state for faster reads
Last checkpoint pointerSmall JSON fileTells readers where the newest checkpoint is
Deletion vectorsCompact bitmap filesMark deleted rows without rewriting files
Components of a Delta table on storage

Concurrent writers use optimistic concurrency control. Each writer prepares its changes, then attempts to create the next commit file atomically. If another writer committed first, the late writer checks for logical conflicts and either retries or fails, which guarantees serializable or snapshot isolation depending on configuration.

Key features and table operations

Delta Lake turns a folder of immutable files into a table that supports database-style operations. Because Parquet files cannot be edited in place, updates and deletes are implemented by writing new files and logically removing old ones in a single commit.

FeatureWhat it doesPractical benefit
ACID transactionsCommits all file changes atomicallyNo partial writes visible to readers
Time travelQueries earlier versions by number or timestampAudits, reproducible reports, rollback
Schema enforcementRejects writes that do not match the schemaPrevents silent data corruption
Schema evolutionAdds columns when explicitly allowedAccommodates changing sources
MERGECombines inserts, updates and deletesEfficient upserts and change data capture
Change data feedExposes row-level changes between versionsIncremental downstream processing
OPTIMIZE and clusteringCompacts small files and co-locates related dataFaster scans and better data skipping
Main Delta Lake features and what they enable

Delta Lake also collects column-level statistics such as minimum and maximum values for each data file. Query engines use them for data skipping, reading only files that can contain matching rows, which often reduces scan volume dramatically on selective filters.

Delta Lake compared with Parquet and Apache Iceberg

Delta Lake is frequently confused with Parquet, but the two operate at different layers. Parquet is a file format for storing columnar data in one file, while Delta Lake is a table format that coordinates many Parquet files through a log. Apache Iceberg is the closest competitor, solving the same problem with a different metadata design.

AspectPlain ParquetDelta LakeApache Iceberg
LayerFile formatTable formatTable format
TransactionsNone across filesACID via transaction logACID via metadata snapshots
Metadata structureFooter inside each fileSequential JSON log with checkpointsMetadata files, manifest lists, manifests
Partition handlingDirectory conventionsExplicit partitions or liquid clusteringHidden partitioning with transforms
Time travelNot supportedSupported by version or timestampSupported by snapshot
Strongest ecosystemUniversalApache Spark and DatabricksMulti-engine, broad vendor support
How Delta Lake differs from related formats

The practical gap between Delta Lake and Iceberg has narrowed. Both now offer row-level deletes, schema evolution and broad engine support, and Delta Lake provides UniForm, which writes Iceberg-compatible metadata alongside the Delta log so that Iceberg readers can query the same files. The choice is often driven by the existing platform rather than by core capabilities.

When to use Delta Lake

Delta Lake is most valuable when many processes write to the same data or when data must be corrected after it lands. For write-once archives that are rarely queried, plain Parquet may be simpler and sufficient.

SituationRecommendationReason
Streaming and batch jobs write one tableUse Delta LakeTransactions keep readers consistent
Frequent upserts from operational systemsUse Delta Lake with MERGEAvoids rewriting whole partitions manually
Regulatory audits require historyUse Delta Lake with tuned retentionTime travel reproduces past states
Static export shared with external toolsPlain Parquet may sufficeNo coordination overhead needed
Several engines share tables equallyCompare Delta Lake with UniForm and IcebergEngine support decides interoperability
Deciding whether Delta Lake fits a workload

A common design is the medallion architecture, in which raw data lands in a bronze layer, is cleaned into a silver layer and is aggregated into a gold layer, with each layer stored as Delta tables. This structure makes data lineage easier to follow and lets teams reprocess a layer from the one below it.

Limitations and operational pitfalls

Delta Lake removes many data lake problems, but it introduces maintenance duties that teams sometimes overlook. Most production issues stem from file layout and retention settings rather than from the format itself.

  • Small file accumulation. Frequent small writes, especially from streaming, create thousands of tiny files; regular OPTIMIZE runs or auto-compaction are needed.
  • VACUUM and time travel conflict. VACUUM deletes unreferenced files older than the retention threshold, 7 days by default, after which older versions can no longer be queried.
  • Over-partitioning. Partitioning by a high-cardinality column produces many tiny partitions; liquid clustering or coarser partitions usually perform better.
  • Protocol versions. Enabling newer table features raises the reader and writer protocol version, which can make the table unreadable by older engines.
  • Not a database server. Delta Lake has no built-in query service or indexes in the database sense; performance depends on the engine and on data layout.

Storage costs can also grow unexpectedly, because every update retains old files until vacuumed. Teams should set retention periods deliberately, balancing audit requirements against storage spending.

The short version

Imagine a large binder of pages where many people add, replace and remove pages at the same time. Without rules, someone could read the binder halfway through a change and get a confusing mix of old and new pages.

Delta Lake adds a numbered logbook to the front of the binder. Every change is written in the logbook in one step, so readers always see a complete, consistent version, and they can even look up what the binder contained last week. The pages themselves are ordinary Parquet files; the logbook is what makes them behave like one reliable table.

Where it makes a difference

An online marketplace ingests about 40 million order events per day from a streaming platform into its data lake, while analysts run revenue dashboards on the same data every 15 minutes. With plain Parquet folders, dashboards occasionally showed partial batches, and correcting late refunds meant rewriting entire daily partitions overnight.

After converting the order data to Delta Lake, the streaming job appends events in transactional commits, and a MERGE job applies refunds and status changes every hour. Dashboards now always read a complete, consistent table version. When a faulty deployment wrote incorrect currency values, engineers used time travel to compare versions and restored the table to the last correct version within minutes, avoiding a full reload of three days of data. A nightly OPTIMIZE job keeps file counts manageable, and VACUUM retention is set to 30 days to match the finance team’s audit window.