A lakehouse is a data architecture that combines the low-cost, open file storage of a data lake with the reliability and management features of a data warehouse, such as ACID transactions, schema enforcement and fast SQL queries. In data engineering, the term describes platforms in which analytical tables live as open columnar files, usually Parquet, in object storage, while a table format layer turns those files into tables that many engines can query safely.
The lakehouse emerged to remove the classic two-tier pattern, in which raw data landed in a lake and a curated copy was then loaded into a separate warehouse. Keeping one copy of the data for business intelligence, data science and machine learning reduces duplication, pipeline complexity and the delay between ingestion and analysis.
How a lakehouse is built
A lakehouse is a stack of independent layers rather than a single product. Storage, table management, metadata and compute are separated, so each can be scaled or replaced on its own. This separation of storage and compute is the main architectural difference from a traditional warehouse appliance.
| Layer | Typical technology | Role |
|---|---|---|
| Object storage | Cloud object stores or on-premises equivalents | Cheap, durable storage of all files |
| File format | Parquet, sometimes ORC or Avro | Columnar, compressed data files |
| Open table format | Delta Lake, Apache Iceberg, Apache Hudi | Transactions, snapshots, schema rules |
| Catalog | Hive Metastore, Iceberg REST catalogs, Unity Catalog | Table names, locations and permissions |
| Compute engines | Apache Spark, Trino, Apache Flink, DuckDB | Batch, streaming and interactive SQL |
The open table format is the decisive component. It keeps a transaction log or metadata tree that lists which data files belong to each version of a table. A write creates new files and then commits a new version atomically, so readers always see a consistent snapshot even while data is being added.
Warehouse features on top of the data lake
Table formats add the features that plain data lakes lacked. Each capability solves a concrete problem that made earlier lakes unreliable, often described as data swamps.
| Capability | Mechanism | Problem solved |
|---|---|---|
| ACID transactions | Atomic metadata commits with optimistic concurrency | Half-written or conflicting updates |
| Schema enforcement and evolution | Schema stored in table metadata | Incompatible files and silent type changes |
| Time travel | Queries against older table snapshots | Audits, reproducibility and rollback |
| Row-level changes | Copy-on-write or merge-on-read files | Updates, deletes and MERGE operations |
| Data skipping | Column statistics and partition metadata | Scanning files that cannot match a query |
Row-level deletes matter beyond analytics, because regulations such as the GDPR right to erasure require removing individual records. In a plain lake this meant rewriting whole partitions by hand; a table format performs the operation as one transactional MERGE or DELETE.
Lakehouse compared with data lake and data warehouse
The lakehouse sits between its two predecessors and borrows from each. The comparison below summarizes the practical differences.
| Aspect | Data lake | Data warehouse | Lakehouse |
|---|---|---|---|
| Storage | Open files in object storage | Proprietary managed storage | Open files in object storage |
| Transactions | None or limited | Full ACID | ACID through table formats |
| Data types | Structured and unstructured | Mainly structured | Both, with tables for structured data |
| Engine choice | Many engines | Usually one vendor engine | Many engines on the same tables |
| Typical workloads | Data science, raw archives | BI and reporting | BI, data science, ML and streaming |
Modern cloud warehouses have narrowed the gap by reading open table formats directly, so the boundary is now a spectrum rather than a strict divide. The lakehouse still differs in principle: the data stays in open formats under the owner’s control, and no single engine is required to read it.
Designing pipelines in a lakehouse
Lakehouse pipelines usually follow ELT rather than ETL: raw data is loaded first and transformed inside the platform. A common way to organize these steps is the medallion architecture, which refines data through three quality layers.
| Layer | Contents | Typical consumers |
|---|---|---|
| Bronze | Raw data as ingested, with load metadata | Data engineers, replay and audit |
| Silver | Cleaned, deduplicated, conformed records | Analysts and data scientists |
| Gold | Aggregated, business-level tables | Dashboards, reports and applications |
Streaming and batch jobs can write to the same tables, since the table format coordinates concurrent commits. Partitioning should follow common filters, such as event date, with file sizes of roughly 128 MB to 1 GB as a widely used target that balances parallelism against metadata overhead.
Limitations and operational pitfalls
A lakehouse shifts work from a managed warehouse to the engineering team. Its typical weaknesses are well known and need deliberate maintenance.
- Small-file proliferation from frequent streaming commits slows queries; regular compaction jobs rewrite files into larger ones.
- Unbounded snapshot history increases storage costs; snapshots must be expired and orphaned files removed on a schedule.
- Concurrency conflicts occur when many writers update the same partitions, because optimistic concurrency retries or fails commits.
- Interactive latency for small queries is often higher than in a tuned warehouse, so caching or a serving layer may still be needed.
- Format and catalog fragmentation can limit interoperability; teams should confirm which engines support their chosen table format version.
Governance is not automatic either. Access control, lineage and data quality checks require a catalog and processes; without them a lakehouse can turn into the same unmanaged data swamp it was meant to replace.
Put simply
Many companies used to keep data in two places. A cheap storage room held everything, messy and unsorted. A tidy, expensive showroom held a cleaned copy for reports. Moving data between the two took time and effort.
A lakehouse keeps everything in the cheap storage room but adds the order of the showroom. Labels, an inventory list and strict rules about changes make the stored data reliable. One organized copy then serves everyone, from report builders to data scientists.
In practice
A mid-sized insurance company runs a nightly ETL job that copies claims data from its data lake into a separate warehouse. Reports are always a day old, and data scientists train fraud models on the lake copy, which sometimes disagrees with the warehouse figures.
The data engineering team moves the claims data into open table format tables on the existing object storage and organizes them into bronze, silver and gold layers. A streaming job updates the silver claims table every few minutes, while a nightly job compacts small files. Dashboards query the gold tables through a SQL engine, and data scientists read the silver tables directly. Reports become near real time, and both groups now work from the same numbers, which ends the reconciliation meetings.