A data lake is a centralized repository for raw data that keeps large volumes in their native format, structured, semi-structured and unstructured, usually on inexpensive object storage, so that it can be processed and analyzed later for many purposes. In data engineering, the data lake is defined by schema-on-read: data is landed first and given structure only when a job or query reads it.
The concept emerged with distributed file systems in the early 2010s and now typically runs on cloud object storage with open file formats such as Parquet, ORC and Avro. It sits beside the data warehouse in most analytics architectures and, through open table formats, has evolved into the lakehouse.
How a data lake stores and organizes data
A data lake stores data as files in object storage, organized by path conventions rather than by database tables. Compute engines such as Spark, Trino or cloud query services read those files directly, which separates storage from compute: storage grows cheaply, and processing capacity is added only when jobs run.
Most lakes are organized into zones or layers that reflect how refined the data is. Raw data is kept unchanged so that any downstream result can be recomputed from the original, while cleaner layers serve analysts and applications.
| Zone | Content | Typical format | Main consumers |
|---|---|---|---|
| Raw or bronze | Data exactly as received from sources | JSON, CSV, logs, original files | Data engineers, reprocessing |
| Cleaned or silver | Validated, deduplicated, typed records | Parquet or an open table format | Data engineers, data scientists |
| Curated or gold | Aggregated, business-ready datasets | Parquet or an open table format | Analysts, dashboards, models |
| Sandbox | Experimental and temporary datasets | Any | Individual analysts and scientists |
Within each zone, data is usually partitioned by date or another key in the folder structure, for example by ingestion day. Query engines skip folders that cannot match a filter, which keeps scans affordable on datasets of many terabytes.
Key design parameters of a data lake
Data lake performance and cost depend on a handful of physical design choices that are easy to get wrong early and expensive to change later.
| Parameter | Common choice | What it affects |
|---|---|---|
| File format | Columnar Parquet or ORC for analytics | Scan speed, compression, column pruning |
| File size | Roughly 128 MB to 1 GB per file | Too small files slow listing and planning |
| Partition key | Event or ingestion date | How much data each query skips |
| Table format | Apache Iceberg, Delta Lake or Apache Hudi | Transactions, schema evolution, time travel |
| Catalog | Hive-compatible or open REST catalog | Discovery, permissions, table definitions |
The small files problem is the most common performance failure: streaming ingestion that writes thousands of kilobyte-sized files per hour makes every query spend most of its time opening files. Regular compaction jobs merge them into larger files.
Data lake compared with warehouse and lakehouse
A data lake, a data warehouse and a lakehouse differ mainly in when structure is enforced and what guarantees the storage layer gives.
| Aspect | Data lake | Data warehouse | Lakehouse |
|---|---|---|---|
| Schema | On read | On write | On write for tables, raw data still possible |
| Data types | Any, including images and logs | Mostly structured | Any |
| Transactions | None on plain files | Full ACID | ACID through table format |
| Storage cost | Low, object storage | Higher, managed storage | Low, object storage |
| Typical users | Engineers, data scientists | Analysts, business users | Both groups |
The lake also shapes integration patterns. Because raw data can be landed before it is modeled, lakes favor ELT, where transformation happens after loading, over classic ETL, where data is transformed before it reaches the warehouse. A data pipeline then moves data between zones on a schedule or as a stream.
When a data lake is the right choice
A data lake is the right choice when an organization must retain diverse data cheaply before it knows every future question. It is a weaker choice when the main need is fast, governed reporting on a stable set of business entities, which a warehouse serves with less engineering effort.
| Situation | Recommended approach | Reason |
|---|---|---|
| Large volumes of logs, telemetry or files | Data lake as the primary store | Object storage keeps long history affordable |
| Machine learning on raw or unstructured data | Data lake with curated zones | Models need detail that warehouses often discard |
| Stable financial and sales reporting | Data warehouse, optionally fed from the lake | Strict schemas and fast SQL for business users |
| Mixed analytics and reporting needs | Lakehouse with an open table format | One copy of data serves both workloads |
Teams should also weigh operational skills. A lake shifts work from the database vendor to the data engineering team, which must manage file layout, compaction, catalogs and permissions that a managed warehouse handles automatically.
Governance and the data swamp risk
Without governance, a data lake degrades into a data swamp: a large collection of files nobody can find, understand or trust. Cheap storage encourages landing everything, and schema-on-read postpones the questions of meaning and quality until someone tries to use the data.
- Register every dataset in a catalog with owner, description, schema and refresh schedule.
- Apply access control at table or column level, not only on storage buckets, and mask personal data.
- Track data lineage from raw files to curated datasets, so errors can be traced and fixed at the source.
- Define retention and deletion rules, which privacy regulations such as GDPR require for personal data.
- Run data quality checks when data moves from raw to cleaned zones.
Deleting individual records is harder in a lake than in a database, because plain files are immutable and must be rewritten. Open table formats make row-level deletes practical, which is one of the main reasons organizations adopt them.
The short version
A data lake is like a large storage hall where an organization keeps every kind of information it collects. Spreadsheets, website logs, photos and sensor readings all go in, in their original form. Nothing has to be sorted on arrival.
This makes collecting data cheap and flexible. The risk is that the hall becomes a mess. Labels, an index and clear owners are what keep it useful instead of overwhelming.
Where it makes a difference
A logistics company collects GPS pings from 12,000 delivery vehicles, warehouse scanner events and customer support emails. Its data warehouse could hold only the structured order data, so the telemetry was discarded after 30 days.
The data engineering team builds a data lake on object storage. Raw telemetry lands as compressed JSON, nightly jobs convert it to Parquet partitioned by day in an Iceberg table, and a catalog documents each dataset. Two years of history now cost a fraction of warehouse storage. Data scientists use it to train a delivery-time prediction model, and curated aggregates flow back into the warehouse for the operations dashboards, giving both teams one consistent source of telemetry.