Data aggregation is the process of combining many detailed records into summary values, such as totals, counts, averages or percentiles, usually grouped by one or more dimensions like date, product or region. In data processing, it turns large volumes of raw events or transactions into compact figures that people and systems can read, compare and act on.
Aggregation is both a query operation and a pipeline stage. Analysts aggregate on the fly with a SQL GROUP BY, while data platforms precompute aggregates in batch or streaming jobs so that dashboards and reports do not have to scan billions of rows each time. In both cases, the essential trade-off is the same: detail is exchanged for speed and clarity.
How data aggregation works
Data aggregation works by partitioning records into groups according to grouping keys and then applying an aggregate function to a measure within each group. Grouping by day and store, for example, produces one output row per day and store with the sum of sales for that combination.
In distributed engines such as Apache Spark, aggregation runs in two phases. Each worker first computes partial aggregates on its local data, and then partial results with the same key are shuffled to one worker and merged into the final value. This pattern, inherited from MapReduce, keeps network traffic small because only partial results move between machines.
The table below lists the most common aggregate functions and how they behave in distributed processing.
| Function | Result | Mergeable from partial results |
|---|---|---|
| SUM | Total of a measure | Yes, sum of partial sums |
| COUNT | Number of records | Yes, sum of partial counts |
| AVG | Arithmetic mean | Yes, if sum and count are kept |
| MIN and MAX | Extreme values | Yes, minimum or maximum of partials |
| COUNT DISTINCT | Number of unique values | Not exactly; sketches approximate it |
| MEDIAN and percentiles | Position-based statistics | Not exactly; approximate digests used |
Additive, semi-additive and non-additive measures
Whether an aggregate can be rolled up further depends on the type of measure, and this classification determines which precomputed summaries remain correct. Ignoring it is one of the most common causes of wrong totals in reports.
| Measure type | Example | Safe aggregation |
|---|---|---|
| Additive | Sales amount, units sold | Sum across all dimensions |
| Semi-additive | Account balance, inventory level | Sum across accounts, not across time |
| Non-additive | Ratios, averages, distinct counts | Recompute from components |
Averages illustrate the problem clearly. The average of daily averages is not the monthly average when days have different numbers of transactions. The correct approach is to store the sum and count separately and divide only at the final level. Likewise, distinct users per day cannot be added to get distinct users per month, because the same person may appear on many days.
Levels and timing of aggregation
Aggregation can happen at different points in the data flow, from the moment data is captured to the moment a report is displayed. The choice affects freshness, cost and how much detail survives for later analysis.
| Stage | Typical approach | Trade-off |
|---|---|---|
| At query time | SQL GROUP BY on detailed tables | Always fresh, slower on large data |
| Batch precomputation | Nightly or hourly summary tables | Fast reads, data as fresh as last run |
| Materialized views | Database maintains stored results | Automatic upkeep, refresh overhead |
| Stream processing | Windowed aggregates over events | Near real-time, complex late-data handling |
| At the source | Devices send summaries instead of events | Lowest volume, detail lost forever |
In streaming, aggregates are computed over windows: fixed tumbling windows of, for example, one minute, overlapping sliding windows, or session windows that close after a period of inactivity. Because events can arrive late, stream processors use watermarks to decide when a window is complete enough to emit.
Aggregation compared with related processing steps
Data aggregation is often discussed together with other processing steps, but it has a distinct purpose: it reduces the number of rows by summarizing them. Other steps keep the row count roughly the same or change it for different reasons.
| Step | Effect on records | Purpose |
|---|---|---|
| Data aggregation | Many rows become one per group | Summarize for analysis and reporting |
| Data transformation | Rows reshaped or recalculated | Standardize formats and values |
| Data enrichment | Columns added from other sources | Add context such as geography |
| Deduplication | Duplicate rows removed | Ensure each entity is counted once |
| Feature engineering | Model inputs derived, often by aggregating | Prepare data for machine learning |
The order of steps matters. Deduplication and data cleaning must precede aggregation, because duplicates and invalid values are invisible once they have been summed into totals.
Risks and common mistakes
Aggregation discards information irreversibly, so its errors are hard to detect from the results alone. A total that looks plausible can still be built on double counting or on a misleading grouping.
- Join fan-out: joining a fact table to a one-to-many table before summing multiplies rows and inflates totals.
- Simpson’s paradox: a trend visible in every subgroup can reverse when the groups are combined, because group sizes differ.
- Time zone boundaries: daily aggregates computed in different time zones assign the same events to different days.
- Null handling: most aggregate functions ignore nulls, so an average may silently exclude many records.
- Small-group disclosure: aggregates over very few people can reveal individual data, so privacy rules often require minimum group sizes.
Keeping the detailed source data available, documenting the grouping keys and measure definitions, and reconciling aggregates with source totals after each run prevents most of these problems from reaching reports.
In simple terms
Data aggregation means adding things up into a summary. A shop does not read every single receipt to know how the month went; it looks at total sales per week and per department.
The summary is much easier to understand, but some detail disappears along the way. That is why careful teams keep the original receipts and make sure each one is counted exactly once before the totals are calculated.
A practical example
A national delivery company records about 30 million parcel scan events per day, each with a parcel identifier, depot, timestamp and status. Managers want a dashboard of daily deliveries, on-time rates and average delivery times for each of 150 depots.
The data team builds a nightly batch job that first removes duplicate scans, then aggregates events into a summary table with one row per depot and day. Instead of storing average delivery times, it keeps the total delivery minutes and the number of parcels, so weekly and regional averages can be computed correctly.
The dashboard now reads about 55,000 summary rows per year instead of billions of events and loads in under a second. When a region asks for on-time rates by hour, the team adds an hourly aggregate from the preserved detailed data rather than trying to split the daily totals.