Data wrangling, also called data munging, is the process of transforming raw data into a clean, consistent and well-structured dataset that is fit for a specific analysis or application. In data processing, it combines several activities, including discovering, structuring, cleaning, enriching and validating data, and it typically precedes analytics, reporting and machine learning.
Wrangling is usually iterative and purpose-driven: the same source can be wrangled differently for a dashboard, a statistical model or a data product. It is often the most time-consuming part of a data project, because real data arrive with inconsistent formats, missing values, duplicates and structures designed for operations rather than analysis.
The stages of data wrangling
Data wrangling is commonly described as a sequence of stages, although practitioners move back and forth between them as new issues appear. Each stage answers a different question about the data.
| Stage | Key question | Example activities |
|---|---|---|
| Discovery | What is in the data? | Profiling columns, counting values and nulls |
| Structuring | Is the shape right for the task? | Parsing, reshaping, splitting columns |
| Cleaning | Are the values correct and consistent? | Fixing types, standardizing codes, handling nulls |
| Enriching | Is anything missing for the task? | Joining reference data, deriving fields |
| Validating | Does the result meet expectations? | Rule checks, row counts, range tests |
| Publishing | How will others use it? | Writing outputs, documenting changes |
Discovery comes first because decisions about cleaning depend on what is actually wrong. Skipping it leads to rules written for imagined problems while real ones, such as a date column stored as text in three formats, go unnoticed.
Structuring data into a usable shape
Structuring changes the layout of the data without changing its meaning. A widely used target is the tidy data layout: each variable forms a column, each observation forms a row and each type of observational unit forms its own table.
Source data often violate this layout. Spreadsheets put years in column headers, logs pack several values into one string, and nested JSON holds lists inside records. The table below lists the core reshaping operations that resolve these patterns.
| Operation | What it does | Typical use |
|---|---|---|
| Pivot longer (melt) | Turns columns into rows | Year columns into a year variable |
| Pivot wider | Turns rows into columns | One column per measured attribute |
| Split and extract | Separates combined values | Parsing codes or full names |
| Flatten or explode | Unnests lists and nested fields | Order records with line item arrays |
| Join | Combines tables on keys | Adding customer attributes to orders |
| Aggregate | Summarizes rows by group | Daily totals from event records |
Joins deserve particular care. A join on a key that is not unique on one side silently multiplies rows, so checking row counts before and after every join is one of the most effective wrangling habits.
Cleaning and standardizing values
Cleaning corrects values that are wrong, inconsistent or unusable. It overlaps with data cleaning as a separate discipline, but within wrangling it is driven by what the target analysis needs.
- Type conversion: parsing dates, numbers and booleans stored as text, with explicit formats and time zones.
- Standardization: unifying codes, units, spelling variants and capitalization, such as country names and currencies.
- Missing data handling: deciding between leaving nulls, flagging them, imputation or exclusion.
- Deduplication: removing exact duplicates and resolving near-duplicates with record linkage.
- Outlier review: separating data entry errors from genuine extreme values.
Every cleaning rule encodes an assumption. Replacing missing incomes with the median, for example, narrows the apparent spread and can bias models. Good practice keeps the original value, records the rule and adds a flag column that marks altered rows.
Data wrangling compared with related processes
Data wrangling is a broad, exploratory activity that overlaps with several narrower or more formalized processes. The distinctions mostly concern scope, automation and who performs the work.
| Process | Scope | Typical setting |
|---|---|---|
| Data wrangling | Discovery to validated dataset | Analysts and data scientists, often interactive |
| Data cleaning | Correcting values and errors | A part of wrangling or data quality work |
| Data transformation | Changing format, structure or values | Any pipeline step, often automated |
| ETL | Extract, transform and load | Scheduled production pipelines |
| Feature engineering | Creating model inputs | Machine learning after wrangling |
In practice, wrangling often starts as interactive exploration in notebooks or spreadsheets and is later hardened into automated pipelines once the steps are stable. The transition is where documentation and tests become essential.
Tools, reproducibility and pitfalls
Common tools range from dataframe libraries such as pandas, Polars and dplyr to SQL, distributed engines such as Apache Spark for large volumes, and visual preparation tools for non-programmers. The choice depends on data volume, team skills and whether the result must run repeatedly.
| Pitfall | Consequence | Prevention |
|---|---|---|
| Manual edits in files | Steps cannot be repeated | Scripted transformations under version control |
| Silent row loss in joins | Incomplete results | Row-count checks and anti-joins |
| Ambiguous date parsing | Day and month swapped | Explicit formats and validation |
| Overwriting raw data | No way to audit or redo | Keep raw data immutable |
| Undocumented assumptions | Results misinterpreted | Change logs and data dictionaries |
The most important principle is reproducibility: the path from raw input to final dataset should be rerunnable from code, so that corrected source data or a new month of records can be processed with the same documented steps.
A plain-language summary
Data wrangling is like preparing ingredients before cooking. Vegetables arrive muddy, in different sizes and sometimes spoiled. Someone has to wash, sort, cut and measure them before any recipe can work.
Raw data are the same. Dates are written in different ways, some boxes are empty and some entries appear twice. Wrangling turns that mess into a tidy, reliable table that people and programs can actually use.
An example from practice
A public health analyst receives monthly vaccination records from 14 regional offices. Some send spreadsheets with one column per month, others send CSV exports with dates written as day-month-year or month-day-year, and two offices use their own codes for vaccine types.
The analyst writes a script that reshapes all files into one row per region, month and vaccine, parses dates with explicit per-office formats, maps local codes to a standard list and removes about 2% of records that appear twice after offices resubmitted corrections. Validation rules check that dose counts are non-negative and that every region reports every month.
The first run takes a week to build; later months take minutes instead of days. When one office corrects last quarter’s figures, the analyst simply reruns the script, and the national dashboard updates with a documented, auditable trail of every change.