Missing data are values that should exist in a dataset but were not recorded, were lost or cannot apply, appearing as nulls, empty strings, sentinel codes or absent rows. In data processing, missing data is not only a cleaning nuisance: the reason values are missing determines whether analyses built on the remaining data are trustworthy.
Every real dataset has gaps. Survey respondents skip questions, sensors drop readings, integrations fail and optional form fields stay empty. The core task is to detect missing values reliably, understand their mechanism and choose a treatment such as deletion, flagging or imputation that does not distort results. Handled carelessly, missing data silently biases averages, breaks joins and weakens models.
Missingness mechanisms
Statistics distinguishes three missingness mechanisms, which describe how the probability of a value being missing relates to the data. The mechanism, more than the amount of missing data, decides which treatments are valid.
| Mechanism | Definition | Example | Consequence |
|---|---|---|---|
| Missing completely at random (MCAR) | Missingness unrelated to any data | A random sample of lab tubes breaks in transport | Deletion loses precision but not validity |
| Missing at random (MAR) | Missingness depends only on observed variables | Older users skip an online question more often, and age is recorded | Valid with methods that use the observed variables |
| Missing not at random (MNAR) | Missingness depends on the missing value itself | High earners decline to report income | Bias remains unless the mechanism is modeled |
MCAR can be partly tested by comparing records with and without missing values. MAR and MNAR cannot be distinguished from the data alone, because the evidence needed is exactly what is missing. Deciding between them requires domain knowledge about how the data were collected, and careful analyses report a sensitivity check under different assumptions.
How missing values appear in data
Missing values rarely announce themselves consistently. A key data processing step is normalizing every representation of missing into one explicit form before analysis.
| Representation | Where it occurs | Risk |
|---|---|---|
| NULL | Databases and warehouses | Excluded silently from aggregates and comparisons |
| NaN | Numeric arrays and data frames | Propagates through arithmetic |
| Empty string | CSV exports, text fields | Counted as a valid value |
| Sentinel codes | Legacy systems using 0, -1, 999 or 1900-01-01 | Treated as real numbers or dates |
| Absent rows | Event logs, time series with gaps | Gaps invisible unless expected rows are generated |
Sentinel values are especially dangerous. A temperature of -999 or a birth date of 1900-01-01 passes type checks but distorts means and ranges dramatically. Profiling distributions for impossible or suspiciously frequent values catches most of them.
Treatment options
There is no universally correct treatment. The choice depends on the mechanism, the share of missing values, the role of the variable and the purpose of the analysis.
| Treatment | How it works | Suitable when | Main drawback |
|---|---|---|---|
| Listwise deletion | Drops every record with any missing value | Small MCAR share | Loses data, biased under MAR and MNAR |
| Pairwise deletion | Uses available values per calculation | Correlation matrices, exploratory work | Inconsistent sample sizes across results |
| Simple imputation | Fills with mean, median, mode or constant | Quick baselines, few missing values | Shrinks variance, weakens correlations |
| Indicator flag | Adds a column marking missingness | Missingness itself is informative | More features, needs a fill value too |
| Model-based imputation | Predicts values from other variables | MAR data with good predictors | Complexity, risk of leakage |
| Multiple imputation | Creates several plausible datasets and pools results | Statistical inference under MAR | More computation and careful pooling |
Multiple imputation is the standard for inference because it reflects the uncertainty of the filled values. Single imputation treats guesses as facts, which makes standard errors too small and confidence intervals too narrow.
Missing data in pipelines and machine learning
In production pipelines, missing data should be measured continuously, not only cleaned once. A sudden rise in the null rate of a column often signals an upstream failure, such as a changed source field or a broken integration.
- Track null rates per column and alert when they exceed an expected range.
- Distinguish structural from accidental missing: a cancellation date is legitimately empty for active customers.
- Fit imputation on training data only and apply the same parameters to validation and production data, to avoid data leakage.
- Keep the original value alongside the imputed one, so analysts can see what was filled.
- Document the treatment in the dataset description, because downstream users cannot otherwise tell observed from imputed values.
In feature engineering, a missingness indicator is often among the most predictive features. A customer who never filled in a phone number may differ systematically from one who did, and that difference is information rather than noise.
Common mistakes with missing data
Most errors come from default behaviors of tools rather than from deliberate choices. Aggregation functions skip nulls, joins drop unmatched rows and model libraries may reject or silently fill missing values.
| Mistake | Effect | Correction |
|---|---|---|
| Filling with zero by default | Zero is mistaken for a real measurement | Use an explicit null or a justified value |
| Ignoring nulls in averages | Average describes only respondents | Report the share missing with every metric |
| Deleting rows without checking | Remaining sample is biased | Compare complete and incomplete records first |
| Mean imputation for inference | Overconfident results | Use multiple imputation or model-based methods |
| Treating structural gaps as errors | Valid empty values are overwritten | Define which fields may be legitimately empty |
A practical rule is to report missingness as part of every result. A satisfaction score based on 40% of customers means something different from the same score based on 95%.
Put simply
Missing data are blanks in a table where an answer should be. A form was left half empty, or a sensor stopped working for an hour.
The important question is why the blanks are there. If answers are missing purely by accident, ignoring them is usually fine. If people skipped a question for a reason, such as embarrassment about their income, the remaining answers no longer represent everyone. Filling blanks with guesses can help, but guesses must never be treated as real facts.
In practice
A city utility analyzes household water consumption from smart meters to plan tariffs. About 6% of daily readings are missing. A first analysis simply drops incomplete days and finds lower average consumption than billing data suggest.
A closer look shows that readings fail more often in large, multi-story houses where the radio signal is weak, and those households use more water. The data are missing at random given building type, not completely at random. Dropping the gaps had removed heavy users disproportionately.
The team adds building type and past consumption to a model-based imputation, flags every imputed reading and reports results with and without imputed values. The corrected average matches billing data closely, and the utility schedules antenna upgrades for the buildings with the most missing readings.