Semi-structured data is data that does not conform to a fixed, predefined schema such as the columns of a relational table, yet contains self-describing markers, such as keys, tags or attributes, that identify its elements and organize them into hierarchies. Among data fundamentals, it occupies the space between structured data in tables and unstructured data such as free text, images and audio.
Common examples are JSON documents, XML files, YAML configurations, log events and email headers. Each record carries its own field names, so two records in the same collection can contain different fields, nested objects and lists of varying length, a property often described as schema-on-read rather than schema-on-write.
What makes data semi-structured
The defining feature is that structure travels with the data. A JSON record such as {"name": "Ana", "orders": [101, 102]} states field names explicitly, so a program can interpret it without a separate schema definition. This self-description is what separates semi-structured data from unstructured content, where meaning must be extracted by analysis.
Semi-structured data is typically modeled as a tree or graph rather than as rows and columns. Objects contain other objects, arrays hold repeated elements, and the depth of nesting can vary from record to record. Fields may be missing, repeated or of different types across records, which gives flexibility but shifts the burden of validation to the consumer.
| Property | Structured data | Semi-structured data | Unstructured data |
|---|---|---|---|
| Schema | Fixed, defined before loading | Implicit in the data, flexible | None in machine-readable form |
| Organization | Rows and columns | Nested keys, tags, arrays | Free-form content |
| Typical formats | Relational tables, CSV with fixed columns | JSON, XML, YAML, Avro, Parquet with nesting | Text, images, audio, video |
| Query approach | SQL on known columns | Path expressions, flattening, document queries | Search, parsing, machine learning |
| Change of structure | Requires schema migration | New fields appear without migration | Not applicable |
Common formats for semi-structured data
Several formats dominate, and they differ in readability, size, typing and schema support. Text formats are easy to inspect and exchange, while binary formats with schemas are faster and safer for large-scale processing.
| Format | Encoding | Schema support | Typical use |
|---|---|---|---|
| JSON | Text | Optional, for example JSON Schema | Web APIs, document databases, event payloads |
| XML | Text | XML Schema, DTD | Document exchange, industry standards, configuration |
| YAML | Text | Optional | Configuration files, infrastructure definitions |
| Avro | Binary | Required, stored with the data | Streaming pipelines, schema evolution |
| Parquet with nested types | Binary, columnar | Required | Analytical storage in data lakes |
Binary formats such as Avro and Parquet show that the boundary is not absolute. They preserve nested, flexible structures but attach an explicit schema, which combines the flexibility of semi-structured data with the type safety and compression of structured storage.
Storing and querying semi-structured data
Systems handle semi-structured data in three main ways. Document databases store each record natively and index fields inside it. Relational databases offer dedicated column types, such as JSON or binary JSON, with functions for extracting and indexing paths. Data warehouses and lakehouses store nested data in columnar formats and let analysts query it with SQL extensions.
Querying relies on path expressions that navigate the hierarchy, such as selecting the city inside a customer’s address object. Arrays require additional operations to unnest or flatten them into rows before aggregation. Standards and dialects include JSONPath, the SQL/JSON functions defined in the SQL standard, XPath and XQuery for XML.
| Situation | Suitable approach | Reason |
|---|---|---|
| Records vary widely and are read whole | Document database | Native storage, no mapping layer |
| Mostly relational data with a few flexible attributes | JSON column in a relational table | Keeps joins and transactions |
| Large volumes of event data for analysis | Columnar files or warehouse with nested types | Compression and fast scans |
| Raw data of unknown future use | Data lake landing zone | Preserves the original structure |
Schema-on-read and schema evolution
With schema-on-read, structure is imposed when the data is used, not when it is written. Producers can add fields at any time, which speeds up development and integration. The trade-off is that every consumer must handle missing fields, unexpected types and renamed keys, and errors surface late, often in reports or downstream jobs.
Mature pipelines therefore introduce controlled schema evolution. Contracts describe expected fields, schema registries track versions, and compatibility rules state which changes are allowed, for example adding an optional field is usually safe, while renaming or retyping a field breaks consumers. Validation at ingestion turns silent drift into visible, actionable errors.
Limitations and common pitfalls
Flexibility is the main strength of semi-structured data and also its main risk. Without discipline, collections accumulate inconsistent field names, mixed types and deeply nested structures that are expensive to query and hard to trust, which directly harms data quality.
- Type drift: the same field holds a number in some records and a string in others.
- Naming inconsistency, such as customer_id, customerId and custID for the same concept.
- Excessive nesting, which makes analytical queries slow and complex.
- Large repeated arrays inside one document, which can exceed size limits and slow updates.
- Lost metadata about meaning and units, because self-describing keys name fields without explaining them.
A useful rule is to keep raw semi-structured data unchanged in a landing area for traceability and data lineage, and to derive curated, consistently typed tables for reporting. This preserves flexibility at the edge while giving analysts a stable structure.
A plain-language summary
A spreadsheet forces every row to have the same columns. A pile of letters has no fixed layout at all. Semi-structured data sits in between: each record is like a filled-in form where every answer is labeled, but different forms may ask slightly different questions.
Because the labels are always there, a computer can still find a name, a date or a list of purchases in each record. The flexibility makes it easy to collect varied information, but someone must keep the labels consistent so the data stays easy to use.
An example from practice
A mobile app company collects user interaction events as JSON from millions of devices. Each event has a common core, with event type, timestamp and device identifier, but different event types carry different extra fields, and new fields appear with almost every app release.
The data team lands the raw events unchanged in cloud storage, registers a versioned schema for each event type, and runs a nightly job that validates the events and loads them into a warehouse table with a nested properties column. Analysts query common fields as ordinary columns and reach rare fields through path expressions. When a release accidentally changes a numeric field to text, validation flags the drift within a day instead of corrupting a month of reports.