spaCy

On this page 7

spaCy is an open-source Python library for natural language processing (NLP) that turns raw text into structured linguistic annotations such as tokens, part-of-speech tags, dependency trees and named entities. Among AI frameworks & tools, it occupies the niche of production-grade text processing: it is designed to process large volumes of documents quickly and predictably rather than to serve as a research sandbox.

The library is written in Python and Cython, released under the MIT license and maintained by the company Explosion. Its central idea is the processing pipeline, an ordered chain of components that each add annotations to a shared document object, so that a single call converts text into data that downstream code, search indexes or machine learning models can use.

How the spaCy processing pipeline works

A spaCy pipeline is a Language object, usually named nlp, that first runs a rule-based tokenizer and then passes the resulting Doc through each registered component in order. The tokenizer splits text using language-specific prefix, suffix and infix rules plus exception lists, which is why it handles abbreviations and contractions consistently across millions of documents.

Every component reads and writes the same `Doc` container, which stores tokens as a compact array and exposes them through Token and Span views. Strings are mapped to integer hashes in a shared `Vocab` and `StringStore`, which keeps memory use low and makes annotations cheap to compare. The table below lists the core data structures.

ObjectWhat it representsTypical use
Language (nlp)The loaded pipeline with its settingsProcessing text, adding components
DocOne processed text with all annotationsIterating tokens, entities, sentences
TokenA single word, punctuation mark or symbolReading tags, lemmas, heads
SpanA slice of a Doc, such as an entityEntities, noun chunks, matches
VocabShared lexicon and string hashesMemory-efficient lookups
DocBinSerialized collection of DocsStoring training data as .spacy files
Core spaCy data structures

Statistical components share an embedding layer called tok2vec or, in transformer pipelines, a transformer component. Sharing one encoder across the tagger, parser and entity recognizer saves computation, although it also means that retraining one component can affect the others that listen to the same layer.

Pipeline components and trained models

spaCy ships trained pipelines as installable Python packages, named by language, genre and size, for example en_core_web_sm. Trained pipelines exist for roughly two dozen languages, while tokenization support covers more than 70 languages. The following table summarizes the most common components.

ComponentOutputApproach
tagger and morphologizerPart-of-speech tags and morphological featuresStatistical
parserDependency tree, sentence boundaries, noun chunksTransition-based statistical
nerNamed entities such as PERSON, ORG, DATETransition-based statistical
lemmatizerBase word formsRule-based or lookup tables
entity_ruler and span_rulerEntities or spans from patternsRule-based
textcatDocument-level categoriesStatistical classifier
spancatOverlapping labeled spansStatistical span classifier
Common spaCy pipeline components

Package size reflects a speed and accuracy trade-off. Small packages contain no static word vectors, medium and large packages add vectors, and trf packages use a transformer encoder that raises accuracy but needs far more compute, ideally a GPU.

PipelineApproximate sizeRelative speedBest fit
en_core_web_smAbout 12 MBFastest on CPUHigh-volume batch jobs, prototypes
en_core_web_mdAbout 40 MBFast on CPUSimilarity with word vectors
en_core_web_lgAbout 400 MBFast on CPUBetter vectors, larger memory budget
en_core_web_trfAbout 450 MBSlow on CPU, acceptable on GPUHighest accuracy on parsing and entities
Trade-offs between core web pipeline sizes

Training and configuring custom spaCy models

Since version 3, every training run is defined by a single `config.cfg` file that declares the pipeline, the model architectures, the optimizer, batch sizes and the paths to data. This configuration system, built on the Thinc machine learning library, makes experiments reproducible because the entire setup lives in one versionable text file.

A typical workflow converts annotated examples into DocBin files, runs spacy train with the configuration, and evaluates the result with spacy evaluate. Projects files chain these steps into repeatable workflows. Annotation itself usually happens in a separate tool, and the quality of those labels determines the ceiling of model accuracy far more than hyperparameter tuning does.

Custom components can be registered with a decorator and inserted anywhere in the pipeline. This extensibility lets teams combine statistical models with deterministic rules, for example a pattern-based ruler that guarantees product codes are always tagged, placed before or after the statistical entity recognizer depending on which source should win.

spaCy is an opinionated, application-focused library, whereas neighboring tools either target teaching and research or focus on transformer models in general. The comparison below shows where each one fits.

LibraryPrimary focusStrengthLimitation
spaCyProduction NLP pipelinesSpeed, consistent API, rule integrationOne recommended algorithm per task
NLTKTeaching and linguistic researchMany algorithms and corporaSlower and less suited to production
Hugging Face TransformersPretrained transformer modelsState-of-the-art accuracy, large model hubHeavier compute, less linguistic structure
LangChain and LlamaIndexLLM application orchestrationPrompting, retrieval, agentsNo native tagging or parsing
spaCy compared with neighboring tools

These tools are often complementary rather than competing. The spacy-transformers package plugs Hugging Face Transformers models into a spaCy pipeline, and the spacy-llm package lets a large language model act as a pipeline component, which is useful for zero-shot entity extraction before enough labeled data exists to train a smaller model.

Practical use, limitations and common mistakes

spaCy performs best in information extraction: finding entities, relations and key phrases in contracts, support tickets, clinical notes or news. For throughput, process texts with `nlp.pipe` in batches instead of calling nlp once per text, and disable components that the task does not need.

SymptomLikely causeFix
Slow processingCalling nlp per text, all components enabledUse nlp.pipe, disable unused components
Poor entities in a niche domainPretrained model trained on general web textFine-tune on in-domain annotations
Accuracy dropped after retrainingCatastrophic forgetting of earlier labelsMix original examples into training data
High memory in long servicesGrowing string store from unique tokensReload the pipeline periodically
Wrong sentence splitsUnusual formatting such as lists or logsAdd a rule-based sentencizer or custom rules
Frequent problems and their fixes

The main limitation is that pretrained pipelines reflect their training corpus and label scheme; entity types outside that scheme require custom training. spaCy also does not generate text and does not replace a language model for open-ended reasoning, so it is best treated as a reliable structuring layer around such models.

A non-technical view

spaCy is a toolkit that reads text the way a careful editor would. It splits sentences into words, notes which words are names, places, dates or amounts, and records how the words relate to one another. The result is no longer just text but organized information that a computer can sort, search and count.

Think of it as a fast assembly line for documents. Each station adds one kind of label, and at the end every document carries the same set of labels. The key point is consistency at scale: spaCy can label millions of documents in the same way, which a team of people could never do.

A concrete example

An insurance company receives about 40,000 claim emails per week and wants to route each one to the right department. Its data team builds a spaCy pipeline that starts from en_core_web_md, adds a pattern-based ruler for policy numbers in a fixed format, and fine-tunes the entity recognizer on 3,000 annotated emails so that it recognizes vehicle models, injury types and repair shops.

A text classifier trained on the same emails assigns each message to one of six claim categories. Running with nlp.pipe on ordinary CPU servers, the pipeline processes a full week of email in well under an hour. Manual triage time fell sharply, and because policy numbers come from deterministic rules, every extracted policy number can be audited against the original message.