NLP, short for natural language processing, is the branch of artificial intelligence that enables computers to analyze, interpret and generate human language. It covers tasks from splitting text into words and tagging their grammatical roles to translating documents, answering questions and holding open-ended conversations.
Within speech and language AI, NLP is the text-centered core that other components connect to. Speech-to-text turns audio into words, NLP extracts meaning or produces a response, and text-to-speech turns that response back into audio. Since the late 2010s, the field has been reshaped by pretrained transformer models, which now underpin most practical NLP systems.
How NLP turns text into computable form
NLP turns language into computable form by breaking text into tokens and mapping each token to a numeric vector that a model can process. Modern systems use subword tokenization, such as byte-pair encoding, so that rare or new words are split into known fragments instead of being treated as unknown.
The resulting vectors, called embeddings, place words and phrases with similar meanings close together in a high-dimensional space. A transformer model then refines each embedding using self-attention, which lets every token weigh information from every other token in the context, so that “bank” in “river bank” and “bank loan” receives different representations.
| Stage | What happens | Example output |
|---|---|---|
| Normalization | Encoding fixes, casing and whitespace cleanup | Consistent UTF-8 text |
| Tokenization | Text split into words or subword units | “unhappiness” may become “un”, “happi”, “ness” |
| Embedding | Tokens mapped to dense vectors | Vectors of several hundred to thousands of dimensions |
| Contextual encoding | Transformer layers mix information across tokens | Context-aware token representations |
| Task head or decoding | Classification, tagging or text generation | A label, spans or a generated answer |
Core NLP tasks and how they are measured
NLP tasks are usually grouped by what the system must output: a label for the whole text, labels for individual spans, or new text. Each group has established evaluation metrics, and choosing the wrong metric is a frequent source of misleading results.
| Task | Output type | Typical metric | Example use |
|---|---|---|---|
| Text classification | One label per document | Accuracy, macro F1 | Routing support tickets |
| Named entity recognition | Labeled text spans | Entity-level F1 | Extracting names, dates, amounts |
| Sentiment analysis | Polarity or score | Accuracy, F1 | Monitoring product reviews |
| Machine translation | Text in another language | BLEU, chrF, COMET | Localizing documentation |
| Summarization | Shorter text | ROUGE, human rating | Condensing reports |
| Question answering | Span or generated answer | Exact match, F1 | Answering from manuals |
Overlap metrics such as BLEU and ROUGE reward shared word sequences with a reference text and therefore underrate correct paraphrases. For generative tasks, teams increasingly combine them with learned metrics, human ratings or model-based judges.
From rules to large language models
NLP has moved through three broad methodological eras, and each one is still in use where its strengths fit. Understanding them explains why a modern system may combine regular expressions, a small classifier and a large language model in one product.
| Approach | Core technique | Strength | Main limitation |
|---|---|---|---|
| Rule-based | Hand-written grammars and patterns | Predictable, auditable, no training data | Brittle, costly to extend |
| Statistical | N-gram models, hidden Markov models, CRFs | Learns from data, efficient | Needs feature engineering, short context |
| Neural pretrained | Transformers such as BERT and GPT-style models | Strong transfer across tasks and domains | Compute cost, opacity, hallucination |
The transformer architecture, introduced in 2017, made large-scale pretraining practical. Encoder models such as BERT excel at understanding tasks like classification and extraction, while decoder models in the GPT family generate text and, at sufficient scale, perform many tasks from instructions alone.
NLP within speech and language AI
NLP is the meaning layer of speech and language AI, and neighboring technologies handle conversion between audio and text or between languages. The boundaries matter when diagnosing errors, because a wrong answer from a voice assistant may originate in recognition, understanding or synthesis.
| Technology | Input | Output | Relationship to NLP |
|---|---|---|---|
| Speech-to-text | Audio | Text transcript | Feeds text into NLP |
| NLP | Text | Labels, structure or text | Interprets and produces language |
| Translation | Text in one language | Text in another language | A core NLP task |
| Text-to-speech | Text | Synthetic audio | Voices NLP output |
| Voice cloning | Short voice samples | A synthetic voice model | Customizes text-to-speech output |
Subfields are often described as natural language understanding (NLU), which extracts meaning, and natural language generation (NLG), which produces text. Large language models blur this division because one model performs both.
Limitations and common mistakes
The main limitations of NLP stem from ambiguity, uneven data coverage and the gap between fluency and correctness. A system can produce grammatical, confident text that is factually wrong, and evaluation must be designed to catch this.
- Hallucination: generative models may state unsupported facts, so high-stakes uses need grounding in source documents and verification.
- Low-resource languages receive far less training data, and quality can drop sharply outside widely spoken languages.
- Domain shift: a model trained on news or web text often misreads legal, clinical or technical vocabulary.
- Bias in training text can surface as stereotyped or unequal outputs across groups.
- Tokenization effects: languages with complex morphology or non-Latin scripts may need more tokens per word, which raises cost and shortens the usable context.
A common mistake is to reach for a large generative model when a small classifier suffices. For narrow, high-volume tasks with clear labels, a fine-tuned compact model is usually cheaper, faster and easier to evaluate.
Stripped to the essentials
NLP is the technology that lets computers work with ordinary human language. It is what allows a phone to understand a spoken question, an email system to spot spam or a translation tool to turn a menu into another language.
The computer does not understand words the way people do. It turns them into numbers and learns patterns from enormous amounts of text. The key takeaway: NLP is very good at patterns in language, but its output still needs checking whenever accuracy really matters.
Where it is applied
An insurance company receives about 40,000 customer emails a month, and staff spend hours sorting them by hand. The company fine-tunes a compact transformer classifier on 25,000 historical emails labeled with 12 categories, such as claim status, address change and complaint, and adds a named entity recognition step that extracts policy numbers and dates.
On a held-out test set of 3,000 emails, the classifier reaches a macro F1 of about 0.90. Emails with a confidence below a set threshold, around 15% of traffic, go to human reviewers. The remaining 85%, roughly 34,000 emails a month, are routed automatically with the extracted fields prefilled, and the outcome is that average first-response time falls from two days to the same day while staff concentrate on complex cases.