Keyword extraction is the task of automatically identifying the words and multi-word phrases that best summarize a document, ranked by how central they are to its content. In speech & language AI, a branch of artificial intelligence, it turns unstructured text or transcribed speech into a compact set of descriptors that support search indexing, tagging, content routing and analytics.
The output is usually a ranked list of keyphrases with scores, such as “supply chain disruption” or “interest rate”. Unlike text classification, which assigns labels from a fixed set, keyword extraction draws its candidates from the text itself, although modern systems also generate absent phrases that do not appear verbatim.
How keyword extraction selects candidate phrases
Keyword extraction works as a two-stage pipeline: first it generates candidate phrases, and then it scores and ranks them. Candidate generation typically applies tokenization, part-of-speech tagging and stop word removal, keeping noun phrases that match patterns such as zero or more adjectives followed by one or more nouns.
Scoring then estimates how representative and specific each candidate is. Statistical methods reward terms that are frequent in the document but rare across a reference corpus. Graph methods reward terms that co-occur with many other important terms. Embedding methods reward phrases whose vector representation is semantically close to the whole document.
A final step usually applies lemmatization or stemming to merge variants such as “network” and “networks”, removes near-duplicates and returns the top-k phrases, where k between 5 and 15 is common for articles and reports.
| Stage | Typical operation | Effect on the result |
|---|---|---|
| Preprocessing | Tokenization, lowercasing, text normalization | Consistent units for counting |
| Candidate generation | POS patterns, n-gram windows, noun chunks | Defines which phrases can win |
| Scoring | TF-IDF, graph centrality, embedding similarity | Ranks candidates by importance |
| Post-processing | Deduplication, diversity reranking, top-k cut | Removes redundancy and noise |
Main families of keyword extraction methods
Keyword extraction methods fall into unsupervised, supervised and generative families, and each one trades data requirements against accuracy and control. The choice depends mainly on whether labeled keyphrase data exists for the domain.
| Method family | Representative techniques | Strength | Main weakness |
|---|---|---|---|
| Statistical | TF-IDF, YAKE | Fast, transparent, no training | Misses meaning and synonyms |
| Graph-based | TextRank, SingleRank, TopicRank | Works on single documents | Favors frequent generic terms |
| Embedding-based | KeyBERT, EmbedRank | Captures semantic relevance | Needs a good encoder model |
| Supervised sequence labeling | BIO tagging with transformers | High accuracy in domain | Requires annotated corpora |
| Generative | Sequence-to-sequence models, LLM prompting | Produces absent keyphrases | Can hallucinate or drift |
TF-IDF multiplies term frequency by inverse document frequency, so it needs a background corpus. YAKE uses local features such as position, casing and dispersion and works without any corpus. TextRank builds a co-occurrence graph and applies a PageRank-style random walk. KeyBERT embeds the document and candidates with a transformer encoder and selects phrases with the highest cosine similarity, often combined with Maximal Marginal Relevance to keep results diverse.
Keyword extraction compared with related language tasks
Keyword extraction is related to several neighboring tasks but answers a different question: which phrases characterize this document. Confusing it with entity recognition or topic modeling leads to wrong evaluation choices and disappointing results.
| Task | Output | Scope | Typical use |
|---|---|---|---|
| Keyword extraction | Ranked keyphrases | One document | Indexing, tagging, SEO analysis |
| Named entity recognition | Typed spans such as person or place | Every mention | Information extraction |
| Topic modeling | Word distributions per topic | A whole corpus | Exploring collections |
| Text summarization | Sentences or a paragraph | One or more documents | Reading support |
| Text classification | Labels from a fixed set | One document | Routing and filtering |
Entities often appear among keyphrases, yet a keyphrase can be a concept with no entity type, such as “carbon pricing”. Topic models describe a collection, whereas keyword extraction describes a single document and can run on one text in isolation.
Keyword extraction from speech and noisy text
Applying keyword extraction to speech requires a speech-to-text front end, and transcription errors propagate directly into the keyphrases. A word error rate of 15 to 25 percent, common in noisy call recordings, can corrupt rare domain terms, which are exactly the words that carry the most information.
Practitioners mitigate this by using confidence-weighted scoring, custom vocabularies in the recognizer, punctuation restoration before candidate generation and speaker diarization so that phrases can be attributed to the agent or the customer. Disfluencies such as “you know” must be filtered, because they inflate frequency counts.
Evaluating and choosing a keyword extraction approach
Keyword extraction is usually evaluated with precision, recall and F1 at k against human-assigned keyphrases, often after stemming both sides. Exact matching is strict and penalizes valid paraphrases, so many teams add partial matching or embedding-based similarity, together with human review of samples.
Human agreement on keyphrases is notoriously low, frequently below 50 percent overlap between annotators, which caps achievable scores. Benchmark results should therefore be read as relative rankings rather than absolute quality.
| Situation | Recommended approach | Reason |
|---|---|---|
| No labels, short documents | YAKE or KeyBERT | No training, works per document |
| Large in-house corpus | TF-IDF with noun phrase filter | Corpus statistics reveal specificity |
| Annotated domain data available | Fine-tuned transformer tagger | Highest precision in domain |
| Controlled vocabulary required | Extraction plus entity linking | Maps phrases to approved terms |
| Abstractive tags acceptable | LLM prompting with validation | Generates absent keyphrases |
Common pitfalls in keyword extraction
- Generic high-frequency terms such as “system” or “result” dominate when no corpus statistics or specificity weighting is used.
- Redundant variants like “neural network” and “neural networks” crowd the top-k list without deduplication.
- Overlong phrases reduce reuse as tags; most useful keyphrases contain one to four words.
- Language mismatch occurs when stop word lists and taggers built for English are applied to other languages without language identification.
- Unverified LLM output can invent plausible keyphrases that the document never supports.
Put simply
Keyword extraction is like asking a careful reader to underline the few phrases that tell you what a text is about. Instead of reading a whole report, you glance at five or ten words and know its subject.
A computer looks for words that appear often in this text but rarely elsewhere. It also looks for words linked to many others. Some methods pick phrases closest in meaning to the whole document. The result is a short list of labels that helps people and search tools find, sort and compare documents quickly.
In practice
A customer support team at a mid-sized internet provider records about 4,000 calls per week. Each call is transcribed, and a keyword extraction step runs on the transcript after filler words and greetings are removed. The team uses an embedding-based method with a custom vocabulary that includes product names and error codes.
Each call receives eight ranked keyphrases, such as “router firmware update” or “billing address change”. A weekly dashboard aggregates them, and within two weeks analysts notice that “firmware update” and “connection drop” rise together after a specific release. Engineering rolls back the update, and the share of calls about dropped connections falls to its earlier level within days, without anyone having to listen to recordings manually.