Keyword extraction

On this page 8

Keyword extraction is the task of automatically identifying the words and multi-word phrases that best summarize a document, ranked by how central they are to its content. In speech & language AI, a branch of artificial intelligence, it turns unstructured text or transcribed speech into a compact set of descriptors that support search indexing, tagging, content routing and analytics.

The output is usually a ranked list of keyphrases with scores, such as “supply chain disruption” or “interest rate”. Unlike text classification, which assigns labels from a fixed set, keyword extraction draws its candidates from the text itself, although modern systems also generate absent phrases that do not appear verbatim.

How keyword extraction selects candidate phrases

Keyword extraction works as a two-stage pipeline: first it generates candidate phrases, and then it scores and ranks them. Candidate generation typically applies tokenization, part-of-speech tagging and stop word removal, keeping noun phrases that match patterns such as zero or more adjectives followed by one or more nouns.

Scoring then estimates how representative and specific each candidate is. Statistical methods reward terms that are frequent in the document but rare across a reference corpus. Graph methods reward terms that co-occur with many other important terms. Embedding methods reward phrases whose vector representation is semantically close to the whole document.

A final step usually applies lemmatization or stemming to merge variants such as “network” and “networks”, removes near-duplicates and returns the top-k phrases, where k between 5 and 15 is common for articles and reports.

StageTypical operationEffect on the result
PreprocessingTokenization, lowercasing, text normalizationConsistent units for counting
Candidate generationPOS patterns, n-gram windows, noun chunksDefines which phrases can win
ScoringTF-IDF, graph centrality, embedding similarityRanks candidates by importance
Post-processingDeduplication, diversity reranking, top-k cutRemoves redundancy and noise
Stages of a typical keyword extraction pipeline

Main families of keyword extraction methods

Keyword extraction methods fall into unsupervised, supervised and generative families, and each one trades data requirements against accuracy and control. The choice depends mainly on whether labeled keyphrase data exists for the domain.

Method familyRepresentative techniquesStrengthMain weakness
StatisticalTF-IDF, YAKEFast, transparent, no trainingMisses meaning and synonyms
Graph-basedTextRank, SingleRank, TopicRankWorks on single documentsFavors frequent generic terms
Embedding-basedKeyBERT, EmbedRankCaptures semantic relevanceNeeds a good encoder model
Supervised sequence labelingBIO tagging with transformersHigh accuracy in domainRequires annotated corpora
GenerativeSequence-to-sequence models, LLM promptingProduces absent keyphrasesCan hallucinate or drift
Comparison of keyword extraction method families

TF-IDF multiplies term frequency by inverse document frequency, so it needs a background corpus. YAKE uses local features such as position, casing and dispersion and works without any corpus. TextRank builds a co-occurrence graph and applies a PageRank-style random walk. KeyBERT embeds the document and candidates with a transformer encoder and selects phrases with the highest cosine similarity, often combined with Maximal Marginal Relevance to keep results diverse.

Keyword extraction is related to several neighboring tasks but answers a different question: which phrases characterize this document. Confusing it with entity recognition or topic modeling leads to wrong evaluation choices and disappointing results.

TaskOutputScopeTypical use
Keyword extractionRanked keyphrasesOne documentIndexing, tagging, SEO analysis
Named entity recognitionTyped spans such as person or placeEvery mentionInformation extraction
Topic modelingWord distributions per topicA whole corpusExploring collections
Text summarizationSentences or a paragraphOne or more documentsReading support
Text classificationLabels from a fixed setOne documentRouting and filtering
How keyword extraction differs from neighboring tasks

Entities often appear among keyphrases, yet a keyphrase can be a concept with no entity type, such as “carbon pricing”. Topic models describe a collection, whereas keyword extraction describes a single document and can run on one text in isolation.

Keyword extraction from speech and noisy text

Applying keyword extraction to speech requires a speech-to-text front end, and transcription errors propagate directly into the keyphrases. A word error rate of 15 to 25 percent, common in noisy call recordings, can corrupt rare domain terms, which are exactly the words that carry the most information.

Practitioners mitigate this by using confidence-weighted scoring, custom vocabularies in the recognizer, punctuation restoration before candidate generation and speaker diarization so that phrases can be attributed to the agent or the customer. Disfluencies such as “you know” must be filtered, because they inflate frequency counts.

Evaluating and choosing a keyword extraction approach

Keyword extraction is usually evaluated with precision, recall and F1 at k against human-assigned keyphrases, often after stemming both sides. Exact matching is strict and penalizes valid paraphrases, so many teams add partial matching or embedding-based similarity, together with human review of samples.

Human agreement on keyphrases is notoriously low, frequently below 50 percent overlap between annotators, which caps achievable scores. Benchmark results should therefore be read as relative rankings rather than absolute quality.

SituationRecommended approachReason
No labels, short documentsYAKE or KeyBERTNo training, works per document
Large in-house corpusTF-IDF with noun phrase filterCorpus statistics reveal specificity
Annotated domain data availableFine-tuned transformer taggerHighest precision in domain
Controlled vocabulary requiredExtraction plus entity linkingMaps phrases to approved terms
Abstractive tags acceptableLLM prompting with validationGenerates absent keyphrases
Decision guide for selecting a method

Common pitfalls in keyword extraction

  • Generic high-frequency terms such as “system” or “result” dominate when no corpus statistics or specificity weighting is used.
  • Redundant variants like “neural network” and “neural networks” crowd the top-k list without deduplication.
  • Overlong phrases reduce reuse as tags; most useful keyphrases contain one to four words.
  • Language mismatch occurs when stop word lists and taggers built for English are applied to other languages without language identification.
  • Unverified LLM output can invent plausible keyphrases that the document never supports.

Put simply

Keyword extraction is like asking a careful reader to underline the few phrases that tell you what a text is about. Instead of reading a whole report, you glance at five or ten words and know its subject.

A computer looks for words that appear often in this text but rarely elsewhere. It also looks for words linked to many others. Some methods pick phrases closest in meaning to the whole document. The result is a short list of labels that helps people and search tools find, sort and compare documents quickly.

In practice

A customer support team at a mid-sized internet provider records about 4,000 calls per week. Each call is transcribed, and a keyword extraction step runs on the transcript after filler words and greetings are removed. The team uses an embedding-based method with a custom vocabulary that includes product names and error codes.

Each call receives eight ranked keyphrases, such as “router firmware update” or “billing address change”. A weekly dashboard aggregates them, and within two weeks analysts notice that “firmware update” and “connection drop” rise together after a specific release. Engineering rolls back the update, and the share of calls about dropped connections falls to its earlier level within days, without anyone having to listen to recordings manually.