Repetition penalty

On this page 7

A repetition penalty is a decoding parameter in language models that reduces the logits of tokens that have already appeared in the context, so that the model becomes less likely to select them again. It is applied at inference time, after the network has produced its scores and before sampling or greedy selection, and it changes no model weights.

The technique addresses a well-known failure of autoregressive generation: once a phrase appears, the model often assigns it an even higher probability on the next pass, producing degenerate loops such as the same sentence printed again and again. A modest penalty breaks these loops, while an excessive one damages grammar, terminology and factual accuracy.

How a repetition penalty rescales logits

A repetition penalty works by adjusting the raw score of every previously seen token before the probability distribution is computed. At each step, the decoder collects the set of token IDs present in the prompt and the generated text, or in a recent window of it, and modifies their logits by a factor θ (theta).

In the multiplicative form popularized by the CTRL model in 2019 and adopted by most open-source inference libraries, a positive logit is divided by θ and a negative logit is multiplied by θ. Both operations push the token toward a lower probability, because dividing a positive number shrinks it and multiplying a negative number makes it more negative. A value of θ = 1.0 disables the penalty.

The adjustment is binary with respect to occurrence: a token seen once receives the same penalty as a token seen fifty times. This distinguishes the classic repetition penalty from count-based penalties, which grow with every additional occurrence. The table below summarizes the main parameters that control the mechanism.

ParameterTypical valuesWhat it controls
Penalty factor θ1.0 to 1.3Strength of the logit reduction
Penalty window64 to 2,048 tokensHow far back seen tokens are collected
Prompt inclusionOn or offWhether prompt tokens are also penalized
Application orderBefore temperature and truncationWhich scores the sampler finally sees
Core parameters of a repetition penalty

Repetition penalty versus frequency and presence penalties

The repetition penalty is one of several anti-repetition controls, and they differ in arithmetic and in scope. Hosted chat APIs usually expose an additive frequency penalty and presence penalty instead, while local inference engines usually expose the multiplicative repetition penalty.

An additive presence penalty subtracts a fixed amount from every token that has appeared at least once, which makes it the closest relative of the repetition penalty. An additive frequency penalty subtracts an amount proportional to the occurrence count, so frequent tokens are suppressed more strongly than rare ones. The comparison below shows how these methods behave.

MethodArithmeticDepends on countTypical range
Repetition penaltyDivide or multiply logit by θNo1.0 to 1.3
Presence penaltySubtract a constantNo0 to 2.0
Frequency penaltySubtract constant times countYes0 to 2.0
No-repeat n-gram blockSet logit to minus infinityExact n-gram matchn = 2 to 4
Sequence-aware penaltyPenalize continuation of a repeated spanSpan lengthVaries by implementation
Comparison of common anti-repetition methods

A hard n-gram block guarantees that no sequence of n tokens appears twice, which suits beam search in summarization but can forbid legitimate repeats such as a person’s full name. Sequence-aware samplers target only the extension of an already repeated span, which leaves common single tokens untouched and is therefore gentler on grammar.

Effects of different penalty strengths

The effect of a repetition penalty is strongly non-linear in practice, because function words, punctuation and code syntax appear in almost every context. A small increase in θ can therefore change the output style far more than the number suggests.

Penalty factorObserved behaviorMain risk
1.0Unmodified model distributionLoops in long or greedy outputs
1.05 to 1.15Fewer loops, natural phrasingMinimal for most tasks
1.2 to 1.3Noticeably varied wordingSynonym drift and altered terms
Above 1.5Avoidance of common wordsBroken grammar and incoherence
Typical behavior at different penalty settings

The danger at high values comes from the fact that the penalty cannot distinguish harmful repetition from necessary repetition. Articles, variable names, units and the subject of a document must recur, and penalizing them forces the model into unnatural synonyms or invented spellings. In code, an aggressive setting may rename a variable halfway through a function, which produces a program that does not compile.

Configuring a repetition penalty in practice

The right configuration depends on the decoding strategy, the task and the model’s alignment. Instruction-tuned and reasoning models trained with modern preference optimization loop far less than older base models, so they often need no penalty at all or only a very light one.

SituationRecommended settingReason
Greedy decoding of long text1.1 to 1.2Greedy search loops most easily
Creative writing with sampling1.05 to 1.15Adds variety without harming style
Code generation1.0 or a narrow windowIdentifiers must repeat exactly
Structured output such as JSON1.0Keys and brackets must recur
Small or older base model1.1 to 1.2 with a short windowWeak models repeat more often
Decision guide for typical generation tasks

A limited penalty window is one of the most effective refinements. Restricting the penalty to the last 64 to 256 tokens prevents the model from avoiding words simply because they appeared in a long system prompt. Practitioners also tune one anti-repetition control at a time, because stacking a repetition penalty with frequency and presence penalties multiplies their side effects.

Limitations and common mistakes

A repetition penalty treats a symptom rather than the cause of degenerate text. Loops usually indicate a weak model, a poorly formatted prompt, a missing chat template or overly deterministic decoding, and fixing those causes often removes the need for any penalty.

  • Penalizing the prompt by default: some implementations include all input tokens, so key terms from the instructions become harder to use in the answer.
  • Treating the factor as linear: a step from 1.1 to 1.3 is not a small change, because it affects nearly every common token.
  • Ignoring tokenization: the penalty acts on subword tokens, so suppressing a fragment can also suppress unrelated words that share it.
  • Assuming it improves accuracy: discouraging repeated facts can push the model toward less likely and sometimes incorrect alternatives.

Stripped to the essentials

When a language model writes text, it chooses one word piece at a time. Sometimes it gets stuck and repeats the same phrase over and over, like a record with a scratch.

A repetition penalty is a small rule that makes words already used a little less attractive for the next choice. A gentle penalty keeps the text fresh. A strong one forces the model to avoid even ordinary words, and the writing becomes strange. The key is a light touch: just enough to stop the loops, not so much that normal language suffers.

Where it is applied

A support team runs a small open-weight model on its own server to draft long product descriptions. With greedy decoding and no penalty, roughly one draft in ten ends with the same feature sentence repeated until the token limit is reached.

The engineers set the repetition penalty to 1.1 with a window of the last 128 tokens, and they exclude the long system prompt from the penalized set. Looping drafts almost disappear, and product names and units still appear consistently. A trial at 1.4 is rejected because the model began replacing the product name with awkward synonyms. The team keeps the lighter setting as the production default.