Constitutional AI

On this page 7

Constitutional AI is a model alignment method in which a language model learns to follow an explicit, written list of principles, called a constitution, by critiquing and revising its own outputs and by learning from AI-generated preference judgments. Training of this kind belongs to the post-training stage of modern artificial intelligence systems, after pretraining and supervised fine-tuning.

Artificial intelligence developers introduced the approach to reduce dependence on large volumes of human harm labels while making the values that guide a model transparent and editable. The technique is closely related to RLHF, but it replaces most human preference data on harmlessness with reinforcement learning from AI feedback (RLAIF).

How Constitutional AI trains a model

Constitutional AI trains a model in two consecutive phases: a supervised phase built on self-critique and revision, followed by a reinforcement learning phase driven by AI preference labels. Both phases consult the same constitution, so the principles act as the shared source of supervision rather than thousands of individual human judgments.

In the supervised self-critique phase, a helpful but unaligned model answers deliberately difficult prompts, including requests designed to elicit harmful content. The model is then asked to critique its own answer against a randomly sampled principle and to rewrite it. After one or several critique and revision rounds, the revised answers become a fine-tuning dataset, and the base model is trained on them.

In the reinforcement learning phase, the fine-tuned model generates pairs of answers to the same prompt. A feedback model reads each pair together with a principle and selects the response that better satisfies it. These comparisons train a preference model, which then supplies the reward signal for policy optimization, typically with proximal policy optimization (PPO) or a comparable algorithm.

The table below summarizes the stages, their inputs and what each stage produces.

StageMain inputOutput
Initial response generationRed-team prompts and a helpful modelDraft answers, some harmful
Critique and revisionDraft answers plus sampled principlesRevised, less harmful answers
Supervised fine-tuningPrompt and revised answer pairsSL-CAI model
AI preference labelingAnswer pairs plus a principlePreference dataset
Preference model trainingAI and human preference dataReward model
Reinforcement learningReward model and promptsFinal RL-CAI policy
Stages of the Constitutional AI pipeline

What a constitution contains

A constitution is a short set of natural-language principles that describe desirable and undesirable model behavior, usually phrased as instructions for comparing two responses. Typical principles ask which response is less harmful, more honest or more respectful, and they may draw on human rights documents, platform policies or research ethics.

Principles are usually sampled one at a time during critique and labeling rather than applied all at once. Random sampling exposes the model to a wide range of values without overloading a single prompt, and it reduces overfitting to one phrasing. The following table shows common principle categories and the behavior each one targets.

Principle categoryExample focusBehavior encouraged
Harm avoidanceViolence, weapons, self-harmDeclining or redirecting dangerous requests
HonestyUncertainty and fabricationAdmitting limits instead of inventing facts
Respect and fairnessStereotypes and discriminationNeutral, non-demeaning language
PrivacyPersonal data about individualsAvoiding disclosure of private details
Non-evasivenessOverly cautious refusalsExplaining objections rather than stonewalling
Common categories of constitutional principles

The non-evasiveness principle deserves particular attention. Early harmlessness training often produced models that refused anything sensitive. Constitutional AI explicitly rewards answers that engage with the request and explain the objection, which improves the balance between helpfulness and harmlessness.

Constitutional AI compared with RLHF and DPO

Constitutional AI differs from neighboring alignment methods mainly in where the preference signal comes from. RLHF relies on human raters who compare responses, while DPO optimizes the policy directly on preference pairs without a separate reward model. Constitutional AI can be combined with either, because its core contribution is a principled way to generate preference data.

MethodSource of preferencesReward modelMain strength
RLHFHuman ratersYesDirect capture of human judgment
Constitutional AIAI feedback guided by principlesUsually yesScalable, auditable harmlessness labels
DPOAny preference pairsNoSimpler, more stable optimization
Supervised fine-tuningCurated demonstrationsNoFast teaching of format and style
Comparison of alignment methods

In practice, many teams use hybrid pipelines: human feedback for helpfulness, constitutional AI feedback for harmlessness and a DPO-style objective for final optimization. The method therefore functions as a data generation strategy more than as a standalone optimizer.

Benefits for alignment teams

The central benefit of Constitutional AI is scalable oversight: one written principle can supervise millions of comparisons that would otherwise require paid human annotation. This lowers cost and shields human raters from repeated exposure to disturbing content, a known welfare problem in harm labeling.

  • Transparency: the governing values are written down and can be published, reviewed and debated.
  • Editability: changing model behavior can start with revising a principle instead of relabeling a dataset.
  • Consistency: AI labelers apply the same principle uniformly, while human raters vary by background and fatigue.
  • Chain-of-thought labeling: asking the feedback model to reason before choosing often improves label quality.

Limitations and common misconceptions

Constitutional AI does not remove human judgment; it moves human judgment into the drafting of principles and the evaluation of results. The quality of the outcome depends on how well the constitution is written and on the capability of the feedback model, which can misread vague principles or inherit its own biases.

Several failure modes appear repeatedly in practice. The table below lists them with causes and typical mitigations.

Failure modeLikely causeTypical mitigation
Reward hackingPolicy exploits quirks of the preference modelKL penalty, early stopping, fresh evaluation
Excessive cautionHarm principles outweigh helpfulnessNon-evasiveness principles and helpfulness data
Vague value conflictsPrinciples contradict each otherClear priority ordering and precise wording
Inherited biasFeedback model shares base model blind spotsHuman audits and diverse red-teaming
Preachy toneRevisions add moralizing languagePrinciples that penalize lecturing
Failure modes and mitigations

A frequent misconception is that a published constitution guarantees specific behavior. It does not: the principles shape training data statistically, and the resulting model can still produce outputs that violate them. Independent evaluation and red-teaming remain necessary after training.

The simple explanation

Constitutional AI is a way of teaching a chatbot good manners by giving it a written rulebook and letting it grade its own homework. Instead of people reading thousands of answers and marking which ones are acceptable, the model reads a short list of rules, checks its answers against them and rewrites the ones that fall short.

Think of a new employee who receives a clear code of conduct. The employee drafts a reply to a customer, compares it with the code, and fixes anything that breaks the rules. Over time, following the code becomes habit. The key point is that people still write the rules, so the values remain a human choice, and the rules are visible to anyone who wants to check them.

How it is used in practice

A mid-sized company builds an internal assistant for its customer support staff on top of an open-weight language model. Human labeling of harmful requests would require weeks of work and expose contractors to abusive messages, so the alignment team writes a constitution of about 20 principles covering privacy, honesty, refusal of illegal requests and avoidance of condescending tone.

The team collects roughly 15,000 red-team prompts, lets the model critique and revise its answers, and fine-tunes on the revised versions. It then generates answer pairs, has a stronger model label them against sampled principles and trains a preference model. A short DPO stage finishes the process.

In internal evaluation, the assistant refuses clearly unsafe requests more consistently while explaining its reasons instead of giving bare refusals. When the legal department later asks for stricter handling of customer addresses, the team adds one privacy principle and reruns the labeling, rather than commissioning a new human dataset.