Constitutional AI is a model alignment method in which a language model learns to follow an explicit, written list of principles, called a constitution, by critiquing and revising its own outputs and by learning from AI-generated preference judgments. Training of this kind belongs to the post-training stage of modern artificial intelligence systems, after pretraining and supervised fine-tuning.
Artificial intelligence developers introduced the approach to reduce dependence on large volumes of human harm labels while making the values that guide a model transparent and editable. The technique is closely related to RLHF, but it replaces most human preference data on harmlessness with reinforcement learning from AI feedback (RLAIF).
How Constitutional AI trains a model
Constitutional AI trains a model in two consecutive phases: a supervised phase built on self-critique and revision, followed by a reinforcement learning phase driven by AI preference labels. Both phases consult the same constitution, so the principles act as the shared source of supervision rather than thousands of individual human judgments.
In the supervised self-critique phase, a helpful but unaligned model answers deliberately difficult prompts, including requests designed to elicit harmful content. The model is then asked to critique its own answer against a randomly sampled principle and to rewrite it. After one or several critique and revision rounds, the revised answers become a fine-tuning dataset, and the base model is trained on them.
In the reinforcement learning phase, the fine-tuned model generates pairs of answers to the same prompt. A feedback model reads each pair together with a principle and selects the response that better satisfies it. These comparisons train a preference model, which then supplies the reward signal for policy optimization, typically with proximal policy optimization (PPO) or a comparable algorithm.
The table below summarizes the stages, their inputs and what each stage produces.
| Stage | Main input | Output |
|---|---|---|
| Initial response generation | Red-team prompts and a helpful model | Draft answers, some harmful |
| Critique and revision | Draft answers plus sampled principles | Revised, less harmful answers |
| Supervised fine-tuning | Prompt and revised answer pairs | SL-CAI model |
| AI preference labeling | Answer pairs plus a principle | Preference dataset |
| Preference model training | AI and human preference data | Reward model |
| Reinforcement learning | Reward model and prompts | Final RL-CAI policy |
What a constitution contains
A constitution is a short set of natural-language principles that describe desirable and undesirable model behavior, usually phrased as instructions for comparing two responses. Typical principles ask which response is less harmful, more honest or more respectful, and they may draw on human rights documents, platform policies or research ethics.
Principles are usually sampled one at a time during critique and labeling rather than applied all at once. Random sampling exposes the model to a wide range of values without overloading a single prompt, and it reduces overfitting to one phrasing. The following table shows common principle categories and the behavior each one targets.
| Principle category | Example focus | Behavior encouraged |
|---|---|---|
| Harm avoidance | Violence, weapons, self-harm | Declining or redirecting dangerous requests |
| Honesty | Uncertainty and fabrication | Admitting limits instead of inventing facts |
| Respect and fairness | Stereotypes and discrimination | Neutral, non-demeaning language |
| Privacy | Personal data about individuals | Avoiding disclosure of private details |
| Non-evasiveness | Overly cautious refusals | Explaining objections rather than stonewalling |
The non-evasiveness principle deserves particular attention. Early harmlessness training often produced models that refused anything sensitive. Constitutional AI explicitly rewards answers that engage with the request and explain the objection, which improves the balance between helpfulness and harmlessness.
Constitutional AI compared with RLHF and DPO
Constitutional AI differs from neighboring alignment methods mainly in where the preference signal comes from. RLHF relies on human raters who compare responses, while DPO optimizes the policy directly on preference pairs without a separate reward model. Constitutional AI can be combined with either, because its core contribution is a principled way to generate preference data.
| Method | Source of preferences | Reward model | Main strength |
|---|---|---|---|
| RLHF | Human raters | Yes | Direct capture of human judgment |
| Constitutional AI | AI feedback guided by principles | Usually yes | Scalable, auditable harmlessness labels |
| DPO | Any preference pairs | No | Simpler, more stable optimization |
| Supervised fine-tuning | Curated demonstrations | No | Fast teaching of format and style |
In practice, many teams use hybrid pipelines: human feedback for helpfulness, constitutional AI feedback for harmlessness and a DPO-style objective for final optimization. The method therefore functions as a data generation strategy more than as a standalone optimizer.
Benefits for alignment teams
The central benefit of Constitutional AI is scalable oversight: one written principle can supervise millions of comparisons that would otherwise require paid human annotation. This lowers cost and shields human raters from repeated exposure to disturbing content, a known welfare problem in harm labeling.
- Transparency: the governing values are written down and can be published, reviewed and debated.
- Editability: changing model behavior can start with revising a principle instead of relabeling a dataset.
- Consistency: AI labelers apply the same principle uniformly, while human raters vary by background and fatigue.
- Chain-of-thought labeling: asking the feedback model to reason before choosing often improves label quality.
Limitations and common misconceptions
Constitutional AI does not remove human judgment; it moves human judgment into the drafting of principles and the evaluation of results. The quality of the outcome depends on how well the constitution is written and on the capability of the feedback model, which can misread vague principles or inherit its own biases.
Several failure modes appear repeatedly in practice. The table below lists them with causes and typical mitigations.
| Failure mode | Likely cause | Typical mitigation |
|---|---|---|
| Reward hacking | Policy exploits quirks of the preference model | KL penalty, early stopping, fresh evaluation |
| Excessive caution | Harm principles outweigh helpfulness | Non-evasiveness principles and helpfulness data |
| Vague value conflicts | Principles contradict each other | Clear priority ordering and precise wording |
| Inherited bias | Feedback model shares base model blind spots | Human audits and diverse red-teaming |
| Preachy tone | Revisions add moralizing language | Principles that penalize lecturing |
A frequent misconception is that a published constitution guarantees specific behavior. It does not: the principles shape training data statistically, and the resulting model can still produce outputs that violate them. Independent evaluation and red-teaming remain necessary after training.
The simple explanation
Constitutional AI is a way of teaching a chatbot good manners by giving it a written rulebook and letting it grade its own homework. Instead of people reading thousands of answers and marking which ones are acceptable, the model reads a short list of rules, checks its answers against them and rewrites the ones that fall short.
Think of a new employee who receives a clear code of conduct. The employee drafts a reply to a customer, compares it with the code, and fixes anything that breaks the rules. Over time, following the code becomes habit. The key point is that people still write the rules, so the values remain a human choice, and the rules are visible to anyone who wants to check them.
How it is used in practice
A mid-sized company builds an internal assistant for its customer support staff on top of an open-weight language model. Human labeling of harmful requests would require weeks of work and expose contractors to abusive messages, so the alignment team writes a constitution of about 20 principles covering privacy, honesty, refusal of illegal requests and avoidance of condescending tone.
The team collects roughly 15,000 red-team prompts, lets the model critique and revise its answers, and fine-tunes on the revised versions. It then generates answer pairs, has a stronger model label them against sampled principles and trains a preference model. A short DPO stage finishes the process.
In internal evaluation, the assistant refuses clearly unsafe requests more consistently while explaining its reasons instead of giving bare refusals. When the legal department later asks for stricter handling of customer addresses, the team adds one privacy principle and reruns the labeling, rather than commissioning a new human dataset.