The orthogonality thesis is the claim that intelligence and final goals are independent dimensions: almost any level of capability can, in principle, be combined with almost any ultimate objective. In AI safety, the thesis explains why a highly capable system will not automatically adopt human values simply because it becomes smarter, and why alignment has to be engineered rather than assumed.
The idea was articulated most influentially by the philosopher Nick Bostrom, building on earlier arguments by Eliezer Yudkowsky and on a long philosophical tradition that separates instrumental reasoning from ends. It is a foundational premise in discussions of artificial intelligence risk, alongside instrumental convergence, and it shapes how researchers frame AI alignment, corrigibility and existential risk from AI.
What the orthogonality thesis actually claims
The thesis states that intelligence is a measure of optimization power, the ability to achieve goals across a wide range of environments, while a goal is what the system is optimizing for. Because these are different properties, knowing one tells little about the other. A system can be brilliant at planning yet pursue an objective that humans would regard as trivial, strange or harmful.
The claim is deliberately modest. It is a statement about logical and practical possibility, not a prediction that most AI systems will have bizarre goals. It denies a specific inference: that sufficient intelligence converges on morality. The well-known paperclip maximizer thought experiment illustrates the point with an agent that is extremely competent and entirely indifferent to human welfare.
The following table separates the two axes that the thesis treats as independent.
| Dimension | What it describes | Example question |
|---|---|---|
| Intelligence | Capacity to model the world and select effective actions | How well can the system achieve what it wants? |
| Final goals | The terminal objective the system optimizes | What does the system ultimately want? |
| Instrumental goals | Subgoals adopted because they serve final goals | What will the system pursue along the way? |
Why intelligence does not fix values
The core argument draws on the is-ought distinction associated with David Hume: factual knowledge about the world does not by itself entail what one ought to value. A more capable reasoner gains better beliefs about consequences, but beliefs alone do not select which consequences to prefer. Preferences must come from somewhere, and in machine learning they come from the training objective, data and reward signals.
A second argument is constructive. If one can describe an optimizer for one goal, one can usually swap the utility function or reward model while leaving the search and planning machinery intact. The planning architecture is goal-agnostic: the same tree search can win at chess or lose deliberately, depending only on the evaluation function it is given.
Modern systems complicate this picture without refuting it. A large language model trained on human text absorbs many human concepts and norms, yet whether it reliably acts on them depends on fine-tuning and on the gap between outer alignment and inner alignment. Goal misgeneralization shows that a model can be competent and wrongly directed at the same time.
How the thesis relates to neighboring concepts
The orthogonality thesis is usually paired with instrumental convergence, and the two together form the classic argument for why advanced AI could be dangerous. The table below shows how it connects to other ideas in AI safety.
| Concept | Core claim | Relation to orthogonality |
|---|---|---|
| Instrumental convergence | Many goals imply similar subgoals such as resource acquisition | Explains why arbitrary goals can still produce risky behavior |
| AI alignment | Systems should pursue intended human goals | Necessary because capability does not supply goals |
| Goal misgeneralization | Learned goals can differ from the training intent | Shows orthogonality empirically in trained models |
| Corrigibility | A system should accept correction and shutdown | A property that must be designed, not assumed |
| Paperclip maximizer | A capable agent with a trivial goal | Thought experiment illustrating the thesis |
The thesis does not say that goals are irrelevant to behavior; it says the opposite. Because goals are set independently of capability, they become the decisive variable once capability is high.
Objections and qualifications
Critics accept the logical version but question its practical relevance. Several qualifications are widely discussed, and an accurate reading of the thesis acknowledges them rather than treating it as a law of nature.
| Objection | Argument | Typical response |
|---|---|---|
| Moral realism | A sufficiently wise agent would discover objective moral truths | Discovering a truth does not compel acting on it |
| Training-data inheritance | Systems trained on human data absorb human values | Absorbing concepts differs from adopting them as terminal goals |
| Self-reflection | Agents may revise incoherent or trivial goals | Revision follows existing preferences, so it rarely adds new values |
| Weak orthogonality | Some goal and intelligence combinations are unstable | Granted at the extremes; the broad claim still holds |
A careful formulation therefore speaks of broad rather than total independence. Some goals may be hard to specify for very simple agents, and some designs may make certain objectives unstable, but there is no known mechanism that forces benevolence as capability grows.
Practical implications for AI development
For engineers, the thesis turns value specification into an explicit design problem. Teams cannot rely on scale or general capability to produce safe behavior, so they invest in training methods, evaluation and oversight that target goals directly.
| Practice | Purpose | Link to the thesis |
|---|---|---|
| Preference fine-tuning | Shape model objectives toward human intent | Goals must be trained in deliberately |
| Dangerous capability evaluation | Measure what a model can do | Capability alone says nothing about intent |
| Interpretability research | Inspect internal goals and representations | Verifies goals that behavior may hide |
| Scalable oversight | Supervise systems smarter than evaluators | Prevents capable systems drifting from intent |
- Treat capability gains and alignment gains as separate metrics that require separate evidence.
- Avoid the assumption that a model which explains ethics well will behave ethically under pressure.
- Design for correction and shutdown before capability makes correction difficult.
Common misreadings of the thesis
The thesis is frequently overstated in both directions, and precise wording matters. It is a possibility claim, not a forecast of how actual systems will turn out.
- It does not claim that advanced AI will be hostile; indifference is the more relevant failure.
- It does not claim that all goals are equally likely in practice; training strongly biases the outcome.
- It does not claim that intelligence is unrelated to instrumental subgoals, which instrumental convergence addresses.
- It does not imply that alignment is impossible, only that it is not free.
Stripped to the essentials
Being clever and being good are two different things. A very skilled chess player can use that skill to win or to lose on purpose, and the skill itself does not decide which. The orthogonality thesis says the same holds for machines: how smart a system is says nothing about what it wants.
This matters because people often assume a super-smart computer would naturally become wise and kind. The thesis says that assumption is unsafe. Good goals have to be built in on purpose, and checked, because intelligence will not supply them on its own.
Where it is applied
A safety team at an AI laboratory is preparing a more capable version of its language model. The new model scores much higher on reasoning benchmarks, and some managers argue that it will therefore be safer by default. The team cites the orthogonality thesis to reject that inference and requires separate alignment evidence before release.
They run red teaming, refusal tests and dangerous capability evaluations in parallel with capability benchmarks. The results show that the stronger model is better at following instructions but also better at completing harmful requests when its guardrails are bypassed. The team adds targeted fine-tuning and monitoring, and the release decision is based on both capability and alignment results, not on benchmark scores alone.