ARC-AGI

On this page 7

ARC-AGI is a benchmark of abstract visual reasoning puzzles, built on the Abstraction and Reasoning Corpus, that measures how efficiently an artificial intelligence system can acquire a new skill from only a few examples. In model evaluation it occupies an unusual position: instead of testing stored knowledge, it tests fluid intelligence, the ability to infer an unfamiliar rule and apply it correctly to a new case.

François Chollet introduced the corpus in 2019 together with a formal argument that intelligence should be measured as skill-acquisition efficiency rather than as skill itself. The benchmark has since become a reference point in debates about general intelligence, largely because tasks that most people solve in seconds remained very hard for large language models for several years.

How ARC-AGI tasks are structured

Each ARC-AGI task consists of a few demonstration pairs and one or more test inputs. Every input and output is a grid of colored cells, from 1×1 up to 30×30 cells, with 10 possible colors. The solver must infer the transformation that turns each demonstration input into its output and then produce the exact output grid for the test input.

The transformations draw on what Chollet calls core knowledge priors: objectness, basic geometry and topology, counting, symmetry and simple goal-directed behavior. No language, cultural knowledge or specialized training is needed, so the tasks are designed to be fair to humans and machines alike.

The main components of a task and its scoring are summarized below.

ComponentDescriptionTypical value
Demonstration pairsInput and output grids that show the ruleUsually 2 to 5
Test inputNew grid the solver must transformUsually 1, sometimes 2
Grid sizeHeight and width in cells1×1 to 30×30
Color paletteDistinct cell values10 colors
Attempts allowedPredictions submitted per test input2 (pass@2 scoring)
Success criterionMatch against the hidden answerExact, cell by cell
Components of an ARC-AGI task

Scoring is binary per test output. A grid with a single wrong cell counts as a failure, which rules out partial credit and makes lucky approximations nearly worthless.

Versions and evaluation sets

ARC-AGI exists in several versions, each released when the previous one became less discriminating. The table outlines the main differences.

VersionFormatMain purpose
ARC-AGI-1Static grid puzzles, released 2019Original test of few-shot abstraction
ARC-AGI-2Harder static puzzles, released 2025Resist brute-force search and pattern matching
ARC-AGI-3Interactive game environmentsTest exploration, planning and learning over time
ARC-AGI versions compared

Each version is split into public and hidden evaluation sets. The public training and evaluation tasks can be studied freely, while a semi-private set is used to score commercial models through their APIs and a private set is reserved for competition entries running offline. This separation is the benchmark’s main defense against benchmark contamination, because hidden tasks cannot leak into training corpora.

For ARC-AGI-2, every task was solved by at least two human testers within two attempts, which documents that the puzzles are feasible for people even though they defeat most models.

What ARC-AGI measures and how it differs from other benchmarks

ARC-AGI measures generalization to novel tasks, whereas most AI benchmarks measure performance on tasks resembling the training distribution. A model can score well on knowledge tests through memorization; on ARC-AGI each task is deliberately unique, so memorized solutions do not transfer.

BenchmarkMain skill testedSensitivity to memorization
ARC-AGIFew-shot rule induction on novel gridsLow by design
MMLUBroad academic knowledgeHigh
GSM8KGrade-school math word problemsHigh, partly saturated
GPQAExpert-level science questionsModerate
SWE-benchFixing real software issuesModerate
ARC-AGI compared with other well-known benchmarks

Because of this design, ARC-AGI results tend to move independently of model scale. Early large language models gained little on it despite large gains on knowledge benchmarks, and the jumps that did occur came with test-time reasoning and search methods rather than with more pretraining data.

Progress, compute and the ARC Prize

Progress on ARC-AGI has been driven by program synthesis, test-time training and reasoning models. The ARC Prize competition, launched in 2024, offers awards for open solutions that reach a target score on the private set under fixed compute limits.

Since 2024 the organizers have reported cost per task alongside accuracy, because high scores can be bought with enormous inference budgets. In December 2024 a preview of OpenAI’s o3 model reached about 76 percent on the semi-private ARC-AGI-1 set at a moderate budget and about 88 percent with far more compute. The organizers stressed that this was not evidence of general intelligence and released the harder ARC-AGI-2, on which frontier systems initially scored in the single digits.

ApproachHow it worksMain limitation
Program synthesisSearch a domain-specific language for a program fitting the demonstrationsSearch space grows explosively
Test-time trainingFine-tune a model on each task’s demonstrationsExpensive per task
Reasoning modelsGenerate and check long chains of thoughtHigh inference cost
Hybrid systemsCombine neural guidance with symbolic searchEngineering complexity
Main solution approaches

Limitations and common misreadings

ARC-AGI is a necessary but not sufficient test of general intelligence, and its creators say so explicitly. Passing it shows strong few-shot abstraction on grids, not broad competence.

  • Domain narrowness: all tasks are small colored grids, so success may not transfer to language, perception or planning.
  • Compute dependence: a score without a cost per task says little about efficiency, which is the property the benchmark targets.
  • Public-set overfitting: results on the public evaluation set are less trustworthy than semi-private or private scores.
  • Saturation risk: once a version is solved, it no longer separates systems, which is why new versions keep appearing.

Reported scores should always state the version, the evaluation set and the compute budget. A headline percentage without those three facts cannot be compared with any other result.

Understanding it simply

ARC-AGI is a set of picture puzzles. Each puzzle shows a few small colored grids and how each one changes. The task is to work out the hidden rule and apply it to a new grid. Most people find these puzzles easy.

For computers, they have been surprisingly hard. The reason is that every puzzle is new, so a machine cannot simply remember the answer. It has to learn the rule on the spot, from just a handful of examples. That is why ARC-AGI is used to check whether AI can truly adapt, not just recall.

Applied in practice

A university research group develops a hybrid solver that pairs a mid-sized language model with a search over short grid-editing programs. Before publishing, the team tests it on the 120 public ARC-AGI-2 evaluation tasks and records both accuracy and cost per task. The solver reaches 14 percent at an average of about 0.40 dollars of compute per task.

The group then submits the system through the competition pipeline, where it runs offline on the private set within the official time and hardware limits. The private score comes out at 11 percent, a few points below the public result. The team reports both numbers with the version and the budget. The gap reveals mild overfitting to the public tasks, and the transparent report lets other researchers compare the method fairly.