LMArena

Also known as: Chatbot Arena

On this page 7

LMArena is a public evaluation platform that ranks large language models and other generative AI systems by collecting crowdsourced pairwise votes between anonymous models and converting them into a statistical leaderboard. Originally launched in 2023 as Chatbot Arena by researchers associated with LMSYS and the University of California, Berkeley, it became one of the most cited references in model evaluation for artificial intelligence because it measures human preference on open-ended tasks rather than accuracy on a fixed test set.

Unlike static benchmarks such as MMLU or GPQA, LMArena has no fixed answer key. Each user writes a prompt, receives two answers from models whose names are hidden, and votes for the better one. The ranking therefore reflects which model people prefer in practice, with all the strengths and biases that such a signal carries.

How an LMArena battle works

The core unit of LMArena is the anonymous battle: two randomly selected models answer the same user prompt side by side, and the user chooses the left answer, the right answer, a tie, or that both are bad. Model identities are revealed only after the vote, which prevents brand loyalty from influencing the judgment.

The steps below show how a single battle becomes part of the leaderboard.

StageWhat happensWhy it matters
Prompt submissionUser writes any question or taskCovers real, diverse use cases
Model samplingPlatform pairs two models, often weightedBalances vote counts across models
Blind comparisonAnswers shown as model A and model BRemoves name and brand bias
VoteA better, B better, tie or both badProduces a pairwise preference label
FilteringDuplicate, abusive or identity-revealing votes removedProtects data quality
AggregationVotes fitted into a rating modelProduces scores and confidence intervals
Stages of a single battle

Battles that reveal the model identity in the answer, for example when a model states its own name, are typically excluded from the ranking, because the vote is then no longer blind.

How LMArena scores are computed

LMArena scores are estimated with the Bradley-Terry model, a statistical model that assigns each competitor a strength so that the probability of one beating another depends on the difference between their strengths. The early leaderboard used an online Elo rating borrowed from chess, but Elo is sensitive to the order of games, so the platform moved to a Bradley-Terry fit over all votes, reported on an Elo-like scale.

On this scale, a difference of about 100 points corresponds to the stronger model being preferred in roughly 64 percent of battles. Each score carries a bootstrap confidence interval, and models whose intervals overlap should be read as statistically tied rather than strictly ordered.

Score differenceExpected win rate of stronger modelPractical reading
0 points50 percentIndistinguishable
25 pointsAbout 54 percentBarely noticeable to users
50 pointsAbout 57 percentSmall but measurable edge
100 pointsAbout 64 percentClear preference
200 pointsAbout 76 percentLarge quality gap
Score gap and expected win rate on an Elo-like scale

LMArena also publishes style-controlled rankings, which statistically adjust for response length and Markdown formatting such as headers and bullet lists. This control was introduced because voters tend to favor longer, more heavily formatted answers even when the content is not better.

Leaderboard categories and arenas

LMArena splits its data into categories and separate arenas so that a single overall score does not hide differences between task types. Prompts are classified automatically, which allows rankings for coding, mathematics, creative writing, longer queries, multi-turn conversations and individual languages.

  • Text arena: the original chat leaderboard, with filters such as hard prompts, coding and instruction following.
  • Vision arena: models answer prompts that include images.
  • WebDev arena: models build small web applications that users compare visually.
  • Text-to-image and image editing arenas: users compare generated or edited images.
  • Search arena: models with web access compare grounded answers.

Category rankings often differ markedly from the overall list. A model can rank near the top in creative writing while sitting lower in hard coding prompts, so practitioners should read the category that matches their use case.

LMArena compared with other evaluation approaches

LMArena complements, rather than replaces, static benchmarks and LLM-as-a-judge evaluation. Each approach answers a different question, and serious model selection usually combines several of them.

ApproachSignalMain strengthMain weakness
LMArenaCrowdsourced pairwise human votesReal prompts, open-ended qualityVoter bias, shallow judgments
Static benchmarkAccuracy on fixed questionsReproducible and cheapContamination and saturation
LLM-as-a-judgeModel-graded comparisonsFast and scalableJudge model bias
Expert human evaluationSpecialist ratings on defined tasksDepth and domain accuracyExpensive, small samples
Internal task evaluationMetrics on company dataDirectly relevant to the productNot comparable across teams
LMArena and related evaluation methods

A key advantage of LMArena is resistance to benchmark contamination: because prompts arrive continuously from users, a model cannot simply memorize a published test set. Its key weakness is that most voters judge a response in seconds, so the signal favors answers that look convincing over answers that have been carefully verified.

Criticisms and limitations

LMArena faces criticism mainly about voter bias, provider gaming and representativeness. The platform has responded with style control, published methodology and policy changes, but the core limitations of crowdsourced preference remain.

  • Style over substance: voters reward length, formatting and a confident tone, which style control reduces but does not remove.
  • Private testing: large providers can test many unreleased variants and publish only the best, which a 2025 academic critique argued inflates their scores.
  • Tuned variants: a model version optimized for arena voting may differ from the version that is publicly released.
  • Unrepresentative prompts: the user base skews toward technically curious English speakers, so everyday enterprise tasks are underrepresented.
  • Limited verification: voters rarely check facts, code execution or long documents, so subtle errors go unpenalized.

The responsible reading is therefore to treat LMArena as one strong signal of general user preference, check confidence intervals before claiming a ranking, and confirm any decision with task-specific evaluation.

Put simply

LMArena works like a blind taste test for AI chatbots. You type a question, two hidden chatbots answer, and you pick the answer you like more. Only then do you learn which chatbots they were.

Millions of these small votes are added up into a ranking. The result shows which models ordinary users tend to prefer, but it does not prove that a model is always more accurate, because people often choose the answer that simply sounds better.

In practice

A product team choosing a model for a coding assistant looks at the LMArena leaderboard and shortlists the three models with the highest coding-category scores. Two of them differ by only 12 points, and their confidence intervals overlap, so the team treats them as tied rather than picking the nominal leader.

The team then runs its own evaluation on 400 real tickets from its codebase, measuring whether the generated patches pass the existing unit tests. The second-ranked model on LMArena passes noticeably more tests and costs less per request, so it is selected. The arena ranking narrowed the field quickly, while the internal test made the final decision.