LMArena is a public evaluation platform that ranks large language models and other generative AI systems by collecting crowdsourced pairwise votes between anonymous models and converting them into a statistical leaderboard. Originally launched in 2023 as Chatbot Arena by researchers associated with LMSYS and the University of California, Berkeley, it became one of the most cited references in model evaluation for artificial intelligence because it measures human preference on open-ended tasks rather than accuracy on a fixed test set.
Unlike static benchmarks such as MMLU or GPQA, LMArena has no fixed answer key. Each user writes a prompt, receives two answers from models whose names are hidden, and votes for the better one. The ranking therefore reflects which model people prefer in practice, with all the strengths and biases that such a signal carries.
How an LMArena battle works
The core unit of LMArena is the anonymous battle: two randomly selected models answer the same user prompt side by side, and the user chooses the left answer, the right answer, a tie, or that both are bad. Model identities are revealed only after the vote, which prevents brand loyalty from influencing the judgment.
The steps below show how a single battle becomes part of the leaderboard.
| Stage | What happens | Why it matters |
|---|---|---|
| Prompt submission | User writes any question or task | Covers real, diverse use cases |
| Model sampling | Platform pairs two models, often weighted | Balances vote counts across models |
| Blind comparison | Answers shown as model A and model B | Removes name and brand bias |
| Vote | A better, B better, tie or both bad | Produces a pairwise preference label |
| Filtering | Duplicate, abusive or identity-revealing votes removed | Protects data quality |
| Aggregation | Votes fitted into a rating model | Produces scores and confidence intervals |
Battles that reveal the model identity in the answer, for example when a model states its own name, are typically excluded from the ranking, because the vote is then no longer blind.
How LMArena scores are computed
LMArena scores are estimated with the Bradley-Terry model, a statistical model that assigns each competitor a strength so that the probability of one beating another depends on the difference between their strengths. The early leaderboard used an online Elo rating borrowed from chess, but Elo is sensitive to the order of games, so the platform moved to a Bradley-Terry fit over all votes, reported on an Elo-like scale.
On this scale, a difference of about 100 points corresponds to the stronger model being preferred in roughly 64 percent of battles. Each score carries a bootstrap confidence interval, and models whose intervals overlap should be read as statistically tied rather than strictly ordered.
| Score difference | Expected win rate of stronger model | Practical reading |
|---|---|---|
| 0 points | 50 percent | Indistinguishable |
| 25 points | About 54 percent | Barely noticeable to users |
| 50 points | About 57 percent | Small but measurable edge |
| 100 points | About 64 percent | Clear preference |
| 200 points | About 76 percent | Large quality gap |
LMArena also publishes style-controlled rankings, which statistically adjust for response length and Markdown formatting such as headers and bullet lists. This control was introduced because voters tend to favor longer, more heavily formatted answers even when the content is not better.
Leaderboard categories and arenas
LMArena splits its data into categories and separate arenas so that a single overall score does not hide differences between task types. Prompts are classified automatically, which allows rankings for coding, mathematics, creative writing, longer queries, multi-turn conversations and individual languages.
- Text arena: the original chat leaderboard, with filters such as hard prompts, coding and instruction following.
- Vision arena: models answer prompts that include images.
- WebDev arena: models build small web applications that users compare visually.
- Text-to-image and image editing arenas: users compare generated or edited images.
- Search arena: models with web access compare grounded answers.
Category rankings often differ markedly from the overall list. A model can rank near the top in creative writing while sitting lower in hard coding prompts, so practitioners should read the category that matches their use case.
LMArena compared with other evaluation approaches
LMArena complements, rather than replaces, static benchmarks and LLM-as-a-judge evaluation. Each approach answers a different question, and serious model selection usually combines several of them.
| Approach | Signal | Main strength | Main weakness |
|---|---|---|---|
| LMArena | Crowdsourced pairwise human votes | Real prompts, open-ended quality | Voter bias, shallow judgments |
| Static benchmark | Accuracy on fixed questions | Reproducible and cheap | Contamination and saturation |
| LLM-as-a-judge | Model-graded comparisons | Fast and scalable | Judge model bias |
| Expert human evaluation | Specialist ratings on defined tasks | Depth and domain accuracy | Expensive, small samples |
| Internal task evaluation | Metrics on company data | Directly relevant to the product | Not comparable across teams |
A key advantage of LMArena is resistance to benchmark contamination: because prompts arrive continuously from users, a model cannot simply memorize a published test set. Its key weakness is that most voters judge a response in seconds, so the signal favors answers that look convincing over answers that have been carefully verified.
Criticisms and limitations
LMArena faces criticism mainly about voter bias, provider gaming and representativeness. The platform has responded with style control, published methodology and policy changes, but the core limitations of crowdsourced preference remain.
- Style over substance: voters reward length, formatting and a confident tone, which style control reduces but does not remove.
- Private testing: large providers can test many unreleased variants and publish only the best, which a 2025 academic critique argued inflates their scores.
- Tuned variants: a model version optimized for arena voting may differ from the version that is publicly released.
- Unrepresentative prompts: the user base skews toward technically curious English speakers, so everyday enterprise tasks are underrepresented.
- Limited verification: voters rarely check facts, code execution or long documents, so subtle errors go unpenalized.
The responsible reading is therefore to treat LMArena as one strong signal of general user preference, check confidence intervals before claiming a ranking, and confirm any decision with task-specific evaluation.
Put simply
LMArena works like a blind taste test for AI chatbots. You type a question, two hidden chatbots answer, and you pick the answer you like more. Only then do you learn which chatbots they were.
Millions of these small votes are added up into a ranking. The result shows which models ordinary users tend to prefer, but it does not prove that a model is always more accurate, because people often choose the answer that simply sounds better.
In practice
A product team choosing a model for a coding assistant looks at the LMArena leaderboard and shortlists the three models with the highest coding-category scores. Two of them differ by only 12 points, and their confidence intervals overlap, so the team treats them as tied rather than picking the nominal leader.
The team then runs its own evaluation on 400 real tickets from its codebase, measuring whether the generated patches pass the existing unit tests. The second-ranked model on LMArena passes noticeably more tests and costs less per request, so it is selected. The arena ranking narrowed the field quickly, while the internal test made the final decision.