OLMo

Also known as: Open Language Model

On this page 7

OLMo, short for Open Language Model, is a family of large language models developed by the Allen Institute for AI (Ai2) and released as a fully open research artifact: model weights, the complete pretraining dataset, training code, intermediate checkpoints, training logs and evaluation tools are all public. Among AI models & assistants, OLMo stands out less for leading benchmark scores than for end-to-end transparency, which allows researchers to study how a language model’s behavior arises from its data and training process.

The first OLMo models appeared in early 2024 at 1 and 7 billion parameters, followed by OLMo 2 and OLMo 3 generations that scaled to 32 billion parameters and added instruction-tuned and reasoning variants. The models are distributed under the permissive Apache 2.0 license.

What makes OLMo fully open

OLMo is fully open because it publishes every component needed to reproduce the model, not only the final weights. Most widely used models labeled open, such as Llama, Gemma or Qwen, are better described as open-weight models: anyone can download and run them, but the training data and much of the training recipe remain undisclosed.

The table below lists the components Ai2 releases and why each matters for research and auditing.

ComponentWhat is releasedWhy it matters
Pretraining dataDolma and later curated data mixesEnables data attribution and contamination checks
Training codeFull codebase and configuration filesAllows exact reproduction and modification
CheckpointsHundreds of intermediate snapshotsShows how abilities emerge during training
Training logsLoss curves and run metricsExposes instabilities and design decisions
Post-training recipeTülu data, code and settingsExplains instruction and preference tuning
Evaluation suiteStandardized benchmarks and scriptsMakes reported scores verifiable
Components released with OLMo models

Ai2 also introduced OLMoTrace, a tool that matches spans of a model’s output to verbatim passages in its training data. Such tracing is only possible when the full corpus is public and indexed, which is a capability closed and open-weight models cannot offer.

OLMo model generations and variants

OLMo has evolved through three main generations, each improving data quality, training stability and downstream performance while keeping the same openness policy. Alongside dense models, Ai2 released OLMoE, a mixture-of-experts variant that activates roughly 1 billion of about 7 billion total parameters per token.

GenerationReleasedSizesNotable changes
OLMo 120241B, 7BFirst fully open release with Dolma
OLMoE2024About 7B total, 1B activeSparse mixture-of-experts design
OLMo 22024 to 20251B, 7B, 13B, 32BStability fixes and mid-training data stage
OLMo 320257B, 32BLonger context and Think reasoning variants
Main OLMo releases

Each generation typically ships as a base model for further training, an Instruct model tuned for chat and tool use, and, from OLMo 3 onward, a Think model trained to produce explicit reasoning traces. Exact token counts and context lengths differ by release, so practitioners should consult the model card of the specific version they deploy.

How OLMo is trained

OLMo is trained as a decoder-only transformer in stages that Ai2 documents in detail. Pretraining on several trillion tokens of filtered web text, code, scientific papers and books builds general knowledge; a shorter mid-training stage on high-quality and mathematical data sharpens specific skills; and post-training aligns the model to follow instructions.

  • Pretraining uses deduplicated, quality-filtered corpora derived from Common Crawl and curated sources, with the exact mixture published.
  • Mid-training, introduced with OLMo 2 under the name Dolmino, anneals the learning rate on curated data to improve math and reasoning.
  • Supervised fine-tuning teaches the model to answer prompts in a conversational format.
  • Preference tuning with direct preference optimization aligns responses with human-like judgments.
  • Reinforcement learning with verifiable rewards improves math, coding and instruction following by rewarding answers that can be checked automatically.

Ai2 also published architectural stability changes, such as reordered layer normalization and QK-norm, which reduced loss spikes during long training runs. These findings are useful beyond OLMo because they come with logs that demonstrate the effect.

OLMo compared with other open models

OLMo differs from other openly available models mainly in how much of the development process is disclosed, while its raw capability generally sits close to, but not always ahead of, open-weight models of similar size. The comparison below focuses on openness rather than benchmark rankings, which change with every release.

Model familyWeightsTraining dataTraining code
OLMoOpen, Apache 2.0Fully publicFully public
LlamaOpen under a custom licenseNot releasedPartial
GemmaOpen under usage termsNot releasedPartial
QwenMostly Apache 2.0Not releasedPartial
gpt-ossOpen, Apache 2.0Not releasedInference code only
Openness of OLMo and comparable model families

Other fully open efforts exist, such as models from research collectives that publish data and code, but OLMo is among the most complete and actively maintained of them. For teams that need the strongest possible performance at a given size, a leading open-weight model may still score higher; for teams that need auditability and reproducibility, OLMo is often the stronger choice.

Typical uses and limitations of OLMo

OLMo is used primarily in research, education and regulated settings where knowing exactly what a model was trained on is more valuable than marginal benchmark gains. The following table matches common needs with the reason OLMo fits or does not fit.

SituationSuitabilityReason
Training dynamics researchStrong fitIntermediate checkpoints and logs are public
Benchmark contamination studiesStrong fitTraining corpus can be searched directly
On-premises assistantGood fitPermissive license and local deployment
Frontier-level performanceWeaker fitLargest models outperform it on hard tasks
Many non-English languagesLimited fitTraining data is predominantly English
When OLMo is a suitable choice

Users should note that public training data does not remove risk. The corpus contains web text with errors, bias and potentially sensitive content, and OLMo models can hallucinate like any language model. Openness makes these problems easier to investigate, not absent, so standard safeguards such as output review and domain evaluation still apply.

Put simply

OLMo is an AI language model, similar in purpose to well-known chatbots, but built with a different goal: nothing about how it was made is hidden. Its creators share not only the finished model but also the text it learned from, the instructions used to train it and snapshots taken along the way.

Compare it to a restaurant that publishes the full recipe, the list of suppliers and video of the kitchen, instead of only serving the meal. You might find a tastier dish elsewhere, but here you can check every ingredient. That openness is why scientists, teachers and cautious organizations value OLMo.

In practice

A university research group wants to know whether a language model truly solves grade-school math problems or simply remembers them from training. With a closed or open-weight model, the group cannot check what the model saw during training, so any conclusion remains speculative without data access.

Using the 7B OLMo model, the researchers search the published pretraining corpus for each test problem and find that about one in twenty appears in close paraphrase. They then evaluate the model separately on the clean subset and on the overlapping subset, and they repeat the analysis across a dozen intermediate checkpoints to see when the skill emerges.

The result is a verifiable finding: accuracy is noticeably higher on the overlapping problems, but the model still performs well on unseen ones, and the ability appears steadily rather than suddenly. Because every step uses public artifacts, other groups can reproduce the study exactly.