Falcon · language model

Also known as: Falcon LLM

On this page 7

Falcon is a family of open-weight large language models developed by the Technology Innovation Institute (TII) in Abu Dhabi, first released in 2023 and continued through several generations. Within artificial intelligence, and specifically among AI models & assistants, Falcon is the language model line best known for showing that carefully filtered web data alone can train models competitive with those built on curated mixtures.

The name, sometimes written Falcon LLM, covers dense Transformer models, a pure state space model, hybrid architectures and small edge-oriented variants. Falcon is a set of base and instruction-tuned models that developers download and run, not a consumer chat product in the way that Claude or Gemini are offered.

Falcon generations and model sizes

The Falcon family has evolved through distinct generations, each changing the training data scale, the architecture or the target hardware. The table below lists the main releases and what distinguished them.

ReleaseSizesDefining feature
Falcon 7B and 40B (2023)7B, 40BTrained mostly on RefinedWeb, Apache 2.0 license
Falcon 180B (2023)180BAbout 3.5 trillion training tokens, restricted license
Falcon 2 (2024)11B, 11B VLMMultilingual focus and a vision-language variant
Falcon Mamba (2024)7BAttention-free state space architecture
Falcon 3 (2024)1B to 10BSmall models trained on about 14 trillion tokens
Falcon-H1 (2025)0.5B to 34BHybrid of attention and Mamba layers
Main Falcon releases

When Falcon 40B appeared in mid-2023, it led public open-model leaderboards for several weeks, and Falcon 180B was among the largest openly downloadable models at its release. Later generations shifted toward smaller, efficient models that run on a single GPU or on laptops, reflecting wider industry interest in deployment cost rather than raw scale.

Architecture and the RefinedWeb dataset

Falcon models are decoder-only Transformers in their early generations, with design choices aimed at fast inference and efficient training. The original models used multi-query attention, in which all attention heads share one key and value projection, which shrinks the key-value cache and speeds up generation.

The distinctive contribution was the RefinedWeb dataset, built from Common Crawl through aggressive URL filtering, text extraction, heuristic quality rules and large-scale fuzzy and exact deduplication. TII publicly released a portion of roughly 600 billion tokens, which helped other teams study how much quality web text can replace curated sources such as books and papers.

ComponentChoicePractical effect
AttentionMulti-query or grouped-query attentionSmaller memory use during generation
Position encodingRotary position embeddingsGood handling of sequence order
Block layoutParallel attention and feed-forward pathsHigher training throughput
Sequence mixing in Falcon-H1Attention combined with Mamba-2 layersLong contexts at lower cost
Architectural choices in the Falcon family

The Falcon Mamba and Falcon-H1 models mark a clear departure. A state space model processes tokens with a fixed-size recurrent state, so memory use does not grow with context length, while the hybrid design keeps some attention layers for precise recall.

How Falcon compares with other open model families

Falcon belongs to the group of open-weight model families that also includes Llama, Mistral, Qwen, Gemma and OLMo. The families differ mainly in licensing terms, training transparency and the range of sizes offered.

FamilyDeveloper typeNotable trait
FalconGovernment-funded research instituteWeb-data focus and hybrid architectures
LlamaLarge technology companyBroad ecosystem and community fine-tunes
MistralCommercial startupEfficient models and mixture-of-experts designs
QwenLarge technology companyStrong multilingual and coding variants
OLMoNonprofit research instituteFully open data, code and checkpoints
Falcon among open-weight model families

Benchmark positions change quickly, so leaderboard rank at release says little about current standing. In practice, Falcon models are chosen for permissive licensing on many variants, Arabic language support in dedicated releases, and the availability of small hybrid models for constrained hardware.

Licensing and deployment considerations

Falcon licensing varies by release, which is the most frequent source of confusion. Falcon 7B and 40B were moved to the Apache 2.0 license, while Falcon 180B and later generations use TII-specific licenses based on Apache terms with an acceptable use policy and, for some versions, limits on offering the model as a hosted service.

  • Check the license per checkpoint before commercial use, because terms differ between generations.
  • Size the hardware to the model: Falcon 180B needs several hundred gigabytes of accelerator memory at 16-bit precision, while Falcon 3 models of 1B to 10B parameters fit on a single consumer GPU, especially when quantized to 4 or 8 bits.
  • Use instruction-tuned variants for chat and task following, and base models only as a starting point for fine-tuning.

Falcon checkpoints are distributed in standard formats and run with common open inference stacks such as Hugging Face Transformers, vLLM and llama.cpp. Teams usually add retrieval, guardrails and evaluation on their own data before production use, because general benchmark scores do not predict accuracy on a specific domain. Self-hosting also shifts responsibility for security patches, monitoring and acceptable use enforcement to the deploying organization, which a managed service would otherwise handle.

Limitations and common misconceptions

Like other language models, Falcon can produce fluent but incorrect statements, reflect biases in web data and follow harmful instructions if deployed without safeguards. The early models also had short context windows of about 2,048 tokens, which limits their use for long documents compared with later generations.

  • Misconception: Falcon is one model. It is a family with very different architectures and licenses.
  • Misconception: open weights mean open data. Only part of the training corpus was released.
  • Misconception: bigger is always better. Newer small Falcon models can outperform the original 40B model on many benchmarks.

The simple explanation

Falcon is a set of AI text models made by a research institute in Abu Dhabi. Anyone can download many of them and run them on their own computers. They can answer questions, write and summarize text, and help with code.

Picture a library of engines in different sizes. The biggest one is powerful but needs a data center. The smallest ones fit in a laptop. The main point is that Falcon lets organizations keep the model and their data in-house instead of sending text to an outside service.

How it is used in practice

A regional hospital network wants to summarize discharge notes without sending patient text to an external provider. Its IT team downloads an instruction-tuned Falcon 3 model of 7B parameters, quantizes it to 4-bit precision and runs it on a single server GPU with 24 GB of memory inside the hospital network.

The team fine-tunes the model on a few thousand approved, anonymized note and summary pairs, then requires a clinician to review every generated summary. After a three-month pilot, clinicians report that drafting a discharge summary takes roughly half the previous time, and the network confirms that the license terms of the chosen checkpoint permit internal commercial use.