IBM Granite

Also known as: Granite models

On this page 7

IBM Granite is a family of open foundation models developed by IBM for business applications, covering language, code, vision, speech, embeddings, time series and safety classification. Among AI models & assistants, the Granite models stand out less for raw benchmark leadership than for their enterprise orientation: permissive licensing, documented training data, small and efficient sizes, and governance features that matter to regulated organizations.

In the wider field of artificial intelligence, Granite occupies the space of compact open-weight models alongside families such as Llama, Mistral, Gemma, Phi and Qwen. Most Granite releases are published under the Apache 2.0 license, can run locally or in the cloud, and are also offered through IBM watsonx with contractual intellectual property protection for customers.

How the Granite family developed

Granite began in 2023 as a set of proprietary models inside IBM watsonx and became an open-weight family in 2024, starting with the Granite Code models. Each subsequent generation broadened the scope from text to code, vision, speech and specialist tasks while keeping parameter counts deliberately modest.

The table below summarizes the main generations and the change each one introduced. Exact model lists vary by release, so the entries describe the defining features rather than every checkpoint.

GenerationApproximate periodDefining change
Granite 13B2023First proprietary language models on watsonx
Granite Code2024Open code models from 3B to 34B parameters
Granite 3.0 to 3.3Late 2024 to 2025Dense and mixture-of-experts models, 128K context, vision, speech and a reasoning toggle
Granite 4.0Late 2025Hybrid Mamba-2 and transformer architecture for lower memory use
Main generations of IBM Granite

Throughout this history, IBM has emphasized transparency about training data and filtering, publishing technical reports that describe data sources, deduplication and governance reviews. This openness is a deliberate contrast with models whose training corpora remain undisclosed.

Architecture and the hybrid design of Granite 4.0

Granite 4.0 combines Mamba-2 state space layers with a smaller share of transformer attention layers. State space layers process a sequence with memory that grows linearly with length, whereas attention compares every token with every other token and needs a key-value cache that grows with context. Mixing the two keeps most of the quality of attention while sharply reducing memory use on long inputs.

Some Granite 4.0 variants also use mixture-of-experts routing, in which only a fraction of the parameters is active for each token. The result is a model that stores more knowledge than it computes per step. The table below compares the main design elements.

Design elementWhat it doesPractical benefit
Dense transformerAll parameters active for every tokenPredictable behavior and broad tool support
Mixture of expertsRoutes each token to a subset of expertsLower compute per token at a given capacity
Mamba-2 layersLinear-time sequence processingMuch smaller memory footprint on long context
Hybrid stackMostly Mamba-2 with periodic attentionLong-context efficiency with retained precision
Design elements used across Granite models

IBM reports that the hybrid design can cut inference memory by more than half compared with conventional transformers of similar size, especially with long documents and many concurrent sessions. The actual saving depends on context length, batch size and the serving engine.

Model types within the family

Granite is not a single model but a portfolio of task-specific models that share naming and licensing. Organizations typically combine several of them in one system, for example a language model for generation, an embedding model for retrieval and a guardian model for safety checks.

BranchPurposeTypical use
Granite language modelsGeneral instruction following and tool callingAssistants, summarization, agents
Granite CodeCode generation and explanationDeveloper tools, legacy code modernization
Granite GuardianDetection of harmful content and hallucinationInput and output filtering in RAG systems
Granite EmbeddingDense vector representationsSemantic search and retrieval
Granite VisionImage and document understandingTables, charts and scanned forms
Granite SpeechSpeech recognition and translationTranscription pipelines
Granite Time SeriesCompact forecasting modelsDemand, sensor and capacity forecasts
Specialist branches of the Granite family

The Granite Guardian models deserve particular attention, because they act as classifiers for risk rather than as chat models. They flag jailbreak attempts, harmful content, and answers that are not supported by retrieved context, which makes them useful even next to models from other vendors.

How Granite compares with other open model families

Granite competes mainly in the small and mid-size open-weight segment. Its positioning differs from families optimized primarily for benchmark rankings or consumer chat.

FamilyMain strengthLicensing emphasis
IBM GraniteEnterprise governance and efficiencyApache 2.0 with optional indemnity on watsonx
LlamaLarge ecosystem and community toolingCustom community license with conditions
MistralStrong small models and European presenceMix of Apache 2.0 and commercial licenses
QwenBroad multilingual and coding strengthMostly Apache 2.0 for open releases
Phi and GemmaVery compact models for local usePermissive or custom terms by release
Granite compared with selected open model families

On public leaderboards, Granite models generally perform competitively within their size class rather than leading the frontier. Their advantage appears in total cost of operation and compliance, where license clarity, documented data and certified management processes reduce legal and procurement friction.

Governance, licensing and limitations

IBM positions Granite as a governance-first model family. The company states that Granite 4.0 was covered by an ISO/IEC 42001 certified AI management system, the international standard for managing artificial intelligence responsibly, and model checkpoints are cryptographically signed so that users can verify their origin.

  • Capability ceiling: the smaller Granite models trail frontier systems on complex reasoning, advanced mathematics and open-ended creative writing.
  • Language coverage: support is strongest in English and a set of major languages, and quality drops for low-resource languages.
  • Indemnity scope: intellectual property protection applies to use through IBM offerings under contract, not to every self-hosted copy.
  • Tooling maturity: the hybrid architecture requires serving engines that support Mamba-2 layers, which older inference stacks may lack.

A frequent mistake is to treat governance documentation as a guarantee of correct output. Granite models can still hallucinate, so production systems should combine them with retrieval, evaluation and output checks.

Stripped to the essentials

IBM Granite is a set of AI models that IBM gives away for free use, including in commercial products. Some models write text, some write code, some read documents or listen to speech, and some check whether an answer is safe.

They are not the biggest or flashiest models available. Their appeal is that they are small enough to run cheaply, their rules of use are simple, and IBM explains how they were trained. For a company that must justify every tool to auditors, that clarity can matter more than a few extra points on a leaderboard.

Where it is applied

A regional insurer wants an internal assistant that answers claims handlers using policy documents, but its compliance team will not send customer data to an external service. The IT department deploys a small Granite language model on its own servers, pairs it with Granite Embedding for document search and places Granite Guardian in front of every answer to flag responses that are not supported by the retrieved policy text.

Because the models carry an Apache 2.0 license and published training documentation, the procurement review takes weeks rather than months. In the pilot, handlers use the assistant to locate clauses and draft summaries, while every flagged answer is routed to a senior reviewer, which keeps the error rate within the limits agreed with the compliance team.