PaLM

Also known as: Pathways Language Model · PaLM 2

On this page 7

PaLM, short for Pathways Language Model, is a family of large language models developed by Google and first announced in April 2022, best known for a 540-billion-parameter dense transformer that set new results in reasoning, code and multilingual tasks at the time. Among AI models & assistants, PaLM marks the generation of Google models that came between LaMDA and Gemini, and its successor PaLM 2 powered the Bard assistant and the first Google generative AI developer APIs.

In artificial intelligence history, PaLM matters for three reasons: it demonstrated training at very large scale across thousands of accelerators, it popularized evidence that chain-of-thought prompting unlocks multi-step reasoning in large models, and it served as the base for specialized variants in medicine, robotics and security.

How PaLM was built and trained

PaLM is a decoder-only transformer trained to predict the next token, the same basic design as most generative language models. Its distinguishing feature was not a new objective but engineering at scale: the largest model was trained on 6,144 TPU v4 chips spread across two pods, coordinated by Google’s Pathways system, which gave the model its name.

The original paper described several architectural refinements that later became common in other model families. The table below lists the most important ones and why they were chosen.

Design choiceWhat it doesPractical benefit
SwiGLU activationGated activation in the feed-forward layersBetter quality per unit of compute
Parallel layersAttention and feed-forward computed side by sideRoughly 15 percent faster training at large scale
Multi-query attentionKeys and values shared across attention headsCheaper and faster autoregressive decoding
Rotary position embeddingsPosition encoded by rotating query and key vectorsStronger handling of longer sequences
Large SentencePiece vocabularyAbout 256,000 tokens covering many languages and codeEfficient tokenization beyond English
No bias termsBiases removed from dense and normalization layersImproved training stability
Architectural choices in the original PaLM

The training corpus contained about 780 billion tokens drawn from filtered web pages, books, Wikipedia, news, conversations and source code. In hindsight this was relatively few tokens for 540 billion parameters, a point that later compute-optimal scaling research made clear and that shaped the design of PaLM 2.

PaLM model sizes and variants

PaLM refers to a lineage of models, not one checkpoint. The original release came in three sizes, and Google then adapted the base models for instruction following, specialized domains and multimodal input.

ModelYearKey characteristic
PaLM 8B, 62B and 540B2022Base models used to study scaling behavior
Flan-PaLM2022Instruction-tuned on a large collection of tasks
Med-PaLM2022Adapted for medical question answering
PaLM-E2023Embodied model combining language with robot sensor input
PaLM 22023Smaller, compute-optimal, more multilingual successor
Med-PaLM 2 and Sec-PaLM2023Domain variants for clinical and security work
Main members of the PaLM family

PaLM 2 was offered in four sizes with internal names Gecko, Otter, Bison and Unicorn, from a model small enough to run on mobile devices up to the largest tier. Developers met it mainly as text-bison and chat-bison endpoints in the PaLM API and in Google Cloud Vertex AI, and as the engine behind Bard during 2023.

What PaLM demonstrated about scaling

PaLM became a reference point because it showed discontinuous gains on some tasks as model size increased. On parts of the BIG-bench evaluation suite, the jump from 62 billion to 540 billion parameters produced improvements far larger than a smooth trend would predict, which fed the debate about emergent abilities in large models.

Its most cited result concerned chain-of-thought prompting. When the 540B model was shown examples that spelled out intermediate reasoning steps, its accuracy on grade-school math word problems rose sharply and approached that of fine-tuned systems. This helped establish prompting technique as a lever for reasoning quality, not only model size.

Later research qualified both findings. Some apparent emergence depends on how the metric is defined, and PaLM 2 showed that a smaller model trained on more data can match or exceed a much larger undertrained one. The broader lesson is that parameter count alone is a poor guide to capability.

PaLM compared with other model families

PaLM is best understood in relation to the models around it, both inside Google and across the industry. The comparison below summarizes the main differences.

Model familyDeveloperMain distinction from PaLM
GeminiGoogle DeepMindNatively multimodal successor that replaced PaLM 2
T5GoogleEncoder-decoder design focused on transfer learning
BERTGoogleEncoder-only model for understanding, not generation
GPT familyOpenAICompeting decoder-only line with chat products
LlamaMetaOpenly released weights, unlike closed PaLM weights
GemmaGoogleSmall open-weight models derived from Gemini research
PaLM and related model families

A practical difference for users is access. PaLM weights were never released publicly; the models were reachable only through Google products and paid APIs. This contrasts with open-weight families, where organizations can host and fine-tune the model themselves.

Status, limitations and common misconceptions

PaLM is now a legacy model family. After Gemini launched at the end of 2023, Google moved Bard and its developer platforms to Gemini, and the PaLM API was deprecated in favor of the Gemini API during 2024. Existing applications built on PaLM endpoints had to migrate.

  • PaLM is not the same as Pathways. Pathways is the distributed training and serving system; PaLM is a model trained with it.
  • PaLM 2 was not simply a bigger PaLM. It was smaller in its largest form and gained quality through better data mixtures and compute-optimal scaling.
  • Benchmark scores from 2022 do not reflect current standards. Modern models surpass PaLM on most reasoning and coding evaluations.
  • Known weaknesses included hallucinated facts, sensitivity to prompt wording and biases inherited from web-scale training data.

The table below summarizes how teams that still encounter PaLM references should interpret them today.

SituationRecommended interpretationReason
Older code calls text-bisonPlan migration to a Gemini modelPaLM endpoints are deprecated
Research paper cites PaLMTreat results as a 2022 to 2023 baselineLater models changed the frontier
Product mentions Med-PaLMRead it as a research systemClinical use requires separate validation
Discussion of scaling lawsPair PaLM with compute-optimal studiesPaLM was undertrained for its size
Interpreting PaLM references today

A non-technical view

PaLM was one of Google’s big text-writing AI systems before the Gemini models arrived. Like other chat-style AI, it learned by reading an enormous amount of text and practicing how to guess the next word.

What made it notable was its size and what that size revealed. When it was asked to show its working step by step, much like a student writing out a math problem, it solved harder questions far more often. Its successor, PaLM 2, ran the early Bard chatbot. Today PaLM has been retired and replaced by Gemini, so it is mostly a milestone in the story of modern AI rather than a tool people choose.

A concrete example

In mid-2023, a mid-sized insurance company built an internal assistant that summarized claim notes for adjusters. The developers used the text-bison model through the PaLM API, sending each claim file with a prompt asking for a five-point summary and a list of missing documents. Adjusters reported that the summaries saved several minutes per file, although staff still checked dates and amounts because the model occasionally invented details not present in the notes.

When Google announced the deprecation of PaLM endpoints, the team ran the same set of several hundred archived claims through a Gemini model and compared outputs against adjuster-approved summaries. The new model produced fewer factual errors and handled longer files, and the migration required changes mainly to the API client and prompt formatting. The outcome illustrates PaLM’s typical role today: a first-generation foundation for enterprise prototypes that were later moved to its successor.