PaLM, short for Pathways Language Model, is a family of large language models developed by Google and first announced in April 2022, best known for a 540-billion-parameter dense transformer that set new results in reasoning, code and multilingual tasks at the time. Among AI models & assistants, PaLM marks the generation of Google models that came between LaMDA and Gemini, and its successor PaLM 2 powered the Bard assistant and the first Google generative AI developer APIs.
In artificial intelligence history, PaLM matters for three reasons: it demonstrated training at very large scale across thousands of accelerators, it popularized evidence that chain-of-thought prompting unlocks multi-step reasoning in large models, and it served as the base for specialized variants in medicine, robotics and security.
How PaLM was built and trained
PaLM is a decoder-only transformer trained to predict the next token, the same basic design as most generative language models. Its distinguishing feature was not a new objective but engineering at scale: the largest model was trained on 6,144 TPU v4 chips spread across two pods, coordinated by Google’s Pathways system, which gave the model its name.
The original paper described several architectural refinements that later became common in other model families. The table below lists the most important ones and why they were chosen.
| Design choice | What it does | Practical benefit |
|---|---|---|
| SwiGLU activation | Gated activation in the feed-forward layers | Better quality per unit of compute |
| Parallel layers | Attention and feed-forward computed side by side | Roughly 15 percent faster training at large scale |
| Multi-query attention | Keys and values shared across attention heads | Cheaper and faster autoregressive decoding |
| Rotary position embeddings | Position encoded by rotating query and key vectors | Stronger handling of longer sequences |
| Large SentencePiece vocabulary | About 256,000 tokens covering many languages and code | Efficient tokenization beyond English |
| No bias terms | Biases removed from dense and normalization layers | Improved training stability |
The training corpus contained about 780 billion tokens drawn from filtered web pages, books, Wikipedia, news, conversations and source code. In hindsight this was relatively few tokens for 540 billion parameters, a point that later compute-optimal scaling research made clear and that shaped the design of PaLM 2.
PaLM model sizes and variants
PaLM refers to a lineage of models, not one checkpoint. The original release came in three sizes, and Google then adapted the base models for instruction following, specialized domains and multimodal input.
| Model | Year | Key characteristic |
|---|---|---|
| PaLM 8B, 62B and 540B | 2022 | Base models used to study scaling behavior |
| Flan-PaLM | 2022 | Instruction-tuned on a large collection of tasks |
| Med-PaLM | 2022 | Adapted for medical question answering |
| PaLM-E | 2023 | Embodied model combining language with robot sensor input |
| PaLM 2 | 2023 | Smaller, compute-optimal, more multilingual successor |
| Med-PaLM 2 and Sec-PaLM | 2023 | Domain variants for clinical and security work |
PaLM 2 was offered in four sizes with internal names Gecko, Otter, Bison and Unicorn, from a model small enough to run on mobile devices up to the largest tier. Developers met it mainly as text-bison and chat-bison endpoints in the PaLM API and in Google Cloud Vertex AI, and as the engine behind Bard during 2023.
What PaLM demonstrated about scaling
PaLM became a reference point because it showed discontinuous gains on some tasks as model size increased. On parts of the BIG-bench evaluation suite, the jump from 62 billion to 540 billion parameters produced improvements far larger than a smooth trend would predict, which fed the debate about emergent abilities in large models.
Its most cited result concerned chain-of-thought prompting. When the 540B model was shown examples that spelled out intermediate reasoning steps, its accuracy on grade-school math word problems rose sharply and approached that of fine-tuned systems. This helped establish prompting technique as a lever for reasoning quality, not only model size.
Later research qualified both findings. Some apparent emergence depends on how the metric is defined, and PaLM 2 showed that a smaller model trained on more data can match or exceed a much larger undertrained one. The broader lesson is that parameter count alone is a poor guide to capability.
PaLM compared with other model families
PaLM is best understood in relation to the models around it, both inside Google and across the industry. The comparison below summarizes the main differences.
| Model family | Developer | Main distinction from PaLM |
|---|---|---|
| Gemini | Google DeepMind | Natively multimodal successor that replaced PaLM 2 |
| T5 | Encoder-decoder design focused on transfer learning | |
| BERT | Encoder-only model for understanding, not generation | |
| GPT family | OpenAI | Competing decoder-only line with chat products |
| Llama | Meta | Openly released weights, unlike closed PaLM weights |
| Gemma | Small open-weight models derived from Gemini research |
A practical difference for users is access. PaLM weights were never released publicly; the models were reachable only through Google products and paid APIs. This contrasts with open-weight families, where organizations can host and fine-tune the model themselves.
Status, limitations and common misconceptions
PaLM is now a legacy model family. After Gemini launched at the end of 2023, Google moved Bard and its developer platforms to Gemini, and the PaLM API was deprecated in favor of the Gemini API during 2024. Existing applications built on PaLM endpoints had to migrate.
- PaLM is not the same as Pathways. Pathways is the distributed training and serving system; PaLM is a model trained with it.
- PaLM 2 was not simply a bigger PaLM. It was smaller in its largest form and gained quality through better data mixtures and compute-optimal scaling.
- Benchmark scores from 2022 do not reflect current standards. Modern models surpass PaLM on most reasoning and coding evaluations.
- Known weaknesses included hallucinated facts, sensitivity to prompt wording and biases inherited from web-scale training data.
The table below summarizes how teams that still encounter PaLM references should interpret them today.
| Situation | Recommended interpretation | Reason |
|---|---|---|
| Older code calls text-bison | Plan migration to a Gemini model | PaLM endpoints are deprecated |
| Research paper cites PaLM | Treat results as a 2022 to 2023 baseline | Later models changed the frontier |
| Product mentions Med-PaLM | Read it as a research system | Clinical use requires separate validation |
| Discussion of scaling laws | Pair PaLM with compute-optimal studies | PaLM was undertrained for its size |
A non-technical view
PaLM was one of Google’s big text-writing AI systems before the Gemini models arrived. Like other chat-style AI, it learned by reading an enormous amount of text and practicing how to guess the next word.
What made it notable was its size and what that size revealed. When it was asked to show its working step by step, much like a student writing out a math problem, it solved harder questions far more often. Its successor, PaLM 2, ran the early Bard chatbot. Today PaLM has been retired and replaced by Gemini, so it is mostly a milestone in the story of modern AI rather than a tool people choose.
A concrete example
In mid-2023, a mid-sized insurance company built an internal assistant that summarized claim notes for adjusters. The developers used the text-bison model through the PaLM API, sending each claim file with a prompt asking for a five-point summary and a list of missing documents. Adjusters reported that the summaries saved several minutes per file, although staff still checked dates and amounts because the model occasionally invented details not present in the notes.
When Google announced the deprecation of PaLM endpoints, the team ran the same set of several hundred archived claims through a Gemini model and compared outputs against adjuster-approved summaries. The new model produced fewer factual errors and handled longer files, and the migration required changes mainly to the API client and prompt formatting. The outcome illustrates PaLM’s typical role today: a first-generation foundation for enterprise prototypes that were later moved to its successor.