Program-aided language model

Also known as: PAL · program-aided prompting

On this page 7

A program-aided language model (PAL) is a prompting technique in which a language model translates a problem into executable code, usually Python, and an external interpreter runs that code to produce the final answer. In prompting for artificial intelligence systems, PAL is the standard example of dividing labor between a model and a deterministic tool: the model handles understanding and decomposition, while the interpreter handles calculation.

The approach was introduced in 2022 research on arithmetic and symbolic reasoning and is also called program-aided prompting. Its central insight is that large models often reason correctly but calculate poorly, so moving the calculation into code removes a major source of error.

How a program-aided language model solves a problem

A PAL pipeline works by asking the model for a program instead of a prose answer. The prompt contains a few worked examples in which each word problem is followed by Python code whose variable names and comments mirror the reasoning steps. The model imitates this pattern for the new question, and the answer is whatever the code returns.

The model therefore never states the final number itself. It declares quantities, relations and operations, and the interpreter performs the arithmetic exactly. The table below traces the stages of a typical run.

StagePerformed byOutput
Few-shot promptDeveloperExamples pairing questions with code
Code generationLanguage modelPython function with commented steps
ExecutionPython interpreterComputed value or an exception
Answer extractionApplication codeReturned value formatted for the user
Optional retryApplication code and modelCorrected program after an error
Stages of a PAL run

Code is a useful intermediate language because it is unambiguous and checkable. A variable such as apples_left = apples_bought - apples_eaten exposes the model’s interpretation of the problem, which makes errors visible and debuggable in a way that free-form reasoning text is not.

Where PAL performs best

PAL is strongest on tasks where the difficulty lies in exact computation or state tracking rather than in world knowledge. Benchmarks used to evaluate it include grade-school math word problems, a variant with large, awkward numbers, date arithmetic, object counting and symbolic manipulation.

Gains are largest when numbers are big or when many intermediate values must be tracked, because chain-of-thought text accumulates small arithmetic slips at each step. On problems that need commonsense judgment or open-ended explanation, PAL offers little or no advantage.

Task typeBenefit from PALReason
Multi-step arithmeticHighInterpreter removes calculation errors
Date and time arithmeticHighLibraries handle calendars correctly
Counting and state trackingHighVariables hold exact running totals
Symbolic and logic puzzlesModerateDepends on clean formalization
Commonsense questionsLowNo computation to offload
Open-ended writingNoneOutput is not a computed value
Suitability of PAL by task type

PAL differs from chain-of-thought prompting because reasoning is written as code and computation is delegated. In chain-of-thought, the model writes and computes everything in natural language. Program of thoughts is a closely related technique published at nearly the same time, with a similar idea and a stronger emphasis on financial and numerical datasets.

TechniqueReasoning formatWho computes the answer
Chain-of-thoughtNatural-language stepsThe language model
Program-aided language modelCode with commentsAn external interpreter
Program of thoughtsCode, often with less proseAn external interpreter
Self-consistencySeveral sampled reasoning pathsMajority vote over model outputs
Tool-calling agentsStructured function callsPredefined tools or APIs
PAL and neighboring reasoning techniques

These techniques combine well. Sampling several programs and taking a majority vote over executed results applies self-consistency to PAL and usually improves accuracy further. Modern assistants with a built-in code interpreter apply the PAL principle automatically whenever they decide to run code.

Designing an effective PAL prompt

A good PAL prompt makes the model write readable, self-contained programs. Examples should use descriptive variable names, one comment per reasoning step and a single function that returns the answer, because the model copies the structure of the demonstrations closely.

  • Provide three to eight worked examples that cover the variety of expected problems.
  • Ask for a function with a fixed name, such as solution(), to simplify answer extraction.
  • Restrict the program to the standard library or an explicit allowlist of modules.
  • Return the raw error message to the model for one or two repair attempts, not unlimited retries.

Instruction-tuned models can often produce PAL-style code with zero examples if the instruction is explicit. Few-shot examples remain valuable, however, when a domain has specific conventions, such as rounding rules, currency handling or units of measurement.

Limitations, risks and failure modes

PAL guarantees correct calculation, not correct understanding. If the model misreads the problem, the interpreter will compute the wrong quantity precisely. Executing model-generated code also introduces a serious security concern, so production systems must run it in an isolated environment.

SymptomLikely causeRemedy
Exact but wrong answerMisinterpreted problemBetter examples or self-consistency voting
Runtime exceptionInvalid or incomplete codeFeed the error back for a repair attempt
Program never endsUnbounded loopEnforce timeouts and resource limits
File or network accessUnsafe generated codeSandbox with no network or file system
Answer in wrong unitsMissing domain conventionsState units and rounding in examples
Common PAL failures and remedies

PAL also adds latency and infrastructure: an interpreter, a sandbox and error handling. For simple questions, a direct answer from a strong model may be cheaper and fast enough, so the technique is best reserved for computation-heavy workloads.

In everyday language

Language models understand questions well. They can still make careless mistakes with numbers. A program-aided language model acts like a sensible student. It writes down the calculation and lets a calculator do the math.

The model does not answer directly. It writes a short piece of code that lists the steps, and a computer runs that code. The main takeaway is simple: the model plans and the computer calculates. The arithmetic is exact, but the plan itself still has to be right.

Seen in the real world

A logistics company builds an internal assistant that answers dispatcher questions such as how many pallets fit on a truck after mixed loads, weight limits and partial returns. Early tests with plain chain-of-thought prompting show that roughly one answer in eight contains an arithmetic slip, usually in multi-step unit conversions.

The team switches to a PAL design. The model receives six examples that convert each question into a Python function with named variables for weights, dimensions and limits, and the generated code runs in a sandbox with a two-second timeout and no network access. Runtime errors are returned to the model for a single repair attempt.

In follow-up testing, arithmetic errors nearly disappear, and the remaining mistakes come from misread questions, which reviewers can spot quickly by reading the variable names in the generated code.