Code LLM

Also known as: code model

On this page 8

A code LLM is a large language model specialized for source code: it is trained or fine-tuned on programs, documentation and developer conversations so that it can generate, complete, explain, translate and repair code. Among language models in artificial intelligence, a code LLM (often simply called a code model) is distinguished less by its architecture than by its training data, tokenizer choices and evaluation targets, which are tuned to the strict syntax and executable semantics of programming languages.

Code LLMs power inline completion in editors, chat assistants that answer programming questions, automated review tools and coding agents that edit repositories and run tests. Their output can be checked by a compiler or a test suite, which makes code one of the few domains where model quality can be measured objectively at scale.

How a code LLM learns to write programs

A code LLM learns programs the same way a general LLM learns prose: through next-token prediction on a very large corpus, usually with a decoder-only transformer. The difference lies in the corpus, which is dominated by permissively licensed public repositories, issue threads, commit messages, technical documentation and question-and-answer pages, typically filtered for duplicates, generated files and low-quality code.

Several training objectives are specific to code. Fill-in-the-middle (FIM) training rearranges a file into prefix, suffix and middle segments so the model learns to insert code at the cursor while seeing the text after it. Repository-level training concatenates related files in dependency order, teaching the model how a function defined in one file is called in another. Later stages add instruction tuning on paired requests and solutions and, increasingly, reinforcement learning from execution feedback, where the reward depends on whether generated code compiles and passes unit tests.

The following table summarizes the main training stages and what each contributes to coding ability.

StageData usedCapability gained
PretrainingMixed code and natural language, often trillions of tokensSyntax, idioms, library knowledge
Fill-in-the-middleFiles split into prefix, middle and suffixCursor-aware completion in editors
Long-context trainingWhole repositories or concatenated filesCross-file references and project structure
Instruction tuningRequest and solution pairs, chat transcriptsFollowing natural-language coding requests
Execution-based reinforcement learningProblems with unit tests or verifiersHigher functional correctness, self-correction
Typical training stages of a code LLM

Tokenization and context in code models

Tokenization matters more for code than for prose because whitespace and symbols carry meaning. Code-oriented tokenizers, usually variants of byte-pair encoding, add tokens for runs of spaces and tabs, common operators and frequent identifiers, which reduces the number of tokens per file and keeps indentation in languages such as Python intact.

The context window determines how much of a project the model can see at once. Completion tools assemble a prompt from the current file, open tabs and retrieved snippets, while agents read files on demand. Even with windows of 100,000 tokens or more, a large monorepo does not fit, so practical systems combine the model with retrieval, code search and symbol indexes. Models also show the lost-in-the-middle effect, attending less reliably to material buried in the center of a long prompt.

Tasks a code LLM performs

Code LLMs serve a range of tasks that differ in latency budget, context size and how errors are caught. The table below compares the most common ones.

TaskTypical latency needHow errors are caught
Inline completionUnder about 300 msDeveloper reads and accepts or rejects
Function generationA few secondsUnit tests, type checker, review
Code explanationA few secondsDeveloper judgment only
Language translationSeconds to minutesBehavioral tests on both versions
Bug fixing and refactoringSeconds to minutesRegression test suite, static analysis
Agentic repository tasksMinutes to hoursContinuous integration, human pull request review
Common code LLM tasks and their requirements

Latency drives model choice. Inline completion favors small, fast models served close to the editor, often with a few billion parameters, while multi-step agentic work favors larger reasoning models that spend more test-time compute planning, running tools and revising their own output.

How code LLMs compare with general LLMs

A code LLM differs from a general-purpose LLM mainly in emphasis rather than kind. Modern frontier general models are trained on large amounts of code and often match or exceed dedicated code models, while specialized code models remain attractive when cost, latency, privacy or self-hosting matter.

AspectSpecialized code LLMGeneral-purpose LLM
Training mixMostly code and technical textBroad text with substantial code
Completion supportNative fill-in-the-middleOften chat-only interface
Model sizeFrequently small to mid-sizedOften very large
DeploymentCommon as open-weight models on local hardwareMostly hosted APIs
Non-coding knowledgeLimitedBroad domain and world knowledge
Specialized code model versus general-purpose LLM

The practical consequence is that teams often run two tiers: a small local model for keystroke-level completion and a large hosted model for design questions, complex refactoring and agentic work.

Measuring code LLM quality

Code LLMs are evaluated chiefly by functional correctness: generated code is executed against hidden tests. The standard metric is pass@k, the probability that at least one of k sampled solutions passes all tests; pass@1 reflects single-attempt reliability, while pass@10 reflects how useful the model is when several candidates can be filtered.

Benchmarks progressed from short function-level problems, such as HumanEval and MBPP, to repository-level benchmarks such as SWE-bench, in which the model must resolve real issues in open-source projects. Older benchmarks are now largely saturated and contaminated, because their solutions have leaked into training corpora, so results on them overstate real-world ability.

ApproachWhat it measuresMain weakness
Function-level benchmarksCorrectness on short, self-contained problemsSaturation and data contamination
Repository-level benchmarksResolving real issues across many filesExpensive to run, flaky test environments
Acceptance rate in editorsShare of suggestions developers keepRewards plausible rather than correct code
Internal task suitesPerformance on the team’s own codebaseSmall samples, costly to maintain
Evaluation approaches for code models

Limitations and risks of code LLMs

The central risk is that a code LLM produces plausible code that is subtly wrong. Because output is fluent and idiomatic, defects in edge cases, concurrency and error handling are easy to miss during review.

  • Hallucinated APIs and packages: the model invents functions or dependency names; attackers can register invented package names, a risk known as package hallucination squatting.
  • Insecure patterns: training data contains vulnerable code, so models can reproduce SQL injection, weak cryptography or missing input validation.
  • Outdated knowledge: the knowledge cutoff means newer library versions and deprecations may be unknown to the model.
  • Licensing and provenance: generated snippets can resemble licensed code, which concerns organizations with strict compliance rules.
  • Skill and review erosion: heavy reliance can reduce careful reading of generated changes, so mandatory review and automated tests remain essential.

A non-technical view

A code LLM is a writing assistant for software. It has read an enormous amount of existing code, so when a programmer starts typing or describes what is needed, it suggests the next lines or a whole working piece. It predicts likely code, not guaranteed correct code.

Think of it as a fast, well-read junior colleague. It saves a great deal of typing and can explain unfamiliar code, but its work must still be checked, ideally by running tests, before anyone relies on it.

A concrete example

A small backend team maintains a payment reconciliation service of about 80,000 lines of Python. They run a small open-weight code model on a local server for inline completion, so that proprietary code never leaves the company network, and use a larger hosted model for refactoring tasks with strict data-handling terms.

When the team migrates from one database library version to the next, an engineer asks the larger model to update 40 query functions. The model rewrites them in minutes, but the test suite fails on three functions where the model used a parameter that no longer exists in the new version. The engineer feeds the error output back, the model corrects the calls, and all tests pass.

The outcome is a migration finished in one day instead of roughly a week, with every change reviewed in a pull request. The team keeps the rule that no generated code merges without passing tests.