AI test generation

Also known as: AI-generated tests

On this page 8

AI test generation is the use of machine learning models, most often large language models, to write software tests automatically from source code, requirements, user stories or recorded application behavior. Among practical AI applications in software engineering, it sits next to AI code generation and AI code review: instead of producing the product itself, the model produces the checks that verify the product.

The output of AI test generation, sometimes called AI-generated tests, ranges from small unit tests for a single function to full end-to-end browser scripts. The value lies in faster coverage of routine cases, while the main risk is that a model writes tests that pass without proving anything. Skilled teams therefore treat generated tests as drafts that a human engineer reviews, runs and keeps only when they detect real defects.

How AI test generation produces tests

AI test generation works by giving a model the code under test plus context and asking it to propose inputs, expected outputs and assertions. The context usually includes the function signature, its documentation, neighboring tests that show the project’s conventions, and the testing framework in use, such as pytest, JUnit or Jest.

Mature tools run a generate, execute and refine loop. The model drafts a test, the tool compiles and runs it, and any compilation error, failing assertion or low coverage result is fed back to the model for another attempt. Tests that still fail after several rounds are discarded or flagged, because a failing generated test may reveal either a real bug or a wrong expectation.

The following table lists the main inputs that a generator can draw on and what each one contributes to the resulting tests.

InputWhat it contributesTypical limitation
Source codeBranches, edge values, exception pathsEncodes current behavior, including bugs
Requirements or user storiesIntended behavior and acceptance criteriaOften vague or incomplete
Existing test suiteNaming style, fixtures, mocking patternsCan copy weak assertions
Runtime traces or recordingsRealistic inputs and call sequencesMay contain personal or sensitive data
API schemasValid and invalid request shapesSilent on business rules
Inputs used by AI test generators

Types of tests that models can generate

AI test generation covers several levels of the testing pyramid, and its reliability falls as the scope of the test grows. Unit tests are the strongest case because the model sees the whole unit at once, while end-to-end tests depend on a running environment the model cannot fully observe.

Test levelTypical generated artifactFit for generation
Unit testsAssertions on one function or classHigh, with execution feedback
Integration testsCalls across modules, databases or servicesMedium, needs fixtures and mocks
API testsRequests built from an OpenAPI schemaHigh for structure, low for rules
End-to-end UI testsBrowser scripts with selectors and stepsMedium, prone to brittle selectors
Property-based testsInvariants checked over random inputsMedium, needs careful human framing
Test levels and how well generation fits them

Some tools also generate test data, such as boundary values, malformed strings and synthetic records, and others translate manual test cases written in natural language into executable scripts. Both uses save typing, yet both still require someone to confirm that the data is representative and that the scripted steps match the intended scenario.

Comparison with classical automated test generation

Automated test generation existed long before language models, through search-based tools, symbolic execution and random fuzzing. These classical techniques explore program paths systematically and can reach coverage targets reliably, but they tend to produce tests that are hard for people to read and that lack meaningful names or assertions.

TechniqueMain strengthMain weakness
LLM-based generationReadable tests that follow project styleMay assert wrong expectations confidently
Search-based generationSystematic branch coverageUnnatural inputs and opaque tests
Symbolic executionPrecise path conditionsScales poorly on large programs
FuzzingFinds crashes and security flawsWeak at checking functional correctness
Manual test writingGrounded in domain knowledgeSlow and unevenly applied
AI test generation compared with related techniques

In practice the techniques are complementary rather than competing. A language model can name and structure tests well, while fuzzing or search-based tools can supply unusual inputs that a model would not think to try. Hybrid pipelines that combine both approaches tend to reach higher coverage with readable output.

The test oracle problem and test quality

The central difficulty of AI test generation is the test oracle problem: deciding what the correct output should be. When a model reads only the implementation, it tends to describe what the code does, not what it should do. A function with an off-by-one error will often receive a test that confirms the wrong value, which turns a bug into a protected behavior.

For this reason, line coverage is a poor quality signal for generated tests. A more demanding measure is mutation testing, in which a tool introduces small deliberate faults into the code and checks whether the tests catch them. A generated suite with high coverage but a low mutation score contains assertions that are too weak to detect real defects.

  • Specification-driven prompts reduce the oracle problem by describing intended behavior before the model sees the code.
  • Assertion review focuses human attention on expected values rather than on setup code.
  • Flakiness checks rerun each new test several times to reject results that depend on timing or ordering.

Adopting AI test generation in a development workflow

Teams usually introduce AI test generation in the editor or the pull request, where a developer requests tests for new code and reviews them before committing. Fully autonomous generation in the continuous integration pipeline is less common, because unreviewed tests can inflate the suite and slow every build.

The decision table below summarizes where generation helps most and where it needs extra safeguards.

SituationRecommended approachReason
Legacy code without testsGenerate characterization tests firstLocks current behavior before refactoring
New feature with clear requirementsPrompt from the specificationAvoids copying implementation bugs
Security-sensitive logicCombine with fuzzing and expert reviewModels miss adversarial inputs
Large UI regression suiteGenerate drafts, stabilize selectors manuallyGenerated selectors break easily
Proprietary or regulated codeUse self-hosted or approved modelsSource code leaves the organization otherwise
When to use AI test generation

Limitations and common mistakes

The most frequent mistake is accepting tests because they pass. A passing generated test proves only that it agrees with the current code. Other recurring problems are excessive mocking, which tests the mocks instead of the logic, and duplicated tests that add maintenance cost without new protection.

There are also governance concerns. Sending source code to an external model may breach confidentiality rules, and recorded production traffic used as input can contain personal data. Generated tests also carry a long-term cost: each one must be maintained when the code changes, so a smaller suite of meaningful, reviewed tests is usually worth more than a large suite of shallow ones.

The simple explanation

Software needs checks that confirm it still works after every change. Writing those checks by hand is slow, so AI test generation lets a computer model suggest the checks automatically. It reads the program and proposes small experiments: give this input, expect that answer.

The catch is that the model often guesses the expected answer by looking at the program itself. If the program is wrong, the check can be wrong too. It is a bit like asking a student to grade their own homework. That is why a person still has to review every generated test and keep only the ones that truly prove something.

How it is used in practice

A payments team inherits a ten-year-old billing module of about 8,000 lines with almost no tests, and it must refactor the module to support a new tax rule. Before touching the code, engineers use an AI test generator to draft characterization tests for the invoice calculation functions, running each draft through a generate, execute and refine loop.

The tool proposes around 300 tests, and reviewers keep roughly 180 after removing duplicates and tests that only checked mocks. A mutation testing run shows that the kept tests catch most injected faults in the rounding and discount logic. During review, one generated test asserts a negative invoice total for a fully refunded order, which turns out to be a long-standing bug rather than intended behavior. The team fixes the bug, corrects the test and completes the refactoring with a safety net it did not have before.