Property-based testing

On this page 7

Property-based testing is a software testing technique in which the developer states general rules that must always hold for a piece of code, and a framework checks those rules against hundreds or thousands of automatically generated inputs. In testing, it complements example-based unit tests: instead of asserting that one specific input produces one specific output, it asserts a property, such as “sorting a list never changes its length”, across a wide input space.

When a generated input violates a property, the framework shrinks it to the smallest input that still fails and reports that minimal counterexample. The approach originated with the QuickCheck library for Haskell and is now available in most mainstream programming languages.

How property-based testing finds failures

A property-based test runs a loop of generate, check and shrink. The framework uses generators (also called arbitraries or strategies) to build random values of the required type, calls the code under test, evaluates the property and repeats, typically 100 runs by default in many frameworks.

If any run fails, shrinking begins: the framework tries smaller or simpler variants of the failing input, such as a shorter list, a smaller integer or an empty string, and keeps any variant that still fails. The result is a counterexample that a developer can understand at a glance, rather than a random 300-element list. Most frameworks also print a random seed so that the exact failing run can be reproduced.

ComponentRoleExample
GeneratorProduces random inputs of a given shapeLists of integers up to 100 items
PropertyBoolean rule that must hold for every inputReversing twice returns the original
RunnerExecutes many trials and collects failuresRuns 100 to 1,000 cases
ShrinkerReduces a failing input to a minimal oneTurns a long list into a two-item list
SeedMakes a random run reproducibleReplays the failing case in CI
Components of a property-based test

Common patterns for writing properties

The hardest part of property-based testing is finding good properties, because a property must be true for every valid input without simply reimplementing the code. Experienced practitioners rely on a set of recurring property patterns that fit most functions.

PatternRule it expressesTypical target
Round tripDecoding an encoded value returns the originalSerializers, parsers, compression
InvariantSome quantity never changesSorting keeps length and elements
IdempotenceApplying an operation twice equals onceNormalization, deduplication
Test oracleResult matches a simpler trusted versionOptimized algorithms
CommutativityOrder of independent steps does not matterSet operations, merges
Metamorphic relationChanging input predictably changes outputSearch ranking, numeric code
Recurring property patterns

A round-trip property is often the best starting point because it is easy to state and catches many encoding bugs. The oracle pattern is especially powerful when a slow but obviously correct implementation exists, for example comparing a hand-optimized search against a linear scan. Metamorphic relations help when the exact correct output is unknown but the relationship between two outputs is known.

Property-based testing compared with other test styles

Property-based testing differs from neighboring techniques mainly in who chooses the inputs and what the assertion describes. In a classic unit test, the developer chooses a few inputs and checks exact outputs; in a property test, the framework chooses the inputs and the assertion describes a general rule.

TechniqueInput sourceAssertionMain strength
Example-based unit testChosen by the developerExact expected outputClear documentation of cases
Property-based testGenerated by the frameworkGeneral rule over all inputsFinds unanticipated edge cases
FuzzingMutated or random bytes, often coverage-guidedNo crash or sanitizer errorFinds memory and parser faults
Mutation testingExisting tests on altered codeTests detect injected faultsMeasures test suite quality
Property-based testing and related techniques

The boundary between property-based testing and fuzzing is blurring, because modern frameworks can use coverage guidance and long-running campaigns. The practical difference remains that property tests run as part of the regular test suite, use typed generators and check semantic rules, while fuzzers usually run for hours and look for crashes. The two styles work best together with example-based tests, which document the known important cases explicitly.

Frameworks and stateful testing

Mature property-based testing frameworks exist for most languages, and they share the same core ideas while differing in shrinking strategy and integration. Choosing one usually follows the project language and test runner.

LanguageFrameworkNotable feature
HaskellQuickCheckOriginal design of the technique
PythonHypothesisIntegrated shrinking and example database
JavaScript and TypeScriptfast-checkWorks with common test runners
JavajqwikJUnit 5 engine integration
Erlang and ElixirPropEr and StreamDataStrong stateful testing support
Widely used frameworks by language

Beyond pure functions, stateful property-based testing generates random sequences of commands against a system, such as a cache, a queue or a database API, and compares the real system with a simple model after every step. This technique has uncovered subtle concurrency and ordering bugs in production databases and storage systems, because a random sequence of operations explores interleavings that hand-written scenarios rarely cover.

Limitations and common mistakes

Property-based testing is not a proof of correctness: it samples the input space, so rare inputs may never be generated. Its value depends on generator quality, because a generator that rarely produces empty lists, boundary integers or unusual Unicode will miss exactly the cases that break code.

  • Restating the implementation as the property, so that the test repeats the same bug.
  • Filtering too aggressively with preconditions, which discards most generated inputs and slows runs.
  • Ignoring flaky failures instead of recording the seed and adding a regression example.
  • Testing slow code with too many runs, which makes the suite impractical in continuous integration.
  • Weak properties such as “does not throw”, which pass even when results are wrong.

A good practice is to pin every discovered counterexample as an ordinary unit test, so that regression testing keeps the bug from returning even if random generation never finds it again.

The idea in brief

Normal tests check a few hand-picked cases, like trying a calculator with 2 + 2. Property-based testing instead states a rule that should always be true, for example that adding zero never changes a number, and lets the computer try that rule on thousands of random inputs.

When the computer finds an input that breaks the rule, it keeps simplifying it until it finds the smallest example that still fails. The developer then sees a short, clear case that explains the bug.

A practical scenario

A developer maintains a library that converts configuration objects to a text format and back. The existing suite has 40 example-based unit tests, all passing. She adds one round-trip property: for any generated configuration object, parsing the serialized text must return an object equal to the original.

On the first run, after 63 generated cases, the framework reports a failure and shrinks it to a configuration with a single key whose value is a string containing only a newline character. The serializer had not escaped newlines, so the parser split the value into two lines. The developer fixes the escaping, adds the shrunk input as a permanent unit test and keeps the property running with 500 cases per build. Over the next months, the same property catches two more escaping bugs introduced by new features, each reported as a minimal, readable counterexample.