Restoration test

On this page 7

A restoration test is a planned exercise in which a backup is actually restored to a working system and the result is verified for completeness, consistency and usability. In the backup and recovery practice of servers, it is the only reliable proof that a backup can fulfill its purpose, because a backup job that reports success says nothing about whether the data can be brought back.

A restoration test also measures how long recovery takes and how much data would be lost. It therefore produces the achieved recovery time and recovery point, which can be compared with the agreed RTO and RPO targets of a service.

What a restoration test verifies

A restoration test answers several questions at once. It checks that the backup media are readable and complete, that the backup software and its catalog still work, that encryption keys and credentials are available, and that the restored system starts and serves correct data.

The test is meaningful only when the restored data is checked against an independent reference. Typical checks compare row counts in key tables, file counts and checksums, the date of the newest record and the result of application-level operations such as logging in or opening a recent document.

Many backup failures stay invisible until restoration is attempted. Silent media corruption, missing files excluded by a filter, database dumps taken while writes were in progress, and backups that depend on a retired server or an expired license are all discovered only by restoring and inspecting the result.

Levels and types of restoration tests

Restoration tests range from a quick file check to a full disaster recovery exercise. Each level covers more failure modes but costs more time and infrastructure, so mature operations combine several levels at different frequencies.

Test levelWhat is restoredWhat it provesTypical frequency
Integrity verificationNothing, only checksums are readBackup files are readable and unchangedAfter every backup
File-level restoreSelected files or foldersCatalog and media workWeekly to monthly
Database restoreA full database to a test instanceData is consistent and usableMonthly
Full server restoreOperating system, applications and dataBare-metal or VM recovery worksQuarterly
Disaster recovery exerciseAn entire service in another sitePeople, process and RTO holdOnce or twice a year
Common levels of restoration testing

Integrity verification alone is not a restoration test. It confirms that stored bytes match what was written, yet it cannot detect a backup that captured the wrong data or an inconsistent database state.

How backup types affect the test

The structure of a backup chain determines how many pieces a restore needs, and the test should exercise exactly that chain. A restore that uses only the latest full backup does not prove that the incremental or differential steps work.

Backup typePieces needed to restoreMain test riskWhat to test specifically
Full backupOne backup setLarge size and long restore timeRestore duration against RTO
Incremental backupLast full plus every later incrementalOne broken link breaks the chainRestore from the end of a long chain
Differential backupLast full plus latest differentialGrowing differential sizeRestore just before the next full
SnapshotSnapshot plus its base storageSnapshot lives on the same systemRestore from an off-site copy
Restoration requirements by backup type

Snapshots deserve special attention, because a snapshot on the same storage array protects against logical mistakes but not against loss of the array itself. A test that restores only from local snapshots does not prove recoverability after a hardware or site failure.

Running a restoration test safely

A restoration test should always run in an isolated environment, never over the production system. Restored servers can send email, run scheduled jobs or connect to payment gateways, so the test network must block outbound side effects and use different hostnames.

  1. Select a backup by age and type in advance, preferably at random rather than the most recent one.
  2. Restore it to isolated infrastructure using only the documented procedure and stored credentials.
  3. Record the elapsed time from start to a usable system.
  4. Run predefined verification checks on data, application functions and timestamps.
  5. Document the result, any deviations and the corrective actions, then remove the test system.

Restored copies of production data are as sensitive as the originals. The test environment needs the same access controls and data protection as production, and the restored data should be deleted once the test is complete.

Typical findings and pitfalls

Most failed restoration tests reveal a gap in process rather than in technology. The table lists findings that appear repeatedly in practice, with their usual cause and the remedy.

FindingUsual causeRemedy
Restore fails to startMissing encryption key or credentialsStore keys separately with tested access
Database will not openBackup taken without a consistent stateUse native dump tools or quiesced snapshots
Data incompleteExclusion rules or new paths not coveredReview backup scope after every change
Restore far slower than expectedThroughput of media or network unknownMeasure and adjust RTO or infrastructure
Nobody knows the stepsProcedure undocumented or outdatedMaintain a runbook and rotate testers
Frequent problems found by restoration tests

A test that always uses the same small file or the newest backup creates false confidence. Varying the backup, the operator and the target environment exposes the weaknesses that a real incident would encounter.

A non-technical view

Owning a spare tire is only useful if it holds air and the jack is in the car. A restoration test is like actually changing the tire on a quiet afternoon instead of discovering a problem on a dark highway.

With backups, the test means taking a stored copy and rebuilding the system from it, then checking that everything is there and works. A backup that has never been restored is only a hope, not a guarantee.

A concrete example

A hosting company backs up a customer database server every night with a full backup on Sundays and incrementals on the other days. The backup console has shown green status for months. As part of a quarterly restoration test, an engineer restores the Saturday state into an isolated virtual network, which requires the Sunday full backup and six incrementals.

The restore fails at the fourth incremental, which turns out to be unreadable because of a storage fault that the backup job never reported. Further checks show that the database dump also omitted a newly created schema because of an old exclusion rule.

The team fixes the storage, enables checksum verification after each job and updates the backup scope. A repeated test succeeds in 2 hours and 10 minutes, within the agreed recovery time of four hours, and the procedure is added to the monthly schedule.