Backups Are Not Recovery: How to Run a Restore Test That Works

TL;DR

A successful backup job proves that a backup operation completed—not that your application can be recovered. Only a controlled restore test can verify whether databases, objects, configuration, secrets, and business workflows can be brought back after data loss or an incident. This guide provides a repeatable, production-safe restore drill, practical success criteria, and a checklist.

A green backup job is not a recovery guarantee

Your nightly backup reports success. Yet recovery may still fail: storage credentials have expired, a database dump is inconsistent, an encryption key is unavailable, or the application starts but cannot process customer orders.

The real question is not “Do we have a backup?” It is “Can we restore our critical business service to a usable state within the agreed time and data-loss limits?”

1. Define RPO and RTO before testing

Two objectives define success:

• Recovery Point Objective (RPO): The maximum tolerable amount of lost data, expressed as time. A 15-minute RPO requires backup or replication mechanisms capable of meeting that recovery-point target.
• Recovery Time Objective (RTO): The maximum acceptable time to resume service. A two-hour RTO leaves little room for verification if importing the database already takes 110 minutes.

These are business requirements, not promises automatically made by a backup product. Define targets per service before running the drill.

2. Recover the entire application, not just its database

For a typical web application using PostgreSQL, Kubernetes, and object storage, you need to restore multiple layers:

1. Data: Database backups, uploaded objects, and other persistent application state.
2. Configuration: Kubernetes manifests, Helm values, infrastructure definitions, DNS, and routing settings.
3. Identities and keys: Secrets, certificates, encryption keys, and recovery access, preferably stored with independent safeguards.
4. Dependencies: Compatible database versions, container images, external APIs, queues, and storage endpoints.
5. Business functionality: Login, reads and writes, critical workflows, and background jobs.

A Kubernetes cluster backup does not necessarily include a consistent backup of an external database. Likewise, restoring PostgreSQL will not bring back missing S3 objects.

3. A seven-step restore drill

Step 1: Define the scenario and success criteria

Choose a realistic incident, such as a production database becoming unavailable. Agree on RPO, RTO, roles, start time, data scope, and functional checks. For your first drill, restore into an isolated recovery environment, never directly over production.

Step 2: Verify backup availability independently

Confirm the required backup exists, is complete, and can be read using currently available permissions and keys. Record its timestamp, format, software version, and checksums where appropriate. A successful upload status is not sufficient.

Step 3: Recreate the recovery environment

Provision compute, network policies, storage, and supporting services. Infrastructure as Code and version-controlled configuration help minimize manual dependencies. Block or reroute outbound emails, payments, and webhooks so the test cannot accidentally trigger real-world actions.

Step 4: Restore data and keys

Restore the database and object storage into the isolated environment. Check database extensions, version compatibility, permissions, and consistency between records and stored documents. Handle secrets through controlled access mechanisms and never put their values in test logs.

Step 5: Start the application and validate workflows

A running pod or an HTTP 200 response is not proof of a successful application recovery. Have a test user log in, read and update a record, download a stored document, and process a queue task. Apply appropriate security controls to recovered personal data.

Step 6: Measure recovery time and data loss

Measure the interval from the simulated incident to a functionally usable application, not merely until a database import ends. Determine the latest recovered data point and compare both measurements with RTO and RPO.

Step 7: Document, improve, and repeat

Record gaps, manual workarounds, missing permissions, and bottlenecks. Assign owners and deadlines to corrective actions, update the runbook, and schedule the next drill. A single successful recovery is not permanent evidence.

4. Example: A SaaS application with PostgreSQL and S3

A SaaS company runs its API in Kubernetes, stores customer records in PostgreSQL, and keeps documents in S3-compatible object storage. Its team restores a defined backup into a separate environment.

Assumed objectives: RPO 30 minutes, RTO 4 hours.

In this hypothetical drill, recovery meets both targets: 140 versus 240 minutes for RTO, and 20 versus 30 minutes for RPO. The operational value lies in the report: What was missing? Which steps required manual intervention? Can another team reproduce the result?

5. Five problems a restore test can expose

Missing secrets: Data is available, but recovery keys or credentials are not. Test key management and recovery access separately.

Incompatible versions: The recovered database or application cannot run with the intended versions. Document compatible versions and dependencies.

Incomplete application state: PostgreSQL starts, but objects are missing. Verify the complete service, not one component.

Unsafe test connections: The recovered application sends real emails, payments, or callbacks. Isolate outbound integrations and use test endpoints.

Underreported downtime: Approvals, environment provisioning, failover, DNS changes, and validation are excluded from the stopwatch. Measure the full restoration to intended user access where the scenario requires it.

6. Restore-test checklist

• [ ] Critical applications and data stores inventoried
• [ ] RPO and RTO agreed with business owners
• [ ] Scenario, responsibilities, and communications documented
• [ ] Recovery environment isolated from production
• [ ] Backups, keys, and recovery permissions accessible
• [ ] Database and object storage included
• [ ] External integrations blocked or redirected safely
• [ ] Functional tests and consistency checks prepared
• [ ] Actual recovery time and data gap measured
• [ ] Findings and corrective actions documented
• [ ] Next exercise scheduled

FAQ

How often should you run a restore test?

There is no universal frequency. Base it on business criticality, change rate, and regulatory obligations. Critical systems benefit from recurring drills and additional tests after major architecture or backup changes.

Is restoring a database enough?

No. Without configuration, credentials, dependent data, and working application components, the business service can remain unavailable.

Can restore tests be automated?

Provisioning, imports, integrity checks, and smoke tests can often be automated. Business-process validation, communications, and incident decisions may still need human review.

How do backup, restore tests, and disaster recovery exercises differ?

A backup creates a copy of data. A restore test verifies recoverability. A wider disaster recovery exercise also tests people, escalation, infrastructure failover, and the return to business operations.

Conclusion

A backup is only the beginning. A measured and documented restore test demonstrates whether an application—including its data and dependencies—can become usable again after an incident. Set RPO and RTO, test safely in isolation, validate real workflows, and keep improving the runbooks.

At ayedo, this is a core operational principle: a reliable platform is not just available today; its recoverability must also be testable.