The Backup Exists. The Restore Still Needs Work.

The database restores successfully to 10:42.

The payment provider has transactions through 11:07. Object storage contains invoices created after the restore point. The queue still holds jobs referring to orders the restored database does not remember.

The backup worked. We have not yet recovered the application.

I think this is the useful distinction to make before an incident. Recovery is not one storage operation. It is deciding which facts remain authoritative after different parts of the system have moved to different points in time.

Availability and recovery solve different failures

A replica can keep the service available after one machine fails. It may also faithfully replicate the accidental deletion we are trying to undo.

A backup preserves another state. Point-in-time recovery can reconstruct a database using a base backup and subsequent transaction-log records, within the retained recovery window. Its value depends on those records being available and the restore procedure actually working.

Neither mechanism automatically restores object storage, credentials, queue state, or external services.

I would define the failure being covered: infrastructure loss, accidental deletion, corruption, an incompatible migration, or compromised access. A design that handles the first can still be inadequate for the others.

Put a business meaning behind RPO and RTO

Recovery point objective describes acceptable data loss. Recovery time objective describes acceptable time to restore the required service.

"Fifteen minutes" sounds specific. It still needs a meaning. Can fifteen minutes of submitted support forms disappear? Of inventory reservations? Of charged orders? Those are different promises.

Some operations need reconciliation from another authoritative system even when the database recovery point is acceptable. A payment provider may tell us about a charge the restored database lost. We need a stable reference linking it back to the business operation.

For recovery time, include more than restoring bytes: provisioning, access approval, decryption keys, application startup, verification, reconciliation, traffic switching, and safe resumption of workers.

I would measure the whole exercise. A ten-minute restore inside a two-hour recovery procedure is not a ten-minute RTO.

Restore beside production, not blindly over it

For a logical data mistake, restoring an isolated instance gives us room to inspect the recovered state and compare it with current production. We may be able to repair a specific set of rows instead of rewinding everything.

Keep the recovery environment unable to send real emails, charge payments, consume production queues, or execute live webhooks. A copied database can contain pending work that application startup eagerly resumes.

Use controlled configuration and network access, not merely a promise that nobody will click the dangerous button. Preserve the original state as evidence while deciding which changes to apply.

Sometimes a full cutover is necessary. Even then, the new instance should be verified before traffic moves, with a deliberate plan for connections, endpoints, and old application instances.

DNS changes have the overlap behavior discussed earlier. Applications may also cache connection settings or continue using old pools. Changing a database endpoint is a release step, not a universal disconnect command.

Match data with code and configuration

A database restored to yesterday may have yesterday's schema. Today's application may require a column that did not exist then.

Record the schema migration state alongside artifact and configuration revisions. Retain deployable artifacts, not just source code that could theoretically be rebuilt with today's package registry.

An expand/contract migration gives recovery more options, but it does not remove the need to choose a compatible application version. Running every pending migration automatically against the restored instance may recreate the destructive change we were trying to investigate.

Secrets matter too. A backup encrypted with an unavailable key is not useful. A restored application with revoked credentials cannot necessarily reach its dependencies. Back up and test the recovery of access paths without scattering secret values through documents or logs.

External effects do not rewind with the database

Consider an order that was paid at 10:55 and lost from a database restored to 10:42. Asking the customer to retry can charge them again unless the payment operation has a durable, stable identity.

Reconcile against the provider using idempotency keys, merchant references, and the provider's authoritative transaction state. Do not infer a refund or charge from the absence of a local row alone.

Email is another external effect. A replayed confirmation job may send a second message. An invoice number may already have been issued. A shipment may already be in transit.

The recovery procedure should describe those consequences and the systems that settle them. We cannot restore the world to match a snapshot. We can reconstruct our records so they agree with the world that continued operating.

This is where the identity choices from the API and queue articles become more than implementation conveniences.

Queues and caches need a recovery policy

A restored outbox may contain events already published before the incident. A current queue may contain messages for records absent from the recovered database.

Pause consumers while reconciling. Decide which messages to preserve, replay, quarantine, or discard. Idempotent consumers reduce the cost of duplicates, but they still need a coherent business state to operate on.

Do not delete the queue simply to make its depth look tidy. It may contain the only remaining record of accepted work.

Redis also needs role-specific handling. Rebuild a disposable product cache from the recovered source. For sessions, consider whether invalidating them is safer than restoring older authorization state. For locks or scheduler metadata, old values may be actively misleading.

"Restore every dependency" is not always the right instruction. Some state should be restored, some reconciled, and some deliberately recreated.

Verify meaning, not only row counts

A database that starts and contains the expected number of rows is encouraging. It does not prove the checkout workflow is usable.

Check relationships, critical constraints, recent business operations, object references, account permissions, and one controlled end-to-end flow. Compare representative records with external authoritative systems where necessary.

Keep enough telemetry to tell when corrupted or missing data first appeared. Otherwise choosing a restore point becomes a sequence of guesses, each followed by another expensive restore.

Data integrity checks and recovery logs should avoid publishing private customer content. The people conducting a recovery need controlled access, not an unrestricted data export pasted into the incident channel.

Test the access path that an incident will use

A recovery exercise run by an administrator with every permission can conceal missing privileges in the actual emergency role. Test who can start the restore, decrypt it, deploy compatible code, change routing, and release the isolation controls.

For compromise scenarios, consider independent backup protection: restricted deletion, retention controls, and appropriate account or region separation. If the same credential can destroy production and every backup, the copies share a rather important failure mode.

There is no universal arrangement that makes all of this free. The choices should reflect the actual data, recovery promise, and threat model.

Document who makes the cutover decision and who handles reconciliation. That is operational ownership, not an excuse to collect an impressive runbook nobody has tried.

A useful drill has an inconvenient detail

Try restoring with an old schema, an unavailable primary region, a rotated credential, or a job already delivered once. Choose one complication rather than pretending the exercise must reproduce every disaster at once.

Measure the result, record the steps that required undocumented knowledge, and repair those gaps. A failed rehearsal is useful when it changes the recovery path before the real incident.

A backup preserves data. A rehearsed recovery preserves the ability to use it safely.

That is where I would end this field guide. The database, cache, queue, proxy, and application are not independent checkboxes. Their boundaries become visible when a request fails, a deployment overlaps, or one component returns to an earlier state.

I would be satisfied with an ordinary system whose recovery procedure is unremarkable because it has been tested. That seems a more useful goal than another layer whose failure behavior we will learn later.

If you've attempted a restore rehearsal, what took longer than the database restore itself? I'd be interested in the dependency or decision the backup dashboard could not account for.