The art of recovering well

A system that never fails is an attractive story. A system that exposes a failure clearly and recovers without compounding it is a practical engineering objective.

Solace’s original essay used the body and its immune system as metaphors for resilience. The useful technical thread is straightforward: observe the system, limit the damage, restore service, and retain what the incident taught.

Observe a user’s operation

Begin with a real action: loading a saved design, generating a quote, or completing an order. Identify the components it depends on and what successful completion looks like.

OpenTelemetry’s signals include traces, metrics, and logs. These provide different views of a system’s behavior. A trace can follow a request across components; a metric can show a trend; a log can retain a specific event.

Choose information that helps distinguish causes. More telemetry is not automatically more understanding.

Limit repeated damage

An unavailable dependency should not trigger an uncontrolled storm of retries. Use bounded attempts, delays appropriate to the operation, and an explicit failure state.

Protect consequential actions from duplication. A retried payment, export, or production release must not be treated as a new request simply because the network response was lost.

Decide which parts of the product can continue in a reduced mode and tell the user what remains available.

Check behavior that matters

A process can be running while its essential work is failing. A health endpoint should have a defined purpose: checking process availability is different from checking readiness to serve a particular operation.

Use tests that can reject a plausible mistake. Boundary cases, permission checks, and recovery exercises are often more informative than a high line-coverage total.

Avoid a health check that restarts a service endlessly because a shared external dependency is down. Recovery mechanisms can become a source of damage when they are coupled carelessly.

Practice restoration

Keep a known version and the data needed to restore it. Test that the backup can be read and that the restored service performs a meaningful operation.

A restart does not repair corrupt data. A deployment rollback does not necessarily reverse a database migration. Document those differences before the incident.

For dangerous failure injection, use an isolated environment and a defined scope. “Chaos” is a method of learning, not permission for uncontrolled disruption.

Allocate a retry budget to the whole operation

Google’s SRE guidance on overload explains why clients need to control the additional load created by retries. Nested retries can multiply work: if three layers each make three total attempts, one user request can cause as many as 27 downstream attempts.

Give the user operation a deadline and a shared attempt budget. Retry only failures classified as transient, apply backoff with jitter, and stop when another attempt cannot finish within the remaining deadline. Preserve an operation ID so a late success can be reconciled without creating another order or export.

An isolated recovery exercise should simulate a dependency timeout at each layer. Count actual downstream requests, not just the retries visible in the outer service. Verify that an overloaded service receives less work as failures accumulate.

Preserve the result when the response is lost

A difficult failure happens after a durable change succeeds but before its response reaches the caller. The caller sees uncertainty, not proof that nothing happened.

Persist the operation’s identity and result with the state transition where possible. On retry, resolve the existing operation first. A user-facing status can say that the outcome is being checked while the system reconciles the durable record.

For agent workflows, pin the version of the evolving OpenTelemetry GenAI conventions used by instrumentation, and connect tool attempts to the enclosing operation. Keep the authoritative result in application storage rather than inferring it from a model’s final message.

Test recovery by interrupting the response after commit, restarting the worker, and retrying with the same identity. Success means one durable effect and a readable explanation of its outcome.

Let incidents improve the system

Write a concise account of the trigger, impact, contributing conditions, and correction. Add a regression check where it can catch the same class of failure.

Measure the time and effort needed to detect and recover. The aim is a smaller interruption and a clearer response.

Resilience is accumulated through these ordinary acts. Each one gives the next person a better chance to restore the service without having to rediscover the whole problem.