awesomechecklists
llms.txtGitHub ↗
/

31 — Reliability and Disaster Recovery Checklist#

Verify that critical user journeys remain usable under expected failures and can recover within agreed downtime and data-loss targets. Reliability, backup/restore, disaster recovery, and business continuity must be demonstrated with evidence, not assumed from redundancy alone.

Scope: Backend / API-specific. This file is one part of the 31_Reliability_and_DisasterRecovery.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • Critical service / user journeys and dependencies:
  • Availability target / SLO reference:
  • Recovery time objective (RTO) / recovery point objective (RPO):
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Copy this shared template into project-specific review records; keep all repository checkboxes unchecked.
  2. Mark [x] only after verifying a check through architecture review, actual backup restores, failure tests, resilience drills, observability, or an accountable risk decision. Document failed and genuinely inapplicable checks separately in project review notes or issues using their REL IDs.
  3. Establish business-appropriate availability, RTO, and RPO targets before deciding on redundancy or recovery architecture. Record deviations and accepted risks with named owners.
  4. Evaluate ## API Checks and ## Frontend Checks independently where applicable. For non-web or non-API systems, record the reason for N/A outside this template.
  5. Use safe staged fault injection and restoration exercises, and obtain authorization before any production disruption. Reassess after important architecture, vendor, data, risk, or traffic changes.

Availability objectives#

  • REL-011 Measure critical workflow success from a user perspective instead of relying only on instance or server uptime.
  • REL-013 Record expected regional, network, vendor, identity-provider, and data-store dependencies for each critical journey.

Resilience and redundancy#

  • REL-017 Document fault domains and single points of failure across compute, storage, network, identity, DNS, and external providers.
  • REL-018 Select redundant capacity and isolation boundaries according to actual service objectives rather than assuming multi-region deployment is always required.
  • REL-028 Define queue limits, dead-letter handling, retry limits, and recovery from accumulated backlog for asynchronous systems.
  • REL-029 Protect durable state from accidental loss during restarts, worker relocation, or autoscaling.
  • REL-030 Test database replication, replica promotion, split-brain prevention, and transaction behavior under failures when replication is used.
  • REL-032 Check maintenance/disruption controls such as rolling updates and Kubernetes PodDisruptionBudgets where relevant, recognizing limits on involuntary failures.
  • REL-035 Run controlled fault-injection or game-day scenarios covering representative instance, dependency, network, or storage failures.
  • REL-019 Verify replacement instances or replicas can receive traffic without relying on unavailable local or ephemeral state.
  • REL-020 Use health and readiness checks that reflect ability to serve important requests without restarting healthy instances during dependency outages.
  • REL-021 Test load-balancing and failover behavior when an instance, node, or availability zone becomes unhealthy.
  • REL-022 Check that shared components and control planes do not create hidden correlated-failure paths.
  • REL-023 Define timeouts and deadlines on cross-service calls so partial outages cannot exhaust caller resources indefinitely.
  • REL-024 Use bounded retries with backoff and jitter where appropriate; prevent amplification by stacked retry layers.
  • REL-025 Specify idempotency or deduplication for operations that may be repeated after timeouts or failover.
  • REL-026 Apply concurrency limits, backpressure, circuit breaking, or load shedding to preserve useful service during overload.
  • REL-027 Separate background work, bulk operations, and critical interactive requests where competition could jeopardize reliability.
  • REL-031 Determine how stale reads, eventual consistency, or replication lag affect customer-visible correctness.
  • REL-033 Verify critical third-party dependencies have documented degraded modes, alternative workflows, or accepted outages.
  • REL-034 Provide operational controls for disabling high-risk features, stopping runaway jobs, or reducing load during incidents.
  • REL-036 Measure how quickly normal service is restored after a simulated failure and compare against the relevant target.
  • REL-037 Retest resilience assumptions after major topology, dependency, data architecture, or traffic-pattern changes.

Backup and restoration#

  • REL-038 Inventory business-critical data, database schemas, configuration, object storage, secrets references, and other state required for recovery.
  • REL-043 Encrypt backup data in transit and at rest with controlled key management and recovery access.
  • REL-044 Separate backup administration and deletion privileges from ordinary application and deployment identities.
  • REL-048 Verify consistency across related data stores, queues, object storage, and transactional boundaries at the chosen recovery point.
  • REL-039 Document data that is backed up versus deliberately reproducible from authoritative sources, including rebuild procedures.
  • REL-040 Choose backup frequency and retention to satisfy RPO, legal retention, storage cost, and business risk.
  • REL-041 Automate backup creation and verify job completion rather than assuming a schedule means backups succeed.
  • REL-042 Alert an accountable owner when backups are stale, fail, become inaccessible, or exceed expected completion time.
  • REL-045 Protect recovery copies against accidental or malicious deletion using isolation, immutability, offline copies, or an equivalent risk-based control.
  • REL-046 Document backup location, region/account/provider dependence, replication, and survivability of the relevant failure scenario.
  • REL-047 Capture versioned schema, application configuration, encryption-key availability, and external dependencies needed to use restored data.
  • REL-049 Test point-in-time recovery or equivalent granularity when the business RPO requires it.
  • REL-050 Periodically restore backups to an isolated environment using documented or automated procedures.
  • REL-051 After test restoration, query representative records and execute relevant business operations to validate usability and integrity.
  • REL-052 Measure actual restore duration and recovered data age and compare them with RTO and RPO targets.
  • REL-053 Ensure restore testing does not leak sensitive data into insecure nonproduction environments.
  • REL-054 Test recovery from corrupted, missing, outdated, or compromised backup copies where risk warrants.
  • REL-055 Maintain recoverable backup credentials, key material, and operational documentation even if the primary environment is unavailable.
  • REL-056 Document safe disposal, retention expiry, and post-incident access revocation for recovery copies.
  • REL-057 Record restoration drill results, deviations, corrective actions, and the next scheduled verification in project-specific records.

Disaster recovery and continuity#

  • REL-062 Keep recovery documentation and communication channels accessible when the primary identity, network, or documentation platform is unavailable.
  • REL-063 Confirm recovery staff can obtain approved emergency access without weakening controls or depending on the failed infrastructure.
  • REL-064 Map restoration order across identity, networking, storage, databases, application services, APIs, and end-user interfaces.
  • REL-065 Prepare required infrastructure definitions, machine images, artifacts, configuration, and secrets references outside the affected failure domain.
  • REL-079 Review disaster-recovery exercises, incidents, and infrastructure changes to keep the runbook and target values current.

API Checks#

  • REL-080 Define availability and data-correctness indicators for each business-critical API operation and consumer group.
  • REL-081 Verify API clients have bounded timeouts and can distinguish transient outages from permanent validation or authorization errors.
  • REL-082 Define response codes and stable error representations for overload, unavailable dependencies, and temporary maintenance.
  • REL-083 Specify retry policies with budgets and jitter; document when a failed request must not be retried.
  • REL-084 Ensure repeated or ambiguous write requests use idempotency keys or documented deduplication where necessary.
  • REL-085 Test partial failures when downstream APIs, databases, message brokers, or identity services become slow or unavailable.
  • REL-086 Provide readiness and liveness signals appropriate for routing decisions without disclosing internal secrets.
  • REL-087 Confirm long-running API tasks expose durable status or resumption mechanisms when interruptions are expected.
  • REL-088 Test API behavior when read replicas lag, failover changes leaders, or cached data becomes temporarily stale.
  • REL-089 Verify queues, webhooks, and event handlers tolerate duplicate delivery, delayed delivery, and replay according to their contracts.
  • REL-090 Document API maintenance and degraded-mode semantics for client developers, including relevant retry guidance.
  • REL-091 Test graceful shutdown and connection draining for active requests and streaming responses during deployment and failover.
  • REL-092 Assess rate limiting, traffic prioritization, and load shedding for preserving critical endpoints during overload.
  • REL-093 Validate that failed authentication or authorization dependencies cannot accidentally permit unauthorized API access.
  • REL-094 Test backward compatibility and version negotiation when old and new API instances coexist during failover or phased recovery.
  • REL-095 Retain useful request IDs, trace context, and version metadata for API errors without unnecessarily logging sensitive data.
  • REL-096 Verify API contract-test and failover-test coverage for critical consumer/provider dependencies before sign-off.
  • REL-097 Confirm recovery actions do not silently lose, reorder, or duplicate committed business operations contrary to the documented API contract.

Release gate#

  • Critical service journeys and data have assigned owners and measurable availability, RTO, and RPO targets appropriate to their business impact.
  • Important failure modes and dependency risks are assessed, and resilience controls have been exercised or supported by an explicitly accepted risk-based plan.
  • Required backups are current, access-controlled, and demonstrated restorable through recent drills that verify data integrity and recovery objectives.
  • Disaster-recovery activation, roles, communications, recovery order, cutover, and failback procedures are documented and rehearsed to the level required by service risk.
  • Critical API and frontend flows degrade safely, avoid unintended duplicate actions or data loss, and recover coherently where those interfaces exist.
  • Outstanding reliability risks, failed recovery tests, and objective breaches have a documented disposition, owner, remediation date, and authorized release decision.

Domain-specific approval rule: Block reliability sign-off when an unaccepted failure mode can disable an essential service, when required backups cannot be restored, when recovery targets are demonstrably missed without authorization, or when data integrity and recovery ownership are unresolved. A written runbook or backup job success alone does not prove recoverability.

Trusted references and scope#

These verifiable review questions are informed by authoritative engineering and contingency-planning guidance. They are not copied standards language and do not imply universal compliance or guaranteed availability.

Applicability: Define targets and test depth according to criticality, business impact, and actual platform. Multi-region deployments, active-active replication, Kubernetes disruption budgets, and zero-data-loss objectives are not universally required. A PodDisruptionBudget addresses supported voluntary Kubernetes evictions, not all outages. Avoid fault tests that could harm production users without proper authorization and safeguards.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: network/service failure injection, retries, restore, and recovery objectives; include API and service behavior where applicable.
  • Cross-stack: Toxiproxy + k6.
  • Python: testcontainers-python + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: Testcontainers + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: Testcontainers + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

  • debugging-and-error-recovery — Review failure handling and diagnostics (addyosmani/agent-skills, 116,378 repo stars).
  • python-resilience — Assess Python service retries and fallback behavior where applicable (wshobson/agents, 42,437 repo stars).
  • slo-implementation — Review service objectives and recovery triggers (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.