10 — Reliability and Disaster Recovery Checklist#

Verify that critical user journeys remain usable under expected failures and can recover within agreed downtime and data-loss targets. Reliability, backup/restore, disaster recovery, and business continuity must be demonstrated with evidence, not assumed from redundancy alone.

Scope: Shared / cross-cutting. This file is one part of the 10_Reliability_and_DisasterRecovery.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Critical service / user journeys and dependencies:
• Availability target / SLO reference:
• Recovery time objective (RTO) / recovery point objective (RPO):
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Copy this shared template into project-specific review records; keep all repository checkboxes unchecked.
2. Mark [x] only after verifying a check through architecture review, actual backup restores, failure tests, resilience drills, observability, or an accountable risk decision. Document failed and genuinely inapplicable checks separately in project review notes or issues using their REL IDs.
3. Establish business-appropriate availability, RTO, and RPO targets before deciding on redundancy or recovery architecture. Record deviations and accepted risks with named owners.
4. Evaluate ## API Checks and ## Frontend Checks independently where applicable. For non-web or non-API systems, record the reason for N/A outside this template.
5. Use safe staged fault injection and restoration exercises, and obtain authorization before any production disruption. Reassess after important architecture, vendor, data, risk, or traffic changes.

Availability objectives#

• REL-001 Identify the business-critical services, user journeys, and supporting dependencies that require explicit reliability targets.
• REL-002 Rank business processes and data by the impact of disruption through a documented business impact analysis or equivalent risk assessment.
• REL-003 Set measurable service-level indicators for user-visible availability, successful operations, latency, and correctness where applicable.
• REL-004 Define service-level objectives with calculation methods, target values, review windows, and accountable owners.
• REL-005 Distinguish customer commitments and service-level agreements from internal reliability objectives and alert thresholds.
• REL-006 Set recovery time objectives (RTOs) for each critical service or business process based on acceptable downtime.
• REL-007 Set recovery point objectives (RPOs) for critical data and state based on acceptable data loss.
• REL-008 Ensure RTO and RPO targets explicitly address dependency order, customer obligations, and realistic restoration constraints.
• REL-009 Document planned maintenance windows and whether they are included in availability calculations and customer commitments.
• REL-010 Use appropriate error-budget or equivalent change-risk controls when service-level objectives are defined.
• REL-012 Define failure-severity thresholds and distinguish localized degradation, service incidents, and disaster declarations.
• REL-014 Identify data integrity and consistency expectations during outages, failover, retries, and restoration.
• REL-015 Assign a business owner and technical owner to each critical reliability and recovery target.
• REL-016 Review reliability objectives after material changes to customer impact, criticality, workload, or contractual obligations.

Disaster recovery and continuity#

• REL-058 Define disaster scenarios relevant to the service, including regional outage, data corruption, ransomware, operator error, and critical vendor unavailability.
• REL-059 Choose a recovery strategy (restore, cold standby, warm standby, active-passive, or active-active) with explicit cost and RTO/RPO trade-offs.
• REL-060 Document when an incident qualifies as a disaster and who may declare, escalate, and end the event.
• REL-061 Maintain a step-by-step disaster-recovery runbook with prerequisites, named roles, decision points, validation, and escalation paths.
• REL-066 Verify alternate-site resources, DNS, routing, certificates, monitoring, and permissions are compatible with recovery procedures where applicable.
• REL-067 Test failover traffic routing, DNS propagation, session behavior, and dependent client resolution where relevant.
• REL-068 Verify replicated data and promoted stores meet integrity and RPO expectations before reopening writes.
• REL-069 Use controlled cutover and rollback/forward-recovery criteria to avoid repeated failover or inconsistent simultaneous writers.
• REL-070 Define how customer transactions accepted during degradation or recovery are reconciled and audited.
• REL-071 Include staff coverage, supplier contacts, manual workarounds, and business communications in the continuity plan.
• REL-072 Prepare accurate internal and external outage status updates that can be published through alternate channels when needed.
• REL-073 Rehearse disaster recovery through tabletop exercises and staged restoration drills with appropriate frequency and risk controls.
• REL-074 Perform representative end-to-end recovery exercises safely, including cold-start and loss-of-primary-control-plane scenarios where feasible.
• REL-075 Measure recovery times and data loss in drills against objectives rather than accepting the existence of a plan as proof.
• REL-076 Define a controlled failback sequence, including data reconciliation, traffic migration, and verification of the restored primary site.
• REL-077 Document risks of switching back too early, including replication lag, split-brain, and stale client connections.
• REL-078 Assign owners and deadlines to every failed drill, broken dependency, documentation gap, and unproven recovery assumption.

Release gate#

• Critical service journeys and data have assigned owners and measurable availability, RTO, and RPO targets appropriate to their business impact.
• Important failure modes and dependency risks are assessed, and resilience controls have been exercised or supported by an explicitly accepted risk-based plan.
• Required backups are current, access-controlled, and demonstrated restorable through recent drills that verify data integrity and recovery objectives.
• Disaster-recovery activation, roles, communications, recovery order, cutover, and failback procedures are documented and rehearsed to the level required by service risk.
• Critical API and frontend flows degrade safely, avoid unintended duplicate actions or data loss, and recover coherently where those interfaces exist.
• Outstanding reliability risks, failed recovery tests, and objective breaches have a documented disposition, owner, remediation date, and authorized release decision.

Domain-specific approval rule: Block reliability sign-off when an unaccepted failure mode can disable an essential service, when required backups cannot be restored, when recovery targets are demonstrably missed without authorization, or when data integrity and recovery ownership are unresolved. A written runbook or backup job success alone does not prove recoverability.

Trusted references and scope#

These verifiable review questions are informed by authoritative engineering and contingency-planning guidance. They are not copied standards language and do not imply universal compliance or guaranteed availability.

• Microsoft Azure Well-Architected — Reliability design review checklist — critical-flow prioritization, failure-mode analysis, SLO/RTO/RPO targets, resilience, and recovery testing.
• Microsoft Azure Well-Architected — Disaster recovery strategies — DR architecture, roles and communication, infrastructure, failover, failback, and drills.
• AWS Well-Architected — Reliability design principles — automatic recovery, resilience, and repeatable recovery testing.
• AWS Well-Architected — Disaster recovery objectives — distinguishing recovery time objectives from recovery point objectives.
• AWS Well-Architected — Back up data — backup identification, encryption, automation, and restoration.
• AWS Well-Architected — Test backup recovery — verifying restored integrity and RTO/RPO outcomes.
• AWS Well-Architected — Plan for disaster recovery — documented recovery plans, configuration drift, and recovery drills.
• NIST SP 800-34 Rev. 1 — Contingency Planning Guide — business impact assessment, contingency plans, and exercises.
• Google SRE — Production Services Best Practices — user-centric service-level objectives and error budgets.
• Google SRE — Handling Overload — bounded retries, load shedding, and overload prevention.
• Google SRE — Addressing Cascading Failures — failure propagation, overload recovery, and isolating dependencies.
• Kubernetes — PodDisruptionBudget — protecting against voluntary disruptions where Kubernetes is used; not a guarantee against all failures.

Applicability: Define targets and test depth according to criticality, business impact, and actual platform. Multi-region deployments, active-active replication, Kubernetes disruption budgets, and zero-data-loss objectives are not universally required. A PodDisruptionBudget addresses supported voluntary Kubernetes evictions, not all outages. Avoid fault tests that could harm production users without proper authorization and safeguards.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: network/service failure injection, retries, restore, and recovery objectives; include shared/cross-team controls where applicable.
• Cross-stack: Toxiproxy + k6.
• Python: testcontainers-python + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: Testcontainers + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: Testcontainers + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• debugging-and-error-recovery — Review failure handling and diagnostics (addyosmani/agent-skills, 116,378 repo stars).
• python-resilience — Assess Python service retries and fallback behavior where applicable (wshobson/agents, 42,437 repo stars).
• slo-implementation — Review service objectives and recovery triggers (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the shared / cross-cutting scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.