16 — Reliability and Disaster Recovery Checklist#

Verify that critical user journeys remain usable under expected failures and can recover within agreed downtime and data-loss targets. Reliability, backup/restore, disaster recovery, and business continuity must be demonstrated with evidence, not assumed from redundancy alone.

Scope: Frontend-specific. This file is one part of the 16_Reliability_and_DisasterRecovery.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Critical service / user journeys and dependencies:
• Availability target / SLO reference:
• Recovery time objective (RTO) / recovery point objective (RPO):
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Copy this shared template into project-specific review records; keep all repository checkboxes unchecked.
2. Mark [x] only after verifying a check through architecture review, actual backup restores, failure tests, resilience drills, observability, or an accountable risk decision. Document failed and genuinely inapplicable checks separately in project review notes or issues using their REL IDs.
3. Establish business-appropriate availability, RTO, and RPO targets before deciding on redundancy or recovery architecture. Record deviations and accepted risks with named owners.
4. Evaluate ## API Checks and ## Frontend Checks independently where applicable. For non-web or non-API systems, record the reason for N/A outside this template.
5. Use safe staged fault injection and restoration exercises, and obtain authorization before any production disruption. Reassess after important architecture, vendor, data, risk, or traffic changes.

Frontend Checks#

• REL-098 Define a clear user-facing experience when essential backend services are unavailable or degraded.
• REL-099 Provide understandable loading, retry, offline, maintenance, and recovery states for critical user workflows.
• REL-100 Distinguish transient connectivity failures from account, permission, validation, and permanent application errors.
• REL-101 Prevent repeated submission of high-impact actions when an API outcome is unknown, and explain how the user can verify completion.
• REL-102 Preserve in-progress form content and other recoverable user input during temporary failures when appropriate.
• REL-103 Ensure failed background saves and offline edits are surfaced rather than silently discarded.
• REL-104 Define how stale or cached content is labeled when freshness could change a user decision.
• REL-105 Test deep links and core navigation when an origin server, CDN, or supporting API becomes unreachable.
• REL-106 Verify cached frontend assets and service-worker versions remain compatible with recovered backend contracts.
• REL-107 Provide an accessible alternative to live, auto-updating UI status messages for screen-reader and keyboard users.
• REL-108 Explain temporarily disabled features and offer useful alternatives rather than presenting controls that fail without feedback.
• REL-109 Use bounded client retries and avoid refresh loops or aggressive polling that amplify backend outages.
• REL-110 Test session expiration, authentication-provider outages, and recovery without losing unrelated user state where feasible.
• REL-111 Check that dynamic application content remains coherent during cross-region traffic changes or cached DNS transitions.
• REL-112 Verify that outage banners, status pages, and support links remain accessible using an independently resilient channel when warranted.
• REL-113 Confirm UI behavior when partial data loads, individual widgets fail, or several API requests return inconsistent versions.
• REL-114 Test that primary user journeys recover correctly after service restoration without requiring unsafe manual cache clearing.
• REL-115 Make post-recovery data reconciliation or duplicate transaction warnings visible where the business workflow requires them.

Release gate#

• Critical service journeys and data have assigned owners and measurable availability, RTO, and RPO targets appropriate to their business impact.
• Important failure modes and dependency risks are assessed, and resilience controls have been exercised or supported by an explicitly accepted risk-based plan.
• Required backups are current, access-controlled, and demonstrated restorable through recent drills that verify data integrity and recovery objectives.
• Disaster-recovery activation, roles, communications, recovery order, cutover, and failback procedures are documented and rehearsed to the level required by service risk.
• Critical API and frontend flows degrade safely, avoid unintended duplicate actions or data loss, and recover coherently where those interfaces exist.
• Outstanding reliability risks, failed recovery tests, and objective breaches have a documented disposition, owner, remediation date, and authorized release decision.

Domain-specific approval rule: Block reliability sign-off when an unaccepted failure mode can disable an essential service, when required backups cannot be restored, when recovery targets are demonstrably missed without authorization, or when data integrity and recovery ownership are unresolved. A written runbook or backup job success alone does not prove recoverability.

Trusted references and scope#

These verifiable review questions are informed by authoritative engineering and contingency-planning guidance. They are not copied standards language and do not imply universal compliance or guaranteed availability.

• Microsoft Azure Well-Architected — Reliability design review checklist — critical-flow prioritization, failure-mode analysis, SLO/RTO/RPO targets, resilience, and recovery testing.
• Microsoft Azure Well-Architected — Disaster recovery strategies — DR architecture, roles and communication, infrastructure, failover, failback, and drills.
• AWS Well-Architected — Reliability design principles — automatic recovery, resilience, and repeatable recovery testing.
• AWS Well-Architected — Disaster recovery objectives — distinguishing recovery time objectives from recovery point objectives.
• AWS Well-Architected — Back up data — backup identification, encryption, automation, and restoration.
• AWS Well-Architected — Test backup recovery — verifying restored integrity and RTO/RPO outcomes.
• AWS Well-Architected — Plan for disaster recovery — documented recovery plans, configuration drift, and recovery drills.
• NIST SP 800-34 Rev. 1 — Contingency Planning Guide — business impact assessment, contingency plans, and exercises.
• Google SRE — Production Services Best Practices — user-centric service-level objectives and error budgets.
• Google SRE — Handling Overload — bounded retries, load shedding, and overload prevention.
• Google SRE — Addressing Cascading Failures — failure propagation, overload recovery, and isolating dependencies.
• Kubernetes — PodDisruptionBudget — protecting against voluntary disruptions where Kubernetes is used; not a guarantee against all failures.

Applicability: Define targets and test depth according to criticality, business impact, and actual platform. Multi-region deployments, active-active replication, Kubernetes disruption budgets, and zero-data-loss objectives are not universally required. A PodDisruptionBudget addresses supported voluntary Kubernetes evictions, not all outages. Avoid fault tests that could harm production users without proper authorization and safeguards.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: network/service failure injection, retries, restore, and recovery objectives; include browser and client behavior where applicable.
• Cross-stack: Toxiproxy + k6.
• Python: testcontainers-python + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: Testcontainers + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: Testcontainers + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• debugging-and-error-recovery — Review failure handling and diagnostics (addyosmani/agent-skills, 116,378 repo stars).
• python-resilience — Assess Python service retries and fallback behavior where applicable (wshobson/agents, 42,437 repo stars).
• slo-implementation — Review service objectives and recovery triggers (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the frontend scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.