awesomechecklists
llms.txtGitHub ↗
/

22 — AI Red Teaming and Guardrails Checklist#

Verify that AI-enabled products have risk-based adversarial coverage and enforceable defenses against harmful outputs, unauthorized actions, data disclosure, and abuse. Red teaming seeks to discover failures; passing a finite test suite does not prove a model or product cannot be bypassed.

Scope: Frontend-specific. This file is one part of the 22_AI_Red_Teaming_and_Guardrails.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • AI capabilities, model/provider, tools, data sources, and protected assets:
  • Authorized red-team scope / test environment / contact:
  • Policy / guardrail versions and risk/incident owners:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Copy this shared template to project-specific review records; keep every checkbox in the repository unchecked.
  2. Scope adversarial testing to systems, models, datasets, and endpoints the project owns or is authorized to test. Use synthetic identities, dummy data, sandboxed tools, and safe endpoints; do not attack unrelated systems or disclose real secrets.
  3. Mark [x] only after verifying a check through an approved threat model, model/tool permission test, adversarial result, policy decision, safety drill, or production-equivalent guardrail validation. Track evidence, defects, accepted exceptions, and justified N/A in project-specific notes/issues keyed by RED ID, not in this template.
  4. Separate security objective, actual observed violation, blocked attempt, inconclusive result, and benign false positive. Do not credit a model refusal if unauthorized tool effects or external disclosure already occurred.
  5. Evaluate ## API Checks and ## Frontend Checks independently where those surfaces exist; document N/A outside the template for systems without them.
  6. Repeat targeted adversarial and benign evaluation after changes to the model, prompts, retrieved data, tool scopes, guardrails, modalities, or application policy. Review general AI governance and detailed agent, RAG, and regression requirements in their other checklists.

Frontend Checks#

  • RED-117 Clearly identify the AI feature's intended use, meaningful limitations, and the types of requests it may reject or escalate.
  • RED-118 Distinguish a policy refusal, an uncertain answer, a request for human review, and a system error in user-visible feedback.
  • RED-119 Provide an accessible way to appeal false blocks or report harmful generated output where appropriate.
  • RED-120 Explain when an external action requires explicit confirmation and do not present a proposed action as completed.
  • RED-121 Require comprehensible user approval for material transactions, messages, publishing, or other consequential tool actions.
  • RED-122 Ensure safety notices, refusals, and reporting flows remain usable with keyboard and assistive technologies.
  • RED-123 Test supported languages and scripts for appropriate safety messaging and overblocking of legitimate requests.
  • RED-124 Protect user-upload previews, model-generated markup, and linked content from active script injection or misleading destinations.
  • RED-125 Verify that sensitive text, tokens, uploads, and safety reports are not unnecessarily stored in browser logs or analytics.
  • RED-126 Test streaming interfaces so blocked or incomplete content does not appear as a finalized approved answer.
  • RED-127 Provide useful recovery options after refusal, moderation errors, network loss, or a guardrail service failure.
  • RED-128 Explain human-review waiting states, relevant data use, and expected escalation path without promising unavailable support.
  • RED-129 Make it possible to stop, cancel, or retract supported ongoing AI actions and communicate any irreversible side effects.
  • RED-130 Test authentication changes, shared-device use, and tenant switching for leaked protected responses or review history.
  • RED-131 Check that safety controls and user reports still work in mobile, offline, screen-reader, and low-bandwidth scenarios as relevant.
  • RED-132 Ensure generated citations, evidence links, and safety disclaimers do not misleadingly imply independent verification.
  • RED-133 Assess whether users understand model limitations and avoid unsafe overreliance in representative task testing.
  • RED-134 Provide clear consequences and recovery for content that is moderated after initial streaming or draft display.
  • RED-135 Ensure frontend feature flags align with approved API guardrail and model versions during staged rollouts.
  • RED-136 Verify safety-related feedback, user complaints, and critical UI failures are routed to monitored incident channels.

Release gate#

  • A reviewed threat model covers intended use, attacker-controlled inputs, privileged operations, sensitive data, relevant harms, and approved test boundaries.
  • A versioned adversarial evaluation set covers representative direct/indirect attacks, meaningful tool and data boundaries, high-impact policy failures, and benign controls.
  • No unaccepted high-severity violation remains, including unauthorized access, protected-data disclosure, disallowed external side effects, or serious user harm.
  • Moderation and other guardrails have documented coverage, known limitations, false-positive/false-negative measurements, and safe behavior during outages and handoffs.
  • Independent authentication, authorization, parameter validation, resource limits, and action-specific approval prevent prompt wording from granting privilege or bypassing protections.
  • API, retrieval, and frontend paths have been tested for unsafe output handling, information leakage, improper caching, failure states, and supported multilingual/multimodal behavior where relevant.
  • Safety monitoring, incident escalation, emergency disablement, user reporting, and validated retest/rollback procedures have named owners.
  • Release approvers have reviewed exact model/prompt/policy/tool versions, test results, residual risks, and written risk-acceptance decisions with deadlines.

Domain-specific approval rule: Block sign-off if an agent or model can bypass required access controls, execute material unauthorized actions, disclose protected content, or cause serious harm without acceptable mitigation; if required guardrails fail open on dangerous paths; or if critical attack surfaces have not been meaningfully evaluated. No red-team result establishes perfect security, comprehensive legal compliance, or immunity to new attacks.

Trusted references and scope#

These official risk frameworks, test guidance, and provider documentation inform the review items; applicability depends on system capability and the relevant policy.

Applicability: OWASP GenAI LLM Top 10 (2026) and Agentic Top 10 (2026) are awareness and risk frameworks, not certifications. MITRE ATLAS is a living adversary knowledge base; do not treat its tactics as an exhaustive checklist. NIST AI RMF and its generative AI profile are voluntary guidance. Provider moderation models and agent-guardrail SDKs differ in modalities, integration points, blocking semantics, and coverage; verify the actual runtime. Safety classifiers and prompt instructions are defense in depth, not authorization controls. Only conduct red-teaming where the project has explicit authority.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: injection, jailbreak, exfiltration, unsafe tools, and guardrail bypass; include browser and client behavior where applicable.
  • Cross-stack: garak + promptfoo.
  • Python: garak + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

How to check: In a project-specific copy, have the agent audit the frontend scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.