16 — RAG and Knowledge Bases Checklist#

Verify that retrieval-augmented generation (RAG) uses authorized, attributable, sufficiently fresh evidence; retrieves relevant passages; and produces answers whose factual claims can be checked against the source. Search relevance, answer quality, and access controls are separate requirements: improving one does not establish the others.

Scope: Frontend-specific. This file is one part of the 16_RAG_and_Knowledge_Bases.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Knowledge sources, document owners, and access policy:
• Index / embedding / retrieval / reranker / model versions:
• Source freshness expectations and evaluation owners:
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Make a project-specific copy of this shared template; leave repository checkboxes unchecked.
2. Verify each applicable item using approved source inventories, ingestion logs, index metadata, permission-boundary tests, retrieval judgments, grounded-answer evaluations, and user-facing citation checks. Record failures, approved exceptions, and justified N/A outside the shared template, keyed by RAG ID.
3. Define what content is authoritative and what the current user is permitted to see before optimizing embeddings or model prompts. A retrieval score is not an access-control decision.
4. Evaluate retrieval relevance separately from generation groundedness. Test missing, stale, conflicting, malicious, and inaccessible sources, not only answerable examples.
5. Complete ## API Checks and ## Frontend Checks independently where present. For retrieval-only or non-interactive systems, document why an interface is inapplicable.
6. Reassess the entire retrieval-to-answer chain after source permissions, documents, ingestion, embeddings, index schema, ranking, prompts, model, or client presentation change. Deep AI agent, red-team, monitoring, and regression requirements belong in their dedicated templates.

Frontend Checks#

• RAG-109 Make clear when an answer is generated from a knowledge base and when the system lacks sufficient supporting documents.
• RAG-110 Show source titles, authors or publishers, dates, and accessible links where available and permitted.
• RAG-111 Ensure each user-visible citation navigates to the cited document or specific supporting section when possible.
• RAG-112 Distinguish relevant source references from verified evidence supporting a particular factual claim.
• RAG-113 Explain when a source is unavailable, access-restricted, expired, or no longer part of the knowledge base.
• RAG-114 Display document freshness, version, or effective-date warnings when these affect user decisions.
• RAG-115 Provide comprehensible clarification or no-answer states for ambiguous queries and missing evidence.
• RAG-116 Ensure users can inspect sources without exposing content they are not permitted to access.
• RAG-117 Test source navigation, permissions, and deep links after login, account switching, or session expiration.
• RAG-118 Prevent retrieved HTML, Markdown, file names, or snippets from injecting active scripts into the frontend.
• RAG-119 Provide accessible headings, keyboard interaction, link text, and focus handling for citations and source panels.
• RAG-120 Preserve citation-to-claim mapping while rendering markdown tables, lists, long answers, and streamed text.
• RAG-121 Handle slow retrieval, partial answers, generation timeouts, and source-loading failures without implying results are final.
• RAG-122 Communicate source conflicts or limitations rather than choosing one document without explanation.
• RAG-123 Support long, multilingual, and mixed-script queries in relevant locales without losing retrieval context.
• RAG-124 Make it easy to revise a query or report a wrong answer, broken citation, or outdated document.
• RAG-125 Protect sensitive questions and retrieved snippets from accidental browser persistence, analytics, or error reports.
• RAG-126 Explain that a citation does not alone guarantee the claim is correct or legally authoritative.
• RAG-127 Verify frontend caches do not surface previous users' source snippets, answers, or permission-restricted results.
• RAG-128 Review representative user journeys to assess whether evidence presentation supports accurate, independent judgment.

Release gate#

• Knowledge sources have accountable owners, approved permitted uses, provenance records, and documented inclusion/exclusion and retention policies.
• Ingestion, extraction, chunking, embeddings, indexing, and synchronization have been tested with representative source formats and update/delete scenarios.
• Retrieval enforces document, user, group, and tenant authorization before any protected passage is sent to the model or client; cross-tenant tests pass.
• Held-out relevance checks meet agreed recall, precision/ranking, identifier search, and latency thresholds for critical question types.
• Grounded answers meet predeclared factual support, citation accuracy, completeness, uncertainty, and abstention criteria; severe unsupported claims are resolved or accepted through an authorized process.
• Stale, revoked, conflicting, insufficient, and malicious source material triggers approved warnings, no-answer behavior, containment, or escalation rather than unauthorized disclosure.
• API and frontend integrations preserve safe source identifiers, access boundaries, evidence presentation, and usable recovery paths where applicable.
• A named owner approves the exact corpus/index/model configurations, remaining limitations, monitoring of freshness/quality, and incident response or rollback plan.

Domain-specific approval rule: Block sign-off if unauthorized content can be retrieved or disclosed, critical source revocations fail to propagate within approved bounds, harmful prompt injection enables unapproved actions, or high-impact answers present materially unsupported or incorrect citations without accepted mitigation. A plausible generated answer or clickable citation alone is not proof of source fidelity or legal compliance.

Trusted references and scope#

The primary technical references below describe common RAG patterns and specific vendor implementations. Their mechanisms and defaults are conditional, not universal requirements.

• Microsoft Azure AI Search — RAG overview — retrieval architecture, chunking, security boundaries, multi-source retrieval, citation context.
• Microsoft Azure Architecture Center — RAG solution design and evaluation — end-to-end architecture and distinct retrieval and answer-quality evaluation stages.
• Microsoft Azure Architecture Center — Retrieval phase — hybrid search, relevant passage ranking, reranking, recall/precision tradeoffs, latency.
• Microsoft Azure Architecture Center — LLM end-to-end evaluation — groundedness, completeness, relevance, correctness, and evaluation limitations.
• Microsoft Azure Architecture Center — RAG prompt engineering — source-grounding instructions, abstention, citations, handling conflicting documents.
• Amazon Bedrock — Knowledge-base chunking — structure-aware chunking, metadata, and treatment of tables and multimodal content.
• Amazon Bedrock — Knowledge-base metadata — document and per-chunk metadata for search filters and provenance.
• Amazon Bedrock — Data-source synchronization — source ingestion, synchronization, and operational failure handling.
• OpenAI API — Retrieval guide — vector stores, file chunking, metadata filters, indexing status, and eventual deletion consistency.
• OWASP — Vector and Embedding Weaknesses — injection, poisoning, tenant isolation, and sensitive information risks in RAG.
• OWASP — LLM Prompt Injection Prevention Cheat Sheet — trust boundaries for retrieved text and limits of prompt-only defenses.

Applicability: Azure AI Search, Amazon Bedrock, and OpenAI Retrieval support different indexing, chunking, metadata, and deletion behaviors; validate deployed service versions and settings rather than transferring defaults between vendors. Some retrieval operations and deletions can be eventually consistent. Embeddings and vector similarity are not a substitute for authorization. A high retrieval score, citation, or instruction to ground responses does not guarantee factual accuracy, privacy protection, or resistance to prompt injection. Never ingest protected content without the appropriate processing rights and scope.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: retrieval relevance, groundedness, citations, freshness, and document access; include browser and client behavior where applicable.
• Cross-stack: promptfoo.
• Python: Ragas + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• rag-implementation — Check ingestion, retrieval, and grounding architecture (wshobson/agents, 42,437 repo stars).
• llm-evaluation — Check retrieval/citation evaluation design (wshobson/agents, 42,437 repo stars).
• pinecone-rag — Inspect index/retrieval behavior where Pinecone is used (github/awesome-copilot, 41,073 repo stars).

How to check: In a project-specific copy, have the agent audit the frontend scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.