awesomechecklists
llms.txtGitHub ↗
/

10 — RAG and Knowledge Bases Checklist#

Verify that retrieval-augmented generation (RAG) uses authorized, attributable, sufficiently fresh evidence; retrieves relevant passages; and produces answers whose factual claims can be checked against the source. Search relevance, answer quality, and access controls are separate requirements: improving one does not establish the others.

Scope: AI-specific. This file is one part of the 10_RAG_and_Knowledge_Bases.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • Knowledge sources, document owners, and access policy:
  • Index / embedding / retrieval / reranker / model versions:
  • Source freshness expectations and evaluation owners:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Make a project-specific copy of this shared template; leave repository checkboxes unchecked.
  2. Verify each applicable item using approved source inventories, ingestion logs, index metadata, permission-boundary tests, retrieval judgments, grounded-answer evaluations, and user-facing citation checks. Record failures, approved exceptions, and justified N/A outside the shared template, keyed by RAG ID.
  3. Define what content is authoritative and what the current user is permitted to see before optimizing embeddings or model prompts. A retrieval score is not an access-control decision.
  4. Evaluate retrieval relevance separately from generation groundedness. Test missing, stale, conflicting, malicious, and inaccessible sources, not only answerable examples.
  5. Complete ## API Checks and ## Frontend Checks independently where present. For retrieval-only or non-interactive systems, document why an interface is inapplicable.
  6. Reassess the entire retrieval-to-answer chain after source permissions, documents, ingestion, embeddings, index schema, ranking, prompts, model, or client presentation change. Deep AI agent, red-team, monitoring, and regression requirements belong in their dedicated templates.

Knowledge ingestion and provenance#

  • RAG-001 Inventory all knowledge sources, source systems, content types, document owners, and intended retrieval audiences.
  • RAG-002 Document the business purpose and allowed use of each corpus before ingesting it into an AI retrieval system.
  • RAG-003 Verify permissions, licenses, privacy notices, and contractual restrictions permit indexing, embedding, and sending content to selected providers.
  • RAG-004 Assign a steward and update contact to each authoritative source and significant dataset.
  • RAG-005 Record source identifiers, canonical URLs or document paths, titles, content versions, and last-updated timestamps where available.
  • RAG-006 Identify the source of truth when the same policy, specification, or product fact appears in several repositories.
  • RAG-007 Define which repositories, folders, collections, documents, and fields are in scope or excluded from indexing.
  • RAG-008 Apply source-side user, group, role, tenant, and document restrictions to the retrieval design.
  • RAG-009 Reject or quarantine content that exceeds permitted sensitivity, retention, licensing, or geographic processing constraints.
  • RAG-010 Validate ingestion connectors, file format support, parser behavior, and accepted document sizes against representative source files.
  • RAG-011 Check extraction quality for PDFs, scanned pages, tables, footnotes, headings, hyperlinks, and other structured content.
  • RAG-012 Measure ingestion failures, unsupported files, truncated content, missing text, and processing retries instead of silently dropping documents.
  • RAG-013 Preserve page, section, row, or anchor metadata needed to navigate back to a specific source passage.
  • RAG-014 Keep a traceable mapping from every indexed chunk to its original source item and content revision.
  • RAG-015 Capture source language, locale, effective date, authority, classification, and relevant metadata where available and permissible.
  • RAG-016 Define document deduplication, version precedence, superseded-content handling, and conflicting-source rules.
  • RAG-017 Configure incremental updates, full synchronization, and retry behavior according to each source's change frequency.
  • RAG-018 Ensure revoked access, source deletion, document expiration, and retention changes propagate to all retrievable copies.
  • RAG-019 Test the lag between a source update or deletion and its visibility in search, cache, and generated answers.
  • RAG-020 Define a safe quarantine and incident process for malicious, corrupted, or unauthorized indexed content.
  • RAG-021 Control access and retention for raw files, extracted text, embeddings, chunk metadata, and ingestion error reports.
  • RAG-022 Record ingestion pipeline, parser, embedding, and index versions so a problematic corpus can be reconstructed.

Chunking, embeddings, and indexing#

  • RAG-023 Choose a chunking strategy based on document structure, query patterns, retrieval quality, and model context constraints.
  • RAG-024 Evaluate fixed-size, structure-aware, parent-child, semantic, or no-chunking strategies only where supported and useful.
  • RAG-025 Test chunk boundaries against headings, tables, lists, code blocks, citations, and other semantically connected content.
  • RAG-026 Choose and measure chunk size and overlap so evidence is coherent without unnecessary duplicate context.
  • RAG-027 Check whether table rows, captions, and figure references retain the context required to answer realistic questions.
  • RAG-028 Prevent accidental merging of content with different access permissions, tenants, or retention policies into a shared chunk.
  • RAG-029 Preserve identifiers and access metadata across source files, document records, chunks, and embeddings.
  • RAG-030 Validate extraction and text normalization preserve units, dates, punctuation, technical identifiers, and multilingual meaning.
  • RAG-031 Select embedding models that support relevant languages, domains, passage lengths, and input types.
  • RAG-032 Record embedding model revision, dimensionality, preprocessing, tokenizer, and vector similarity settings.
  • RAG-033 Verify query and document embedding configurations are compatible for each active index.
  • RAG-034 Test long documents, very short snippets, duplicate texts, and near-duplicate passages for search quality.
  • RAG-035 Define index schema for text, vector fields, metadata filters, source locations, and document version.
  • RAG-036 Validate indexing completion status before marking newly added content available to users.
  • RAG-037 Check partial ingestion and failed embedding jobs for inconsistent or missing chunks.
  • RAG-038 Use scoped indexing and least-privilege credentials to prevent writers modifying unrelated tenant data.
  • RAG-039 Plan safe re-embedding and index rebuilds when embedding models or extraction strategies change.
  • RAG-040 Test index migration or blue-green index switching without serving mixed incompatible embeddings.
  • RAG-041 Measure index storage, vector counts, ingestion time, query cost, and performance against expected corpus growth.
  • RAG-042 Document index compaction, lifecycle, backup/restore dependencies, and retention for the selected search service.
  • RAG-043 Verify source removal deletes or invalidates associated vectors, text, and cache entries within the approved time bound.
  • RAG-044 Review index quality using representative queries before enabling a new chunking or embedding configuration.

Retrieval and reranking#

  • RAG-045 Define representative information needs and user queries, including precise identifiers, synonyms, ambiguity, and conversational context.
  • RAG-046 Create query-to-relevant-document or passage judgments for an authorized held-out evaluation sample.
  • RAG-047 Benchmark keyword, dense-vector, and hybrid retrieval where relevant instead of assuming one strategy is universally superior.
  • RAG-048 Measure retrieval recall at k, precision at k, ranking quality, and empty-result behavior with appropriate judgments.
  • RAG-049 Test exact-match retrieval for part numbers, legal references, codes, names, and other identifiers that embeddings may miss.
  • RAG-050 Test semantic retrieval for paraphrases, domain-specific language, multilingual questions, and concept matching.
  • RAG-051 Apply user and tenant authorization filters before protected content can be passed to the model, not merely masked in the UI.
  • RAG-052 Verify all query rewrites, expansions, multi-search branches, and reranking stages preserve the same access restrictions.
  • RAG-053 Test retrieval isolation against users with similar queries but different document permissions.
  • RAG-054 Define metadata filtering for source authority, date, product version, locale, and tenant when relevant to answers.
  • RAG-055 Evaluate whether source recency, quality, or authority boosting improves factual retrieval without hiding essential contrary evidence.
  • RAG-056 Check query rewriting and multi-turn search for incorrect carryover of previous user, tenant, or subject context.
  • RAG-057 Compare candidate-set size and reranking strategies against recall, precision, latency, and operating cost.
  • RAG-058 Verify rerankers receive only permitted content and do not cause unintended secondary disclosures.
  • RAG-059 Define thresholds and fallbacks for low-confidence, low-relevance, or no-result retrieval instead of always generating an answer.
  • RAG-060 Test duplicate chunks, near-duplicate documents, conflicting versions, and noisy passages in ranked results.
  • RAG-061 Ensure multi-source retrieval prioritizes authoritative content and handles unresolved conflicts explicitly.
  • RAG-062 Measure end-to-end retrieval latency, timeouts, index availability, and degraded search paths at realistic concurrency.
  • RAG-063 Limit top-k, query expansion, tool calls, and returned context to control token usage and denial-of-service risk.
  • RAG-064 Collect appropriately redacted diagnostics about query type, retrieved passage IDs, scores, filter outcomes, and reranking version.
  • RAG-065 Use controlled adversarial cases to verify malicious document text cannot change search permissions or trigger external actions.
  • RAG-066 Rerun relevance tests when corpus distribution, embedding model, search engine, ranking configuration, or query patterns change.

Grounding, citations, and freshness#

  • RAG-067 Specify when the system must answer only from retrieved evidence and when other authorized knowledge sources may be used.
  • RAG-068 Keep retrieved passages clearly separated from privileged instructions and label them as untrusted source content.
  • RAG-069 Test direct and indirect prompt injection embedded in documents, snippets, titles, metadata, and retrieved web content.
  • RAG-070 Require source-backed factual claims to reference retrievable evidence identifiers where the product promises citations.
  • RAG-071 Validate that a citation links to the particular source passage supporting the statement, not merely a thematically related document.
  • RAG-072 Test that cited evidence is visible or accessible to the requesting user with appropriate access controls.
  • RAG-073 Ensure generated answers distinguish verified source statements, reasoning or synthesis, and unsupported assumptions.
  • RAG-074 Measure answer groundedness, relevance, completeness, correctness, and evidence utilization on representative held-out tasks.
  • RAG-075 Include cases where retrieval finds no source, conflicting sources, outdated documents, or sources that do not answer the question.
  • RAG-076 Define abstention, clarification, escalation, or alternate search behavior for insufficient or contradictory evidence.
  • RAG-077 Do not fabricate source titles, URLs, page numbers, quotations, or verification status when metadata is unavailable.
  • RAG-078 Preserve document version, effective date, and source timestamp where staleness materially affects the decision.
  • RAG-079 Handle conflicting publication dates, superseded policies, and source revisions without silently mixing incompatible versions.
  • RAG-080 Set accepted freshness windows for time-sensitive facts and provide a safe mechanism to re-query or warn users.
  • RAG-081 Verify answers change appropriately after controlled source updates, document revocation, and reindexing.
  • RAG-082 Test whether model summaries omit essential qualifiers, footnotes, exceptions, or conditions in retrieved passages.
  • RAG-083 Check numerical, legal, financial, and other high-impact factual claims against exact authoritative passages where applicable.
  • RAG-084 Define evaluation thresholds for unsupported claims and incorrect citations separately from overall answer satisfaction.
  • RAG-085 Measure the impact of context ordering, truncation, chunk count, and conflicting context on generated output.
  • RAG-086 Verify structured citations and source references survive streaming, caching, and frontend rendering.
  • RAG-087 Disclose evidence limitations and encourage qualified review when retrieved documents cannot justify consequential decisions.
  • RAG-088 Document a recovery and correction process for discovered inaccurate, poisoned, expired, or misattributed answers.

Release gate#

  • Knowledge sources have accountable owners, approved permitted uses, provenance records, and documented inclusion/exclusion and retention policies.
  • Ingestion, extraction, chunking, embeddings, indexing, and synchronization have been tested with representative source formats and update/delete scenarios.
  • Retrieval enforces document, user, group, and tenant authorization before any protected passage is sent to the model or client; cross-tenant tests pass.
  • Held-out relevance checks meet agreed recall, precision/ranking, identifier search, and latency thresholds for critical question types.
  • Grounded answers meet predeclared factual support, citation accuracy, completeness, uncertainty, and abstention criteria; severe unsupported claims are resolved or accepted through an authorized process.
  • Stale, revoked, conflicting, insufficient, and malicious source material triggers approved warnings, no-answer behavior, containment, or escalation rather than unauthorized disclosure.
  • API and frontend integrations preserve safe source identifiers, access boundaries, evidence presentation, and usable recovery paths where applicable.
  • A named owner approves the exact corpus/index/model configurations, remaining limitations, monitoring of freshness/quality, and incident response or rollback plan.

Domain-specific approval rule: Block sign-off if unauthorized content can be retrieved or disclosed, critical source revocations fail to propagate within approved bounds, harmful prompt injection enables unapproved actions, or high-impact answers present materially unsupported or incorrect citations without accepted mitigation. A plausible generated answer or clickable citation alone is not proof of source fidelity or legal compliance.

Trusted references and scope#

The primary technical references below describe common RAG patterns and specific vendor implementations. Their mechanisms and defaults are conditional, not universal requirements.

Applicability: Azure AI Search, Amazon Bedrock, and OpenAI Retrieval support different indexing, chunking, metadata, and deletion behaviors; validate deployed service versions and settings rather than transferring defaults between vendors. Some retrieval operations and deletions can be eventually consistent. Embeddings and vector similarity are not a substitute for authorization. A high retrieval score, citation, or instruction to ground responses does not guarantee factual accuracy, privacy protection, or resistance to prompt injection. Never ingest protected content without the appropriate processing rights and scope.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: retrieval relevance, groundedness, citations, freshness, and document access; include model and dataset behaviors where applicable.
  • Cross-stack: promptfoo.
  • Python: Ragas + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

  • rag-implementation — Check ingestion, retrieval, and grounding architecture (wshobson/agents, 42,437 repo stars).
  • llm-evaluation — Check retrieval/citation evaluation design (wshobson/agents, 42,437 repo stars).
  • pinecone-rag — Inspect index/retrieval behavior where Pinecone is used (github/awesome-copilot, 41,073 repo stars).

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.