awesomechecklists
llms.txtGitHub ↗
/

18 — AI Monitoring and Drift Checklist#

Verify that production AI systems are observable, that distribution changes are separated from demonstrated model-quality regressions, and that emerging user harm or operational failures trigger accountable responses. A drift alarm alone does not establish a decline in accuracy, fairness, or safety.

Scope: AI-specific. This file is one part of the 18_AI_Monitoring_and_Drift.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • Deployed model/provider, prompt, data, and monitoring revisions:
  • Monitoring baselines, signal owners, and alert thresholds:
  • User-harm, security, privacy, and incident escalation owners:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Copy this reusable checklist into project-specific monitoring review records; leave the shared repository checkboxes unchecked.
  2. Define meaningful production quality, risk, availability, latency, cost, and data-change indicators using the actual AI task and deployment environment. Distinguish feature skew, data drift, model-quality drift, and delayed or absent outcome labels.
  3. Mark [x] only when verified using approved dashboards, trace samples, monitoring jobs, drift reports, ground-truth comparisons, alert tests, incident exercises, or qualified monitoring and governance decisions. Record evidence, failed checks, accepted exceptions, and justified N/A in project-specific notes/issues keyed by DRIFT ID; do not add evidence tables to this shared template.
  4. Apply ## API Checks and ## Frontend Checks independently where the corresponding product interfaces exist; document why any surface does not apply.
  5. Do not infer task correctness or safe impact from low drift scores or endpoint uptime. Sampling gaps, changing cohorts, delayed labels, and provider/model updates can invalidate simple comparisons.
  6. Reassess baselines and alert thresholds after major data, population, model, prompt, retrieval, tool, provider, or regulatory changes. Coordinate incident response and evaluation with AI Governance, AI Evaluation and Regression, MLOps, and Monitoring and Observability checklists.

Production performance signals#

  • DRIFT-001 Define the monitored AI task, served model or provider, intended users, and accountable production monitoring owner.
  • DRIFT-002 Map every inference path, retrieval service, tool call, batch job, streaming endpoint, and fallback path that can affect user outcomes.
  • DRIFT-003 Establish pre-release baselines for task completion, policy/safety failures, model errors, latency, throughput, and resource cost.
  • DRIFT-004 Set availability and responsiveness objectives for the AI-powered user journey rather than only the underlying model endpoint.
  • DRIFT-005 Measure request success, model refusal, guardrail block, incomplete response, timeout, and canceled operation as distinct outcomes.
  • DRIFT-006 Record model, prompt, retrieval, tool, evaluator, and safety-policy revisions with approved low-cardinality identifiers.
  • DRIFT-007 Trace a representative request through orchestration, inference, retrieval, tool execution, and any human-approval step.
  • DRIFT-008 Monitor end-to-end latency, model service latency, time to first output, streaming duration, and queue or batch processing delays.
  • DRIFT-009 Track p50, p95, and p99 latency under normal, peak, and degraded traffic with meaningful sample counts.
  • DRIFT-010 Monitor request, token, batch, image, audio, or other billable usage consistent with the actual provider pricing model.
  • DRIFT-011 Track provider quotas, rate limiting, timeouts, transient errors, model loading, cold starts, and failover events.
  • DRIFT-012 Measure CPU, GPU, accelerator memory, cache, retrieval index, storage, and queue saturation for self-hosted components where applicable.
  • DRIFT-013 Monitor the share of traffic handled by each active model, experiment, region, tenant category, and rollout cohort where permitted.
  • DRIFT-014 Verify that online and batch inference monitoring report job completion, failed records, retry counts, and data freshness.
  • DRIFT-015 Set sampling and telemetry completeness targets so measured behavior remains representative without collecting prohibited data.
  • DRIFT-016 Detect missing tracing spans, dropped metrics, pipeline outages, and failed dashboard refreshes independently of actual model health.
  • DRIFT-017 Validate that request identifiers enable incident correlation without exposing user identity, prompts, credentials, or private outputs by default.
  • DRIFT-018 Review metric-label cardinality and retention costs before adding tenant IDs, document IDs, prompts, or other high-cardinality attributes.
  • DRIFT-019 Define operational runbooks for provider outages, inference overload, guardrail failure, and model-specific latency regression.
  • DRIFT-020 Compare actual deployed model/configuration versions with the approved release manifest and detect unexpected alias changes.
  • DRIFT-021 Test monitoring under partial provider failure, network partition, canary rollback, and simultaneous model versions.
  • DRIFT-022 Document telemetry ownership, storage locations, access controls, deletion obligations, and time synchronization.

Input and output drift#

  • DRIFT-023 Distinguish training-serving skew, input distribution drift, output distribution drift, and concept or relationship drift in monitoring definitions.
  • DRIFT-024 Choose a relevant reference window or training baseline for each feature, model, population, and deployment context.
  • DRIFT-025 Record baseline dataset version, data transformations, feature schema, time period, and population composition.
  • DRIFT-026 Define feature- and task-specific drift tests suited to numerical, categorical, text, image, embedding, or other applicable data.
  • DRIFT-027 Specify drift thresholds, comparison windows, minimum sample counts, and investigation criteria before reviewing alerts.
  • DRIFT-028 Test how chosen drift statistics behave with small samples, sparse categories, missing values, and changing class prevalence.
  • DRIFT-029 Monitor missingness, unexpected nulls, schema changes, type violations, outliers, and feature ranges separately from statistical drift.
  • DRIFT-030 Check that production feature extraction matches the approved training or evaluation preprocessing steps.
  • DRIFT-031 Track shifts in input languages, prompt length, document categories, modalities, user cohorts, and request complexity where relevant and lawful.
  • DRIFT-032 Distinguish legitimate seasonality, campaigns, product rollouts, and market changes from unexpected distribution shifts.
  • DRIFT-033 Monitor output-class frequencies, confidence-score distributions, refusal rates, answer lengths, and response formats where meaningful.
  • DRIFT-034 Detect new or unexpected labels, hallucination patterns, tool-use decisions, citation types, and guardrail outputs for relevant AI applications.
  • DRIFT-035 Evaluate embedding or semantic drift with task-specific reference data rather than assuming vector distance directly measures accuracy.
  • DRIFT-036 Monitor retrieval index version, source freshness, coverage, ranking quality, and source deletion propagation for RAG workloads.
  • DRIFT-037 Compare feature attribution or explanation patterns where supported and appropriately validated, noting attribution changes are not proof of causality.
  • DRIFT-038 Evaluate drift separately for justified subpopulations and languages, subject to data rights and minimum cohort-size safeguards.
  • DRIFT-039 Investigate whether observed shift results from upstream collection bugs, missing identifiers, schema changes, or provider-model updates.
  • DRIFT-040 Keep production and reference data comparable in units, time zones, filtering, data quality, and sampling rules.
  • DRIFT-041 Test monitoring pipelines against known synthetic shifts and non-drift controls to establish alert sensitivity and false-positive behavior.
  • DRIFT-042 Record changes in label-availability windows and account for delayed or biased ground truth before inferring quality trends.
  • DRIFT-043 Use independent outcome or adjudicated labels to confirm actual model-quality degradation where the task makes them available.
  • DRIFT-044 Document cases where labels are unavailable and clearly mark distribution and proxy metrics as indirect evidence.
  • DRIFT-045 Set rebaselining approval rules so a degraded new distribution is not silently accepted as healthy.
  • DRIFT-046 Version drift reports and baseline changes so regressions can be traced across model, data, and workflow releases.
  • DRIFT-047 Define ongoing task-quality measures that connect model outputs to real user outcomes and high-impact failure categories.
  • DRIFT-048 Establish how delayed ground-truth labels are collected, joined to predictions, validated, and protected.
  • DRIFT-049 Measure label delay, missing outcomes, selective feedback, and survivorship bias before interpreting production accuracy estimates.
  • DRIFT-050 Run periodic representative offline or shadow evaluations against recent authorized production-like cases.
  • DRIFT-051 Compare current task-quality results against approved baseline and previous production model revisions.
  • DRIFT-052 Track RAG groundedness, evidence relevance, citation support, abstention, and no-answer outcomes independently where RAG is used.
  • DRIFT-053 Track agent task completion against confirmed tool state, including unauthorized or duplicate side effects, where agents are used.
  • DRIFT-054 Monitor policy refusal accuracy and legitimate-task false blocks separately from true unsafe-content passes.
  • DRIFT-055 Monitor severe errors as separately actionable events even if average model-quality metrics remain stable.
  • DRIFT-056 Watch for model output schema violations, invalid tool arguments, parsing failures, and downstream contract regressions.
  • DRIFT-057 Measure relevant subgroup, language, region, and accessibility impacts without drawing conclusions from inadequate samples or unlawful sensitive-data collection.
  • DRIFT-058 Compare user-reported harmful or incorrect outputs with evaluator findings and investigate disagreement.
  • DRIFT-059 Check for benchmark or evaluator drift after judge model, policy, rubric, or evaluation data changes.
  • DRIFT-060 Track prompt, input, output, cache, reasoning, and other metered token categories according to supported provider usage data.
  • DRIFT-061 Report cost per successful task and failed/canceled-task costs rather than only aggregate inference spend.
  • DRIFT-062 Detect anomalous token growth, recursive agent loops, excessive retries, repeated retrieval, and unnecessary tool fan-out.
  • DRIFT-063 Monitor differences in quality, latency, and cost between baseline, canary, and fallback models by comparable traffic segments.
  • DRIFT-064 Separate infrastructure outages, provider throttling, input errors, safety blocks, and quality regressions in incident dashboards.
  • DRIFT-065 Set review rules for low-volume but high-consequence harms that do not produce reliable aggregate-rate alarms.
  • DRIFT-066 Review whether changing demographics or input tasks invalidate historical baselines or evaluation coverage.
  • DRIFT-067 Track temporal performance trends and business outcomes with explicit metric definitions and known measurement limitations.
  • DRIFT-068 Require investigation and owner sign-off before using drift or quality signals to trigger retraining, model switching, or other material action.
  • DRIFT-069 Do not infer fairness, safety, or factual correctness solely from a stable input feature distribution.
  • DRIFT-070 Document how production incidents and verified failures become sanitized, versioned regression-evaluation cases.

Alerts and feedback loops#

  • DRIFT-071 Assign owners, severity, escalation contacts, and response windows for monitoring coverage and material AI risks.
  • DRIFT-072 Define actionable alert thresholds for availability, response latency, critical failure rates, quality and drift, and cost exposure.
  • DRIFT-073 Separate immediate page-worthy user harm or service loss from trend investigation and routine analytical review.
  • DRIFT-074 Require alerts to identify affected model versions, cohorts, services, time window, and available investigation context.
  • DRIFT-075 Set minimum sample counts, aggregation windows, and hysteresis or persistence rules to reduce alert noise.
  • DRIFT-076 Test that alerts fire on controlled latency, drift, invalid-schema, guardrail, and provider-failure scenarios.
  • DRIFT-077 Detect failed monitoring jobs, absent metrics, stale baselines, and dropped telemetry so lack of alerts is not mistaken for health.
  • DRIFT-078 Document a triage procedure to distinguish true model drift from pipeline defects, seasonal traffic, and insufficient data.
  • DRIFT-079 Define containment options including rollout freeze, feature disablement, provider switch, model rollback, and human-only fallback.
  • DRIFT-080 Test emergency shutdown and rollback after unsafe generation, unexpected tool actions, or confirmed protected-data leakage.
  • DRIFT-081 Keep a responsible human approval gate for automated retraining or promotion when risk policy requires it.
  • DRIFT-082 Define severity-based communication plans for customer support, security, product, compliance, and relevant external stakeholders.
  • DRIFT-083 Provide approved feedback intake channels for harmful, inaccurate, discriminatory, or frustrating AI outcomes.
  • DRIFT-084 Validate feedback labeling, review sampling, complaint escalation, and protected-data handling before operational use.
  • DRIFT-085 Review false positive alerts and false negative incidents periodically and adjust thresholds with documented rationale.
  • DRIFT-086 Record actual negative impacts, near misses, root-cause hypotheses, mitigations, owners, and retest evidence in incident records.
  • DRIFT-087 Use canary and experiment guardrails to halt traffic growth when observed user harm or severe regression exceeds approved thresholds.
  • DRIFT-088 Schedule recurring post-deployment quality reviews including fairness, privacy, security, robustness, and user value as applicable.
  • DRIFT-089 Review alert routing and coverage after staffing, vendor, model, telemetry, or supported-market changes.
  • DRIFT-090 Ensure owner dashboards and escalation runbooks remain accessible during a telemetry platform or cloud-provider outage.
  • DRIFT-091 Retain reviewable history of alert acknowledgments, corrective actions, approved rebaselines, and lifecycle decisions.
  • DRIFT-092 Define decommissioning criteria for models whose residual performance or safety risk exceeds approved tolerance.

Release gate#

  • Critical AI user journeys, production components, responsible teams, and actual serving model/configuration versions have verifiable monitoring coverage.
  • Availability, error, tail-latency, workload, cost, safety, and high-impact task-quality indicators have approved baselines, thresholds, and accountable escalation paths.
  • Feature input quality, training-serving skew, output changes, and where feasible ground-truth quality are tracked with suitable reference populations, sampling, and freshness controls.
  • Monitoring distinguishes observed distribution drift from confirmed quality degradation, accounts for missing or delayed labels, and identifies unreliable/inconclusive comparisons.
  • Production privacy, security, access, sampling, cardinality, and retention controls protect telemetry, feedback, and evaluation records.
  • Alerts and monitoring-failure detection have been exercised; critical incidents trigger tested containment, human response, emergency disablement, or model rollback as applicable.
  • Relevant API and frontend monitoring detects errors, incomplete/unsafe outputs, user-visible failures, and problematic model or feature-rollout changes.
  • A named risk/release approver has reviewed remaining monitoring blind spots, observation windows, accepted residual risk, and the plan for drift investigation and remediation.

Domain-specific approval rule: Block monitoring readiness when serious user harm or unauthorized tool effects cannot be detected or escalated appropriately, monitoring is materially blind on critical workflows, an essential safety guardrail can fail open unnoticed, or required privacy controls for monitored data are absent. Stable feature distributions and passing infrastructure health checks do not certify model quality or the safety of the user experience.

Trusted references and scope#

The following primary and authoritative sources informed the monitoring questions. Vendor-specific services and algorithms are examples rather than universal requirements.

Applicability: The NIST AI RMF is voluntary guidance. Statistical drift test behavior depends on baseline size, measurement type, missing values, thresholds, and cohort composition; drift may warrant investigation but is not proof of concept drift or degraded ground-truth performance. AWS states SageMaker Model Monitor is no longer open to new customers as of this review; its older model-quality documentation applies to existing deployments, and the availability-change page describes recommended alternatives. Google Cloud feature-skew/drift features depend on supported deployment types and configurations. In 2026, OpenTelemetry GenAI semantic conventions moved to the dedicated open-telemetry/semantic-conventions-genai repository; verify schema and instrumentation support before adopting fields. Monitor sensitive content only where approved and avoid exporting raw prompts, protected documents, or personally identifying labels by default.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: drift thresholds, sampling, alert routing, and inference telemetry; include model and dataset behaviors where applicable.
  • Cross-stack: Prometheus promtool + Grafana.
  • Python: Evidently + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: OpenTelemetry + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: OpenTelemetry + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

  • distributed-tracing — Check inference trace and context propagation (wshobson/agents, 42,437 repo stars).
  • llm-evaluation — Check quality changes over time (wshobson/agents, 42,437 repo stars).
  • agent-platform-eval-flywheel — Review continuous evaluation where Google Agent Platform is used (google/skills, 24,012 repo stars).

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.