12 — AI Inference and Serving Checklist#

Verify that production AI inference is deployable, secure, performant, cost-bounded, and recoverable under realistic workloads. Serving improvements must be measured against user-visible quality and latency; faster execution or higher aggregate throughput does not by itself establish a release-ready system.

Scope: AI-specific. This file is one part of the 12_AI_Inference_and_Serving.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Model/provider, artifact, serving runtime, and configuration versions:
• Workload type, expected concurrency, throughput, latency, and cost budgets:
• Deployment regions, tenancy, data handling, and fallback policy:
• Engineering / on-call / security / risk owners:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Make a project-specific copy of this shared checklist; keep all reusable repository boxes unchecked.
2. Set realistic workload distributions, concurrency, task-quality requirements, availability objectives, and latency/cost budgets before benchmarking. Review production and fallback variants against the exact supported interfaces.
3. Verify checks through approved model and runtime manifests, load testing, authorization and privacy controls, safe streaming/batching/caching tests, deployment exercises, and incident runbooks. Track evidence, failures, accepted exceptions, and justified N/A outside this shared template, keyed by SERVE ID.
4. Apply ## API Checks and ## Frontend Checks separately where appropriate. Non-interactive batch, on-device, and managed-provider workloads may have different applicability; document inapplicable checks in project records.
5. Validate serving optimizations and resource controls against the actual model, inference engine, region, hardware, and workload. Do not infer that features offered by one runtime are available in every other stack.
6. Reassess after model, adapter, tokenizer, context length, inference runtime, scheduler, cache, scaling policy, provider, endpoint, or policy changes. Coordinate deeper performance, resilience, model-lifecycle, safety, and AI-evaluation review with their dedicated checklists.

Serving architecture and capacity#

• SERVE-001 Define the supported inference tasks, input modalities, expected users, deployment regions, and responsible service owner.
• SERVE-002 Choose real-time, asynchronous, batch, streaming, or hybrid serving based on documented latency and completion requirements.
• SERVE-003 Document whether inference uses a managed provider endpoint, self-hosted runtime, on-device model, or provider routing.
• SERVE-004 Identify the approved model artifact or provider snapshot, tokenizer, runtime, and relevant prompt and safety configurations.
• SERVE-005 Verify model weight, adapter, quantization, and runtime compatibility before loading the production artifact.
• SERVE-006 Record GPU or accelerator type, CPU, memory, storage, interconnect, and regional capacity requirements for each deployment.
• SERVE-007 Measure peak accelerator memory including model weights, temporary buffers, key-value caches, and realistic concurrent contexts.
• SERVE-008 Validate readiness and liveness behavior during model downloads, compilation, initialization, warm-up, and degraded health.
• SERVE-009 Document cold-start latency, scale-from-zero consequences, minimum warm capacity, and prewarming requirements where applicable.
• SERVE-010 Size replicas, model instances, partitions, or parallel workers using measured request distributions and available resources.
• SERVE-011 Evaluate tensor, pipeline, data, or expert parallelism only when the chosen runtime and hardware support them reliably.
• SERVE-012 Define request routing across providers, model versions, hardware pools, and regions with explicit authority and fallback constraints.
• SERVE-013 Set capacity reservations and autoscaling minima and maxima based on realistic peaks and provider quotas.
• SERVE-014 Use suitable scaling signals such as queue depth, tokens per second, active sequences, or accelerator utilization when supported.
• SERVE-015 Test scaling lag, resource provisioning, model startup, and warm-up against burst and sustained-load scenarios.
• SERVE-016 Define ingress, egress, network isolation, credential scope, and regional processing controls for all serving components.
• SERVE-017 Separate public, internal, tenant-specific, and privileged inference routes with appropriate authorization.
• SERVE-018 Verify the serving stack accepts only approved model packages and rejects untrusted artifact paths or unsafe loading methods.
• SERVE-019 Record deployment artifact digests, model aliases, container images, configuration, and ownership in the release manifest.
• SERVE-020 Estimate baseline and peak operating costs, capacity costs, provider quotas, and the economics of reserved versus elastic serving.
• SERVE-021 Document operating requirements for high availability, regional isolation, maintenance windows, and infrastructure dependencies.
• SERVE-022 Test deployment reproducibility in a production-equivalent environment and reconcile running versions with the approved manifest.

Latency and throughput#

• SERVE-023 Define application-specific p50, p95, and p99 latency, time-to-first-output, throughput, and task-success budgets.
• SERVE-024 Measure end-to-end user latency separately from serving-only compute latency and provider network delay.
• SERVE-025 Instrument queue wait, preprocessing, model prefill, decode, postprocessing, streaming, and downstream validation times where supported.
• SERVE-026 Use realistic distributions of input tokens, output tokens, image sizes, audio duration, and concurrent requests in load tests.
• SERVE-027 Report prefilling and decoding performance separately for autoregressive models where these dominate latency.
• SERVE-028 Calculate tokens per second and successful requests per second with precise denominators and counting rules.
• SERVE-029 Test minimum, typical, long-context, maximum-supported, and malformed request sizes against actual runtime limits.
• SERVE-030 Benchmark steady-state, burst, ramp-up, and sudden-spike traffic, including cold starts and scale events.
• SERVE-031 Measure latency under saturation and identify the throughput knee where queueing or error rates become unacceptable.
• SERVE-032 Include real preprocessing, tokenization, retrieval, content filtering, and structured-output checks when testing total workload.
• SERVE-033 Compare models, quantization, compilation, and optimization variants using equivalent datasets and acceptable task-quality thresholds.
• SERVE-034 Test queue scheduling, request priorities, fairness, and starvation risk across long and short generations.
• SERVE-035 Set bounded waiting time, admission control, and queue-length policies to avoid memory exhaustion under overload.
• SERVE-036 Measure cancel/disconnect cleanup so abandoned requests stop occupying expensive resources where the runtime supports cancellation.
• SERVE-037 Check autoscaling targets against token-based serving capacity rather than relying solely on generic CPU metrics where insufficient.
• SERVE-038 Validate inference behavior under concurrent batches, multiple model instances, and shared accelerator resources.
• SERVE-039 Use separate load profiles for interactive, batch, background, and high-priority requests where they share infrastructure.
• SERVE-040 Record performance test hardware, software, runtime flags, model versions, warm-up, load-generator settings, and sample counts.
• SERVE-041 Compare tail latency and cost per completed task before approving a change that raises aggregate throughput.
• SERVE-042 Test provider rate limits, local overload thresholds, backpressure, and client retry behavior at expected peak demand.
• SERVE-043 Validate performance metrics are traceable to exact model and serving configuration rather than mixed unlabelled deployments.
• SERVE-044 Rebenchmark after model weights, context limits, batch scheduler, cache design, routing, or infrastructure change.
• SERVE-045 Monitor inference quality alongside performance, because a faster but materially degraded response is not a successful optimization.
• SERVE-046 Document realistic capacity headroom and response plans when forecast growth exceeds safe serving limits.

Batching, streaming, and caching#

• SERVE-047 Select dynamic, static, continuous, sequence-aware, or no batching according to model state and supported serving-runtime behavior.
• SERVE-048 Test maximum batch size, scheduler delay, and queue deadlines against both throughput gains and tail-latency budgets.
• SERVE-049 Check that incompatible tensor shapes, modalities, model revisions, or request parameters are not incorrectly batched together.
• SERVE-050 Verify stateful sequences retain correct request order and session isolation when sequence batching is used.
• SERVE-051 Measure batching behavior with mixed prompt lengths and output sizes instead of only uniform synthetic payloads.
• SERVE-052 Test fairness and priority behavior when short interactive requests compete with long batch generations.
• SERVE-053 Ensure canceling a request in a batch does not corrupt or prematurely stop unrelated requests.
• SERVE-054 Validate streamed text or media chunk ordering, framing, finalization markers, and partial-output semantics.
• SERVE-055 Test streaming interruptions, client disconnects, network proxies, buffer behavior, and backpressure under load.
• SERVE-056 Distinguish partial streamed content from accepted, complete, validated responses in API and user interfaces.
• SERVE-057 Check tool-triggering or irreversible downstream actions are not authorized solely on partial unvalidated streamed output.
• SERVE-058 Define which caches are allowed: response, embedding, tokenizer, prefix/KV, retrieval, or model artifact caches.
• SERVE-059 Verify cache keys account for model, prompt, policy, tenant, permission scope, and other correctness-relevant versions.
• SERVE-060 Prevent cache reuse from exposing protected prompts, responses, session histories, or retrieval context across tenants.
• SERVE-061 Test cache hit, miss, eviction, warm-up, and invalidation behavior with changing permissions and model versions.
• SERVE-062 Measure cache memory pressure and eviction effects on concurrent requests and long-context serving.
• SERVE-063 Validate whether prefix caching improves prefill performance for the workload; do not assume it accelerates token generation.
• SERVE-064 Review cache persistence, deletion, and data residency against retention and privacy requirements.
• SERVE-065 Check speculative decoding, quantization, or other acceleration options against output quality and compatibility before enabling them.
• SERVE-066 Ensure cached and uncached requests produce acceptable results and correct billing, usage, and attribution metadata.
• SERVE-067 Handle batching and streaming errors so one malformed result does not leak or poison another user's output.
• SERVE-068 Record scheduler, cache, batching, and streaming configuration in the deployment manifest for reproducible tests.
• SERVE-069 Apply independent concurrency, output length, context size, and spend limits even when batching or cache optimizations are enabled.
• SERVE-070 Test batch-job deduplication, checkpoint/restart, partial results, and resumable output for supported offline workloads.

Fallbacks and operational resilience#

• SERVE-071 Specify service-level availability, recovery objectives, and acceptable degraded behavior for each critical inference journey.
• SERVE-072 Identify failure dependencies including model providers, accelerators, storage, networking, authentication, retrieval, and moderation services.
• SERVE-073 Define responses to provider outage, regional loss, model-load failure, OOM, resource exhaustion, and sustained queue overload.
• SERVE-074 Use bounded retries with jitter/backoff and retry budgets that avoid multiplying inference costs or tool side effects.
• SERVE-075 Return explicit overload, timeout, unsupported-input, quota, and temporarily unavailable states through appropriate contracts.
• SERVE-076 Define circuit-breaker, admission-control, load-shedding, and provider failover behavior based on tested failure signals.
• SERVE-077 Verify fallback models support required modalities, output schemas, languages, context length, and safety constraints.
• SERVE-078 Ensure fallback routing does not bypass user authorization, data residency, consent, or approved processing providers.
• SERVE-079 Test failure when a fallback has lower context capacity or different tokenization and formatting semantics.
• SERVE-080 Recheck guardrails and tool permissions after model/provider switching rather than assuming a fallback inherits them.
• SERVE-081 Define safe refusal or non-AI service alternatives when no approved model can complete a request.
• SERVE-082 Test release canary, traffic shifting, rollback, and health gating using the actual serving mode and orchestrator version.
• SERVE-083 Protect existing sessions, streaming responses, and batch jobs during rolling restarts and model replacement.
• SERVE-084 Confirm the previous approved model artifact, runtime dependencies, and routing configuration can be restored when rollback is required.
• SERVE-085 Test abrupt worker termination and recovery without duplicate payments, messages, or other model-driven external actions.
• SERVE-086 Validate emergency disablement, provider credential revocation, and traffic drain under simulated incidents.
• SERVE-087 Monitor safe service identifiers, model revisions, failure rates, accelerator memory, queue depth, throughput, and token usage.
• SERVE-088 Configure alerts for service loss, tail-latency regression, excessive failures, unexpected billable usage, and critical guardrail outages.
• SERVE-089 Maintain runbooks for capacity exhaustion, faulty model rollout, cache corruption, and provider-specific errors.
• SERVE-090 Assign an on-call owner, escalation route, incident severity, and recovery decision authority to each critical inference service.
• SERVE-091 Conduct a production-equivalent failure exercise and record corrective actions, evidence, and retest before launch.
• SERVE-092 Retire unsupported or unsafe serving configurations with dependent-client notices, approved data disposal, and updated registries.

Release gate#

• The exact serving artifact, model/provider version, supported interface, deployment topology, data policy, and responsible owners are documented and approved.
• Capacity, startup/warm-up, saturation, autoscaling, cost, and resource-headroom tests meet the expected traffic envelope and agreed budgets.
• End-to-end p95/p99 latency, first-output latency where applicable, task success, and throughput pass predeclared criteria with production-like request lengths and concurrency.
• Batching, streaming, and caching configurations preserve tenant isolation, response correctness, state integrity, and downstream action safeguards where used.
• Authentication, resource authorization, per-tenant limits, request validation, model-package integrity, and network controls have passed security checks.
• Overload, provider outage, resource exhaustion, cancellation, fallback, regional fault, and rollback scenarios have tested safe outcomes.
• Relevant API and frontend consumers correctly handle model revisions, partial outputs, overload/error states, privacy controls, and degraded-capability messaging.
• Monitoring, on-call escalation, emergency disablement, approved failover, incident recovery, and any remaining risk acceptances are signed off for the exact release.

Domain-specific approval rule: Block release when inference can disclose protected cross-tenant content, invoke unapproved actions, exceed materially unbounded resource limits, fail to meet essential capacity or latency needs, or leave critical users without a tested safe outage/fallback path. Benchmark throughput is not a substitute for task quality, accessibility, compliance review, or service reliability.

Trusted references and scope#

The following primary runtime documentation, operational guidance, and security resources support the project-level review questions. Treat individual serving features as conditional on the version and deployment mode actually used.

• NVIDIA Triton — Dynamic batching — batching, queue delay, sequence behavior, latency-throughput tradeoffs.
• NVIDIA Triton — Rate limiter — model-level scheduling priorities and resource admission limits.
• NVIDIA Triton — Performance Analyzer — concurrency/request-rate benchmarking and performance test setup.
• vLLM — Serving configuration — LLM serving parameters and scheduler, KV-cache, and request configuration.
• vLLM — Automatic prefix caching — shared-prefix KV caching, workload-dependent prefill benefits, and cache limits.
• vLLM — Production metrics — queue, throughput, token, and inference runtime observability.
• KServe — Serving runtimes overview — model-serving architectures, autoscaling, routing, telemetry, and mode-specific capabilities.
• KServe — Predictive inference canary rollout — progressive traffic release and rollback; supported modes vary with KServe version.
• Kubernetes — Horizontal Pod Autoscaling — resource/custom-metric scaling controllers and scale-up/scale-down behavior.
• Amazon SageMaker AI — Real-time inference — managed serving endpoints, autoscaling, runtime configurations, and deployment constraints.
• Google SRE — Handling overload — admission control, overload protection, client throttling, and graceful degradation.
• OpenTelemetry — GenAI inference conventions — model/provider identity, inference tracing, usage metrics, and opt-in sensitive-content capture.
• OWASP API Security Top 10 (2023) — authorization, resource exhaustion, authentication, and API-side protection.

Applicability: Triton dynamic batching applies to supported workload/model types; batching may improve throughput while increasing queueing latency. vLLM prefix/KV caching principally helps shared-prefix prefill, not every workload's decode throughput, and cache isolation must be validated independently. KServe serving modes and canary APIs differ between releases; its documentation for standard versus serverless mode may lag implementation changes, so confirm the exact deployed version and feature gates. Kubernetes HPA alone does not guarantee capacity during long model cold starts. NVIDIA Perf Analyzer documentation notes that genai-perf is being deprecated in favor of AIPerf; check benchmark tool support. OpenTelemetry GenAI semantic conventions have moved to their own repository and remain subject to schema/version changes; prompt and response collection is sensitive and generally opt-in. OWASP's API risk list is guidance rather than certification.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: concurrency, latency, streaming, load, failover, and response contracts; include model and dataset behaviors where applicable.
• Cross-stack: k6 + OpenTelemetry.
• Python: Locust + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: autocannon + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: Artillery + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• performance-optimization — Check latency/throughput and bottlenecks (addyosmani/agent-skills, 116,378 repo stars).
• agent-platform-inference — Review serving behavior if using Google Agent Platform (google/skills, 24,012 repo stars).
• llm-evaluation — Check response correctness across serving changes (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.