18 — AI Inference and Serving Checklist#

Verify that production AI inference is deployable, secure, performant, cost-bounded, and recoverable under realistic workloads. Serving improvements must be measured against user-visible quality and latency; faster execution or higher aggregate throughput does not by itself establish a release-ready system.

Scope: Backend / API-specific. This file is one part of the 18_AI_Inference_and_Serving.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Model/provider, artifact, serving runtime, and configuration versions:
• Workload type, expected concurrency, throughput, latency, and cost budgets:
• Deployment regions, tenancy, data handling, and fallback policy:
• Engineering / on-call / security / risk owners:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Make a project-specific copy of this shared checklist; keep all reusable repository boxes unchecked.
2. Set realistic workload distributions, concurrency, task-quality requirements, availability objectives, and latency/cost budgets before benchmarking. Review production and fallback variants against the exact supported interfaces.
3. Verify checks through approved model and runtime manifests, load testing, authorization and privacy controls, safe streaming/batching/caching tests, deployment exercises, and incident runbooks. Track evidence, failures, accepted exceptions, and justified N/A outside this shared template, keyed by SERVE ID.
4. Apply ## API Checks and ## Frontend Checks separately where appropriate. Non-interactive batch, on-device, and managed-provider workloads may have different applicability; document inapplicable checks in project records.
5. Validate serving optimizations and resource controls against the actual model, inference engine, region, hardware, and workload. Do not infer that features offered by one runtime are available in every other stack.
6. Reassess after model, adapter, tokenizer, context length, inference runtime, scheduler, cache, scaling policy, provider, endpoint, or policy changes. Coordinate deeper performance, resilience, model-lifecycle, safety, and AI-evaluation review with their dedicated checklists.

API Checks#

• SERVE-093 Authenticate inference endpoints and authorize each user, tenant, model, and permitted operation independently of model responses.
• SERVE-094 Validate request modality, schemas, encoding, context length, token limits, and file sizes before scheduling inference.
• SERVE-095 Restrict supported model selection and tool capabilities using server-owned allowlists rather than caller-controlled arbitrary identifiers.
• SERVE-096 Expose documented version, response-schema, input-limit, and compatibility behavior for every supported inference route.
• SERVE-097 Test HTTP and streaming errors for invalid inputs, policy refusal, overload, timeout, provider failure, and canceled requests.
• SERVE-098 Enforce per-user and per-tenant rate, concurrency, token, and spend limits before placing work on expensive serving queues.
• SERVE-099 Apply quota isolation or fairness so one tenant's large requests cannot starve other authorized users.
• SERVE-100 Validate partial streaming chunks and final completion indicators against published API contracts.
• SERVE-101 Keep pending, partial, complete, and failed asynchronous inference job states distinct and retrievable by authorized clients.
• SERVE-102 Use idempotency keys or authoritative reconciliation for retries that could trigger non-idempotent downstream effects.
• SERVE-103 Ensure provider tokens, model registry credentials, private prompts, and sensitive output are not exposed in API diagnostics.
• SERVE-104 Test cache isolation, expiration, deletion, and invalidation across accounts, authorization changes, and model promotions.
• SERVE-105 Verify applied model version, provider, route, token usage, and safe request correlation are observable for incidents.
• SERVE-106 Protect access to health, telemetry, debugging, model-loading, and administrative endpoints according to their sensitivity.
• SERVE-107 Apply TLS and approved network controls between inference gateway, service mesh, model runtimes, and external providers.
• SERVE-108 Test compatibility of batch, synchronous, and streaming clients with rollout and rollback behavior.
• SERVE-109 Prevent model-generated structured outputs or tool calls from bypassing separate schema validation and authorization.
• SERVE-110 Validate abort, timeout, and cancellation semantics free queued or running resources where feasible.
• SERVE-111 Run representative production-equivalent API load and policy tests before deploying a new model or runtime.
• SERVE-112 Confirm API behavior under cross-region failover, fallback model selection, and complete approved-capacity exhaustion.

Release gate#

• The exact serving artifact, model/provider version, supported interface, deployment topology, data policy, and responsible owners are documented and approved.
• Capacity, startup/warm-up, saturation, autoscaling, cost, and resource-headroom tests meet the expected traffic envelope and agreed budgets.
• End-to-end p95/p99 latency, first-output latency where applicable, task success, and throughput pass predeclared criteria with production-like request lengths and concurrency.
• Batching, streaming, and caching configurations preserve tenant isolation, response correctness, state integrity, and downstream action safeguards where used.
• Authentication, resource authorization, per-tenant limits, request validation, model-package integrity, and network controls have passed security checks.
• Overload, provider outage, resource exhaustion, cancellation, fallback, regional fault, and rollback scenarios have tested safe outcomes.
• Relevant API and frontend consumers correctly handle model revisions, partial outputs, overload/error states, privacy controls, and degraded-capability messaging.
• Monitoring, on-call escalation, emergency disablement, approved failover, incident recovery, and any remaining risk acceptances are signed off for the exact release.

Domain-specific approval rule: Block release when inference can disclose protected cross-tenant content, invoke unapproved actions, exceed materially unbounded resource limits, fail to meet essential capacity or latency needs, or leave critical users without a tested safe outage/fallback path. Benchmark throughput is not a substitute for task quality, accessibility, compliance review, or service reliability.

Trusted references and scope#

The following primary runtime documentation, operational guidance, and security resources support the project-level review questions. Treat individual serving features as conditional on the version and deployment mode actually used.

• NVIDIA Triton — Dynamic batching — batching, queue delay, sequence behavior, latency-throughput tradeoffs.
• NVIDIA Triton — Rate limiter — model-level scheduling priorities and resource admission limits.
• NVIDIA Triton — Performance Analyzer — concurrency/request-rate benchmarking and performance test setup.
• vLLM — Serving configuration — LLM serving parameters and scheduler, KV-cache, and request configuration.
• vLLM — Automatic prefix caching — shared-prefix KV caching, workload-dependent prefill benefits, and cache limits.
• vLLM — Production metrics — queue, throughput, token, and inference runtime observability.
• KServe — Serving runtimes overview — model-serving architectures, autoscaling, routing, telemetry, and mode-specific capabilities.
• KServe — Predictive inference canary rollout — progressive traffic release and rollback; supported modes vary with KServe version.
• Kubernetes — Horizontal Pod Autoscaling — resource/custom-metric scaling controllers and scale-up/scale-down behavior.
• Amazon SageMaker AI — Real-time inference — managed serving endpoints, autoscaling, runtime configurations, and deployment constraints.
• Google SRE — Handling overload — admission control, overload protection, client throttling, and graceful degradation.
• OpenTelemetry — GenAI inference conventions — model/provider identity, inference tracing, usage metrics, and opt-in sensitive-content capture.
• OWASP API Security Top 10 (2023) — authorization, resource exhaustion, authentication, and API-side protection.

Applicability: Triton dynamic batching applies to supported workload/model types; batching may improve throughput while increasing queueing latency. vLLM prefix/KV caching principally helps shared-prefix prefill, not every workload's decode throughput, and cache isolation must be validated independently. KServe serving modes and canary APIs differ between releases; its documentation for standard versus serverless mode may lag implementation changes, so confirm the exact deployed version and feature gates. Kubernetes HPA alone does not guarantee capacity during long model cold starts. NVIDIA Perf Analyzer documentation notes that genai-perf is being deprecated in favor of AIPerf; check benchmark tool support. OpenTelemetry GenAI semantic conventions have moved to their own repository and remain subject to schema/version changes; prompt and response collection is sensitive and generally opt-in. OWASP's API risk list is guidance rather than certification.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: concurrency, latency, streaming, load, failover, and response contracts; include API and service behavior where applicable.
• Cross-stack: k6 + OpenTelemetry.
• Python: Locust + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: autocannon + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: Artillery + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• performance-optimization — Check latency/throughput and bottlenecks (addyosmani/agent-skills, 116,378 repo stars).
• agent-platform-inference — Review serving behavior if using Google Agent Platform (google/skills, 24,012 repo stars).
• llm-evaluation — Check response correctness across serving changes (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.