18 — AI Inference and Serving Checklist#

Verify that production AI inference is deployable, secure, performant, cost-bounded, and recoverable under realistic workloads. Serving improvements must be measured against user-visible quality and latency; faster execution or higher aggregate throughput does not by itself establish a release-ready system.

Scope: Frontend-specific. This file is one part of the 18_AI_Inference_and_Serving.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Model/provider, artifact, serving runtime, and configuration versions:
• Workload type, expected concurrency, throughput, latency, and cost budgets:
• Deployment regions, tenancy, data handling, and fallback policy:
• Engineering / on-call / security / risk owners:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Make a project-specific copy of this shared checklist; keep all reusable repository boxes unchecked.
2. Set realistic workload distributions, concurrency, task-quality requirements, availability objectives, and latency/cost budgets before benchmarking. Review production and fallback variants against the exact supported interfaces.
3. Verify checks through approved model and runtime manifests, load testing, authorization and privacy controls, safe streaming/batching/caching tests, deployment exercises, and incident runbooks. Track evidence, failures, accepted exceptions, and justified N/A outside this shared template, keyed by SERVE ID.
4. Apply ## API Checks and ## Frontend Checks separately where appropriate. Non-interactive batch, on-device, and managed-provider workloads may have different applicability; document inapplicable checks in project records.
5. Validate serving optimizations and resource controls against the actual model, inference engine, region, hardware, and workload. Do not infer that features offered by one runtime are available in every other stack.
6. Reassess after model, adapter, tokenizer, context length, inference runtime, scheduler, cache, scaling policy, provider, endpoint, or policy changes. Coordinate deeper performance, resilience, model-lifecycle, safety, and AI-evaluation review with their dedicated checklists.

Frontend Checks#

• SERVE-113 Measure user-perceived time to first meaningful content and total task completion, not only server inference latency.
• SERVE-114 Clearly communicate loading, queued, generating, paused, timed-out, and failed inference states.
• SERVE-115 Show streamed partial output as provisional until the required validation and safety checks complete.
• SERVE-116 Preserve readable streaming order, focus behavior, and screen-reader announcements for dynamic generated content.
• SERVE-117 Offer cancellation and retry controls that actually cancel or deduplicate server work when supported.
• SERVE-118 Explain error and recovery paths for provider outages, rate limits, unsupported inputs, and model fallback.
• SERVE-119 Check that fallback does not misleadingly promise the same model capabilities, language, or quality guarantees.
• SERVE-120 Prevent model credentials, privileged endpoint configuration, and hidden prompt data from appearing in browser assets.
• SERVE-121 Render generated markup, tool results, and attached output files safely without executable script injection.
• SERVE-122 Maintain accurate completion or action-confirmation status when streams disconnect or clients reload.
• SERVE-123 Verify that old browser tabs and cached frontend versions handle current inference schemas and model routing.
• SERVE-124 Test representative mobile, slow-network, and high-latency journeys for progress visibility and usable timeouts.
• SERVE-125 Validate responsive layouts for long outputs, multilingual text, tables, code blocks, and multimodal responses.
• SERVE-126 Check user-accessible controls for maximum upload size, generation length, and cost-sensitive actions where relevant.
• SERVE-127 Prevent browser caches or service workers from exposing another user's private AI responses after account changes.
• SERVE-128 Ensure user feedback and error-reporting interfaces capture safe model/version context without unapproved prompt logging.
• SERVE-129 Keep provenance, uncertainty, and citation indicators accurate after provider or model changes where relevant.
• SERVE-130 Require proper confirmation for consequential external actions regardless of inference speed or streaming progress.
• SERVE-131 Verify UI feature flags track approved model cohorts and support immediate safe disablement and rollback.
• SERVE-132 Include important user-task and accessibility checks in serving release validation, not just inference throughput tests.

Release gate#

• The exact serving artifact, model/provider version, supported interface, deployment topology, data policy, and responsible owners are documented and approved.
• Capacity, startup/warm-up, saturation, autoscaling, cost, and resource-headroom tests meet the expected traffic envelope and agreed budgets.
• End-to-end p95/p99 latency, first-output latency where applicable, task success, and throughput pass predeclared criteria with production-like request lengths and concurrency.
• Batching, streaming, and caching configurations preserve tenant isolation, response correctness, state integrity, and downstream action safeguards where used.
• Authentication, resource authorization, per-tenant limits, request validation, model-package integrity, and network controls have passed security checks.
• Overload, provider outage, resource exhaustion, cancellation, fallback, regional fault, and rollback scenarios have tested safe outcomes.
• Relevant API and frontend consumers correctly handle model revisions, partial outputs, overload/error states, privacy controls, and degraded-capability messaging.
• Monitoring, on-call escalation, emergency disablement, approved failover, incident recovery, and any remaining risk acceptances are signed off for the exact release.

Domain-specific approval rule: Block release when inference can disclose protected cross-tenant content, invoke unapproved actions, exceed materially unbounded resource limits, fail to meet essential capacity or latency needs, or leave critical users without a tested safe outage/fallback path. Benchmark throughput is not a substitute for task quality, accessibility, compliance review, or service reliability.

Trusted references and scope#

The following primary runtime documentation, operational guidance, and security resources support the project-level review questions. Treat individual serving features as conditional on the version and deployment mode actually used.

• NVIDIA Triton — Dynamic batching — batching, queue delay, sequence behavior, latency-throughput tradeoffs.
• NVIDIA Triton — Rate limiter — model-level scheduling priorities and resource admission limits.
• NVIDIA Triton — Performance Analyzer — concurrency/request-rate benchmarking and performance test setup.
• vLLM — Serving configuration — LLM serving parameters and scheduler, KV-cache, and request configuration.
• vLLM — Automatic prefix caching — shared-prefix KV caching, workload-dependent prefill benefits, and cache limits.
• vLLM — Production metrics — queue, throughput, token, and inference runtime observability.
• KServe — Serving runtimes overview — model-serving architectures, autoscaling, routing, telemetry, and mode-specific capabilities.
• KServe — Predictive inference canary rollout — progressive traffic release and rollback; supported modes vary with KServe version.
• Kubernetes — Horizontal Pod Autoscaling — resource/custom-metric scaling controllers and scale-up/scale-down behavior.
• Amazon SageMaker AI — Real-time inference — managed serving endpoints, autoscaling, runtime configurations, and deployment constraints.
• Google SRE — Handling overload — admission control, overload protection, client throttling, and graceful degradation.
• OpenTelemetry — GenAI inference conventions — model/provider identity, inference tracing, usage metrics, and opt-in sensitive-content capture.
• OWASP API Security Top 10 (2023) — authorization, resource exhaustion, authentication, and API-side protection.

Applicability: Triton dynamic batching applies to supported workload/model types; batching may improve throughput while increasing queueing latency. vLLM prefix/KV caching principally helps shared-prefix prefill, not every workload's decode throughput, and cache isolation must be validated independently. KServe serving modes and canary APIs differ between releases; its documentation for standard versus serverless mode may lag implementation changes, so confirm the exact deployed version and feature gates. Kubernetes HPA alone does not guarantee capacity during long model cold starts. NVIDIA Perf Analyzer documentation notes that genai-perf is being deprecated in favor of AIPerf; check benchmark tool support. OpenTelemetry GenAI semantic conventions have moved to their own repository and remain subject to schema/version changes; prompt and response collection is sensitive and generally opt-in. OWASP's API risk list is guidance rather than certification.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: concurrency, latency, streaming, load, failover, and response contracts; include browser and client behavior where applicable.
• Cross-stack: k6 + OpenTelemetry.
• Python: Locust + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: autocannon + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: Artillery + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• performance-optimization — Check latency/throughput and bottlenecks (addyosmani/agent-skills, 116,378 repo stars).
• agent-platform-inference — Review serving behavior if using Google Agent Platform (google/skills, 24,012 repo stars).
• llm-evaluation — Check response correctness across serving changes (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the frontend scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.