14 — AI Fine-tuning and Training Checklist#

Verify that training or fine-tuning a model is justified, reproducible, permissioned, numerically stable, evaluated for unintended behavioral changes, and released under explicit control. Lower training loss or a successful provider job alone does not establish model quality, safety, data rights, or deployment readiness.

Scope: Frontend-specific. This file is one part of the 14_AI_FineTuning_and_Training.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Training objective, base model, adaptation method, and intended use:
• Data / split / preprocessing / tokenizer versions and processing rights:
• Training code, environment, hyperparameters, checkpoint, and evaluator versions:
• ML engineering / data / risk / security / release approvers:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Make a project-specific copy of this shared checklist; leave the repository's reusable checkboxes unchecked.
2. Decide whether fine-tuning is needed by comparing the expected benefit with a baseline, prompt or retrieval improvements, external inference, and the full training/serving cost. Scope checks to the actual model type, adaptation method, modality, training provider, and deployment.
3. Verify applicable checks using versioned training data, dataset rights reviews, training recipes, instrumentation, checkpoint recovery tests, held-out evaluations, registry artifacts, and approved deployment records. Track failures, evidence, justified N/A, and risk acceptances in project-specific records keyed by TRAIN ID, not in this shared template.
4. Keep ## API Checks and ## Frontend Checks separate. Hosted training, offline training, adapter-only tuning, and models without an end-user interface have different applicability; document what does not apply and why outside this file.
5. Validate training data formatting, response-token loss masking, checkpoint compatibility, and exact base/adapter revisions against the chosen framework. Techniques such as LoRA, SFT, DPO, mixed precision, packing, and distributed checkpointing are options, not universal mandates.
6. Reassess training and release gates when source rights, data distributions, model/provider versions, tuning objectives, label rules, evaluation results, training environment, or serving integrations change. Use the separate AI Data Curation, Model Selection, AI Evaluation, MLOps, Model Cards, and AI Red Teaming checklists for deeper domain reviews.

Frontend Checks#

• TRAIN-113 Clearly distinguish model-training controls and fine-tuning data uploads from ordinary inference features in the UI.
• TRAIN-114 Explain who may upload data, start training, view evaluation results, export models, or approve deployment.
• TRAIN-115 Show the training method, base-model revision, dataset, costs, and significant use restrictions before a privileged job starts.
• TRAIN-116 Provide warnings for confidential data, rights restrictions, sensitive labels, and external processor transfers where relevant.
• TRAIN-117 Display actual training states, elapsed progress, checkpoint/evaluation status, and failures without inventing successful completion.
• TRAIN-118 Prevent misleading progress percentages when the underlying provider only supplies coarse asynchronous job states.
• TRAIN-119 Provide accessible controls for start, pause if supported, cancel, retry, and inspect job errors.
• TRAIN-120 Do not present generated or partially validated checkpoints as production-approved models.
• TRAIN-121 Show baseline-versus-candidate evaluation results with meaningful metrics, sample limitations, and documented severe regressions.
• TRAIN-122 Distinguish evaluation outcomes, risk approvals, staged deployments, and actual production model promotion.
• TRAIN-123 Provide explicit confirmation and an authenticated approval path for production promotion or irreversible training-data deletion.
• TRAIN-124 Test stale dashboards, duplicate clicks, disconnected sessions, and delayed job events against authoritative server state.
• TRAIN-125 Restrict dataset examples, private evaluation prompts, model artifacts, and sensitive job logs by role and tenant.
• TRAIN-126 Safely render training filenames, dataset descriptions, generated logs, error messages, and model-card content without script injection.
• TRAIN-127 Ensure training and deployment progress, errors, and confirmation states are accessible via keyboard and screen readers.
• TRAIN-128 Check responsive layouts, supported languages, and long experiment identifiers across relevant desktop and mobile screens.
• TRAIN-129 Communicate model behavior changes, known limitations, and safe fallback to users affected by the newly tuned model.
• TRAIN-130 Offer understandable recovery when training or model promotion fails, expires, or is canceled.
• TRAIN-131 Collect user feedback on new-model output regressions with privacy-conscious version attribution.
• TRAIN-132 Confirm model-switching interfaces show only approved model versions and do not expose administrative credentials or protected configuration.

Release gate#

• The training purpose, permitted base-model use, accountable owners, chosen method, expected improvement, and material privacy/licensing restrictions have been reviewed.
• Versioned datasets, preprocessing, labels and preference examples pass defined quality checks; evaluation holdouts are appropriately separated and contamination assessed.
• The exact training job has a reviewed configuration, identifiable base checkpoint, correct loss/format semantics, approved resource budgets, numerical stability evidence, and reproducible lineage.
• Candidate weights or adapters, checkpoints, and artifact metadata are integrity-checked; required resume and failure-recovery paths are tested for the training environment.
• Held-out task and baseline comparisons meet predeclared quality/safety criteria, with checks for catastrophic forgetting, memorization, behavior regression, and relevant cohort differences.
• API/inference and frontend compatibility have been evaluated for the exact tuned model, including schema validity, latency, costs, fallback, safety, and user-visible limitations as relevant.
• Registry, model card, deployment approval, staged rollout, monitoring, and rollback or emergency disablement are ready for the approved artifact version.
• No unaccepted high-severity privacy, data-rights, unsafe-content, or unauthorized-action failure remains; a qualified human has signed off on the specific release and accepted residual risks.

Domain-specific approval rule: Block model promotion when material training data lack permitted use, evaluation holdouts are compromised, the approved base/adapter artifact cannot be identified or restored, the candidate shows serious unmitigated behavior or safety regression, or deployment bypasses required authorization. A low training loss, efficient fine-tuning method, or model-card publication is not an independent release approval.

Trusted references and scope#

The following primary maintainer documentation and authoritative guidance inform these training-related checks. Methods and configuration flags are conditional on the exact framework and release in use; they are not mandatory for every AI system.

• Hugging Face TRL — Supervised Fine-Tuning Trainer — conversational/prompt-completion data, masking, packing, and supervised training configuration.
• Hugging Face TRL — Direct Preference Optimization Trainer — chosen/rejected preference pair formats, reference policies, and preference optimization.
• Hugging Face PEFT — LoRA — adapter-based parameter-efficient fine-tuning, base-model compatibility, and configuration.
• Hugging Face Transformers — Trainer features — checkpoints, resumable training, optimizer and random state, and interruptions.
• Hugging Face Datasets — File Names and Splits — separating and documenting training, validation, and test data.
• Hugging Face — Dataset Cards — source rights, dataset description, known limitations, and versioned metadata.
• Hugging Face — Model Cards — model intended use, provenance, training and evaluation disclosures.
• PyTorch — Reproducibility — randomness controls and limits of reproducibility across framework/device versions.
• PyTorch — Automatic Mixed Precision — mixed precision, gradient scaling where applicable, and numerical stability.
• PyTorch — Distributed Checkpoint — sharded and distributed state saving, restoration, and topology requirements.
• Amazon SageMaker AI — Training Output Artifacts — managed training artifact output and separation of final model from other job data.
• Google Cloud Architecture Center — MLOps CI/CD/CT — reproducible pipelines, model evaluation, controlled release, and continuous training.
• NIST AI RMF Playbook — Measure — trustworthiness evaluations, dataset splits, robustness, representativeness, and ongoing monitoring.

Applicability: Hugging Face TRL SFT and preference trainers vary across versions in chat-template support, loss masking, packing semantics, and trainer defaults; inspect actual deployed code and dataset format. LoRA/PEFT adapters depend on the exact base model and are not universally interchangeable. PyTorch documents that complete reproducibility cannot be guaranteed across releases and hardware even with identical random seeds. Restoring training exactly may require optimizer, scheduler and RNG states beyond model weights; distributed checkpoints have topology and compatibility constraints. Mixed precision benefits and numerical risks depend on hardware and dtype. AWS SageMaker output behavior applies only to that managed training platform; Google Cloud MLOps examples do not make its architecture universal. NIST AI RMF practices are voluntary guidance, not certification.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: reproducible runs, training splits, checkpoints, and post-training quality; include browser and client behavior where applicable.
• Cross-stack: MLflow + DVC.
• Python: Transformers Trainer + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• checkpoint-promotion — Review promotion conditions and rollback (wshobson/agents, 42,437 repo stars).
• eval-harness-first — Check pre/post-training evaluation (wshobson/agents, 42,437 repo stars).
• dataset-curation — Review training-data quality and split hygiene (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the frontend scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.