12 — AI Data Curation and Labeling Checklist#

Verify that datasets used for AI training, tuning, evaluation, or retrieval are lawfully and responsibly sourced, technically consistent, properly labeled, traceable to origin, and separated to avoid misleading evaluation. A large dataset or high annotator agreement alone does not establish correctness or representativeness.

Scope: Frontend-specific. This file is one part of the 12_AI_Data_Curation_and_Labeling.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Dataset purpose, sources, data stewards, and intended users:
• Dataset / labeling-guideline / transformation versions:
• Applicable privacy, licensing, and retention rules:
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Make a project-specific copy of this reusable template; keep the repository checkboxes unchecked.
2. Verify applicable items from permitted-source registers, dataset manifests and cards, privacy or licensing reviews, quality reports, annotation guidelines, agreement/adjudication results, and reproducible dataset builds. Record evidence, failures, justified N/A, and risk acceptances outside this shared template, keyed by CUR ID.
3. Scope each check to the actual data source, modality, task, jurisdiction, labeling method, and intended consumers. Some products use unlabeled pretraining data, synthetic examples, weak labels, or externally curated evaluation sets; do not assume every label or review workflow applies.
4. Keep ## API Checks and ## Frontend Checks separate. Record N/A and a rationale in project records when a surface does not exist.
5. Reassess after adding new data sources, changing acquisition rights, collection populations, cleaning rules, label taxonomies, split strategies, processors, or deployment use cases.
6. Coordinate specialist privacy, AI evaluation, model cards, training, and monitoring review with related repository checklists. Source rights and personal-data obligations require qualified legal/privacy review where relevant.

Frontend Checks#

• CUR-113 Display project labeling instructions, taxonomy definitions, and contextual examples adjacent to annotation controls.
• CUR-114 Ensure annotators can identify ambiguous, unlabelable, or insufficient-context samples without fabricating a label.
• CUR-115 Give annotators a way to report unsafe content, task defects, confusing instructions, and suspected bias.
• CUR-116 Make assignment, draft, submission, review, rejection, and adjudication states visibly distinguishable.
• CUR-117 Prevent accidental duplicate submissions and warn before overwriting or losing unsaved annotation work.
• CUR-118 Provide accessible keyboard controls, meaningful focus behavior, labels, and status announcements for labeling tools.
• CUR-119 Test supported screen readers, text scaling, high contrast, and non-pointer operation on critical annotation journeys.
• CUR-120 Verify image, audio, video, and text annotation controls preserve correct coordinates, timing, spans, and selection under zoom.
• CUR-121 Show source context, relevant dates, language, and evidence without exposing unrelated protected source data.
• CUR-122 Support the scripts, locale formatting, Unicode input, and right-to-left behavior required by labeling audiences.
• CUR-123 Allow reviewers to compare independent annotations and resolve disagreements without misrepresenting machine suggestions as ground truth.
• CUR-124 Clearly label machine-preannotated content and require explicit human acceptance where the review policy requires it.
• CUR-125 Prevent machine-suggested labels from biasing independent blind reviews where independence is part of the study design.
• CUR-126 Restrict personal-data previews, exports, and image downloads to authorized annotators and reviewers.
• CUR-127 Ensure annotation text, media metadata, and user-supplied source content are safely rendered without active-script injection.
• CUR-128 Display accurate remaining task count, progress, review outcome, and actionable explanations of rejected submissions.
• CUR-129 Test interrupted sessions, temporary offline states, and retry/resume so annotations are not silently lost or duplicated.
• CUR-130 Make annotation and adjudication histories reviewable by authorized staff with appropriate privacy protections.
• CUR-131 Provide accessible interfaces for searching records by status, label class, source, and audit issue as permitted.
• CUR-132 Verify production labeling UI matches the approved taxonomy, guideline version, API schema, and data-access policy.

Release gate#

• The dataset's intended use, source inventories, stewardship, relevant licensing and privacy rights, and prohibited uses have been reviewed by accountable owners.
• Collection scope, sensitive fields, retention, vendor processing, deletion or withdrawal mechanisms, and access restrictions are approved for the actual use case.
• Data-quality checks cover critical schema fields, record counts, duplicates, corrupt examples, preprocessing integrity, and relevant population/language representation.
• Human or machine-assisted labeling follows versioned guidelines, with measured agreement or quality review and resolved high-impact ambiguities where labels are used.
• Training, validation, test, and holdout split rules have been reviewed for entity, temporal, near-duplicate, and synthetic/augmentation leakage that could invalidate evaluation.
• Dataset and label versions have reconstructable provenance, controlled changes, accessible documentation of limitations, and verified export integrity.
• Relevant API and frontend workflows protect annotator and source data, preserve review states, and prevent unauthorized releases or overwritten labels.
• Severe unresolved rights, privacy, contamination, bias, or label-integrity risks have been remediated or explicitly dispositioned by qualified human approvers before dataset use.

Domain-specific approval rule: Block dataset-release sign-off when collection/use rights are unknown for material records, restricted personal data is exposed, severe unresolved labeling defects undermine the target task, evaluation data leaks into training, or required source deletion cannot be honored within accepted controls. No dataset documentation or quality score guarantees freedom from bias, lawful use in every jurisdiction, or a reliable downstream model.

Trusted references and scope#

These primary and authoritative resources support practical dataset-quality, labeling, provenance, and risk review. Some technical examples are vendor-specific and do not establish universal legal obligations or independent certification.

• Google PAIR — Data Collection + Evaluation — responsible data sourcing, representation, labeling workflows, and dataset evaluation.
• Hugging Face — Dataset Cards — dataset composition, use restrictions, licenses, languages, bias, and metadata.
• TensorFlow Data Validation — Guide — schema validation, data anomalies, training-serving skew, and drift.
• TensorFlow Data Validation — Get Started — example-level and aggregate validation, distributions, and typed schema checks.
• HumanSignal Label Studio — Labeling — annotation instructions, labeling interfaces, overlapping work, and machine suggestions.
• HumanSignal Label Studio — Annotation Quality — review workflows, agreement analysis, annotator calibration, and corrections; some features are enterprise-specific.
• HumanSignal Label Studio — Task Agreement — consensus, pairwise agreement, and suitable comparison for complex labels; vendor-specific.
• W3C — PROV Overview — recording source and transformation provenance for trustworthy derived datasets.
• MLCommons — Croissant Dataset Format — interoperable dataset/resource metadata, semantic structure, and responsible ML description.
• UK ICO — AI Security and Data Minimisation — UK privacy and minimization obligations for AI development where applicable.
• NIST — AI Risk Management Framework Playbook — risk-based documentation, fairness, representativeness, and management of AI data limitations.
• Hugging Face — Dataset Card Creation Guide — documentation conventions and metadata fields for released datasets.

Applicability: Hugging Face dataset cards and MLCommons Croissant are useful documentation/metadata conventions, not mandatory formats for every dataset. TensorFlow Data Validation illustrates schema, skew, and drift techniques; equivalent tools are acceptable. HumanSignal Label Studio agreement and quality features can be edition-specific. W3C PROV describes interoperable lineage, not proof of permission to collect data. UK ICO guidance concerns applicable UK data-protection law; evaluate other markets separately. Human labels may be subjective and disagreement can reveal genuine task ambiguity.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: data lineage, nulls, duplicates, labeling agreement, PII, and train/test leakage; include browser and client behavior where applicable.
• Cross-stack: Great Expectations + DVC.
• Python: Pandera + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: Zod + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: Ajv + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• dataset-curation — Inspect quality, splits, provenance, and leakage (wshobson/agents, 42,437 repo stars).
• data-quality-frameworks — Check schema, null, duplicate, and anomaly tests (wshobson/agents, 42,437 repo stars).
• arize-dataset — Assess evaluation-data lifecycle where Arize is used (github/awesome-copilot, 41,073 repo stars).

How to check: In a project-specific copy, have the agent audit the frontend scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.