12 — AI Data Curation and Labeling Checklist#

Verify that datasets used for AI training, tuning, evaluation, or retrieval are lawfully and responsibly sourced, technically consistent, properly labeled, traceable to origin, and separated to avoid misleading evaluation. A large dataset or high annotator agreement alone does not establish correctness or representativeness.

Scope: Backend / API-specific. This file is one part of the 12_AI_Data_Curation_and_Labeling.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Dataset purpose, sources, data stewards, and intended users:
• Dataset / labeling-guideline / transformation versions:
• Applicable privacy, licensing, and retention rules:
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Make a project-specific copy of this reusable template; keep the repository checkboxes unchecked.
2. Verify applicable items from permitted-source registers, dataset manifests and cards, privacy or licensing reviews, quality reports, annotation guidelines, agreement/adjudication results, and reproducible dataset builds. Record evidence, failures, justified N/A, and risk acceptances outside this shared template, keyed by CUR ID.
3. Scope each check to the actual data source, modality, task, jurisdiction, labeling method, and intended consumers. Some products use unlabeled pretraining data, synthetic examples, weak labels, or externally curated evaluation sets; do not assume every label or review workflow applies.
4. Keep ## API Checks and ## Frontend Checks separate. Record N/A and a rationale in project records when a surface does not exist.
5. Reassess after adding new data sources, changing acquisition rights, collection populations, cleaning rules, label taxonomies, split strategies, processors, or deployment use cases.
6. Coordinate specialist privacy, AI evaluation, model cards, training, and monitoring review with related repository checklists. Source rights and personal-data obligations require qualified legal/privacy review where relevant.

API Checks#

• CUR-093 Authenticate and authorize dataset ingestion, upload, annotation, adjudication, export, and deletion endpoints.
• CUR-094 Enforce source, project, annotator role, record, and tenant isolation on every dataset and label API.
• CUR-095 Validate file types, encodings, MIME claims, schema constraints, sizes, and submitted label payloads before persistence.
• CUR-096 Reject unsupported or malicious files and prevent execution of untrusted content during parsing or previews.
• CUR-097 Use stable source-record IDs and idempotent import requests to prevent accidental duplicate data from retries.
• CUR-098 Preserve raw source and derived data identifiers, metadata, and provenance when transferring records across services.
• CUR-099 Ensure API contracts distinguish unannotated, draft, submitted, reviewed, adjudicated, rejected, and withdrawn labels.
• CUR-100 Validate annotation version and optimistic concurrency to prevent reviewers silently overwriting each other's decisions.
• CUR-101 Apply role-based approval for guideline changes, label taxonomy updates, and data release publication.
• CUR-102 Test bulk exports for correct split membership, approved schema versions, source restrictions, and permitted consumers.
• CUR-103 Redact or restrict sensitive annotations, protected data, and annotator identifiers in responses, logs, and telemetry.
• CUR-104 Ensure scoped service accounts and expiring credentials are used for labeling vendors and automated data pipelines as appropriate.
• CUR-105 Expose ingestion, labeling, deduplication, and validation failures through actionable status or error contracts.
• CUR-106 Rate-limit and bound bulk ingest, media processing, auto-labeling, and export operations to protect service availability.
• CUR-107 Validate source deletion, access revocation, and correction APIs against indexed, cached, and downstream data copies.
• CUR-108 Prevent client-supplied fields from marking unreviewed machine-generated labels as human-adjudicated ground truth.
• CUR-109 Make dataset version, schema version, transformation version, and labeling-guideline revision available to authorized clients.
• CUR-110 Test webhook and task-queue retries for duplicate annotations, source update ordering, and finalization correctness.
• CUR-111 Verify API audit trails support reconstructing a release without storing unnecessary personal or distressing raw content.
• CUR-112 Run production-equivalent contract tests for annotation status, privacy controls, invalid inputs, and export integrity.

Release gate#

• The dataset's intended use, source inventories, stewardship, relevant licensing and privacy rights, and prohibited uses have been reviewed by accountable owners.
• Collection scope, sensitive fields, retention, vendor processing, deletion or withdrawal mechanisms, and access restrictions are approved for the actual use case.
• Data-quality checks cover critical schema fields, record counts, duplicates, corrupt examples, preprocessing integrity, and relevant population/language representation.
• Human or machine-assisted labeling follows versioned guidelines, with measured agreement or quality review and resolved high-impact ambiguities where labels are used.
• Training, validation, test, and holdout split rules have been reviewed for entity, temporal, near-duplicate, and synthetic/augmentation leakage that could invalidate evaluation.
• Dataset and label versions have reconstructable provenance, controlled changes, accessible documentation of limitations, and verified export integrity.
• Relevant API and frontend workflows protect annotator and source data, preserve review states, and prevent unauthorized releases or overwritten labels.
• Severe unresolved rights, privacy, contamination, bias, or label-integrity risks have been remediated or explicitly dispositioned by qualified human approvers before dataset use.

Domain-specific approval rule: Block dataset-release sign-off when collection/use rights are unknown for material records, restricted personal data is exposed, severe unresolved labeling defects undermine the target task, evaluation data leaks into training, or required source deletion cannot be honored within accepted controls. No dataset documentation or quality score guarantees freedom from bias, lawful use in every jurisdiction, or a reliable downstream model.

Trusted references and scope#

These primary and authoritative resources support practical dataset-quality, labeling, provenance, and risk review. Some technical examples are vendor-specific and do not establish universal legal obligations or independent certification.

• Google PAIR — Data Collection + Evaluation — responsible data sourcing, representation, labeling workflows, and dataset evaluation.
• Hugging Face — Dataset Cards — dataset composition, use restrictions, licenses, languages, bias, and metadata.
• TensorFlow Data Validation — Guide — schema validation, data anomalies, training-serving skew, and drift.
• TensorFlow Data Validation — Get Started — example-level and aggregate validation, distributions, and typed schema checks.
• HumanSignal Label Studio — Labeling — annotation instructions, labeling interfaces, overlapping work, and machine suggestions.
• HumanSignal Label Studio — Annotation Quality — review workflows, agreement analysis, annotator calibration, and corrections; some features are enterprise-specific.
• HumanSignal Label Studio — Task Agreement — consensus, pairwise agreement, and suitable comparison for complex labels; vendor-specific.
• W3C — PROV Overview — recording source and transformation provenance for trustworthy derived datasets.
• MLCommons — Croissant Dataset Format — interoperable dataset/resource metadata, semantic structure, and responsible ML description.
• UK ICO — AI Security and Data Minimisation — UK privacy and minimization obligations for AI development where applicable.
• NIST — AI Risk Management Framework Playbook — risk-based documentation, fairness, representativeness, and management of AI data limitations.
• Hugging Face — Dataset Card Creation Guide — documentation conventions and metadata fields for released datasets.

Applicability: Hugging Face dataset cards and MLCommons Croissant are useful documentation/metadata conventions, not mandatory formats for every dataset. TensorFlow Data Validation illustrates schema, skew, and drift techniques; equivalent tools are acceptable. HumanSignal Label Studio agreement and quality features can be edition-specific. W3C PROV describes interoperable lineage, not proof of permission to collect data. UK ICO guidance concerns applicable UK data-protection law; evaluate other markets separately. Human labels may be subjective and disagreement can reveal genuine task ambiguity.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: data lineage, nulls, duplicates, labeling agreement, PII, and train/test leakage; include API and service behavior where applicable.
• Cross-stack: Great Expectations + DVC.
• Python: Pandera + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: Zod + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: Ajv + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• dataset-curation — Inspect quality, splits, provenance, and leakage (wshobson/agents, 42,437 repo stars).
• data-quality-frameworks — Check schema, null, duplicate, and anomaly tests (wshobson/agents, 42,437 repo stars).
• arize-dataset — Assess evaluation-data lifecycle where Arize is used (github/awesome-copilot, 41,073 repo stars).

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.