Label with an audit trail.

Install and orient

pip install "git+https://github.com/expectedparrot/labeling.git"

labeling docs show getting-started
labeling --help

Labeling owns study definition and provenance. Labeling never runs model jobs Execution belongs exclusively to the ep CLI.

1. Initialize a project

Suppose comments.csv contains comment_id and comment_text. Keep a stable row ID, expose only task-relevant fields, and hide any existing labels that could leak the answer.

labeling init comments-labeling \
  --csv data/comments.csv \
  --id-column comment_id \
  --columns-used comment_text \
  --copy-csv

cd comments-labeling
labeling inspect

Edit labeling.yaml so every question has one clear judgment and an operational rubric:

questions:
  - name: stance
    type: multiple_choice
    prompt: >-
      Classify the author's stance toward the proposed rule.
      Use unclear when the text contains no interpretable position.
    fields: [comment_text]
    options: [support, oppose, mixed, unclear]

model:
  service: openai
  name: gpt-4o-mini
  temperature: 0

2. Validate and build the full job

labeling validate
labeling generate
ep inspect edsl_jobs/job_a/jobs.ep

labeling generate constructs an EDSL Survey, ScenarioList, and ModelList and serializes them together as edsl_jobs/job_a/jobs.ep. It does not contact a model provider. Inspect the artifact before execution.

3. Sample before spending at scale

labeling sample --n 30 --strategy random
labeling plan sample --latest

The sample command writes both the sampled rows and edsl_jobs/job_a/sample_jobs.ep. The plan returns an argv array equivalent to:

ep run edsl_jobs/job_a/sample_jobs.ep \
  --output data/results.sample.ep

Only run that command after reviewing the model, question wording, scenario count, output path, and cost risk. Then bring the result back into the audit store:

labeling record run --kind sample \
  --results data/results.sample.ep
labeling preview latest --human
labeling approve sample

4. Run the full dataset

labeling plan full --reuse-approved-sample

The returned plan has requires_user_approval: true. After explicit approval, run its exact command:

ep run edsl_jobs/job_a/jobs.ep \
  --output data/results.ep

labeling record run --kind full --results data/results.ep
labeling preview latest --human

Labeling accepts portable Results.ep artifacts and immediately normalizes each recorded run as a source such as full_run. Metrics and export consume that source directly. With approved-sample reuse, the full source records both accepted-sample and remaining-run provenance.

5. Measure quality and export

# Optional gold-standard comparison
labeling gold import data/gold.csv \
  --id-column comment_id --label-columns stance
labeling metrics classification \
  --reference gold --candidate model_v1 --question stance

# Optional multi-rater aggregation
labeling aggregate create --method majority-vote \
  --raters model_a,model_b --question stance

labeling export --final-label-source model_v1 \
  --output data/cooked/labeled_comments.csv
labeling report context
labeling report template

If no gold labels exist, report disagreement and review representative errors instead of presenting agreement as accuracy. Preserve raw source columns and stable row IDs in the final export.

6. Recover and continue

labeling next
labeling docs list
labeling docs search "gold labels"
labeling docs show troubleshooting

labeling.yaml is the editable study definition. .labeling/ is the append-only audit store for specs, samples, Jobs, plans, Results, metrics, exports, and reports. Use the CLI instead of editing audit records by hand.