Urteil, formerly Labeling. Existing commands and project files remain supported. Use urteil for new work; run urteil docs show rename for transition details.

Expected Parrot · Worked tutorial

Evaluate with an audit trail.

Turn a CSV and a labeling rubric into inspectable EDSL Jobs, review a low-cost sample, and preserve every result and decision without hiding model execution.

1 · DefineWrite an explicit rubric in urteil.yaml.

2 · BuildCreate a portable, inspectable Jobs.ep.

3 · RunApprove the plan and hand the job to ep.

4 · VerifyPreview, compare, aggregate, export, and report.

Install and orient

pip install "git+https://github.com/expectedparrot/urteil.git"

urteil docs show getting-started
urteil --help

Urteil owns study definition and provenance. Urteil never runs model jobs Execution belongs exclusively to the ep CLI.

1. Initialize a project

Suppose comments.csv contains comment_id and comment_text. Keep a stable row ID, expose only task-relevant fields, and hide any existing labels that could leak the answer.

urteil init comments-labeling \
  --csv data/comments.csv \
  --id-column comment_id \
  --columns-used comment_text \
  --copy-csv

cd comments-labeling
urteil inspect

Edit urteil.yaml so every question has one clear judgment and an operational rubric:

questions:
  - name: stance
    type: multiple_choice
    prompt: >-
      Classify the author's stance toward the proposed rule.
      Use unclear when the text contains no interpretable position.
    fields: [comment_text]
    options: [support, oppose, mixed, unclear]

model:
  service: openai
  name: gpt-4o-mini
  temperature: 0

Rubric checkpoint. Define inclusion, exclusion, ambiguity, and borderline cases before scaling. A model cannot repair an underspecified label.

2. Validate and build the full job

urteil validate
urteil generate
ep inspect edsl_jobs/job_a/jobs.ep

urteil generate constructs an EDSL Survey, ScenarioList, and ModelList and serializes them together as edsl_jobs/job_a/jobs.ep. It does not contact a model provider. Inspect the artifact before execution.

Tool	Responsibility
`labeling`	Validate the study, build Jobs, sample, plan, record, preview, evaluate, aggregate, export, and report.
`ep`	Inspect costs and execute a reviewed Jobs artifact, producing a portable Results artifact.
Agent + user	Judge the rubric, approve spending, inspect sample quality, and decide whether to scale.

3. Sample before spending at scale

urteil sample --n 30 --strategy random
urteil plan sample --latest

The sample command preserves a portable Jobs artifact. Inspect the plan's commands[0].argv and obtain approval before model execution. Run that exact command, then record_after.argv, which includes the originating --plan. Each plan has a unique results path.

urteil preview latest --human
urteil approve sample

Revise when

Options overlap, rationales reveal inconsistent interpretations, leakage is visible, or important edge cases are missing.

Scale when

The rubric behaves consistently, ambiguous rows are handled intentionally, and the user approves the full-run plan.

4. Run the full dataset

urteil plan full --reuse-approved-sample

The returned plan has requires_user_approval: true. After explicit approval, run its exact command:

Execute commands[0].argv and then record_after.argv from this full plan. It contains only remaining rows; the original full job would rerun approved rows. If the approved sample covers the entire dataset, export it directly.

urteil preview latest --human

Urteil accepts portable Results.ep artifacts and immediately normalizes each recorded run as a source such as full_run. Metrics and export consume that source directly. With approved-sample reuse, the full source records both accepted-sample and remaining-run provenance.

5. Measure quality and export

# Optional gold-standard comparison
urteil gold import data/gold.csv \
  --id-column comment_id --label-columns stance
urteil metrics classification \
  --reference gold --candidate model_v1 --question stance

# Optional multi-rater aggregation
urteil aggregate create --method majority-vote \
  --raters model_a,model_b --question stance

urteil export --final-label-source model_v1 \
  --output data/cooked/labeled_comments.csv
urteil report context
urteil report template

If no gold labels exist, report disagreement and review representative errors instead of presenting agreement as accuracy. Preserve raw source columns and stable row IDs in the final export.

6. Recover and continue

urteil next
urteil docs list
urteil docs search "gold labels"
urteil docs show troubleshooting

urteil.yaml is the editable study definition. .urteil/ is the append-only audit store for specs, samples, Jobs, plans, Results, metrics, exports, and reports. Use the CLI instead of editing audit records by hand.

Urteil builds the evidence pipeline; ep runs the jobs. Start small, inspect everything, and make the approval boundary visible.

Evaluate items with a codebook and scorecard

Rudin is now part of Urteil. Use urteil scoring for rubric construction, scoring anchors, manual or fitted scorecards, EDSL exports, visualizations, and the worked scoring manual. Existing Rudin rubric and scorecard JSON files remain usable.

urteil scoring configure rubric.json --fields memo
urteil generate
# Follow the sample, execution, and recording workflow above.
urteil scoring apply scorecard.json --source SOURCE --output scores.csv

Configuration includes the codebook's definitions and examples in the rating prompts. Applying a scorecard saves criterion contributions and an immutable audit bundle. Changing weights requires no new model calls. Run urteil docs show scoring for the complete workflow and migration guide.