Would an economist apply to this job?

01 Pre-test before publishing

Job posts are hard to pre-test. A hiring team can edit the prose, but it cannot readily observe which qualified readers will misunderstand the requirements, infer an unattractive working environment, or decide not to apply. Interviews and applicant data arrive only after the post has already shaped the candidate pool.

This study creates personas resembling the kinds of job seekers the company hopes to reach. Each persona reads the same post and answers the same questions about application intent, fit, clarity, missing information, and decisive passages. We begin with an Applied Economist opening.

mccall init economist-search --name "Applied Economist search"
cd economist-search

{
  "command": "init",
  "ok": true,
  "project": "/work/economist-search"
}

02 Specify the real role

Record what the company actually needs before evaluating how the advertisement communicates it. Begin with the role’s title, mission, compensation, location, and first-six-month success criterion.

mccall role set \
  --title "Applied Economist" \
  --mission "Improve product and policy decisions with credible causal evidence" \
  --compensation '$155,000–$205,000 base salary' \
  --location "United States remote; quarterly travel" \
  --success-after-six-months \
    "Ship one decision-changing study and establish a reusable measurement design"

{
  "command": "role.set",
  "ok": true,
  "role_id": "applied-economist",
  "updated": ["compensation", "location", "mission", "success_after_six_months", "title"]
}

Add the work itself one responsibility at a time.

mccall role responsibility add "Design experiments and quasi-experimental studies"
mccall role responsibility add "Analyze marketplace and product data"
mccall role responsibility add "Explain findings to product leaders"

{
  "command": "role.responsibility.add",
  "count": 3,
  "ok": true,
  "role_id": "applied-economist"
}

Then distinguish requirements from preferences explicitly.

mccall role qualification add \
  "Graduate training or equivalent applied research experience" --kind required
mccall role qualification add \
  "Causal inference and statistical programming" --kind required
mccall role qualification add \
  "Clear written communication" --kind required
mccall role qualification add \
  "PhD in economics or a related quantitative field" --kind preferred
mccall role qualification add \
  "Marketplace or technology experience" --kind preferred

mccall role show

{
  "command": "role.show",
  "ok": true,
  "role": {
    "id": "applied-economist",
    "responsibilities": [
      "Design experiments and quasi-experimental studies",
      "Analyze marketplace and product data",
      "Explain findings to product leaders"
    ],
    "required_qualifications": [
      "Graduate training or equivalent applied research experience",
      "Causal inference and statistical programming",
      "Clear written communication"
    ],
    "preferred_qualifications": [
      "PhD in economics or a related quantitative field",
      "Marketplace or technology experience"
    ]
  }
}

03 Register the first draft

Our first draft is plausible but poor. It uses inflated language, blurs required and preferred qualifications, says nothing about compensation or location, and gives candidates little sense of daily work or success.

Save that text as posts/original.md, then register it. McCall hashes the exact contents so later edits cannot silently change the experimental condition.

mccall post add posts/original.md \
  --id original \
  --label "Original vague post"

mccall post list

{
  "command": "post.add",
  "ok": true,
  "path": ".mccall/posts/original.json",
  "post_id": "original"
}
{
  "command": "post.list",
  "ok": true,
  "posts": [{
    "id": "original",
    "label": "Original vague post",
    "sha256": "528d…"
  }]
}

04 Build candidate personas

Start with McCall’s broad synthetic panel. It includes a conventional fit, nontraditional fit, career changer, passive candidate, near-fit candidate, and risk-sensitive candidate.

mccall panel preset broad \
  --label "Broad economist candidate panel"

{
  "command": "panel.preset",
  "ok": true,
  "panel_id": "broad",
  "personas": 6
}

Generic stress-test personas are useful, but this study needs occupation-specific variation. Add an economist with an adjacent credential and a qualified passive candidate. Each persona records where it came from.

{
  "name": "Applied policy researcher",
  "source_type": "manual",
  "source_note": "Stress-test persona authored from the role requirements",
  "traits": {
    "career_situation": "Public-policy researcher considering industry",
    "education": "PhD in Public Policy",
    "experience": "Six years using experiments and causal inference",
    "fit": "Meets every required qualification but not the preferred economics PhD",
    "decision_style": "Avoids roles that treat pedigree as a hard requirement"
  }
}

mccall panel add-persona personas/policy-researcher.json --panel broad
mccall panel add-persona personas/passive-tech-economist.json --panel broad
mccall panel list

{
  "command": "panel.add-persona",
  "ok": true,
  "panel_id": "broad",
  "persona_id": "policy-researcher"
}
{
  "command": "panel.add-persona",
  "ok": true,
  "panel_id": "broad",
  "persona_id": "passive-tech-economist"
}
{
  "command": "panel.list",
  "ok": true,
  "panels": [{
    "id": "broad",
    "label": "Broad economist candidate panel",
    "persona_ids": [
      "conventional-fit", "nontraditional-fit", "career-changer",
      "passive-candidate", "near-fit", "risk-sensitive",
      "policy-researcher", "passive-tech-economist"
    ]
  }]
}

05 Run the first study

McCall creates the experiment; ep executes it. The generated Jobs package contains eight questions, the selected post as a scenario, and the candidate panel as agents. It contains no model choice.

mccall study build \
  --panel broad \
  --post original \
  --id original-diagnostic \
  --output original-diagnostic.jobs.ep

{
  "command": "study.build",
  "ok": true,
  "study": {
    "id": "original-diagnostic",
    "post_ids": ["original"],
    "panel_id": "broad",
    "expected_results": 8,
    "jobs_path": "/work/economist-search/original-diagnostic.jobs.ep"
  }
}

ep run \
  --jobs original-diagnostic.jobs.ep \
  --model gemini-2.5-flash \
  --service google \
  --local \
  --output original-diagnostic-results.ep

{
  "status": "ok",
  "data": {
    "meta": {
      "model_count": 1,
      "agent_count": 8,
      "scenario_count": 1,
      "result_count": 8,
      "completed_interview_count": 8,
      "failed_interview_count": 0,
      "saved": {"path": "original-diagnostic-results.ep"}
    }
  }
}

Preserve both packages. The Jobs artifact records exactly what was asked; the Results artifact records answers and model provenance.

mccall results ingest \
  original-diagnostic-results.ep \
  --study original-diagnostic

mccall report build --study original-diagnostic

{
  "command": "results.ingest",
  "complete": true,
  "ok": true,
  "responses": 8,
  "study_id": "original-diagnostic"
}
{
  "command": "report.build",
  "ok": true,
  "path": ".mccall/reports/original-diagnostic.md",
  "study_id": "original-diagnostic"
}

06 Diagnose the first draft

The first run completed eight interviews with Gemini 2.5 Flash. Three candidates selected a positive application decision; five selected a negative decision. No candidate rated the post clear.

The structured decision and rationale conflict for the passive technology economist. The raw response is retained; this inconsistency is visible rather than silently recoded. Across the eight responses, revision suggestions cluster around five changes:

1. Describe the decisions and studies the economist will own.
2. Separate required qualifications from preferred experience.
3. Accept equivalent applied research preparation.
4. Publish compensation, location, and travel expectations.
5. Replace culture slogans with specific working conditions and success criteria.

07 Write a targeted revision

The revision should respond to diagnosed information failures without inventing benefits or relaxing requirements the company truly needs. Every substantive claim below comes from the private role brief.

mccall post add posts/revised.md \
  --id revised \
  --label "Evidence-led revision"

{
  "command": "post.add",
  "ok": true,
  "path": ".mccall/posts/revised.json",
  "post_id": "revised"
}

08 Test both posts together

Build one study containing both posts and the same candidate panel. Rerunning the original costs additional inference, but it reduces a serious comparison problem: model versions, defaults, or panel composition may have changed since the first diagnostic.

mccall study build \
  --panel broad \
  --post original \
  --post revised \
  --id original-vs-revised \
  --output original-vs-revised.jobs.ep

{
  "command": "study.build",
  "ok": true,
  "study": {
    "id": "original-vs-revised",
    "post_ids": ["original", "revised"],
    "panel_id": "broad",
    "expected_results": 16,
    "jobs_path": "/work/economist-search/original-vs-revised.jobs.ep"
  }
}

ep run \
  --jobs original-vs-revised.jobs.ep \
  --model gemini-2.5-flash \
  --service google \
  --local \
  --output original-vs-revised-results.ep

{
  "status": "ok",
  "data": {"meta": {
    "agent_count": 8,
    "scenario_count": 2,
    "result_count": 16,
    "completed_interview_count": 16,
    "failed_interview_count": 0
  }}
}

mccall results ingest \
  original-vs-revised-results.ep \
  --study original-vs-revised

mccall report build --study original-vs-revised

{
  "command": "results.ingest",
  "complete": true,
  "ok": true,
  "responses": 16,
  "study_id": "original-vs-revised"
}
{
  "command": "report.build",
  "ok": true,
  "path": ".mccall/reports/original-vs-revised.md",
  "study_id": "original-vs-revised"
}

09 Compare candidate responses

The controlled run completed 16 interviews: the same eight personas evaluated both posts with Gemini 2.5 Flash.

Feedback on the revised post shifted to team structure, total compensation, examples of equivalent experience, travel details, decision authority, and the workload implied by the six-month goal. The risk-sensitive candidate remained negative because that goal still appeared aggressive.

10 Inspect the study record

economist-search/
├── mccall.json
├── posts/
│   ├── original.md
│   └── revised.md
├── personas/
│   ├── policy-researcher.json
│   └── passive-tech-economist.json
├── original-diagnostic.jobs.ep
├── original-diagnostic-results.ep
├── original-vs-revised.jobs.ep
├── original-vs-revised-results.ep
└── .mccall/
    ├── role/role.json
    ├── posts/{original,revised}.json
    ├── personas/*.json
    ├── panels/broad.json
    ├── studies/{original-diagnostic,original-vs-revised}.json
    ├── results/*.json
    └── reports/*.md

The source Markdown remains easy to edit. Registered post records preserve the exact text and its SHA-256 digest. Study records bind post IDs, persona IDs, panel, expected result count, and the Git-backed EDSL artifact.

The complete experimental loop

mccall post add posts/original.md --id original
mccall panel preset broad
mccall study build --panel broad --post original --id original-diagnostic \
  --output original-diagnostic.jobs.ep
ep run --jobs original-diagnostic.jobs.ep \
  --model gemini-2.5-flash --service google --local \
  --output original-diagnostic-results.ep
mccall results ingest original-diagnostic-results.ep \
  --study original-diagnostic

mccall post add posts/revised.md --id revised
mccall study build --panel broad --post original --post revised \
  --id original-vs-revised --output original-vs-revised.jobs.ep
ep run --jobs original-vs-revised.jobs.ep \
  --model gemini-2.5-flash --service google --local \
  --output original-vs-revised-results.ep
mccall results ingest original-vs-revised-results.ep \
  --study original-vs-revised
mccall report build --study original-vs-revised