Pretesting a workplace-trust instrument

01 What is being validated?

Validation is not a permanent badge attached to a questionnaire. It is an argument about whether particular evidence supports a particular interpretation and use. The same six items might be adequate for exploratory group description and inadequate for deciding which employee receives an intervention.

02 Design the evidence before collecting it

Messick uses an evidence ladder. Each tier answers different questions, and stronger-sounding claims require stronger evidence.

03 Install and ask the package what comes next

Install Messick and EDSL together. EDSL supplies durable object packages and the external ep execution CLI; Messick owns instrument-testing state.

git clone https://github.com/expectedparrot/messick.git
cd messick
python -m pip install -e ".[edsl]"

messick --version
messick doctor
messick --help
messick agent guide

All commands emit a versioned JSON envelope by default. Use --human before the command for a compact human rendering. A coding agent should use JSON and treat messick agent next as the workflow control surface.

messick --project-dir trust-pretest agent next

{
  "schema_version": "1.0",
  "status": "error",
  "errors": [{
    "code": "PROJECT_NOT_FOUND",
    "hint": "Run `messick init --title ...`."
  }]
}

04 Meet the instrument

The maintained example under examples/simulation_only/ has two five-point trust items followed by free text. The second item is intentionally awkward: “Management does not fail to avoid keeping promises.” Its double negation makes reverse scoring hard to reason about and gives the pretest something real to find.

{
  "questions": [
    {"question_name":"trust_1", "question_type":"multiple_choice",
     "question_text":"I can usually trust management and leadership.",
     "question_options":[1,2,3,4,5]},
    {"question_name":"trust_2", "question_type":"multiple_choice",
     "question_text":"Management does not fail to avoid keeping promises.",
     "question_options":[1,2,3,4,5]}
  ]
}

05 Initialize immutable project state

Create a study directory, import the Survey, and declare the intent and scale. Ordinary wording edits become new revisions; raw Survey and evidence artifacts remain immutable.

mkdir trust-pretest
messick --project-dir trust-pretest init \
  --title "Workplace trust instrument pretest"

messick --project-dir trust-pretest instrument import \
  --survey examples/simulation_only/survey.ep \
  --message "Initial two-item teaching instrument"

messick --project-dir trust-pretest intent add \
  --input examples/simulation_only/intent.json

messick --project-dir trust-pretest scale add \
  --input examples/simulation_only/scale.json

The import copies the exact Survey into instruments/instrument_v001.ep and records its SHA-256, question order, per-question hashes, and branch-graph hash. Reimporting the identical artifact is idempotent.

{
  "revision_id": "v001",
  "ordered_question_ids": ["trust_1", "trust_2", "comment"],
  "parent_revision": null,
  "status": "draft"
}

06 Run deterministic inspection before model calls

messick --project-dir trust-pretest inspect
messick --project-dir trust-pretest options analyze
messick --project-dir trust-pretest scoring validate
messick --project-dir trust-pretest branching analyze
messick --project-dir trust-pretest branching paths

The inspection flags the double negation in trust_2. It does not rewrite the question automatically: the researcher must decide whether the issue reflects the intended construct, a deliberate method choice, or simply bad wording.

{
  "summary": {
    "question_count": 3,
    "issue_counts": {"warning": 1}
  },
  "artifacts": {
    "analysis": "analysis/instrument_inspection_0001.json"
  }
}

07 Estimate burden question by question and path by path

Pre-fielding duration is a reproducible estimate, not observed human timing. Messick decomposes each item into reading, comprehension, recall, judgment, response entry, and conditional overhead.

messick --project-dir trust-pretest burden analyze
messick --project-dir trust-pretest burden show --question trust_2

{
  "paths": {
    "count": 1,
    "shortest": {"questions":["trust_1","trust_2","comment"], "seconds":47.86},
    "typical":  {"questions":["trust_1","trust_2","comment"], "seconds":47.86},
    "longest":  {"questions":["trust_1","trust_2","comment"], "seconds":47.86}
  },
  "warning": "Pre-fielding duration is estimated, not observed human completion time."
}

Edit analysis.burden in messick.yaml when the intended population or response mode warrants different assumptions. Every analysis artifact retains the exact configuration snapshot and hash.

08 Build a simulated cognitive pretest

A cognitive pretest asks simulated respondents to paraphrase each item, explain an answer process, identify ambiguity and missing options, surface assumptions or sensitivity, distinguish adjacent constructs, and describe difficulty.

messick --project-dir trust-pretest agents create \
  --input agents.json \
  --output edsl_jobs/agents.ep

messick --project-dir trust-pretest models create \
  --model <exact-model-id> \
  --output edsl_jobs/models.ep

messick --project-dir trust-pretest pretest plan \
  --mode cognitive \
  --agents edsl_jobs/agents.ep \
  --models edsl_jobs/models.ep

messick --project-dir trust-pretest job generate \
  --plan plan_0001 \
  --output edsl_jobs/cognitive_pretest.ep

The package builders create loadable git-backed EDSL artifacts, so agents do not need to guess serialization formats. pretest plan validates the complete execution design before staging immutable artifacts; a rejected plan leaves no partial run state. job generate creates a real git-backed EDSL Jobs package and immediately loads it again to verify portability. It does not call a model. Its JSON response supplies the exact handoff:

ep inspect trust-pretest/edsl_jobs/cognitive_pretest.ep
ep jobs cost trust-pretest/edsl_jobs/cognitive_pretest.ep

# Only after explicit approval of the model and estimated cost:
ep run trust-pretest/edsl_jobs/cognitive_pretest.ep \
  --output trust-pretest/data/results/cognitive_pretest.ep

messick --project-dir trust-pretest results ingest \
  --plan plan_0001 \
  --results trust-pretest/data/results/cognitive_pretest.ep

After ingestion, analyze the source ID returned by Messick:

messick --project-dir trust-pretest pretest analyze --source source_0001
messick --project-dir trust-pretest pretest findings \
  --source source_0001 \
  --limit 20

# Focus without loading the full analysis artifact:
messick --project-dir trust-pretest pretest findings \
  --source source_0001 \
  --question trust_2 \
  --limit 10

Structured probe fields become question-linked candidate findings and issues. The bounded findings command returns pagination metadata and source labels without requiring an agent to load the underlying analysis JSON. Excerpts remain diagnostic hypotheses about possible response processes—not observations of employee cognition.

09 Run a small behavioral pilot

The behavioral mode administers the actual Survey to registered EDSL agents. Start small, inspect distributions and branch traversal, then decide whether a larger simulated sample has enough expected value to justify its cost.

messick --project-dir trust-pretest pretest plan \
  --mode behavioral \
  --agents agents.ep \
  --models models.ep

messick --project-dir trust-pretest job generate \
  --plan plan_0002 \
  --output edsl_jobs/behavioral_pilot.ep

ep inspect trust-pretest/edsl_jobs/behavioral_pilot.ep
ep jobs cost trust-pretest/edsl_jobs/behavioral_pilot.ep
# approve, then ep run ...
messick --project-dir trust-pretest results ingest \
  --plan plan_0002 --results data/results/behavioral_pilot.ep
messick --project-dir trust-pretest pretest analyze --source source_0002

For an entirely local tour, the repository includes examples/simulation_only/results.ep. It contains five deliberately simple fixture rows:

10 Examine scale coherence without overclaiming

messick --project-dir trust-pretest scale analyze \
  --scale workplace_trust \
  --source source_0002

Messick uses the declared 1–5 bounds, reverse-scores trust_2, and computes reproducible classical diagnostics: item means and standard deviations, Pearson inter-item correlations, corrected item-total correlations, Cronbach’s alpha, alpha if deleted, approximate one-factor omega, exploratory eigenvalues, and first-component loadings.

{
  "scale_id": "workplace_trust",
  "source_id": "source_0002",
  "n": 5,
  "item_count": 2,
  "cronbach_alpha": 1.0,
  "mcdonald_omega": 1.0,
  "warnings": [{"code":"SMALL_SAMPLE"}]
}

The perfect fixture result is a warning sign, not a triumph. There are only five constructed rows, and one reverse-scored item is a deterministic mirror of the other. The correct conclusion is that the scoring implementation behaves as expected on a known fixture.

11 Adjudicate the issue and create a new revision

Inspect the issue ledger, record a decision with rationale, revise the EDSL Survey outside the private ledger, and import it as a new immutable revision.

messick --project-dir trust-pretest issue list \
  --question trust_2 --status open

messick --project-dir trust-pretest issue adjudicate issue_0001 \
  --decision revise \
  --rationale "Double negation obscures direction and creates avoidable response-mapping risk."

# Edit/export the EDSL Survey as revised_survey.ep, then:
messick --project-dir trust-pretest instrument import \
  --survey revised_survey.ep \
  --message "Replace trust_2 with direct distrust wording"

messick --project-dir trust-pretest instrument compare \
  --from v001 --to v002
messick --project-dir trust-pretest burden compare \
  --from v001 --to v002

A clearer replacement might be “Management often breaks its promises.” Whether that wording is construct-relevant, sufficiently specific, and culturally appropriate remains a research judgment. Messick preserves the original issue, decision, rationale, evidence IDs, and resulting revision instead of erasing the history.

12 Add human evidence only when the claim warrants it

Human fielding is an optional escalation, not a gate that makes simulation-first pretesting incomplete. Freeze the revision, generate a plan, field through Humanize or another approved process, and ingest the export with sample provenance.

messick --project-dir trust-pretest fielding plan --revision v002

# Deployment and participation happen outside Messick, with authorization.
messick --project-dir trust-pretest responses ingest \
  --source-type human \
  --input humanize_export.csv \
  --instrument-revision v002 \
  --input-format csv \
  --sample-description "N=84 full-time US employees; August 2026 pilot"

messick --project-dir trust-pretest source compare \
  --left source_0002 --right source_0003

The comparison artifact displays estimates separately, includes both sample sizes, records administration differences, and sets pooled: false and equivalence_claimed: false. Timestamped human rows produce observed median and percentile durations separately from earlier burden estimates.

13 Evaluate the declared claim—not the survey globally

messick --project-dir trust-pretest validation evaluate \
  --intent trust_mean
messick --project-dir trust-pretest validate --strict

Possible statuses are supported, challenged, inconclusive, not_evaluated, and requires_human_evidence. A consequential use without suitable human evidence receives the last status; that limits the claim but does not fail a completed simulation-only pretest.

{
  "intent_id": "trust_mean",
  "status": "supported",
  "reasons": [
    "Required evidence exists at the declared tier and no blocking challenge remains."
  ],
  "limitations": [
    "Status is bounded to the declared evidence tier; it is not global instrument validity."
  ]
}

14 Generate bounded context, then author the narrative

messick --project-dir trust-pretest report context \
  --output analysis/messick_report_context.json
messick --project-dir trust-pretest report template \
  --output analysis/messick_report_template.md

The context inventories the current instrument and revision history, intents, scales, evidence sources separated by type, issues, decisions, analyses, validation statuses, limitations, and canonical artifact paths with hashes. Large tables remain in analysis artifacts rather than flooding an agent’s context.

A proportionate simulation-only conclusion would be: “A deterministic review and simulated cognitive pretest identified a candidate response-mapping problem in the reverse-worded item. The item was rewritten and the scoring pipeline reproduced known fixture behavior. Human interpretation and measurement performance have not been established.”

15 Let a coding agent resume from durable state

The README contains a complete copy-and-paste bootstrap prompt. Once installed, the durable loop is intentionally small:

messick --project-dir trust-pretest agent status
messick --project-dir trust-pretest agent next

# Perform or review the returned action, then ask again:
messick --project-dir trust-pretest agent next

Each recommended action includes a complete working directory, argument array, mutation flag, approval flag, reason, and—when it needs a JSON input file—a schema. The same project state produces the same recommendation. Terminal state is explicit.

16 Completion checklist

1. The intended construct, interpretation, population, use, evidence tier, and tolerances are explicit.
2. The exact Survey revision and every evidence artifact have immutable paths and hashes.
3. Static structure, branching, options, scoring, complexity, and burden checks were reviewed.
4. Every severe issue is adjudicated with a rationale and evidence IDs.
5. Any model run used the exact Messick-generated Jobs package through ep after approval.
6. Simulated, human, and benchmark sources remain visibly separate.
7. Scale results are interpreted with sample-size and identifiability limitations.
8. Every active intent has a claim-specific status.
9. The report context names unresolved limitations and the optional next evidence with highest expected value.
10. The final narrative avoids global validity language.

messick --project-dir trust-pretest agent status
messick --project-dir trust-pretest validate --strict
messick --project-dir trust-pretest report context