The worked example
One question. Two forecasters. A record of the whole run.
Will a fictional factory ship by tomorrow’s deadline? We register this question, give each forecaster its own research session, pause while a tool is waiting, resume, freeze the forecasts, and score them after recording an outcome.
The two forecasters are small Python fixtures: arm a always submits 25%; arm b always submits 75%. Each runs twice. Their research returns a fictional readiness report. These authored choices make the mechanics reproducible without API keys. They do not test a model’s forecasting skill.
| Choice | This run | Why it matters |
|---|---|---|
| Instrument | One versioned question; unconditional probability | Every session answers the same event. |
| Replication | Two arms × two whole sessions | The session is the unit of repetition. |
| Research | Independent live evidence; one page receipt and a readiness assessment per session | Evidence arrives during execution and is preserved. |
| Outcome | Authored YES, recorded after finalization | The scoring stage uses a later record. |
In a real study, the workers would call the chosen model and research services. Vorhersage supplies registration, durable execution, evidence records, validation, reports, and evaluation. Configuring a worker is part of the study.
01 / Get ready
Start with the runnable example.
Use Python 3.11 or newer and a checkout of Vorhersage containing session-study. From that checkout’s root, install the package in a virtual environment. If it is already installed in your active environment, continue with the example download.
Install from a source checkout
python3 -m venv .venvsource .venv/bin/activatepython -m pip install -e .Use vorhersage session-study --help to check that this checkout includes the study runner.
Download the two-file example, save it as session-tutorial.zip in your working directory, and extract it:
python -m zipfile -e session-tutorial.zip session-tutorialThe archive contains prepare.py, which writes dated JSON inputs and reads saved receipts, and worker.py, which supplies deterministic model and tool responses. Keep the same working directory and Python environment throughout this tutorial.
The helper creates a new forecast-study directory. Choose a fresh destination; it deliberately refuses to reuse one.
python session-tutorial/prepare.py inputs ./forecast-studyShow command output
Complete captured response. Measured command time: 0.0318 s. Full execution record.
{
"stage": "inputs",
"written": [
"/private/tmp/vorhersage-docs-tutorial-run-v2/forecast-study/question.json",
"/private/tmp/vorhersage-docs-tutorial-run-v2/forecast-study/condition.json"
],
"provider_calls": 0
}Give this walkthrough to an agent
Run the Vorhersage session-study tutorial using the downloaded prepare.py and worker.py. Register both arms before execution, capture the waiting state, resume the same study, audit all sessions, export comparison.html, then record the synthetic outcome and evaluate. Preserve CLI receipts and report actual versus fixture costs separately.02 / Define the event
Make the probability unambiguous.
Initialize the project that will hold the questions, immutable artifacts, session history, and outcomes. The global --project argument selects this same directory for every CLI command.
vorhersage --project ./forecast-study init --name "Forecasting tutorial"Show command output
Complete captured response. Measured command time: 0.0734 s. Full execution record.
{
"schema_version": "1",
"status": "ok",
"command": "init",
"data": {
"project": "/private/tmp/vorhersage-docs-tutorial-run-v2/forecast-study",
"database": "/private/tmp/vorhersage-docs-tutorial-run-v2/forecast-study/.vorhersage/state.sqlite",
"name": "Forecasting tutorial"
},
"warnings": [],
"errors": [],
"next_actions": []
}The helper wrote explicit YES, NO, and void rules and a deadline tomorrow. This is a simulation, with a fictional dispatch source. Register that file to fix version 1 of the question.
vorhersage --project ./forecast-study question add --from ./forecast-study/question.jsonShow command output
Complete captured response. Measured command time: 0.0590 s. Full execution record.
{
"schema_version": "1",
"status": "ok",
"command": "question",
"data": {
"question_id": "readiness",
"version": 1
},
"warnings": [],
"errors": [],
"next_actions": []
}Inspect question.json
{
"id": "readiness",
"text": "Will the fictional factory ship by tomorrow's deadline?",
"yes": "A qualifying shipment is recorded by the deadline.",
"no": "No qualifying shipment is recorded.",
"void": "The fictional event is withdrawn.",
"event_deadline": "2026-09-13T15:23:16.086470+00:00",
"resolve_after": "2026-09-13T15:23:16.086470+00:00",
"resolution_source": "urn:fixture:dispatch",
"event_group": "factory",
"domain": "operations",
"profile": "general",
"kind": "simulation"
}Captured input fileNext register the unconditional condition: no added assumptions about the world. Save the returned envelope so the helper can use the actual condition ID in the study specification.
vorhersage --project ./forecast-study condition add --from ./forecast-study/condition.json > ./forecast-study/condition-result.jsonShow command output
Complete captured response. Measured command time: 0.0632 s. Full execution record.
{
"schema_version": "1",
"status": "ok",
"command": "condition",
"data": {
"condition_id": "condition_b1dbf5055d22a0fb8cef1031"
},
"warnings": [],
"errors": [],
"next_actions": []
}The response shown above is the JSON written to the file by the redirect.
03 / Register the design
Freeze the comparison before it starts.
The study combines a shared question template with two arms and two repetitions. The helper inserts this run’s condition ID, current information cutoff, and absolute paths to the worker. It writes the specification; registration happens in the next native command.
python session-tutorial/prepare.py study ./forecast-studyShow command output
Complete captured response. Measured command time: 0.0333 s. Full execution record.
{
"stage": "study",
"written": [
"/private/tmp/vorhersage-docs-tutorial-run-v2/forecast-study/study.json"
],
"provider_calls": 0
}Inspect study.json (excerpt)
{
"id": "tutorial-study",
"repetitions": 2,
"order_seed": "tutorial-1",
"evidence_policy": "independent_live",
"session_template": {
"id": "template",
"wave": "tutorial",
"forecaster": "fixture",
"protocol": "synthetic-v1",
"repetition": 1,
"mode": "simulation",
"information_as_of": "2026-09-12T15:23:16.315145+00:00",
"questions": [
{
"question_id": "readiness",
"version": 1
}
],
"condition_ids": [
"condition_b1dbf5055d22a0fb8cef1031"
],
"packet_ids": [],
"relation_ids": [],
"numeric_forecasts": [],
"bindings": [],
"provenance": {
"kind": "native",
"source": "Deterministic documentation fixture"
},
"configuration": {}
}
}Captured input fileEach arm permits at most five model calls, five tool attempts, and $1 of reported usage per session. It requires a successful tool receipt, one unique page, and an assessment of readiness. The model worker’s only difference between arms is the authored probability.
Inspect study.json (excerpt)
{
"arms": [
{
"id": "a",
"configuration": {
"fixture_probability": 0.25
},
"execution": {
"evidence_policy": "live",
"defer_bindings": true,
"budget": {
"max_model_calls": 5,
"max_searches": 5,
"max_cost_usd": 1
},
"research": {
"minimum_successful_tools": 1,
"minimum_unique_pages": 1,
"domains": [
"readiness"
]
},
"requirements": [
{
"id": "fixture",
"description": "No real network calls",
"expected": {
"network_calls": 0
}
}
],
"worker": {
"command": [
"/Users/johnhorton/tools/ep/vorhersage/.venv/bin/python",
"/private/tmp/vorhersage-docs-tutorial-run-v2/session-tutorial/worker.py"
],
"config": {
"probability": 0.25
},
"timeout_seconds": 10
},
"tool_worker": {
"command": [
"/Users/johnhorton/tools/ep/vorhersage/.venv/bin/python",
"/private/tmp/vorhersage-docs-tutorial-run-v2/session-tutorial/worker.py"
],
"config": {},
"timeout_seconds": 10
}
}
},
{
"id": "b",
"configuration": {
"fixture_probability": 0.75
},
"execution": {
"evidence_policy": "live",
"defer_bindings": true,
"budget": {
"max_model_calls": 5,
"max_searches": 5,
"max_cost_usd": 1
},
"research": {
"minimum_successful_tools": 1,
"minimum_unique_pages": 1,
"domains": [
"readiness"
]
},
"requirements": [
{
"id": "fixture",
"description": "No real network calls",
"expected": {
"network_calls": 0
}
}
],
"worker": {
"command": [
"/Users/johnhorton/tools/ep/vorhersage/.venv/bin/python",
"/private/tmp/vorhersage-docs-tutorial-run-v2/session-tutorial/worker.py"
],
"config": {
"probability": 0.75
},
"timeout_seconds": 10
},
"tool_worker": {
"command": [
"/Users/johnhorton/tools/ep/vorhersage/.venv/bin/python",
"/private/tmp/vorhersage-docs-tutorial-run-v2/session-tutorial/worker.py"
],
"config": {},
"timeout_seconds": 10
}
}
}
]
}Captured input fileindependent_live lets each session collect its own evidence. A real comparison under this policy can differ because of both model behavior and research results. A frozen evidence study would answer a different methodological question.
vorhersage --project ./forecast-study session-study add --from ./forecast-study/study.json > ./forecast-study/study-result.jsonShow command output
Complete captured response. Measured command time: 0.0572 s. Full execution record.
{
"schema_version": "1",
"status": "ok",
"command": "session-study",
"data": {
"study_id": "session_study_3b7697cdcd7c013c0f338bc3",
"sessions": 4
},
"warnings": [],
"errors": [],
"next_actions": []
}Use the returned data.study_id wherever STUDY_ID appears below. SESSION_ID means the first trial’s session ID from the status response in the next step. Do not copy the captured run’s IDs into your own project.
04 / Research and resume
A waiting tool is part of the record.
Give the runner five controller steps. The seeded schedule visits all four sessions to queue research, then starts the first tool. This fixture deliberately returns waiting on its first tool invocation.
vorhersage --project ./forecast-study session-study run STUDY_ID --max-steps 5Show command output
Selected data fields from the captured response. Measured command time: 0.3261 s. Full execution record.
{
"study_id": "session_study_3b7697cdcd7c013c0f338bc3",
"complete": false,
"next_position": 1,
"usage": {
"searches": 0,
"model_calls": 4,
"cost_usd": 0.04
}
}--max-steps bounds controller visits, including polls; it is separate from each session’s model, tool, and dollar budgets. Inspect the study and the first trial’s session to see what remains unfinished.
vorhersage --project ./forecast-study session-study status STUDY_IDShow command output
Selected data fields from the captured response. Measured command time: 0.0550 s. Full execution record.
{
"complete": false,
"next_position": 1,
"trials": [
{
"id": "session_trial_f5f686ba115e12d7f1021e83",
"arm": "a",
"input_manifest": {
"session_fd5b8a19854f8f02d507c88f": "c835fa29170c4a9de470172e095294f05cf2c23f8fffefbf898cac3a0fa7cf23"
},
"position": 0,
"repetition": 2,
"session_id": "session_fd5b8a19854f8f02d507c88f",
"study_id": "session_study_3b7697cdcd7c013c0f338bc3",
"disposition": "waiting",
"revision": 5,
"cells": 0,
"expected_cells": 1,
"usage": {
"searches": 0,
"model_calls": 1,
"cost_usd": 0.01
},
"unknown_usage_attempts": [
"attempt_0a3baf4021379fcd31f365e8"
],
"protocol_audit": {
"requirements": [
{
"description": "No real network calls",
"expected": {
"network_calls": 0
},
"id": "fixture",
"status": "unverifiable",
"observation": null
}
],
"amendments": [],
"protocol_fidelity": "unverifiable",
"research": {
"counts": {
"minimum_successful_tools": 0,
"minimum_unique_pages": 0,
"minimum_unique_searches": 0,
"minimum_recent_searches": 0,
"minimum_followup_searches": 0
},
"missing": {
"minimum_successful_tools": 1,
"minimum_unique_pages": 1
},
"domains": [
{
"domain": "readiness",
"disposition": "missing"
}
],
"ready": false,
"limitations": [
"Receipts and assessments are caller-reported; counts do not establish research quality.",
"Unknown domain assessments count as explicit coverage, not evidence of knowledge."
]
},
"complete_grid": false,
"disposition": "waiting",
"usage": {
"searches": 0,
"model_calls": 1,
"cost_usd": 0.01
},
"unknown_usage_attempts": [
"attempt_0a3baf4021379fcd31f365e8"
],
"limitations": [
"Matching is against registered requirements and reported observations, not independent certification.",
"Grid completeness, protocol fidelity, and predictive accuracy are separate properties."
]
}
},
{
"id": "session_trial_b94b25d10af8e578a8532e3f",
"arm": "b",
"input_manifest": {
"session_ebbafe6b9278b8850bb9f249": "f67a1eacf25c4956556110aaaf9adac4029be68fafb37d75dc26a2e9203be021"
},
"position": 1,
"repetition": 1,
"session_id": "session_ebbafe6b9278b8850bb9f249",
"study_id": "session_study_3b7697cdcd7c013c0f338bc3",
"disposition": "open",
"revision": 3,
"cells": 0,
"expected_cells": 1,
"usage": {
"searches": 0,
"model_calls": 1,
"cost_usd": 0.01
},
"unknown_usage_attempts": [],
"protocol_audit": {
"requirements": [
{
"description": "No real network calls",
"expected": {
"network_calls": 0
},
"id": "fixture",
"status": "unverifiable",
"observation": null
}
],
"amendments": [],
"protocol_fidelity": "unverifiable",
"research": {
"counts": {
"minimum_successful_tools": 0,
"minimum_unique_pages": 0,
"minimum_unique_searches": 0,
"minimum_recent_searches": 0,
"minimum_followup_searches": 0
},
"missing": {
"minimum_successful_tools": 1,
"minimum_unique_pages": 1
},
"domains": [
{
"domain": "readiness",
"disposition": "missing"
}
],
"ready": false,
"limitations": [
"Receipts and assessments are caller-reported; counts do not establish research quality.",
"Unknown domain assessments count as explicit coverage, not evidence of knowledge."
]
},
"complete_grid": false,
"disposition": "open",
"usage": {
"searches": 0,
"model_calls": 1,
"cost_usd": 0.01
},
"unknown_usage_attempts": [],
"limitations": [
"Matching is against registered requirements and reported observations, not independent certification.",
"Grid completeness, protocol fidelity, and predictive accuracy are separate properties."
]
}
},
{
"id": "session_trial_d6ebf716cb6e36910d186116",
"arm": "b",
"input_manifest": {
"session_533a1185f1d980bc14a2ade3": "071254204f274ff01e17784473bdfe814e645f052d14e7432a5dafb2ca6d3577"
},
"position": 2,
"repetition": 2,
"session_id": "session_533a1185f1d980bc14a2ade3",
"study_id": "session_study_3b7697cdcd7c013c0f338bc3",
"disposition": "open",
"revision": 3,
"cells": 0,
"expected_cells": 1,
"usage": {
"searches": 0,
"model_calls": 1,
"cost_usd": 0.01
},
"unknown_usage_attempts": [],
"protocol_audit": {
"requirements": [
{
"description": "No real network calls",
"expected": {
"network_calls": 0
},
"id": "fixture",
"status": "unverifiable",
"observation": null
}
],
"amendments": [],
"protocol_fidelity": "unverifiable",
"research": {
"counts": {
"minimum_successful_tools": 0,
"minimum_unique_pages": 0,
"minimum_unique_searches": 0,
"minimum_recent_searches": 0,
"minimum_followup_searches": 0
},
"missing": {
"minimum_successful_tools": 1,
"minimum_unique_pages": 1
},
"domains": [
{
"domain": "readiness",
"disposition": "missing"
}
],
"ready": false,
"limitations": [
"Receipts and assessments are caller-reported; counts do not establish research quality.",
"Unknown domain assessments count as explicit coverage, not evidence of knowledge."
]
},
"complete_grid": false,
"disposition": "open",
"usage": {
"searches": 0,
"model_calls": 1,
"cost_usd": 0.01
},
"unknown_usage_attempts": [],
"limitations": [
"Matching is against registered requirements and reported observations, not independent certification.",
"Grid completeness, protocol fidelity, and predictive accuracy are separate properties."
]
}
},
{
"id": "session_trial_499f64167a6ca88e732fa69f",
"arm": "a",
"input_manifest": {
"session_23abaa288b6648a2c5317a9f": "079e789946e0dd952e07b1cf62919745384d4dd33d39d4ad30b21ca868a8a5e1"
},
"position": 3,
"repetition": 1,
"session_id": "session_23abaa288b6648a2c5317a9f",
"study_id": "session_study_3b7697cdcd7c013c0f338bc3",
"disposition": "open",
"revision": 3,
"cells": 0,
"expected_cells": 1,
"usage": {
"searches": 0,
"model_calls": 1,
"cost_usd": 0.01
},
"unknown_usage_attempts": [],
"protocol_audit": {
"requirements": [
{
"description": "No real network calls",
"expected": {
"network_calls": 0
},
"id": "fixture",
"status": "unverifiable",
"observation": null
}
],
"amendments": [],
"protocol_fidelity": "unverifiable",
"research": {
"counts": {
"minimum_successful_tools": 0,
"minimum_unique_pages": 0,
"minimum_unique_searches": 0,
"minimum_recent_searches": 0,
"minimum_followup_searches": 0
},
"missing": {
"minimum_successful_tools": 1,
"minimum_unique_pages": 1
},
"domains": [
{
"domain": "readiness",
"disposition": "missing"
}
],
"ready": false,
"limitations": [
"Receipts and assessments are caller-reported; counts do not establish research quality.",
"Unknown domain assessments count as explicit coverage, not evidence of knowledge."
]
},
"complete_grid": false,
"disposition": "open",
"usage": {
"searches": 0,
"model_calls": 1,
"cost_usd": 0.01
},
"unknown_usage_attempts": [],
"limitations": [
"Matching is against registered requirements and reported observations, not independent certification.",
"Grid completeness, protocol fidelity, and predictive accuracy are separate properties."
]
}
}
]
}vorhersage --project ./forecast-study session show SESSION_IDShow command output
Selected data fields from the captured response. Measured command time: 0.0576 s. Full execution record.
{
"session_id": "session_fd5b8a19854f8f02d507c88f",
"disposition": "waiting",
"revision": 5,
"usage": {
"searches": 0,
"model_calls": 1,
"cost_usd": 0.01
},
"unknown_usage_attempts": [
"attempt_0a3baf4021379fcd31f365e8"
]
}The pending tool has no final usage receipt yet. Its attempt ID and continuation have already been saved. Run the same study again: the runner polls that attempt and completes the remaining work.
vorhersage --project ./forecast-study session-study run STUDY_ID --max-steps 20Show command output
Selected data fields from the captured response. Measured command time: 0.7133 s. Full execution record.
{
"complete": true,
"usage": {
"searches": 4,
"model_calls": 8,
"cost_usd": 0.1
}
}All four sessions now have one forecast cell and a finalization record. Together they report eight fixture model calls and four fixture tool attempts. Polling does not create a second billable attempt in the ledger.
Explore the captured first session
Actual event payloads from this run. Start-event request bodies are omitted in this view. Revisions 5 and 6 share a tool attempt ID: waiting, then completed. Submissions and finalization are separate immutable records.
A worker refusal or exhausted budget remains visible as a terminal disposition. An interrupted attempt with uncertain usage requires reconciliation before more work can be authorized. Resuming a study does not silently change the protocol or increase its budget.
05 / Inspect the research
Completion and protocol fidelity answer different questions.
The audit asks whether the session filled the grid, satisfied its research requirements, and reported observations matching the registered protocol. Inspect the first session:
vorhersage --project ./forecast-study session audit SESSION_IDShow command output
Selected data fields from the captured response. Measured command time: 0.0566 s. Full execution record.
{
"protocol_fidelity": "matched",
"research": {
"counts": {
"minimum_successful_tools": 1,
"minimum_unique_pages": 1,
"minimum_unique_searches": 0,
"minimum_recent_searches": 0,
"minimum_followup_searches": 0
},
"missing": {},
"domains": [
{
"domain": "readiness",
"disposition": "assessed",
"evidence_refs": [
{
"packet_id": "pkt_8aa4f1c9b9baec369ce5db29",
"record_id": "status"
}
],
"rationale": "The synthetic packet establishes readiness in this fixture.",
"event_id": "joint_event_edcb0b66d6118e8ac56c6302"
}
],
"ready": true,
"limitations": [
"Receipts and assessments are caller-reported; counts do not establish research quality.",
"Unknown domain assessments count as explicit coverage, not evidence of knowledge."
]
},
"complete_grid": true,
"disposition": "finalized",
"usage": {
"searches": 1,
"model_calls": 2,
"cost_usd": 0.025
},
"requirements": [
{
"description": "No real network calls",
"expected": {
"network_calls": 0
},
"id": "fixture",
"status": "matched",
"observation": {
"actual": {
"network_calls": 0
},
"basis": "worker_reported",
"evidence_refs": [],
"failed": false,
"note": "Deterministic fixture",
"requirement_id": "fixture",
"event_id": "joint_event_f47c51972316f85afbaa85a3"
}
}
]
}The page receipt references a captured packet and its status record. The readiness assessment references that same evidence. The protocol observation says zero network calls, matching the fixture requirement.
Read “matched” in context: the observation’s basis is worker_reported. Matching that report against the registered requirement is not independent verification. Likewise, counting a page establishes coverage, not research quality. Here the packet explicitly labels its contents synthetic.
For the whole comparison, export a self-contained HTML report. It retains each session’s status, usage, audit, probabilities, and evidence references.
vorhersage --project ./forecast-study session-study report STUDY_ID --output ./forecast-study/comparison.htmlShow command output
Complete captured response. Measured command time: 0.0595 s. Full execution record.
{
"schema_version": "1",
"status": "ok",
"command": "session-study",
"data": {
"path": "/private/tmp/vorhersage-docs-tutorial-run-v2/forecast-study/comparison.html",
"sessions": 4,
"cells": 1
},
"warnings": [],
"errors": [],
"next_actions": []
}Open the captured comparison →
The report’s dollar column contains fixture usage, totaling $0.10. No money was spent on these workers. Its one row represents one shared question–condition cell, with four session forecasts.
06 / Resolve and evaluate
Keep the outcome after the forecasts.
Once every session is finalized, the helper writes a forecast cutoff, then a later synthetic YES outcome. This is an authored simulation resolution; the readiness packet cited by the fixture is not evidence that a real shipment occurred. The outcome file was unavailable to the workers during their run.
python session-tutorial/prepare.py resolution ./forecast-studyShow command output
Complete captured response. Measured command time: 0.0516 s. Full execution record.
{
"stage": "resolution",
"written": [
"/private/tmp/vorhersage-docs-tutorial-run-v2/forecast-study/forecast-cutoff.json",
"/private/tmp/vorhersage-docs-tutorial-run-v2/forecast-study/resolution.json"
],
"provider_calls": 0
}Inspect resolution.json
{
"question_id": "readiness",
"question_version": 1,
"outcome": "yes",
"reason": "Authored synthetic outcome for the tutorial",
"known_at": "2026-09-12T15:23:17.691274+00:00",
"evidence_refs": [
{
"packet_id": "pkt_8aa4f1c9b9baec369ce5db29",
"record_id": "status"
}
],
"previous_resolution_id": null,
"idempotency_key": "tutorial-resolution"
}Captured input fileRecord the outcome with the native resolution command.
vorhersage --project ./forecast-study resolve --from ./forecast-study/resolution.jsonShow command output
Complete captured response. Measured command time: 0.0547 s. Full execution record.
{
"schema_version": "1",
"status": "ok",
"command": "resolve",
"data": {
"resolution_id": "resolution_6ab415a4c9f947d89a1f",
"outcome": "yes"
},
"warnings": [],
"errors": [],
"next_actions": []
}Prepare an evaluation policy selecting all four sessions, the saved forecast cutoff, and an outcome cutoff after resolution. allow_source_reported: false keeps this evaluation tied to locally recorded timing.
python session-tutorial/prepare.py evaluation ./forecast-studyShow command output
Complete captured response. Measured command time: 0.0506 s. Full execution record.
{
"stage": "evaluation",
"written": [
"/private/tmp/vorhersage-docs-tutorial-run-v2/forecast-study/evaluation.json"
],
"provider_calls": 0
}Inspect evaluation.json
{
"session_ids": [
"session_fd5b8a19854f8f02d507c88f",
"session_ebbafe6b9278b8850bb9f249",
"session_533a1185f1d980bc14a2ade3",
"session_23abaa288b6648a2c5317a9f"
],
"cutoff": "2026-09-12T15:23:17.689989+00:00",
"resolution_as_of": "2026-09-12T15:23:17.796669+00:00",
"allow_source_reported": false
}Captured input fileEvaluate the eligible unconditional forecasts on their common resolved question. The evaluator records its policy, selected inputs, and exclusions with the result.
vorhersage --project ./forecast-study session evaluate --from ./forecast-study/evaluation.jsonShow command output
Selected data fields from the captured response. Measured command time: 0.0630 s. Full execution record.
{
"evaluation_id": "session_evaluation_064b478e4b604a039aae",
"arms": [
{
"forecaster": "a",
"repetitions": 2,
"matched_questions": 1,
"mean_session_brier": 0.5625
},
{
"forecaster": "b",
"repetitions": 2,
"matched_questions": 1,
"mean_session_brier": 0.0625
}
],
"exclusions": []
}Explore the scoring rule
| Arm | Probability | Mean session Brier |
|---|---|---|
| a · two sessions | 25% | 0.5625 |
| b · two sessions | 75% | 0.0625 |
With YES, arm b is closer to the outcome.
Changing this control only recalculates the illustration. It does not alter the saved resolution or evaluation.
Under the captured YES outcome, (0.75 − 1)² = 0.0625. Both repetitions in each arm are identical, so averaging their scores leaves the score unchanged. This is one resolved event, not four independent observations. It establishes that the workflow and scoring work; it cannot establish comparative forecasting skill or calibration.
Conditional forecasts need a suitable conditional evaluation design. This evaluator scores unconditional cells; a complete conditional grid is not automatically an accuracy benchmark.
07 / Keep the receipts
Leave an inspectable project behind.
Finish by checking SQLite integrity and the stored artifact hashes.
vorhersage --project ./forecast-study doctorShow command output
Complete captured response. Measured command time: 0.0572 s. Full execution record.
{
"schema_version": "1",
"status": "ok",
"command": "doctor",
"data": {
"ok": true,
"artifacts_checked": 109,
"sqlite_integrity": "ok"
},
"warnings": [],
"errors": [],
"next_actions": []
}This capture checked 109 artifacts. The complete execution took 1.864 seconds, measured from input preparation through the integrity check on September 12, 2026. Installation, downloads, and building this page are outside that timer. Actual provider calls: 0. Actual provider cost: $0.
| Artifact | Purpose |
|---|---|
| Your forecast-study directory | The native project, immutable artifacts, receipts, inputs, and exported report. Keep the whole directory. |
| comparison.html | Portable, interactive comparison of forecasts and execution records. |
| Evaluation receipt | Policy, selected forecasts, exclusions, and arm scores from this run. |
| Capture manifest | Timing, costs, source hashes, fixture hashes, and the displayed event trace. |
To regenerate the published receipts from a checkout, run the capture script with a fresh destination. It executes the same sequence of CLI operations, asserting the waiting state, completed study, and expected scores.
python examples/live_sessions/capture_tutorial.py ./tutorial-captureThen rebuild this static page and its downloadable kit from those receipts:
python examples/live_sessions/build_tutorial.pyNext / Reproduce a real study
Replace the fixture with a declared research protocol.
The AIRO replication motivated this session workflow: a model researches and answers an entire instrument in one continuing conversation. Reproducing it means preserving more than the final probability table. The model settings, conversation, tools, stopping rules, evidence timing, and failures all matter.
| In this tutorial | In the replication |
|---|---|
| One unconditional question | Register the paper’s questions, horizons, conditions, and coherence relations as one instrument. |
| Two fixed Python responders | Declare a model and transport per arm, with its reasoning settings, output limits, and full conversation records. |
| One fictional page | Implement search and page-reading workers; retain queries, results, retrieved text, and source-linked assessments. |
| One readiness requirement | Register the authors’ research and effort requirements before execution; audit observations and record deviations. |
| An authored YES outcome | Separate reproduction of saved analyses from prospective scoring. Many long-horizon outcomes remain unresolved. |
Firecrawl is optional: the package’s worker contract is service-independent. The authors’ protocol uses Tavily. Match that protocol when reproducing their experiment; a different research service defines a changed configuration that should be recorded.
The existing AIRO example and its replication notes cover the saved-output checks and subsequent model runs. Those are separate studies from the synthetic capture on this page. EDSL transport improvements are tracked in reasoning settings, Anthropic streaming, and native session support; this tutorial does not imply that those changes have shipped.