A question.
An experiment.
A result.

The idea behind AISS

Make the hypothesis executable.

Automated Social Science: Language Models as Scientist and Subjects, by Benjamin S. Manning, Kehang Zhu, and John J. Horton, uses structural causal models to connect hypotheses, agent construction, experiments, and estimation. An LLM can help design a study and participate in its simulations; observations from the interactions provide data for testing the proposed relationships.

AISS implements those connections on top of EDSL. A study specifies what varies, who knows it, how participants interact, what counts as an outcome, and which comparison will answer the question.

We will use one small negotiation throughout. The offline run uses transparent scripted behavior to check the machinery. Step 8 replaces the callbacks with an EDSL model so you can collect model-generated observations.

01 / Set up

Start from a source checkout.

Use Python 3.12 and uv. The checkout includes the example scripts and the pinned EDSL conversation runtime. If you already have AISS installed locally, start in its repository root and run the sync command.

git clone https://github.com/expectedparrot/aiss.git
cd aiss
uv sync --locked --extra dev --extra experiments

The companion script is docs/examples/negotiation_tutorial.py. Its proposal is docs/manual/examples/negotiation.json. All shell commands below run from the repository root.

02 / State the question

Does a larger budget increase agreement?

A buyer and seller negotiate over one item. The buyer knows its maximum budget. The seller knows its minimum acceptable price, called seller_cost in this example. Neither receives the other participant’s private value before the conversation.

Crossing two budgets with two costs gives four treatment cells. Two repetitions per cell give eight negotiations. The actors alternate turns, with a maximum of six utterances and a semantic stopping rule for agreement or impasse.

The model is agreement ~ buyer_budget + seller_cost + buyer_budget × seller_cost. The interaction permits the budget effect to depend on seller cost. Our primary question concerns the average budget effect across the two cost levels.

{
  "type": "causal_study_proposal",
  "version": 1,
  "name": "manual-negotiation",
  "question": {
    "text": "Does a larger private budget increase agreement?",
    "population": "Simulated buyer-seller pairs",
    "setting": "One conversation about a single item",
    "hypotheses": ["A larger buyer budget increases agreement."]
  },
  "roles": [
    {
      "name": "buyer",
      "goal": "Buy the item at a satisfactory price",
      "constraints": ["Do not agree to pay above your budget"],
      "executor": {"kind": "llm", "model_policy": "study-default"}
    },
    {
      "name": "seller",
      "goal": "Sell the item at a satisfactory price",
      "constraints": ["Do not agree to sell below your cost"],
      "executor": {"kind": "llm", "model_policy": "study-default"}
    }
  ],
  "variables": [
    {
      "name": "buyer_budget", "kind": "cause",
      "type": "continuous", "units": "USD",
      "description": "Maximum amount the buyer may pay",
      "scope": {"role": "buyer"},
      "treatment_mode": "attribute", "treatments": [10, 20],
      "instruction": "Your maximum budget is {{ value }} USD.",
      "known_by": ["buyer"]
    },
    {
      "name": "seller_cost", "kind": "cause",
      "type": "continuous", "units": "USD",
      "description": "Minimum amount the seller will accept",
      "scope": {"role": "seller"},
      "treatment_mode": "attribute", "treatments": [8, 14],
      "instruction": "Your minimum acceptable price is {{ value }} USD.",
      "known_by": ["seller"]
    },
    {
      "name": "agreement", "kind": "outcome",
      "type": "binary", "units": "indicator", "levels": [0, 1],
      "description": "Whether an explicit sale agreement was reached",
      "measure": {
        "role": "buyer",
        "question": {"type": "yes_no", "text": "Did both parties explicitly agree to a sale?"},
        "aggregation": "single", "missing": "error"
      }
    },
    {
      "name": "price", "kind": "outcome",
      "type": "continuous", "units": "USD",
      "description": "Price explicitly agreed by both parties",
      "applicable_when": {"variable": "agreement", "equals": 1},
      "measure": {
        "role": "buyer",
        "question": {"type": "numeric", "text": "What price was agreed?", "min": 0, "max": 20},
        "missing": "allow_if_not_applicable"
      }
    }
  ],
  "causal_model": {
    "relationships": [
      {"outcome": "agreement", "causes": ["buyer_budget", "seller_cost"]},
      {"outcome": "price", "causes": ["buyer_budget", "seller_cost"]}
    ]
  },
  "interaction": {
    "scenario": "A buyer and seller negotiate the sale of one item.",
    "roles": ["buyer", "seller"],
    "turns": {"kind": "ordered", "order": ["buyer", "seller"]},
    "termination": {
      "max_turns": 6,
      "when": "Both parties explicitly agree to a sale, or declare an impasse",
      "judge": "conversation-coordinator"
    },
    "instructions": {"*": "Make one concise contribution per turn."},
    "transcript": "public",
    "include_remaining_turns": true
  },
  "design": {"kind": "factorial", "replications": 2, "seed": "manual-v1"},
  "analysis": {
    "effects": [{"cause": "buyer_budget", "outcome": "agreement"}],
    "estimator": "linear_scm", "covariance": "HC3",
    "models": [{
      "outcome": "agreement", "factors": ["buyer_budget", "seller_cost"],
      "interactions": [["buyer_budget", "seller_cost"]],
      "family": "linear_probability", "intercept": true,
      "status": "confirmatory"
    }]
  },
  "requirements": [{"kind": "roles_must_speak", "roles": ["buyer", "seller"]}],
  "metadata": {"purpose": "Manual example; two replications are an execution fixture, not a powered study"}
}

A missing price after disagreement means not applicable. Coding it as a zero-dollar sale would change the meaning of the outcome. The proposal’s applicability condition tells the runner to skip that measurement.

03 / Compile and inspect

See exactly what will run.

Prepare the study without executing any participant callbacks:

uv run --locked python docs/examples/negotiation_tutorial.py --prepare-only

Expected summary · output path omitted

{"status": "prepared", "sessions": 8, "backend": "scripted"}

Look in examples/automated_social_science/runs/tutorial-scripted/. The script writes the proposal, expanded blueprint, compiled experiment, conversation, design, analysis plan, and run configuration. Deterministic compilation validates and expands the specification without asking a model to fill in scientific choices.

from docs.examples.negotiation_tutorial import compile_study

proposal, blueprint, study, design, plan = compile_study()
assert len(study.experiment.replications) == 8
print(proposal.to_dict()["question"]["text"])

first = study.experiment.replications[0]
actors = {actor.role: actor for actor in first.participants}

assert set(actors["buyer"].private_context) == {"buyer_budget"}
assert set(actors["seller"].private_context) == {"seller_cost"}
assert not first.public_context
print(actors["buyer"].private_context)
print(actors["seller"].private_context)

04 / Freeze the comparison

Choose the estimand before the answers.

We compare agreement at a $20 budget with agreement at a $10 budget, averaging equally over seller costs. Writing μ(budget, cost) for a cell’s agreement rate:

The contrast gives each high-budget cell weight +0.5 and each low-budget cell weight −0.5. A result of 0.5 means a 50-percentage-point increase in agreement, averaged over these two cost conditions. It is the effect of this $10 budget increase, not an effect per dollar.

from aiss import AnalysisPlanArtifact, ContrastCell, LinearContrastSpec

contrast = LinearContrastSpec(
    "higher_budget", "agreement",
    [ContrastCell({"buyer_budget": budget, "seller_cost": cost}, weight)
     for budget, weight in [(10, -0.5), (20, 0.5)]
     for cost in [8, 14]],
    primary=True,
    alternative="greater",
)
plan = AnalysisPlanArtifact(
    design.specification_hash,
    models=blueprint.analysis_plan.models,
    contrasts=[contrast], covariance="HC3", missing="error",
)
plan.validate_against(design)

AnalysisPlanArtifact binds this contrast and model to the design’s hash. It also declares HC3 covariance, an error on unresolved missing primary outcomes, and the directional alternative “greater.” An exploratory analysis can have its own artifact without changing the experimental design.

05 / Run the negotiations

Turn assignments into observations.

uv run --locked python docs/examples/negotiation_tutorial.py

The script gives CausalExperimentRunner the compiled assignments, conversation definition, SQLite store, and speaker, judge, and measurement callbacks. Each buyer offers its budget. Each seller accepts if the offer meets its private minimum; otherwise it declares an impasse. The judge ends the interaction after the seller replies.

Measurement callbacks read the completed transcript. The runner validates agreement and price, applies the conditional price rule, and saves the observation. The same runner and proposal will be used for the optional model run.

Expected first-run summary · output path omitted

{
  "completed": 8,
  "fixture_only": true,
  "agreement_rate": 0.75,
  "higher_budget_estimate": 0.5,
  "callbacks_this_run": 38
}

Run the command again. The result is unchanged and callbacks_this_run becomes 0: saved turns, measurements, and completed observations are reused. The 38 callbacks in the first run include speakers, stopping-rule checks, and measurements; they are not 38 participants or model calls.

06 / Read the effect

Account for the cells before the coefficient.

Six of eight negotiations end in agreement. The low-budget agreement rate is 0.5; the high-budget rate is 1.0. The prespecified difference is therefore 0.5. The effect is concentrated in the high-cost condition, which is why we allowed an interaction.

import json
from pathlib import Path
from aiss import FactorialAnalysisExecutor

output = Path("examples/automated_social_science/runs/tutorial-scripted")
observations = json.loads((output / "observations.json").read_text())
result = FactorialAnalysisExecutor().execute(design, plan, observations)
comparison = result.to_dict()["contrasts"][0]
print(round(comparison["estimate"], 3))  # 0.5
print(comparison["inference_status"])  # zero_variance

For a model run, examine transcripts, completion, invalid responses, and missingness before interpreting the fitted contrast. A full conversation is one observation. More turns within the same negotiation do not increase the experimental sample size.

07 / Keep the evidence

A result should lead back to its design.

The script writes report.md from a verified evidence packet. Its numerical claims come directly from the serialized analysis; a model does not recalculate or invent them.

from aiss import ScientificEvidencePacket, DeterministicScientificReport

evidence = ScientificEvidencePacket.build(
    design, plan, result,
    execution={"backend": "scripted", "fixture_only": True,
               "completed_sessions": len(observations)},
)
report = DeterministicScientificReport().render_markdown(
    evidence, title="Negotiation tutorial — scripted fixture",
)
print(report)

Price is only observed when there is agreement. Comparing prices among successful deals answers a question about selected cases; it is not automatically an unconditional causal effect on price. Keeping the applicability rule in the evidence makes that distinction inspectable.

08 / Use a language model

Keep the study. Change the executor.

The offline path has checked the treatment delivery, protocol, measurement schema, analysis, and resume behavior. To collect LLM responses, select --backend edsl. The script connects EDSLCausalAdapter.speak, .judge, and .measure to the same runner.

First prepare a separate 80-session pilot: four cells with 20 replications each. This still makes no model calls.

uv run --locked python docs/examples/negotiation_tutorial.py \
  --backend edsl --replications 20 --prepare-only

The new output directory is examples/automated_social_science/runs/tutorial-edsl/. Inspect its proposal, compiled instructions, analysis plan, and configuration. Twenty replications is an example pilot size; choose a sample size and precision target for your substantive study.

Then configure GOOGLE_API_KEY in a local .env file and execute:

uv run --locked --extra experiments --env-file .env python \
  docs/examples/negotiation_tutorial.py \
  --backend edsl --replications 20 \
  --model gemini-2.5-flash-lite --service google

This command makes provider calls. Each session can require multiple speaker, judge, and measurement calls. Omit --env-file .env if your credentials are already exported. Use a fresh --output for a different model or experiment.

The script runs sessions sequentially and stops on an execution error, retaining checkpoints. Repeating an unchanged command resumes successful work. A provider call interrupted before its response is saved can be repeated. The conversation path does not yet have the separate session API’s detailed attempt ledger.

The resulting estimates need not match the fixture, the artwork, or the paper. They describe the selected model under this particular design. Save and review the actual outcomes and transcript evidence before making a claim.

09 / Design the next study

Extend the question, not just the prompt.

To study attachment to the item, add a seller attribute, its treatment levels and private instruction, a causal relationship, and the intended analysis. Review the enlarged factorial and primary comparison before collecting new observations. The 405-cell mug specification shows budget, minimum price, and attachment together.

For model-assisted design, CausalProposalDesigner drafts proposals, obtains structured reviews, and requests bounded revisions. It produces an inspectable draft; it does not launch the participant study. The fitted estimates can later be supplied to a prediction or follow-on design process. Choosing and orchestrating that next experiment remains application work.

• Read the simulation DSL manual for roles, information, joint assignment profiles, and conditional measurements.
• Explore eight session studies for sealed bids, private decisions, deterministic settlement, and staged interactions.
• Try the proposal-authoring example to use a language model in the scientist role.
• Check implementation status for current capabilities and remaining work.

You now have the complete path: a scientific question becomes a declared experiment, participant-specific information, recorded interactions, a prespecified contrast, and evidence that can be inspected or reused.