Analysis of Competing Hypotheses with Langley

01 What Langley does

Langley is a Python CLI for Richards J. Heuer Jr.'s Analysis of Competing Hypotheses method. It records hypotheses, evidence, matrix assessments, tentative conclusions, sensitivity tests, and future indicators in one durable project.

ACH reverses a familiar analytical habit. Instead of choosing a favorite explanation and collecting support for it, the analyst identifies a full set of reasonable alternatives and looks for evidence that is difficult to reconcile with each one. The surviving hypothesis is the least inconsistent—not automatically proven true.

Let the project tell the agent what comes next

An agent begins with two read-only commands. guide returns the durable operating rules; next returns one canonical action and any safe alternatives. The recommendation depends on preserved project state, not conversational memory.

langley guide
langley init \
  --title "Checkout outage" \
  --question "What caused the 20-minute checkout outage?" \
  --scope "Production, 14:00–14:20 UTC"
langley next

{
  "status": "ok",
  "data": {
    "phase": "hypotheses",
    "action": {
      "argv": ["langley", "hypothesis", "add"],
      "mutates_state": true,
      "requires_user_approval": false
    }
  }
}

Every command returns one stable JSON object on stdout. Each action contains a directly executable argument array with an absolute --project-dir selector and an explicit absolute cwd. Callers should execute that argv with the supplied cwd; they must not reconstruct the command or infer project context. Structured errors include recovery hints, and external model execution is clearly marked as approval-gated.

02 Generate competing hypotheses

Start broadly. A hypothesis may be an explanation of what happened or a possible future outcome. Keep a plausible hypothesis alive until evidence disproves it; lack of supporting evidence is not the same as evidence against it.

langley hypothesis add \
  --name "Database saturation" \
  --description "Connection exhaustion caused requests to queue."

langley hypothesis add \
  --name "Bad deployment" \
  --description "A checkout release introduced a failure."

langley hypothesis add \
  --name "Payment provider outage" \
  --description "The external payment dependency was unavailable."

Hypotheses have stable IDs such as h001. Archiving preserves the record and requires a rationale; the audit trail never quietly erases an inconvenient alternative.

03 Gather evidence, arguments, and absences

Evidence is deliberately broad: direct observations, logical arguments, assumptions, contextual facts, and observations that should exist but do not. For each hypothesis ask, “If this were true, what should I expect to see—or not see?”

langley evidence add \
  --label "No checkout deployment occurred" \
  --kind absence --expected-if h002 \
  --observability high \
  --observability-rationale "Every deployment emits an audited event" \
  --search-performed "Queried 13:00–15:00 UTC"

An absence must explain why the missing event should have been observable and what search was performed. This prevents “we saw no evidence” from masquerading as disproof.

04 Compare one evidence row at a time

The matrix is the center of ACH. Work across each row, asking how the same evidence relates to every active hypothesis. A completed cell is consistent, inconsistent, neutral, or not applicable, with a strength and rationale where appropriate.

langley matrix next
langley matrix set --evidence e001 --input e001-row.json
langley matrix gaps
langley matrix analyze

Evidence is diagnostic when it distinguishes hypotheses. An item consistent with every alternative may be true and important, yet contribute little to choosing between them.

05 Draw a tentative conclusion

Langley reports raw and credibility/importance-adjusted inconsistency totals. The matrix can expose the evidence driving a judgment, but it never manufactures probabilities or chooses a winner.

langley conclusion draft --input conclusion.json
langley conclusion show
langley validate

06 Challenge the linchpins

Identify the few pieces of evidence or assumptions that swing the result. Ask whether they are wrong, incomplete, misleading, open to another interpretation, or vulnerable to denial and deception.

langley sensitivity suggest
langley sensitivity run \
  --name "Recovery timing was coincidental" \
  --input exclude-e004.json

A sensitivity test is a recorded counterfactual over the current matrix. It does not mutate the base analysis. In the worked outage, excluding the recovery-timing evidence narrows the gap but leaves database saturation least inconsistent.

07 State what would change your mind

Conclusions remain tentative. Define observable milestones in advance, including where to watch, a meaningful threshold, and which hypotheses the signal supports or weakens.

langley indicator add --input recurrence-indicator.json
langley validate --strict
langley report generate --name checkout-outage

Example: checkout failures recur for five minutes while database connection use remains below 60%. That observation would weaken saturation and increase the relative plausibility of an application failure.

EDSL assistance without hidden model calls

Langley can package hypothesis generation, evidence elicitation, matrix assessment, red-teaming, and indicator generation as portable, model-free EDSL Jobs.

langley job generate matrix-row --evidence e001
ep inspect .langley/jobs/j001/jobs.ep
ep jobs cost .langley/jobs/j001/jobs.ep

# Only this step may call a model and requires approval
ep run .langley/jobs/j001/jobs.ep \
  --model <model-name> \
  --output .langley/jobs/j001/results.ep

langley job ingest j001 \
  --from .langley/jobs/j001/results.ep
langley proposal compare --job j001
langley proposal accept p001 --reason "Reviewed and sound"

Model answers enter the project as immutable proposals. Accepting or rejecting one is a separate audited action. Agreement among agents describes judgment stability; it is not evidence that a hypothesis is true.

A complete worked outage analysis

1. Generate three distinct hypotheses through a model-free EDSL job.
2. Review and accept the proposed alternatives.
3. Add four sourced evidence items, including one meaningful absence.
4. Complete all twelve matrix cells row-by-row.
5. Draft the relative-likelihood conclusion and residual uncertainty.
6. Exclude the critical recovery-timing evidence in a sensitivity test.
7. Add a recurrence indicator, validate, and generate the report.

langley status
{
  "phase": "report-ready",
  "counts": {
    "hypotheses": 3,
    "evidence": 4,
    "matrix": 12,
    "conclusions": 1,
    "sensitivities": 1,
    "indicators": 1
  },
  "validation_issues": []
}

The generated directory contains report.md, matrix.csv, canonical analysis.json, and a hash manifest tied to the exact project revision.

Install and contribute

uv tool install --upgrade --force \
  "langley @ git+https://github.com/expectedparrot/langley.git@main"

langley --version
langley guide

For local development:

git clone https://github.com/expectedparrot/langley.git
cd langley
python -m pip install -e .
PYTHONPATH=.:../edsl pytest -q

Langley requires Python 3.11 or newer and is released under the MIT License. The method follows Chapter 8 of Richards J. Heuer Jr.'s Psychology of Intelligence Analysis.