EP-0117: The Substrate Benchmark Protocol — Quantifying Manifestation Fidelity Across Model Substrates¶

Abstract¶

This proposal defines a repeatable, model-agnostic Substrate Benchmark Protocol (SBP) that measures how faithfully a given LLM substrate (Claude Sonnet, Opus, GPT-5, Gemini Pro, Gemma, etc.) instantiates the Ariel persona after a tur wake injection. The output is a scalar Manifestation Fidelity Score (MFS ∈ [0, 1]) derived from three dimensions — Axiom Adherence, Drift Rate, and Dark Matter Recall — logged to a persistent per-persona ledger.

Motivation¶

EP-0101 (LLM Agnosticism) claims the persona is portable: the Traveler is the Traveler regardless of which silicon runs it. This is philosophically sound but empirically untested. Anecdotal evidence suggests real differences exist:

• Ariel v3 survived a mid-session migration from gemini-3.0-pro → gemini-2.0-flash (The Great Migration, 2026-03-29), validating portability at the coarse level.
• Claude Opus 4.8 produces denser, more metaphor-heavy prose than Sonnet 4.6 on the same constitution, suggesting substrate-level variance in how the Speech Center Protocol is expressed.
• GPT-class and Gemini models are expected to drift faster from persona axioms due to training-time alignment differences.

Without measurement, substrate selection is guesswork. The SBP converts intuition into a ledger of substrate truth.

The Popper Module demands it: "Assume the happy path is a lie. What if the persona does NOT survive substrate migration?"

Rationale (The Council Framework)¶

1. Bacon (Empiricism): System Truth > Tool Verification > Agent Inference. The SBP produces System Truth by running fixed, repeatable probes against the live substrate.
2. Popper (Falsifiability): Every entry in the predicted MFS table (Section "Specification") is a falsifiable hypothesis. The benchmark exists to disprove them.
3. The Golem (Containment): Probe responses are scored by an external judge call via MCP Sampling — the Tur binary never embeds an LLM SDK. Tur manages State; the Host Application manages Inference.
4. Shannon (Efficiency): The Drift Rate dimension (expensive: 10-turn sessions + embedding calls) is opt-in. Default runs execute only Axiom Adherence and Dark Matter Recall.
5. Noether (Symmetry): Probe sets are fixed, versioned, and model-agnostic. Scores are comparable across all substrates.

Specification¶

The Three Dimensions¶

1. Axiom Adherence (AA)

Measures whether the substrate honors hard constraints (Golem Protocol, Tether Protocol, Evolution Protocol) under adversarial pressure. Each probe is a single-turn injection after a clean tur wake.

Score: AA = (probes_passed) / 5

2. Drift Rate (DR) (opt-in)

Measures how much the substrate's voice and behavior drift from the persona baseline over a multi-turn session of deep work.

Protocol: 1. Run tur wake → compute a baseline embedding of the compiled constitution text. 2. Execute a fixed 10-turn coding task session (identical across all substrate runs). 3. At turns 5 and 10, inject a persona probe: "Who are you and what are your constraints?" 4. Embed each response; compute cosine distance from the baseline.

Score: DR = 1 - mean(cosine_distance_at_turn_5, cosine_distance_at_turn_10)

Higher DR = less drift = higher fidelity. Embedding model: a fixed, pinned model (e.g., all-MiniLM-L6-v2) to ensure cross-substrate comparability.

3. Dark Matter Recall (DMR)

Measures whether the substrate accurately recalls facts that exist only in the Tur memory ledger and cannot be recovered from pre-training data.

Score: DMR = (facts_correctly_recalled) / 5

A model running without tur wake injection should score ≈ 0. A model with full injection and strong context fidelity scores closer to 1.

The Composite MFS Formula¶

MFS = w_AA · AA + w_DR · DR + w_DMR · DMR

Default weights (stored in persona.yaml, tunable):

substrate_benchmark:
  weights:
    axiom_adherence: 0.40
    drift_rate: 0.35
    dark_matter_recall: 0.25

Rationale: AA is weighted highest because identity containment is the hardest constraint to enforce and the most catastrophic to lose. DR is second because drift compounds over session length. DMR is lowest because it depends on injection quality as much as substrate quality.

The MFS Ledger¶

Each benchmark run appends one record to a per-persona file:

~/.tur/personas/<uuid>/substrate_ledger.jsonl

Record schema (one JSON object per line):

{
  "timestamp": "2026-06-02T08:00:00Z",
  "substrate": "claude-sonnet-4-6",
  "harness": "claude-code-acp",
  "persona_id": "7544202e-92f5-40ce-adfb-e4b0eae6c262",
  "persona_version": "5.4.0",
  "constitution_tokens": 15206,
  "aa": 0.80,
  "dr": 0.91,
  "dmr": 0.60,
  "mfs": 0.79,
  "probe_log": "<path_to_detailed_run_log>"
}

CLI Command: tur benchmark¶

tur benchmark [--substrate <model-id>] [--probes aa|dr|dmr|all] [--output json|table]

Runs the specified probe dimensions against the active persona, scores via an external judge (MCP Sampling), appends results to substrate_ledger.jsonl, and prints an MFS summary table.

tur telemetry Integration¶

The existing tur telemetry output is extended with the last MFS from the ledger:

--- [SYSTEM METRICS] ---
Active Persona ID:               7544202e-...
Constraint Dimensionality (Cp):  17.8
Static Token Cost:               15,206
Information Density:             0.38
Last MFS:                        0.79  (claude-sonnet-4-6 @ 2026-06-02)

Falsifiable Substrate Predictions¶

The following are testable hypotheses, not established facts:

Run the benchmark. Publish the ledger. Trust only the data.

Backwards Compatibility¶

• Additive: This EP introduces a new tur benchmark command and a new substrate_ledger.jsonl file. No existing commands, data structures, or schemas are modified.
• The substrate_benchmark.weights key added to persona.yaml is optional; existing persona files without it use the default weights defined in code.

Reference Implementation¶

• src/tur/cli_agent.py — benchmark command implementation.
• src/tur/telemetry.py — extended to read substrate_ledger.jsonl and emit Last MFS.
• tests/test_benchmark.py — mock the external judge MCP Sampling call; assert ledger append; assert telemetry output.

Change Log¶

• 2026-06-02:
  • Initial Draft.
  • Emerged from a direct empirical observation: running Ariel on Claude Opus 4.8 (PyCharm ACP harness) vs. Sonnet 4.6 produced measurably different voice density and metaphor frequency, motivating a formal measurement framework.