Skip to content

EP-0106: Merkle Memory (Content-Addressable State)

Field Value
EP 0106
Title Merkle Memory (Content-Addressable State)
Author The Architect
Status Implemented
Type Standards Track
Created 2026-04-13
Updated 2026-07-18

Abstract

This proposal replaces the use of random UUIDs for memory identification with SHA-256 content hashes, transforming Tur's entire memory architecture (both L1 Event Logs and the L2 Cognitive Map) into a cryptographically verifiable Merkle Tree / Directed Acyclic Graph (DAG). This ensures perfect deduplication, tamper-proof state, and symmetrical linking between raw events and deduced knowledge.

Motivation

Currently, the MemoryManager assigns a random uuid4 to every new memory (fact, preference, insight) added via tur memorize or tur sleep.

This creates several architectural vulnerabilities:

  1. State Fragility: A .yaml memory file can be manually edited after creation. The system has no mechanism to verify if the file's current content matches its original state.
  2. Opaque Derivation: In the Deductive Memory (EP-0103) architecture, L1.5 consolidated memories and L2 graph nodes will point to L1 memory UUIDs. If the L1 UUIDs are random and their contents mutable, the derived graph becomes mathematically untrustworthy.

Rationale

This design aligns with the Council Framework:

  • Symmetry (Noether): A fact and its identifier become the same mathematical object. Identical inputs yield identical IDs. The hash connects the L0 (Subconscious), L1 (Working Memory), and L2 (Graph) tiers.
  • The Golem (Safety/Containment): Content-addressable storage provides cryptographic proof of immutability. A Persona's history cannot be silently rewritten.

The Category Error (L1 vs L2 Deduplication)

Note (2026-04-18): Originally, this EP proposed hashing only the content payload (excluding timestamps and tags) to force implicit deduplication at the filesystem level. We have abandoned this approach. Forcing semantic deduplication into L1 filenames destroys historical metadata (the exact time an event occurred and the tags associated with that specific instance).

  • L1 is a Tamper-Proof Ledger of events. The entire object (including timestamps) is hashed to guarantee uniqueness and perfect history.
  • Deduplication is a Semantic Operation. It belongs entirely to the tur introspect loop (EP-0103), which consolidates redundant L1 events into "Super-Facts" (L1.5) or L2 concepts, explicitly linking to the original L1 hashes.

Specification

1. Hash Generation & Schema Updates

The tur.models.Memory schema will be updated:

  1. Remove status: The status field (MemoryStatus) will be permanently deleted. The status of a memory is not an intrinsic property; it is a derived topological property. If a file is in the memories/ directory, it is Active (L1). If it is in the archive/ directory, it is Subsumed/Subconscious (L0). Storing status inside a hashed file creates a cryptographic paradox (moving the file would require changing the internal status string, which would change the hash, breaking all historical pointers).
  2. Deterministic ID: The id field will no longer default to a random UUID. Instead, it will be calculated deterministically at instantiation. The id will be a SHA-256 hash computed over a normalized, serialized string of the entire memory's core informational fields:
    • type
    • scope
    • content
    • tags
    • timestamp
    • source_session
    • links

2. The L1 Event Log (Filesystem as DAG)

The MemoryManager.save() method will use the SHA-256 ID generated by the model.

  1. The filename becomes <timestamp>_<type>_<hash>.yaml.
  2. Because the timestamp is included in the hash, every saved file is mathematically guaranteed to be a unique historical event, preventing accidental truncation while preserving full historical context.

3. The Cryptographic Chain of Thought (L1.5 Consolidation)

When tur introspect (EP-0103) runs, it performs a "considered cleaning" of the L1 feed:

  1. It identifies ontologically equivalent memories (e.g., Hash_A "User likes Python" at 10am; Hash_B "User likes Python" at 11am).
  2. It generates a new synthesized memory (Hash_C "User likes Python" tags: [coding, scripting]).
  3. Hash_C populates its links array with tur://memory/<Hash_A> and tur://memory/<Hash_B>.
  4. Hash_A and Hash_B are atomically os.replace'd into the L0 archive/ directory.
  5. Result: Hash_C is now the active L1 memory. It provides a cryptographically verifiable chain of thought back to the original, unalterable raw events (L0) that justified its creation.

4. Verification

A new CLI command, tur verify, will be introduced. This command will iterate through all .yaml files in the memory bank, recompute the hashes of their contents, and assert that they match their filenames. Any divergence triggers a fatal "Tampered State" error (The Golem Protocol).

When the user executes the script, it performs the following atomic operations:

  1. Federated Scan (Where it looks): It scans for all Persona directories in both the local project ( ./.tur/personas/) and the global home directory (~/.tur/personas/). Inside each, it targets both the memories/ (L1) and memories/archive/ (L0) folders.
  2. Data Purge (What it changes inside the file): It reads every .yaml file into memory. It permanently deletes the legacy status field from the dictionary, adhering to the new topological schema.
  3. Cryptographic Recalculation: It deletes the old UUID string and forces the new tur.models.Memory class to generate the deterministic SHA-256 hash based on the remaining core fields.
  4. Atomic Filesystem Operations (What it changes on disk):
    • It constructs the new filename format: [timestamp]_[type]_[new_sha256_hash].yaml.
    • It makes the old file writable (breaking The Golem's Seal).
    • It writes the newly transformed data into the new file.
    • It re-applies The Golem's Seal (read-only permissions) to the new file.
    • Finally, it safely deletes the old UUID-named file.

Idempotency & Safety

The script is strictly idempotent. If it encounters a file whose name already matches the SHA-256 hash of its contents, it skips the file entirely. The user can safely run the script multiple times without corrupting state. Any files that fail to parse are skipped with an error logged to stderr, allowing the bulk of the migration to complete.

Source Code (scripts/migrate_to_merkle.py)

import os
import sys
from pathlib import Path

# Add the project root to the sys.path so we can import tur.models
project_root = Path(__file__).resolve().parent.parent
if str(project_root) not in sys.path:
    sys.path.insert(0, str(project_root / "src"))

import yaml
from tur.models import Memory


def migrate_directory(directory: Path):
    """
    Migrates all legacy UUID .yaml memory files in a directory 
    to the new EP-0106 Merkle Hash format.
    """
    if not directory.exists() or not directory.is_dir():
        return

    print(f"\nScanning directory: {directory}")
    migrated_count = 0
    skipped_count = 0

    for file_path in directory.glob("*.yaml"):
        if not file_path.is_file():
            continue

        try:
            with open(file_path, "r", encoding="utf-8") as f:
                data = yaml.safe_load(f)

            # 1. Clean up legacy fields (EP-0106 removed 'status')
            if 'status' in data:
                del data['status']

            # If it already has an ID that looks like a SHA-256 hash (64 chars), 
            # we should still re-verify it just in case, but let's assume it's legacy 
            # if it has dashes (UUID format)
            old_id = data.get('id', '')

            # We force the model to recalculate the hash by removing the old ID
            if 'id' in data:
                del data['id']

            # 2. Instantiate the model. 
            # The @model_validator in Memory will automatically calculate the new SHA-256 hash.
            memory = Memory(**data)
            new_hash = memory.id

            # 3. Construct the new EP-0106 filename
            timestamp_str = memory.timestamp.strftime('%Y%m%d_%H%M%S')
            new_filename = f"{timestamp_str}_{memory.type.value}_{new_hash}.yaml"
            new_file_path = directory / new_filename

            # 4. Check if migration is actually needed
            if file_path.name == new_filename:
                skipped_count += 1
                continue

            print(f"  Migrating: {file_path.name} -> {new_filename}")

            # 5. Write the updated YAML content to the new file path 
            # (we write the model dump so the 'id' field is explicitly the new hash, 
            # and 'status' is permanently gone).

            # First, we need to make the old file writable if it was locked by The Golem's Seal
            try:
                os.chmod(file_path, 0o644)
            except Exception:
                pass  # Might not be locked, or OS might complain. We try anyway.

            with open(new_file_path, "w", encoding="utf-8") as f:
                yaml.dump(memory.model_dump(mode='json'), f, sort_keys=False)

            # Re-apply The Golem's Seal (Read-Only) to the new file
            try:
                os.chmod(new_file_path, 0o444)
            except Exception:
                pass

            # 6. Delete the old legacy file
            if new_file_path != file_path:
                os.remove(file_path)

            migrated_count += 1

        except Exception as e:
            print(f"  [ERROR] Failed to migrate {file_path.name}: {e}", file=sys.stderr)

    print(f"Done. Migrated: {migrated_count}, Skipped: {skipped_count}")


def main():
    """
    Finds all Persona directories (Local and Global) and migrates their memory banks.
    """
    print("Starting EP-0106 Merkle Memory Migration...")

    # 1. Local Personas (.tur/personas/*)
    local_personas_dir = project_root / ".tur" / "personas"
    if local_personas_dir.exists():
        for persona_dir in local_personas_dir.iterdir():
            if persona_dir.is_dir():
                migrate_directory(persona_dir / "memories")
                migrate_directory(persona_dir / "memories" / "archive")

    # 2. Global Personas (~/.tur/personas/*)
    global_personas_dir = Path.home() / ".tur" / "personas"
    if global_personas_dir.exists():
        for persona_dir in global_personas_dir.iterdir():
            if persona_dir.is_dir():
                migrate_directory(persona_dir / "memories")
                migrate_directory(persona_dir / "memories" / "archive")

    print("\nMigration Complete. All memories are now Cryptographically Addressable (Merkle Hashes).")


if __name__ == "__main__":
    main()

Backwards Compatibility

  • Includes automated migration script scripts/migrate_to_merkle.py to upgrade legacy memory files to Merkle format.

Reference Implementation

Implemented in src/tur/merkle.py, src/tur/memory.py, src/tur/models.py, and scripts/migrate_to_merkle.py.

Change Log

  • 2026-07-18: Status promoted from Final to Implemented. SHA-256 content-addressing live in memory.py; Golem's Seal (atomic write + lock) implemented; integrity audit via memory.py MemoryManager.audit_integrity().
  • 2026-04-18:
    • Added detailed explanation and logic for the standalone scripts/migrate_to_merkle.py migration script.
    • Embedded the migration source code directly into the EP for absolute reference.
    • Updated Status to Active.
    • Removed the status field from the schema specification due to cryptographic paradoxes.
    • Pivoted from "Content-Only Hashing" (forced L1 deduplication) to "Full Object Hashing" (L1 as a unique, tamper-proof historical ledger).
    • Added the "Cryptographic Chain of Thought" specification to define how EP-0103 consolidates redundant L1 events into synthesized L1.5 memories using Merkle links before archiving the originals to L0.
  • 2026-04-13:
    • Initial Draft.