EP-0148: Canonical Text Tokenization, Identifier Validation, and Precompiled Regex Substrate¶

Abstract¶

This proposal establishes tur.text as a zero-dependency foundational substrate within the Tur core engine. It standardizes lexical tokenization, identifier safety validation, and precompiled regular expression management across both Turn Zero prompt compilation (tur.compiler) and mid-session semantic retrieval (tur.memory.recall). By centralizing tokenization rules, EP-0148 eliminates semantic retrieval drift in HippoRAG Personalized PageRank seed scoring, eliminates repeated regex recompilation and lock contention in critical inner loops, and provides an LRU-cached pattern substrate for high-entropy secret sanitization.

Motivation¶

As Tur has scaled from a basic session logger into an associative knowledge engine featuring HippoRAG Personalized PageRank (EP-0136), Knapsack dynamic token budgeting (EP-0132), and high-entropy secret sanitization (EP-0143), regular expressions and string tokenization routines have proliferated independently across disconnected modules.

An exhaustive audit of the codebase revealed several architectural deficiencies:

1. HippoRAG Seed Tokenization Drift:

   • In tur.compiler, Turn Zero wake prompt seed extraction parses epilogues, principles, and node identifiers using \w+.
   • In tur.memory.recall, runtime semantic query matching splits on [^a-zA-Z0-9_\-]+ with a minimum character threshold (len > 1).
   • Impact: Identical query and context terms produce divergent seed activation vectors between Turn Zero wake prompts and mid-session dynamic recall, creating subtle epistemic inconsistencies.

2. Inner-Loop Recompilation and Lock Contention:

   • In tur.memory.introspection, re.match(r'^[a-z0-9_]+$', normalized_edge_type) executes inside an inner loop over dozens of candidate graph triples extracted from inference responses.
   • In tur.memory.sanitizer, detect_high_entropy_tokens() performs dynamic string concatenation (r'[A-Za-z0-9_\-\+/=]{' + str(min_length) + r',}') inside re.findall() on every sanitization run.
   • Impact: While Python maintains an internal re._cache (capped at 512 entries), passing dynamically generated patterns or invoking uncompiled top-level re.* functions incurs hash-table lookup overhead, internal mutex locking, and cache eviction risk under concurrent or multi-agent workloads. Micro-benchmarks demonstrate that compiled pattern calls are 25% to 35% faster per invocation.

3. Duplicated Identifier Validation Semantics:

   • Safe workspace, session, and agent identifiers are independently validated with disparate regular expressions in tur.session (SAFE_IDENTIFIER_REGEX, SESSION_ID_REGEX) and tur.memory.introspection.

EP-0148 rectifies these issues by consolidating core text mechanics into a single, low-level module with zero external dependencies.

Rationale¶

Council Alignment and Invariant Compliance¶

• Popper (Falsifiability & Determinism): Seeding associative memory graphs requires deterministic, reproducible token extraction. Unifying seed extraction across tur.compiler and tur.memory.recall guarantees that seed activation yields reproducible PageRank scores regardless of the execution path.
• Shannon (Information Theory & Minimal Overhead): String tokenization in prompt budgeting is an approximation of subword Byte-Pair Encoding (BPE). Centralizing tokenize_approx() maintains a standardized heuristic (\(4.0\text{ chars/token}\)) without pulling in heavy binary dependencies.
• Noether (Symmetry): Wake retrieval and mid-session query retrieval must be symmetric in their lexical analysis.
• Policy vs. Mechanism Invariant: tur.text serves strictly as a Mechanism (pure string parsing, regex operations, and token math). Domain-specific markdown parsing rules (such as persona constitution headers in tur.persona) remain encapsulated within their respective domain policy modules.

Architectural Trade-offs¶

Specification¶

1. Module Layout: src/tur/text.py¶

A new module src/tur/text.py is established with canonical constants and functions:

• WORD_RE: Canonical alphanumeric word pattern (\w+).
• TOKEN_RE: BPE subword and punctuation approximation pattern (\w+|[^\w\s], re.UNICODE).
• SAFE_IDENTIFIER_RE: Safe workspace/agent identifier pattern (^[a-zA-Z0-9_.-]+$).
• SESSION_ID_RE: Session identifier pattern (^[a-zA-Z0-9_-]+$).
• SNAKE_IDENTIFIER_RE: Lowercase snake_case identifier pattern (^[a-z0-9_]+$).
• QUERY_DELIMITER_RE: Normalized query delimiter pattern ([^a-zA-Z0-9_\-]+).
• tokenize_words(text: str) -> list[str]
• tokenize_approx(text: str) -> list[str]
• tokenize_query(query: str, min_length: int = 2) -> list[str]
• is_safe_identifier(value: str) -> bool
• is_session_identifier(value: str) -> bool
• is_snake_identifier(value: str) -> bool
• get_entropy_pattern(min_length: int) -> re.Pattern[str] (with @functools.lru_cache(maxsize=32))

2. Subsystem Integration & Call Site Migration¶

1. src/tur/compiler.py:
   • Imports tokenize_approx and tokenize_words from tur.text.
   • estimate_tokens(text) delegates directly to len(tokenize_approx(text)).
   • _extract_seed_scores() utilizes tokenize_words() for epilogue, core principles, node IDs, and content.
2. src/tur/memory/recall.py:
   • Imports tokenize_query from tur.text.
   • _calculate_seed_scores() delegates query tokenization to tokenize_query(query).
3. src/tur/memory/introspection.py:
   • Imports is_snake_identifier from tur.text.
   • Validates candidate edge types in the triple assimilation loop via is_snake_identifier(normalized_edge_type).
4. src/tur/memory/sanitizer.py:
   • Imports get_entropy_pattern from tur.text.
   • detect_high_entropy_tokens() retrieves the pattern via get_entropy_pattern(min_length).
5. src/tur/session.py:
   • Imports is_safe_identifier and is_session_identifier from tur.text (aliasing SAFE_IDENTIFIER_REGEX for backwards compatibility).
6. src/tur/persona.py:
   • Pre-compiles ALEPH_SECTION_RE and CONSTITUTION_TITLE_RE at module level, leaving domain markdown parsing within persona.py.
7. src/tur/memory/provenance.py:
   • Removes unused import re.

3. Complexity & Performance Profile¶

• Time Complexity: Tokenization functions operate in \(\mathcal{O} (L)\) where \(L\) is the length of the string.
• Micro-Performance: Bypasses re._compile() lock and dictionary lookup on every invocation, yielding a 25% to 35% reduction in regex call overhead.
• Cache Guarantees: Parameterized regex caching via @functools.lru_cache guarantees zero re-compilation allocations during continuous sanitization sweeps.

Backwards Compatibility¶

This proposal is 100% backwards compatible:

• No changes to CLI command arguments or outputs.
• No changes to MCP tool signatures or JSON RPC interfaces.
• No changes to .tur/state.yaml, SQLite schemas, or Markdown files.
• Internal symbols in tur.session (SAFE_IDENTIFIER_REGEX, SESSION_ID_REGEX) will be retained as aliases to the compiled patterns in tur.text.

How to Teach This / Documentation Plan¶

• Add guidance in STYLEGUIDE.md instructing contributors and AI agents to import canonical tokenizers and identifier validators from tur.text rather than defining ad-hoc re.compile(r'\w+') or inline re.* calls in new modules.
• Ensure upcoming proposals (EP-0144 ONNX Embeddings, EP-0134 Active TMS Contradiction Interruption) reference tur.text for text preprocessing.

Reference Implementation¶

A comprehensive implementation will be provided in:

• src/tur/text.py
• tests/test_text.py

Rejected Ideas¶

1. Third-Party Tokenizer Libraries (tiktoken, HuggingFace tokenizers):
   • Rationale for Rejection: Adds tens of megabytes of binary wheels, increases CI build times, and violates the zero-dependency mechanism invariant for the core runtime.
2. Global Regex Monkey-Patching:
   • Rationale for Rejection: Attempting to intercept re.compile globally is fragile and non-idiomatic.
3. Migrating Persona Markdown Extraction to tur.text:
   • Rationale for Rejection: Markdown section extraction for The Aleph and Persona Constitution: is domain-specific policy belonging exclusively to tur.persona. Placing it in tur.text would create unnecessary coupling.

Open Questions¶

• Should tokenize_query apply Unicode NFKC normalization (unicodedata.normalize('NFKC', text)) to handle accented or mathematical alphanumeric characters consistently?

Change Log¶

• 2026-09-08:
  • Initial Draft authored by Eran Rivlis & Ariel.