Ask Daniel's CODEX

BUILD SPEC

BUILD SPEC

— Forrest Corpus Expedition, Orchestrated

Version: 1.0 · 2026-07-02 Author: Claude (Axial Refinement Sherpa), at Daniel J. Comp's direction Target file: app.py (IN Lattice server, release 3.0) Status: draft for Grok (Master Reference Sherpa) review + Carbon Steward ratification

1. The core reframe

The current failure is not a model-capability ceiling and not a charter gap. It is an architecture mismatch: Forrest's brief is produced by a single free-form agent turn (run_conversation → AIAgent with max_iterations=25, SOUL-driven) in which the model is asked to run a four-pass dig and assemble a brief in one open-ended request. Given that shape, Grok 4.3 reliably shortcuts to the cheap half (emit a brief) and skips the expensive half (read many summaries, stop and wait, open files, print gates).

Every one of those skipped behaviors is deterministic work that does not belong to the model. The fix is to keep the model for the small bounded judgments it already does reliably (generate terms, score a handful of summaries, pick a scotoma) and move everything else — retrieval, sequencing, file reads, excerpt slicing, gate enforcement — into app.py.

Design invariant: No single model call is ever asked to do more than one bounded transform, and every model call is JSON-in / JSON-out so it is checkable before the next call fires. The dig emerges from orchestration; no call requires held-across-turns effort.

Model-agnostic: because each step is a narrow JSON contract, the underlying model is a config choice. Grok 4.3 should clear every step. If any single step proves shaky, that step (and only that step) can be routed to another model via the existing mcfg config without touching the pipeline.

2. What already exists in app.py (reuse, do

not rebuild)

Confirmed by reading the source:

• Model invocation: _build_bridge_script(...) + _run_bridge_script(container, script, timeout)

docker-exec a Python body into the Hermes container, which runs run_agent.AIAgent(...).run_conversation(msg) and returns text between ===FINAL=== / ===END=== markers. (Around lines 12700–12775.)

• KB is mounted read-only at /opt/kb inside the daniel container; Forrest profile lives there too.

corpus_index.md at /opt/kb/corpus_index.md. Path resolution rule already codified: '/opt/kb/' + file_path.removeprefix('Corpus/').

• Brief handler: brief_button_handler (line 12109). Brief block formatter _format_forrest_brief_block

(line 9977). Brief files land in Agent_Briefs on the Windows host.

• Dachs research loader: _load_dachs_research_for_forrest (line 9641).
• execute_code tool is available to the Forrest agent inside the loop (SOUL note lines 246–247).

Implication: we do NOT need a new HTTP client or model plumbing. We need a new orchestration function that calls the existing bridge multiple times with narrow prompts, does the KB reads in host-side Python (or via a minimal exec helper), and enforces gates in app.py between calls.

3. Two build options — pick one

Option A (recommended):

host-side orchestration, model does judgment only

app.py reads corpus_index.md and the KB files directly on the host (the index is already copyable to the host, or read via a one-shot docker exec cat), runs all search/slicing/gates in Python, and calls the bridge only for the 3 judgment steps (term-gen, summary-scoring, scotoma-pick).

• Pros: retrieval and gates are pure Python — fully deterministic, debuggable, no model can skip

them. Fastest to a reliable result. Excerpt-slicing (body-not-frontmatter) becomes a regex, not a request. This is the architecture that actually ends the loop.

• Cons: app.py needs read access to the KB files. If the KB only exists inside the container,

add a tiny _kb_read(path) helper that does docker exec daniel cat <path> and caches. One helper, ~10 lines.

Option B: in-container orchestration script

Replace Forrest's free-form turn with a single fixed Python script bridged into the container that runs the whole pipeline internally (search + slice + gate in the script, model called via the in-script agent object for the 3 judgments) and prints the finished brief + dig report.

• Pros: all file access stays in-container; no host KB read needed.
• Cons: harder to debug; the deterministic logic lives in a generated script string rather than

first-class app.py code; gates are one exec away from the server's control.

Recommendation: Option A. The whole point is that the un-skippable steps become code the server owns. Spec below is written for Option A; Option B notes appear where they differ.

4. The pipeline — six steps, three model

calls

`` STEP 0 build_expedition_context (app.py, pure) → context dict STEP 1 gen_search_terms (MODEL call #1) → 12 terms [validated] STEP 2 retrieve_candidates (app.py, pure) → scored rows STEP 3 score_summaries (MODEL call #2, batched) → relevance scores [validated] STEP 4 pick_scotoma (MODEL call #3) → 1–2 deep cuts + reasons [validated] STEP 5 extract_excerpts (app.py, pure) → verbatim body excerpts [gate: body-not-frontmatter] assemble_dig_report (app.py, pure) → Dig Report → RETURN TO DANIEL, STOP --- two-turn gate: Daniel confirms/drops/redirects --- STEP 6 assemble_brief (app.py, pure) → brief YAML [gates: voice_anchor, banned-phrase] ``

Three model calls total. Everything else is app.py. Note Step 5 + Dig Report is where the two-turn gate lives: the server returns the report and stops. Brief assembly (Step 6) only runs on the confirming request. The model never decides to stop — the handler simply returns.

5. Step contracts (JSON schemas)

All model calls use the existing bridge. Each prompt ends with a hard instruction: Return ONLY valid JSON matching this schema. No prose, no markdown fences. Parse with a tolerant loader (strip accidental ``json fences, then json.loads`); on parse failure, retry once with the parse error appended, then fail loud with a flag — never fabricate.

STEP 0

— build_expedition_context (pure) Input: the confirmed intake fields + any blaze artifact path. ```python context = { "keyword_primary": str, "title": str, "slug": str, "voice": "Carbon" | "Si-C" | "Silicon", "archetype_primary": str, "monomyth_hint": str | None,

if intake supplied one

"blaze_path": str | None,

role:guidance source, if any

"blaze_claims": [str],

extracted from blaze for term fan-out seed

} `` No model. Pull blaze claims/open_questions with a YAML/regex read if blaze_path` set.

STEP 1

— gen_search_terms (MODEL call #1) Prompt inputs: keyword_primary, title, archetype_primary, blaze_claims. Instruction: derive concept-adjacent search terms — synonyms, causes, effects, physical images, opposites — for finding corpus files that embody this pattern. Output schema: ``json { "terms": ["string", ... 12 items] } `` app.py validation (hard, deterministic):

• exactly 10–12 terms; else retry once.
• ≥6 terms share NO whole word with keyword_primary + title + blaze named phrases

(lowercase, split on non-alpha, stopword-filtered). This is the v2.4 "Pass 1 in costume" guard, enforced in code. If <6 distinct, append the failing terms to the prompt and retry once, then proceed with whatever distinct terms exist and flag [TERM_FANOUT_WEAK].

STEP 2

— retrieve_candidates (pure) No model. For each term, substring-match (case-insensitive) against the summary and keypoints columns of every corpus_index.md row (not just tags). Union the hits. Also run Pass 3 structural filter: rows whose category/archetype and monomyth match context. Produce candidate rows with a cheap prescore = (

terms matched) + (structural match ? 2:

0). Cap the candidate set at ~40 rows (highest prescore) to keep Step 3 bounded. ``python candidates = [ { "file": str, "summary": str, "keypoints": str, "tags": str, "monomyth": str, "prescore": int }, ... ] ``

STEP 3

— score_summaries (MODEL call #2, BATCHED — 10 rows per call) This is the discrimination task, and the reason batching matters: the model only ever sees 10 fully-visible summaries at once. It is never asked to "read 30 and remember" — app.py paginates and concatenates results. Per batch, prompt inputs: the pattern (one sentence from context) + 10 {id, summary} pairs. Output schema: ``json { "scores": [ { "id": "row_3", "embodies": 0-3, "why": "≤12 words" }, ... ] } ` embodies` = how strongly this summary embodies the article's pattern (0 none … 3 exact). app.py validation: every input id returns exactly once; scores in {0,1,2,3}; else retry that batch once. Merge all batches; sort by (embodies desc, prescore desc).

STEP 4

— pick_scotoma (MODEL call #3) Input: the top 8 scored rows only (small, fully visible). Instruction: choose the 1–2 rows that embody the pattern without naming its keyword — the non-obvious deep cut — and state the non-obvious connection in one line each. If none qualify, return empty and say so. Output schema: ``json { "deep_cuts": [ { "id": "row_7", "connection": "one line" }, ... 0-2 items ], "direct_hits": [ "row_2", "row_5" ], // strongest keyword-obvious rows, for the report "pass4_empty": false } ` No new judgment beyond this — app.py` takes it from here.

STEP 5

— extract_excerpts (pure) + BODY-NOT-FRONTMATTER GATE No model. For each chosen file (deep cuts + any direct hits Daniel will see):

1. _kb_read(path) the full file.
2. Split on the first ---\n...\n--- frontmatter fence. **Excerpt is taken only from BELOW the

fence** (the body).

1. Gate — reject digest: if the sliced excerpt's first 200 chars have >0.8 similarity to the

file's own summary frontmatter field OR to the corpus_index summary cell for that file, it's the digest layer — reject, re-slice deeper, flag [EXCERPT_DIGEST_REJECTED:file] if no body prose found.

1. Gate — third-person tell: if excerpt matches ^\s*Daniel\s+(recounts|reflects|records|describes)

it is digest narration → reject, re-slice.

1. Excerpt length: 150–500 words, whole paragraphs (don't cut mid-sentence).

assemble_dig_report (pure) → RETURN + STOP (two-turn gate)

app.py composes and returns to the UI: `` === DIG REPORT === Direct hits: [file] — [one-line reason] ... Deep cuts: [file] — [non-obvious connection] ... Voice anchors:[VoiceLog/transcript files with first-person body prose on this ground] ... Searched: [12 terms] · [n] summaries scored across [k] batches === END DIG REPORT === Reply to confirm, drop, or redirect any candidate before I build the brief. `` Handler returns here. No brief is assembled on this turn. The confirming reply triggers Step 6.

STEP 6

— assemble_brief (pure) + VOICE_ANCHOR + BANNED-PHRASE GATES No model. Build the brief YAML from confirmed candidates. Two hard gates before write:

• voice_anchor gate (Carbon only): at least one confirmed role: lived excerpt must be

verbatim first-person prose. Detect first-person by presence of \bI\b/\bmy\b/\bwe\b in ≥2 paragraphs AND absence of the third-person-Daniel tell. Write the frontmatter line: voice_anchor: {file} — first-person body prose confirmed, {n} paragraphs If none qualifies: write voice_anchor: MISSING and DO NOT save the brief — return the MISSING prompt to Daniel (record a note / point to a file / dictate live). Malformed brief = not saved. This is the schema validator, in code.

• banned-phrase gate: harvest distinctive phrases from every role: guidance excerpt

(blaze + Minyan). Store as banned_phrases in the brief frontmatter so Arnie's downstream Step 3.5 has the exact list. (Forrest copies guidance verbatim — no paraphrase — so the harvest is stable.)

Write brief to Agent_Briefs, reusing existing brief-write logic in brief_button_handler.

6. Where each piece lives in app.py

Forrest's SOUL still governs the judgment prompts (term sense, scotoma taste) — the pipeline injects the relevant SOUL guidance into each narrow prompt rather than loading the whole charter per call. The parts of SOUL v2.4 that describe procedure (four passes, two-turn gate, gates) become code and can be trimmed from the model-facing charter later.

7. Failure handling — never fabricate

• Any model-call JSON parse failure: one retry with error appended, then return

[STEP_n FAILED — {reason}] to Daniel. Never invent output.

• pass4_empty: true: legitimate — Dig Report says "Pass 4 empty; pattern may be new to corpus."
• voice_anchor MISSING: brief not saved; Daniel prompted. This is correct behavior, not an error.
• All flags surface in the Dig Report / handler response, visible to Daniel, never swallowed.

8. Test plan (the honest one)

The machine-shop find in prior runs proves nothing — it was the charter's own worked example. Validate on a fresh brief whose deep cut is not pre-supplied:

1. Pick a new title from SynNA_00_TitleList.
2. Run the orchestrated pipeline. Confirm: 12 terms with ≥6 distinct (Step 1 gate visible);

a Dig Report returns and the handler STOPS (two-turn gate); the deep cut is a file no one named in the intake; the voice anchor is real first-person body prose, not a summary.

1. Confirm → brief assembles with voice_anchor: populated and banned_phrases: present.
2. Hand that brief to Arnie v2.0. If his CARBON_GATE passes on first or second revision and the

prose carries the voice — the loop is closed end to end.

If the orchestrated pipeline produces a real dig where the monolithic turn did not, the problem was never the model — it was asking one turn to do ten things. If a specific step still wobbles, route that one step's model via mcfg and re-test only that step.

9. Migration order (smallest reversible steps)

1. Add _kb_read, _parse_corpus_index — pure, testable alone.
2. Add Step 1 + _validate_fanout_terms; log output, don't wire to brief yet.
3. Add Steps 2–4; print scored candidates to a debug endpoint; eyeball against a known topic.
4. Add Step 5 excerpt slicing + gates; unit-test on a VoiceLog (first-person) and a Formation

file (third-person digest) — confirm the digest is rejected.

1. Wire the two-turn Dig Report into brief_button_handler; keep the old free-form path behind

a flag for A/B.

1. Add Step 6 + gates; flip Forrest to the orchestrated path; retire the free-form turn once the

fresh-brief test passes.

Each step ships independently and is revertible. No big-bang rewrite of a 14.6k-line file.

Ask Daniel's CODEX