Introduction

draft first full version, 2026-07-18. Tracked as sail-0jz4.

SAIL (Socratic AI Learning Lab) is Torsten’s bootstrapped continuation of the SAITO research thread after the SPRIND NFAI, BiTS, and Foresight Berlin funding rejections (all June 2026). This compendium exists because trying to summarize all of it in a single short, personal follow-up email didn’t work — there was too much real material to compress, and the email also needed to stay short and personal. So instead: publish what’s actually real — ideation, scope, refinements, research, and educational material — as one compendium, and point each collaborator at the parts most relevant to them.

The original AIDLE pathway

Before the individual chapters, it’s worth knowing the bigger shape SAITO was originally scoped in, so terms like “the scaffolding model” and “the trajectory model” have a home. AIDLE — the proposal this whole thread descends from — split SAITO into three coupled questions, each with its own named component: WHAT to ask (the SSA, Socratic Scaffold Agent), WHO you’re asking (the STM, Skill Trajectory Model), and WHEN to ask it (the TSS, Temporal Scaffolding Scheduler) — a closed loop, where an outcome measurement weeks later feeds back into updating WHO. The full figure and a precise built/not-built breakdown are in the Appendix; the short version:

• SSA (the scaffolding model) is built and live — sail-judge, the entire subject of chapter 1.
• STM (the trajectory model) is new territory — chapter 5 is a scaffold stub, not a design; nothing exists yet beyond the idea that it should.
• TSS isn’t started at all — no chapter covers it, and a cross-repo asset scan found no precedent for it anywhere, the same greenfield status as GAT enforcement in chapter 1.

Worth being precise about what this framing is and isn’t: it’s the original pathway, kept as a map so the parts of this compendium that exist can be located within it — not a claim that all three legs are underway. Exactly one is.

How this is organized

Each of the first four chapters is loosely framed around a specific collaborator’s own interests, without naming them explicitly in the text — the chapter boundary does the personalizing, not a name-drop. The fifth chapter is deliberately general rather than person-framed, since STM (above) is cross-cutting groundwork rather than one collaborator’s angle.

1. Bandit-arm design, Goodhart taxonomy & GAT — the design/theory thread; also where SSA — the one built leg of the pathway above — is actually explained in depth.
2. Learning-science grounding & joint study/HPC design — the learning-research thread.
3. EEG/fMRI as a human-grounding reward signal — the neuroscience thread, plus a live workshop angle, plus a speculative survey of what might work better than EEG for a live signal.
4. HTP projective-analysis toy example — a newer, still-unscoped neuroscience/projective-analysis thread.
5. Skill Trajectory Models — the STM leg of the pathway above; general, not collaborator-framed; not yet drafted beyond an outline.

Each chapter’s outline and source material is tracked as its own bean — see the chapter’s own “Status” line for the exact bean ID.

What’s real vs. what’s speculative

This compendium tries to keep an honest line between what’s actually running (the sail-judge scaffold agent, live on trullala; the Thompson-sampling notebook; the artifacts/ design-note series) and what’s still an open question, a parked idea, or a toy example that hasn’t been scoped yet. Where a chapter is speculative, it says so — this is meant to read as “here’s what ’s real, not a pitch deck,” the same framing sailing.tilde.style already carries.

Bandit-Arm Design, Goodhart Taxonomy & GAT

v2026.07.3 last finalized 2026-07-19. One marked change below (Open Questions, panproto note) not yet reviewed. Tracked as sail-7ie2.

This chapter is about the design of sail-judge, SAIL’s live scaffolding agent — how it decides what to say, why the first version of that design was too clever by half, and where Generalized Algebraic Theories (GATs) turn up, twice, in ways that turned out to be genuinely different questions wearing the same three letters.

The design arc: from rhetorical styles to research constructs

sail-judge runs in a real chat room (currently on trullala, over chatto.tilde.style) and, on its turn, has to choose how to intervene. The governing framing from day one — SAIL’s own recurring tagline for this: an agent that chooses to answer or ask. A discourse-driver role judges whether to speak at all; an asker role, when it does speak, asks a generative, clarifying question rather than giving an answer. (The underlying facts substrate — a normalized subject/predicate/object/confidence/disputed/source-event table — was itself evaluated against ElectricSQL’s Burn demo before being adopted, not invented from scratch; that’s the same FactStore the Rust port carries as Restate-durable state.) The first working version picked between three rhetorical styles per role — asker: curious / concrete / reflective; discourse-driver: synthesize / contrast / extend — using Thompson sampling over hand-rolled Beta posteriors (Marsaglia–Tsang gamma sampling under the hood, the same algorithm family as Ariel’s graze project for ATProto custom feeds). It worked, in the narrow sense that the mechanism ran end to end against a real conversation and updated its weights correctly.

It was also the wrong thing to be learning. The arms were generic conversational postures, not grounded in anything about how people actually learn — which meant the bandit was free to converge on “asks things that make people reply with more questions,” a pattern indistinguishable from being mildly confusing. Rewarding engagement isn’t the same as rewarding learning, and nothing about the v1 design could tell the two apart.

The fix wasn’t a better reward function bolted onto the same arms — it was redefining what the arms are. The current design grounds each arm in a named construct from the learning-science literature:

• asker arms: generation effect (Slamecka & Graf), retrieval-practice / testing effect (Roediger & Karpicke), desirable difficulty (Bjork), and metacognitive calibration (asking for a confidence estimate before checking).
• discourse-driver arms: Wood, Bruner & Ross’s (1976) three scaffolding functions — marking critical features, reduction in degrees of freedom, direction maintenance.

Worth naming explicitly, because it’s a real design bet rather than an obvious default: splitting “teacher” into an asker role (knowledge-transfer, testing) and a discourse-driver role (process-facilitation) promotes something that, in a human teacher, is usually an informal personality trait — some teachers lean instructor, some lean mentor — to the level of a structural, addressable choice. That’s a deliberate instructor-to-mentor shift, not a neutral engineering decision, and it’s worth saying so plainly rather than presenting the two-role split as if it fell out of the problem on its own.

Goodhart’s Law, made actionable

Naming the constructs doesn’t make the reward signal honest by itself. Each arm still needs a cheap, live-computable proxy for “did this intervention actually work” — and every proxy is a target Thompson sampling will happily overfit to, in whatever way is cheapest. Manheim & Garrabrant’s taxonomy (regressional / extremal / causal / adversarial) gives a vocabulary for how a given proxy is likely to fail, arm by arm, before it fails live:

This isn’t just a theoretical exercise — the failure modes it predicts have already shown up live, in a real running room: a marking-critical-features reward once fired on a coincidentally shared word (“last”) rather than genuine feature-noticing, and a later pass caught the same class of bug again with generic filler words (“earlier”) producing false-positive overlap. Both are exactly the regressional failure the table above predicts for that arm. A confidence-detection regex also had a real, silly bug (a trailing \b right after % that could never match) — a reminder that a proxy can fail for boring implementation reasons well before it fails for interesting Goodhart reasons, and both need checking.

Seven Signals walks through both bugs as they were actually found and fixed live on trullala, plus the coverage this gave afterward (sail-judge.test.mjs’s 43 assertions, a 12/12 forced-arm testbed run) and the parallel judge’s first real disagreement with a regex evaluator — the verification half of this story, not just the two bugs summarized above.

This table is the condensed version of a longer live argument — The Question Mark Problem, reproduced in full in the Appendix’s Artifact Series, walks through how the arms went from rhetorical styles to these named constructs, and how the convergence table above was actually arrived at (two independent framings — an instructor/mentor collapse, and the Goodhart/Ostrom-commons mechanism — converging on the same per-arm classification).

Why not just use an LLM as the judge?

The obvious next move — replace the regex heuristics with an LLM-as-judge reward — doesn’t dodge Goodhart’s Law, it relocates it, in three specific ways:

1. Adversarial: the generator and judge sit inside the same optimization loop, which is exactly RLHF’s documented “reward model overoptimization” problem (Gao et al. 2022) — the policy converges on examples that fool the judge, not on genuinely good outputs.
2. Regressional: ordinary judge biases (verbosity/fluency preference, self-preference if judge and generator share a model family).
3. Causal: judge-approval and real learning gain likely share an upstream cause (surface plausibility) rather than the judge’s approval causing the real outcome.

This connects directly to a tension the AIDLE proposal had already staked a position on before any of this was built: its “Third Axis of Frontier AI” framing contrasts US labs’ RLHF (unverifiable rater preference — the same failure mode as an LLM-judge reward) against the RLVR path (replacing the rater with a checker — a compiler, a unit-test suite, a formal proof verifier), with the proposal’s own honest caveat that “no checker exists for is this the right question to ask this learner now” — RLVR doesn’t reach pedagogy. AIDLE’s proposed third path was a GAT-constrained, symbolic-grounding layer instead of either a rater or a narrow checker. sail-judge’s reward-evaluator design space — cheap regex proxy vs. LLM-judge vs. something GAT/schema-constrained — turns out to be a small, concrete, currently-running instance of that exact three-way tension, not just an analogy to it.

Thompson sampling — one solution, not the only one

However the reward gets fixed, the sampling mechanism choosing between arms is already about as good as it can be. For this problem class (Bernoulli rewards, independent arms), Thompson sampling has a proven regret bound (Agrawal & Goyal, 2012/2013) matching the theoretical lower bound — it converges on the best arm about as fast as is mathematically possible, for whatever reward it’s given. That’s the real lesson from building the Julia/Pluto teaching notebook that replays sail-judge’s actual historical fire/reward sequence: a more sophisticated bandit implementation would still converge just as efficiently on whatever the reward signal actually measures, which may or may not be what SAIL cares about. The lever that matters is the reward signal, not the sampler.

Bandits, Recapped is the non-interactive companion to that notebook — same running example (sail-judge’s actual arms, not an abstract slot machine), theory-only where the notebook is data-grounded and interactive; the notebook’s own intro points readers here first if the mechanism itself needs explaining before the historical replay does.

The notebook itself is now live and genuinely interactive, not a description of one — see the Teaching Notebooks in the Appendix.

Thompson sampling is one well-studied member of a wider family of solutions to the explore/exploit multi-armed-bandit problem, worth naming so it doesn’t read as the only option:

• Upper Confidence Bound (UCB) — deterministic, optimism-under- uncertainty: pick the arm with the highest upper confidence bound on its estimated reward, so under-explored arms get an automatic boost.
• Epsilon-greedy — the simplest baseline: mostly exploit the current best arm, explore a random arm with fixed probability ε. Easy to reason about, generally less sample-efficient than either of the above.

None of these change the argument above — swapping the sampler doesn’t touch what’s actually being optimized for.

Where GAT shows up — two genuinely different questions

“GAT” comes up twice in this thread, and they’re worth keeping separate — conflating them was a real risk early on.

Sense 1 — GAT as a possible answer to the reward-verification problem above. Could a categorical-lens approach (Milewski’s categorical lenses; Patterson & Lynch et al. 2024, arXiv:2404.04837; the panproto/GATlab integration) give a verifiable, schema-checked mapping from raw signal to reward classification — something between a noisy regex heuristic and a black-box LLM judge? This is still an open, speculative research question, not yet started as design work. It would need real GAT expertise to answer honestly rather than improvised from the GATlab paper alone. A separate cross-repo asset scan (2026-07-11, covering everything currently running across haiku.garden, tilde.style/tilde.cards, and tracker) came back with the same honest verdict independently: GAT schema enforcement has no code or spike anywhere yet, and isn’t close to being de-risked by existing assets the way most of the rest of SAIL’s build sequence is. It’s genuinely greenfield, not just under-prioritized.

Sense 2 — GAT as the literal Rust language feature, already shipped (unrelated pun, same three letters). This is the part with an actual formal derivation behind it, worked out in a standalone spike (2026-07-13-persona-role-arm-gat-spike.md) that’s worth walking through in more detail than a one-line summary, since it’s the part of this chapter with the most concrete material for a real GAT conversation — formal signatures Aaron can react to and push back on directly, not just prose.

The fibration, formally

The spike’s starting move: write sail-judge’s actual current mechanism down as a Generalized Algebraic Theory (Cartmell 1986) rather than describing it in prose. RoomState is the conversational state a turn is evaluated against; the two live counters become operations on it; the two threshold constants become constants of the theory. Role is a sort whose terms are produced by a guarded, priority-ordered derivation — not an enumeration handed down from nowhere:

Γ ⊢ s : RoomState    msgsSinceQuestion(s) ≥ ASK_AFTER_NO_QUESTION
──────────────────────────────────────────────────────────────────  (asker-intro)
Γ ⊢ fires(s) : Role                           [fires(s) ≡ asker]

Γ ⊢ s : RoomState    msgsSinceQuestion(s) < ASK_AFTER_NO_QUESTION
                     msgsSinceReply(s) ≥ SPEAK_EVERY
──────────────────────────────────────────────────────────────────  (driver-intro)
Γ ⊢ fires(s) : Role                   [fires(s) ≡ discourseDriver]

driver-intro’s guard carries the negation of asker-intro’s guard verbatim, as part of its own side-condition — which is what makes “asker wins ties” a derived fact about the theory (the two guards are mutually exclusive by construction) rather than a prose note about if/else if ordering. If neither guard holds, fires(s) is simply undefined at s: no term of Role is derivable, and the no-op case sits outside the sort entirely rather than existing as a third term with an empty fiber.

Arm is a dependent sort indexed by Role — Arm(asker) and Arm(discourseDriver) are disjoint object literals in the actual code, two independently-seeded Beta-Bernoulli posteriors, and there is no arm value that’s even well-formed for both roles. One extra wrinkle inside the asker fiber: a scheduled override for desirable-difficulty, which introduces a second term-formation rule targeting the same dependent sort:

Γ ⊢ fires(s) ≡ asker    askerFireCount(s) mod DESIRABLE_DIFFICULTY_EVERY = 0
───────────────────────────────────────────────────────────────────────────  (scheduled-override)
Γ ⊢ arm(s) : Arm(asker)                [arm(s) ≡ desirable-difficulty]

Γ ⊢ fires(s) ≡ asker    askerFireCount(s) mod DESIRABLE_DIFFICULTY_EVERY ≠ 0
───────────────────────────────────────────────────────────────────────────  (bandit-draw)
Γ ⊢ arm(s) : Arm(asker)                [arm(s) ≡ bandits.asker.choose()]

One deterministic-scheduled rule and one stochastic rule, both still confined to the same fiber — neither can produce a term of Arm(discourseDriver). The natural entity underneath all of this is the total space of the fibration, the dependent-sum sort:

Move := Σ_{r : Role} Arm(r)

A term of Move is a pair (r, a) with a : Arm(r) — you cannot construct one without committing to r first, and committing to r immediately restricts which a are even typeable. That’s the actual content behind the intuition that role-choice and arm-choice were never two independent mechanisms glued together after the fact: they’re the base-projection and fiber-selection of building one term of one sort, in two stages. The two-object, two-branch shape of the real code is a presentation choice, not a semantic claim that two separate mechanisms exist.

This matters for more than tidiness, because a morphism between GATs has to preserve the whole dependent family, fiberwise — it can rename or even merge/split fibers, but it can’t flatten Role × Arm into an unstructured product and permute across that flattening, because that’s a map of a strictly weaker theory (a plain product), not this one. Which gives three concrete, checkable invariants any legitimate extension of this design has to respect: the fiber-cardinality profile (3+1 for asker, 3 for discourse-driver); discreteness of the fibration itself; and, as a direct consequence of discreteness rather than a separate design choice, a block-diagonal joint posterior — asker’s and discourseDriver’s arm draws should show zero statistical cross-dependence. That last one is genuinely falsifiable against the real event log (sail-judge-events.jsonl) and hasn’t been checked yet — if it turns out false, the fibration is leaking, which is itself useful information, not just a bug.

Once sail-judge was actually ported to Rust, this stopped being only a paper exercise: Rust’s own associated-type generics now enforce the Move := Σ_{r:Role} Arm(r) structure at compile time — AskerArm/DiscourseArm enums, exhaustively matched — replacing a private duplicated arm-string list that had already drifted out of sync once, and a bare .expect("unknown arm key") that could previously panic on any bad caller-supplied string. A real, already-landed fix, entirely separate from Sense 1’s still-open verification question above.

From formal fibration to mined roles: a slow theory, a fast theory, and a morphism between them

The two hardcoded roles (asker/discourseDriver) are themselves an invented category, the same way v1’s rhetorical arms were — chosen by hand, not mined from anything. The same move that fixed the arms (ground them in something real, not invented) has an analogous, not-yet-built target here: mine Role itself from real observed community behavior rather than hand-inventing it. There’s already real, in-progress infrastructure this could build on rather than starting from a proposal: a firehose-consuming Bluesky scanner is already running (curating a real candidate account list for a haiku/poetry community), built on existing tab/watchtower infrastructure — genuinely available data, not hypothetical.

The first framing of this ran into a real timescale mismatch, worth stating plainly rather than glossing over: the actual ground-truth signal for “this is a genuine connection between two distinct ideas, not just surface word overlap” is slow — a human curator drawing typed, directional links between saved items over weeks (Semble’s own Connection object between two Cards is exactly this relation, already real and queryable; a hand-written “Threads” section in a separate notes file is the same relation under a different name). Role/persona choice, by contrast, has to happen at conversational speed, within a single live exchange. Querying the slow corpus directly, mid-conversation, doesn’t fit that timescale — so this can’t be a live lookup.

The resolution is an offline distillation step, not a live query: extract, offline, from the real slow corpus what actually characterizes a genuine connection between ideas (as opposed to superficial word overlap), and let that extraction’s output be a fast, “synthetic” artifact — a small classifier, an embedding-similarity check, or something else, not yet decided — that runs live. Stated as two GAT signatures and a morphism between them rather than a bare ML pipeline:

SlowTheory                                    (curation timescale — weeks, human-curated)
  Sort  Card
  Dependent sort  Connection(c₁ c₂ : Card)    — only well-typed between two actual Card terms

FastTheory                                    (conversation timescale — the live process)
  Sort  Role,  Dependent sort  Arm(r : Role),  Move := Σ_{r:Role} Arm(r)     — as above
  Dependent sort  PersonaHint(m : Move)        — new: which mined behavioral
                                                  archetype a given Move resembles

Distill : SlowTheory → FastTheory             — computed offline, once;
                                                 its *action* on terms (the
                                                 classifier/similarity-check)
                                                 is what runs live

PersonaHint is indexed by Move, not free-standing, for the same reason Arm is indexed by Role — a persona-hint meaningful for an asker-shaped move need not even be well-typed for a discourseDriver-shaped one. This is the same shape as the categorical-lens idea in Sense 1 above — a versioned, bidirectional schema mapping extracted from observed data, not designed top-down and not a black-box learned judge either. What the GAT framing adds over treating this as a bare ML pipeline: “is the distillation still faithful to the real data” becomes a checkable morphism law — does Distill commute with Connection’s own dependency on two distinct Cards, so that a genuine cross-card connection lands as a within-fiber Move rather than accidentally collapsing two fibers together — rather than only an offline eval metric with no structural guarantee behind it.

What this means concretely for anything that mines raw data (a “Tab-miner”): it only needs to emit SlowTheory-shaped material — a per-record tuple like {did, collection, record_uri, observed_at, text}, enough to stand as a candidate Card term. It does not need to know Role or Arm exist at all. Mining is the slow/real side; Move/ PersonaHint are the fast/synthetic side; the two connect through exactly one designed morphism — a real layering boundary, useful for scoping any future collaboration on this piece specifically.

Three candidate roles, offered as hypotheses to argue with

None of these are commitments — they’re deliberately concrete guesses, offered so a real mining pass (or a conversation with Aaron) has something to react to or falsify, rather than starting from nothing:

• Synthesizer — notices when two different people (not the same speaker across turns) have made related-but-unconnected points, and names the connection. Structurally, this is the cross-person analogue of Semble’s Connection object, not anything the existing marking-critical-features arm already covers (which only operates on one person’s most recent utterance). A real, checkable reward construct falls out for free: did the two people actually engage with each other’s point afterward, not just individually with the bot.
• New-arrival greeter — fires specifically on a new participant’s first message in the room, oriented at onboarding rather than Socratic pressure. Low-status, side-task engagement; needs fast orientation, not more friction.
• Persistent-thread minder — notices a topic recurring across sessions (a multi-day horizon), not just within one conversation window — a genuinely different need from the moment-to-moment roles above.

Worth being explicit about the source of the pattern, not the content: these were drafted by analogy to a persona workshop for a sibling project, which is a useful structural precedent (real, observable axes — need, context, frequency of engagement, characteristic frustration — rather than invented rhetorical labels) but not itself a source of new SAIL roles. Synthesizer/New-arrival-greeter/Persistent-thread-minder are SAIL’s own hypotheses, sized to be testable once real mined data exists, not imports.

Distill, concretely — candidate implementations

Not drafted yet — a placeholder for a real follow-up pass, not a design.

• Candidate shapes for Distill’s actual implementation, none chosen: a trained embedding-similarity classifier over (Card, Card) pairs; a smaller symbolic rule-set extracted directly from Semble’s Connection metadata (its type/direction fields); or a hybrid of the two.
• What “checking the morphism law” would concretely look like as a test: sample known Connection pairs, run Distill’s action on them, verify the resulting Move-level tags don’t cross fibers — an actual test to write, not just a property to assert.
• The real data-volume question, unanswered: how many genuine Connection examples exist today (Semble plus riddles.md’s Threads section) versus how many a first classifier attempt would plausibly need to be worth trying.
• Not yet scoped: whether this becomes its own bean, or stays folded under sail-i9j3 until there’s enough mined data to make it concrete.

What happens if the arm space grows a lot? A real collapse question

This subsection’s namesake: The GAT-Collapse Question is the discussion-opener version of everything below, written to be argued with directly rather than read passively — every claim here traces to 2026-07-13-persona-role-arm-gat-spike.md, and that artifact is the fuller account, including a fibration diagram and the SlowTheory/FastTheory thread only summarized in passing here.

A question worth asking before scaling this design up rather than after: if the arm space grows drastically, does the role/arm split eventually collapse into one thing? The answer turns out to depend entirely on how the growth happens, and there are two structurally different ways to do it, with opposite answers.

Case A — fiber-preserving expansion: no collapse. Grow each role’s arm list — Arm(asker) and Arm(discourseDriver) go from 3–4 named, literature-grounded terms each to 50 — but stay what they are today: two separate object literals, two separately-seeded posteriors. Nothing qualitative moves; Move is still exactly the Σ-type above, just over bigger fibers, and the block-diagonal posterior property still holds exactly as stated. Worth stating plainly because it’s the intuitive-but- wrong expectation — “more arms” alone doesn’t push the design anywhere near collapse.

Case B — fiber-merging expansion: real collapse, and a specific kind. Collapse requires the added dimensionality to come from moving arms into one shared continuous space used by both roles — arms become points in a common embedding, ℝᵏ, rather than two disjoint named lists. Here the dependency direction actually reverses:

roleOf : ArmSpace → Role      — Role becomes a DERIVED quotient of Arm,
                                 recovered from which region of ℝᵏ a chosen
                                 arm-point falls in, not a prior index that
                                 constrains which arms are even typeable

This is a genuine reversal of base and fiber, not the same fusion as the Σ-type above (which kept Role as base throughout) — Role becomes epiphenomenal, recoverable post-hoc from a unified arm-choice, rather than something chosen before the arm and constraining it.

The realistic middle ground more plausibly sits between the two: Role acting as a soft prior or mixing weight over regions of arm-space, closer to a mixture-of-experts architecture than either a hard partition or a fully-collapsed readout. Concretely, this replaces both the guard cascade and the two per-role bandits with a single gating mechanism:

κ : Context → Dist(K)          — a gating function over K shared experts,
                                  not necessarily aligned with asker/
                                  discourseDriver at all
roleOf : ArmSpace → Dist(Role)  — Role as a DERIVED RANDOM VARIABLE, not a
                                  hard quotient — the fibration itself goes
                                  "soft"

This genuinely isn’t a plain GAT anymore — a strict GAT’s term-formation is all-or-nothing (the guards above are crisp), but a gating function assigns a distribution over experts, not a term. The natural home for this is a GAT enriched over a probability monad (the Giry monad / the Kleisli category of distributions) rather than plain Set. Turning that soft quotient’s peakedness all the way up recovers Case A (today’s design) as a limit; turning it fully deterministic — while keeping it derived, not primary — recovers Case B.

Why this is worth taking seriously rather than filing as a tangent: the two previously-considered options for replacing the current triggering mechanism (a two-level Thompson bandit, or a smarter deterministic eligibility signal) both risked or explicitly avoided the same failure — two nested discrete learning layers both trying to converge on the same sparse live-conversation data volume. A single joint gating function sidesteps that particular risk (one learning problem instead of two nested ones) — but it doesn’t sidestep data-hunger in general: fitting any continuous gate over a richer context typically needs more data than two tiny, extremely sample-efficient Beta-Bernoulli bandits over 3–4 discrete arms each. A trade, not a strict improvement, and shouldn’t be oversold as one.

And it connects back to the distillation morphism above rather than competing with it: Distill’s natural training source (the slow, real curation corpus — Semble connections, hand-written thread notes) could just as well produce this gating kernel as its output, rather than a hard classifier label — PersonaHint(m : Move) becomes the kernel itself, deployed frozen (or slowly re-trained) at conversation speed. Read this way, mixture-of-experts isn’t a rival architecture to the Tab-miner proposal; it’s a candidate answer to what Distill’s output shape should actually be.

Pushing the probabilistic-GAT tail further

Not drafted yet — a placeholder for a real follow-up pass, not a design.

• What would it take to state the Giry-monad / probabilistic-GAT extension above precisely enough to be checkable, rather than gestured at — which categorical-probability formalism actually fits, and does it match how GATlab/panproto already handle probabilistic extensions, or is this a genuinely different move?
• Concretely, what would κ : Context → Dist(K) look like as code — a small mixture-of-experts head over a shared embedding, something built on an existing probabilistic-programming library, or neither?
• How would this actually get tested against the discreteness-violation check already flagged above (cross-fiber dependence in the real event log) — would evidence of leakage there directly motivate building this, or is that too small a signal to justify a probabilistic extension at all?
• A real cost/benefit question, not yet answered: is this worth prototyping before there’s a live use for more than two roles / more than a handful of arms per role — i.e., before the discrete design actually starts straining?

A concrete near-term experiment this feeds: the SAITO ablation

Everything above — the Goodhart taxonomy, the Third-Axis framing, the two senses of GAT — is either live-but-small (the actual prompt-selection bandit) or still speculative (the GAT questions). There’s a third piece, designed but not yet built, that sits between them and is worth Aaron seeing specifically: an actual ablation that trains model weights, rather than selecting between prompts over a frozen large model, to test the Third-Axis claim directly rather than just argue for it.

The claim under test, stated precisely: a policy optimized for a delayed, outcome-based reward (did the learner retain/transfer the material later) behaves differently — and is harder to Goodhart — than a policy optimized for an immediate, rater-style reward (did this reply seem helpful right now), even at small scale. GPT-2-scale (124M–410M), not a literal 2019 GPT-2 checkpoint — a modern small open model (SmolLM2, Pythia, or Qwen2.5-0.5B) is the actual candidate.

Environment. No real human RL loop is feasible at this budget, so this runs in a synthetic tutoring environment: a frozen “student” model (not trained) plays a learner with deliberately incomplete knowledge, conversing for a short multi-turn exchange with the policy model actually being tuned. Afterward the student is quizzed on held-out items from the same topic — reusing renate.chat’s already-validated Zerbe Top-10 item set as real content rather than inventing quiz items. Quiz performance is the downstream-competency signal; a separate same-turn preference judgment is the immediate-satisfaction signal. Only the policy needs gradient updates, so the frozen student can be a stronger model than the policy without inflating training cost.

Method: DPO, not PPO. Direct Preference Optimization needs only preference pairs (chosen vs. rejected trajectory) trained against a closed- form loss against the base checkpoint — far less infrastructure than a live reward model plus value function plus on-policy rollout loop. Concretely: sample several candidate tutor replies per turn from the frozen base checkpoint, roll out the full dialogue + quiz for each candidate branch, score every branch two ways (quiz score, immediate preference), and build two separate preference-pair datasets from the same rollouts — one ranked by competency outcome, one by immediate preference. Two independent DPO fine-tunes from the same base checkpoint follow, one per dataset: Policy-Competency and Policy-Satisfaction.

Evaluation. Three conditions on held-out topics: Policy-Competency, Policy-Satisfaction, and an untrained baseline (the base checkpoint, zero-shot-prompted with sail-judge’s actual ASKER_ARMS/DISCOURSE_ARMS templates as the control — direct reuse, not a new prompt design). The predicted result, if the Third-Axis argument holds: Policy-Satisfaction scores higher on immediate preference but lower on downstream quiz score than Policy-Competency — the Goodhart divergence, demonstrated rather than argued. Either outcome is reportable; a null result (no divergence) is a real, publishable finding too. One genuinely open design decision worth flagging directly: whether the immediate-satisfaction signal itself should be a cheap heuristic or an LLM-as-judge — using the LLM-judge here arguably is the point, since the ablation is trying to show that reward’s failure mode, not avoid it.

Explicit limitation, stated up front rather than glossed over: this measures model-to-model dynamics in a synthetic environment, not real human learning — a mechanism pilot, not a claim about real students.

Where it stands. Design-complete, not started — no code, no data yet. Compute sizing has already been corrected once: the original bootstrap plan’s “~2,000 GPU-hours” figure was sized for the full €3M-grant version of this work, not this pilot; a DPO fine-tune at this scale realistically needs low tens of A100-hours per policy variant, sized into a single short ZIH Alpha Centauri allocation rather than the original grant-era number carried forward unexamined. Blocked on compute landing (either ZIH or Cerebras Ambassador credits), independent of the human mini-study — same deliberate separation the original proposal made between the SAITO bet and the LEARN/benchmark bet.

Open questions

• Is GAT-constrained reward verification (Sense 1) worth building for real, and does it actually solve the adversarial/regressional/causal risks better than the alternatives, or just relocate them again? Sharpened (2026-07-22): given panproto’s actual mechanism is structural schema-checking, not free-text semantic verification, the prior question is now less “is this worth building” than “is this even the right tool for the question” — open in a different way than before.
• What would an “observatory” layer — scanning reward-proxy behavior through versioned, lens-mapped vocabularies rather than either a fixed regex or a learned judge — need to become concrete rather than a design sketch?
• Does the discrete-fibration model (Sense 2) actually hold in the real event log — is there already measurable cross-role dependence between asker’s and discourseDriver’s arm draws that would mean the model is leaking? Not checked yet.
• Distill’s concrete implementation — classifier, embedding-similarity, the gating-kernel shape above, or something else entirely — is genuinely undecided, not just unbuilt.
• Which of the three dimensionality-scaling regimes (fiber-preserving / fiber-merging / the probabilistic middle ground) does the live system currently sit closest to, once there’s enough real event volume to ask?
• For the SAITO ablation: should the immediate-satisfaction signal be a cheap heuristic or an LLM-as-judge (arguably the more honest choice, since the point is to demonstrate that reward’s failure mode), and which small base checkpoint actually fits whatever compute allocation lands first?

Supplementary reading

Five self-contained design notes were written about this material as the work happened — see the Artifact Series in the Appendix, not repeated here.

Learning-Science Grounding & Joint Study/HPC Design

draft first full version, 2026-07-18. Tracked as sail-wm67.

The bandit-arms-gat chapter covers what sail-judge actually does and how its reward signal can fail. This chapter is about the other half: where the arm constructs themselves came from, why they’re the right ones (or aren’t yet), and the actual, live ask that follows from taking that seriously — a joint pre-registered study and a shared compute application, not a hypothetical.

Why these constructs, specifically

The v1 arms (curious/concrete/reflective; synthesize/contrast/extend) were rhetorical postures — plausible-sounding, ungrounded in anything about how people actually learn. The current arms replace each one with a named construct from the learning-science literature, chosen deliberately, not just relabeled:

• Generation effect (Slamecka & Graf) — people retain material better when they generate an answer themselves rather than being shown it. Grounds the asker’s generation-effect arm directly.
• Retrieval-practice / testing effect (Roediger & Karpicke) — the act of recalling something, not just re-exposure to it, is what strengthens memory. Grounds retrieval-practice.
• Desirable difficulty (Bjork) — conditions that make learning feel harder in the moment can produce better long-term retention than smooth, easy practice. Grounds desirable-difficulty — and, as the bandit-arms-gat chapter’s Goodhart table already shows, is exactly the construct that breaks any live reward proxy, by definition: a proxy that rewards smooth immediate success is rewarding the opposite of what this construct is.
• Metacognitive calibration — asking for a confidence estimate before checking an answer, tied directly to the mini-study’s own KPI (below).
• Wood, Bruner & Ross (1976)’s scaffolding functions — marking critical features, reduction in degrees of freedom, direction maintenance — ground the discourse-driver’s three arms. This is a facilitation paper, not an instruction paper, which is part of why the discourse-driver role reads as the “mentor” half of the instructor/mentor split named in the previous chapter.

None of this is a settled taxonomy — it’s a first, citable pass, offered specifically so it can be argued with by someone whose actual field this is.

Where the current design is weakest

Two separate weaknesses, not one:

• The trigger model — when either role fires — is still just a raw ‘?’-count and a fixed turn interval. It never got the same research-grounding treatment the arm-selection side did. Nobody has proposed what a learning-science-grounded triggering signal would even look like yet. The Question Mark Problem (Appendix: Artifact Series) walks through where that raw count came from and why nobody has revisited it — this weakness and that history are the same open question, not two separate ones.
• The reward system — covered in full in the bandit-arms-gat chapter’s Goodhart table, but worth restating the punchline from a learning-science angle specifically: five of the seven arms are regressional or causal risks (a cheap proxy can drift from the real construct without anyone noticing), two are extremal risks (the proxy holds only until optimization pushes hard against it), and one — desirable-difficulty — has no honest live proxy by construction, not as an engineering gap to close. That last one can only ever be validated against a delayed, real outcome measure. Which is exactly what the mini-study below exists to provide.

Grounding further: Hummel (2025)

Sandra Hummel’s own completed study — Higher Education Under Generative AI: Biographical Orientations of Democratic Learning and Teaching, Education Sciences 15(12):1572 (n=151: 122 students, 29 lecturers, grounded-theory analysis of written articulations) — is real data to build on, not just a citation for KPI-realism. Her five reconstructed orientations: pragmatic (coping with workload), adaptive (learning under opacity), relational (authority and resonance), ambiguous (improvisation and fragility), recognition (voice and visibility) — synthesized into three axes: temporal sovereignty, epistemic opacity/accountability, recognition ecologies.

Honest status: this grounds the next arm-taxonomy pass, not the current one. Nobody has yet checked whether these five orientations map onto the asker/discourse-driver arms more directly than the generation-effect/ testing-effect/Wood-Bruner-Ross taxonomy already does, or whether they operate at a different level entirely — learner disposition rather than agent action — and should inform something else (the reward signal, or a future persona model) instead of the arms themselves. Real, unresolved, not glossed over.

The actual ask: a joint study and a compute application

Two concrete, live pieces, not a hypothetical collaboration:

• A pre-registered scaffold-vs-answer mini-study, to Hummel’s own KPI-realism standard: d ≥ 0.4, a 2-week retention follow-up, a Zerbe-style item set, n in the dozens of vocational trainers. This is the study that would actually settle whether desirable-difficulty (and the rest of the arm taxonomy) tracks real learning gain, not just a proxy for it.
• A joint HPC compute application, naming Hummel as PI on her ScaDS.AI Young Investigator standing (which resolves an eligibility gap a solo application would have hit), scoped to a concrete, modest ask: a short Alpha Centauri (A100) allocation for a GPT-2-scale SAITO ablation — training actual model weights via DPO to compare a downstream-competency reward against an immediate-satisfaction reward, reusing sail-judge’s own prompt templates and Goodhart-variant table as the interpretive template for whatever divergence shows up. Not the €3M version of this work — a small, real pilot sized to what a bootstrap can actually run.

This is a real, sent ask, not a pitch — the compendium link went out alongside it, this chapter is what that link points to.

Open questions

• Do Hummel’s five orientations map onto the existing asker/discourse-driver arms, or do they belong somewhere else in the design (the reward signal, a future learner model)?
• What would a learning-science-grounded triggering signal (replacing the raw ‘?’-count) actually look like — is there a construct as citable as the ones already grounding arm selection?
• Is the pre-registered mini-study’s power (n in the dozens) actually sufficient to detect d ≥ 0.4 given real-world attrition over a 2-week follow-up — worth a real power calculation before this goes further, not an assumption.

Sources

Slamecka & Graf (generation effect); Roediger & Karpicke (testing effect); Bjork (desirable difficulties); Wood, Bruner & Ross (1976, scaffolding functions); Hummel, S. (2025), Education Sciences 15(12):1572 — references/hummel-2025-biographical-orientations.pdf.

EEG/fMRI as a Human-Grounding Reward Signal

v2026.07.1 reviewed by Torsten 2026-07-18, as good as this can be at this stage. Includes the "beyond EEG" speculative survey (originally parked as sail-4nwc). Not yet sent to Teon. Tracked as sail-zplm.

Honest framing, up front

This angle wasn’t part of the July synthesis run that produced everything else in this compendium. Hummel’s own KPI-realism review dropped EEG from the mini-study’s KPI line, and the bootstrap plan put passive BCI off the critical path — not because the idea is wrong, but because it’s hardware- and IRB-heavy in a way nothing else here currently is, and this whole thread is deliberately staying zero-cost and fast-moving. Worth saying plainly rather than overstating current involvement: nothing in this chapter is running. It’s a candidate direction, grounded in real, already-built work — this collaborator’s own — not a live component of sail-judge.

What already exists: BrainWaves

BrainWaves is a real, already-designed K-12 classroom EEG tool (design deck, November 2018, built around the EMOTIV EPOC-family headset — the device picker in the actual designs shows an EPOCPLUS-2378 pairing over USB) — not a sketch, a full four-stage application:

1. Review Design — pick a premade paradigm (a “Faces/Houses” experiment is the shipped example: two image categories, framed around why face perception matters for quickly reading trust and emotion) or build a Custom one. The custom builder is genuinely hypothesis-driven at the UI level: it has explicit Research Question, Hypothesis, and Methods fields before it lets you define stimuli, not just an image uploader. Two named stimulus groups, each with its own image folder and a mapped “correct” key response; trial timing set via ITI-duration and image-duration sliders (0.5–2s in the shown example) — a classic, discrete-trial two-alternative design, not a continuous-monitoring one.
2. Collect — headset pairing, then a live per-electrode scalp map color-coded strong/mediocre/weak/no-signal, next to the live raw trace. A built-in tutorial teaches signal-quality practice directly to whoever is running the session (saturate the sensors in saline, re-seat the headset with tension, sweep hair aside) and teaches noise recognition the same way (try closing your eyes, try clenching your jaw — does the signal change?) before a single real trial runs.
3. Clean — a dedicated editor to reject segments contaminated by exactly those noise sources, with its own short tutorial (“before you collected data… you learned how clenching one’s jaw or blinking can affect quality… now you need to reject segments that might be the result of one of these actions, not the stimulus”).
4. Analyze — event-related potential (ERP) output: per-condition averaged waveforms at each electrode, compared side by side (the shown example: Faces vs. Houses, 78 trials each, at a topographic electrode map).

This is real methods pedagogy embedded directly in a classroom tool, not just a wrapper around a headset SDK — worth naming as a serious, mature piece of design work in its own right, independent of whether it connects to anything SAIL is doing.

The honest gap: discrete trials vs. a live conversation

The concrete point that actually matters for SAITO: BrainWaves is built around stimulus-locked, repeated, discrete trials — a brief image shown for a fixed short duration, repeated dozens of times per condition, averaged afterward to pull a clean ERP out of noisy single-trial data (78 trials per condition in the shown example, and that’s for a simple two-category visual task with decades of established ERP literature behind it). It is not built for, and doesn’t obviously generalize to, continuous monitoring of an open, free-running, multi-turn Socratic conversation, where an asker or discourse-driver arm fires once, at an unpredictable moment, with no fixed short duration and no repeated identical stimulus to average against.

That gap is the actual research question here, not “can EEG detect something interesting” (it demonstrably can, per BrainWaves’ own ERP output) — it’s whether SAITO’s live interventions can be recast as something ERP-shaped at all. A candidate framing, not yet attempted: treat each moment a given arm fires (say, every generation-effect prompt across many conversations) as a stimulus onset, capture a fixed short window of EEG around it, and average across many firings of that same arm — literally reusing BrainWaves’ own Collect → Clean → Analyze pipeline, just swapping “Faces vs. Houses” for “generation-effect vs. retrieval-practice vs. …”. Whether that produces a clean, distinguishable ERP the way two visual categories do is a genuinely open empirical question, not a given — SAITO’s arms don’t share anything like the tight timing and stimulus-repetition structure a visual ERP paradigm relies on.

Worth being precise about what a positive result there would and wouldn’t give: even a clean, distinguishable per-arm ERP would only show that different arms produce distinguishable neural responses on average, not automatically hand back a usable live reward value for Thompson sampling. Averaging across dozens of trials happens offline, after the fact — this is exactly the same slow-signal/fast-runtime split already named for role-mining (sail-i9j3’s “we are building synthetic agent roles from real data,” and its SlowTheory/FastTheory/Distill morphism from the bandit-arms-gat chapter). If this is ever pursued, it’s worth framing from the start as another instance of that same pattern — a slow, real, offline-computed signal, distilled into whatever runs live — rather than assuming EEG could feed the bandit’s reward directly and in real time.

Why the fact-store is already shaped for this

sail-judge’s internal state was deliberately designed as a normalized fact store (subject/predicate/object/confidence/disputed/source_event) rather than a flat transcript, specifically so a future signal — EEG or otherwise — could be added later as another source-event type without redesigning anything. That’s a real, already-shipped design decision, not a promise. The door was left open on purpose; it’s just not open this month.

Nearby lab context

This collaborator’s own K-12 work at Gotham Data Clinic (BrainWaves, above) is a nearby, complementary lab context — a place already doing real EEG-in-education work, not itself SAIL’s infrastructure, and not something SAIL has any claim on.

The stretch idea: IOSP workshops

IOSP workshops (iosp.science) may already be silently applying Socratic ideas — multi-disciplinary researchers in the room is exactly the setting that could seize this kind of cross-collaborator connection in person, rather than it staying a compendium chapter. sail-i55b (the ATProto Bots hacking station, bandit.mjs+fact-store.mjs extracted to github.com/ATProto-Science/iosp-hacking-stations) is the concrete foothold already in place there.

Speculative: beyond EEG, what could actually work live?

Everything above establishes what EEG/ERP is good at (a clean, validated signal, per condition, averaged offline) and where it doesn’t fit (anything continuous and live). That still leaves the question genuinely open rather than closed: what, if anything, could work as a live, continuous human-grounding signal for a real Socratic conversation? None of the following is designed, built, or validated — offered as candidates for a real conversation, the same spirit as the three role hypotheses in the bandit-arms-gat chapter, not a plan.

Webcam-based, no new hardware beyond a camera:

• Eye-tracking — WebGazer.js (Brown University): pure JavaScript, runs entirely client-side, infers gaze point from an ordinary webcam, no video leaves the machine. Mature, actively maintained, privacy-friendly by construction.
• Pupillometry specifically — pupil dilation, distinct from gaze location, is a real, targeted marker of cognitive load/arousal in the literature; Jeeliz Pupillometry does this live, browser-only, off a webcam feed. Worth flagging as unusually well-aimed at a specific, already-named gap: sail-wqs3’s desirable-difficulty arm is the one construct with no honest live proxy at all, and pupillometry-as- cognitive-load is a plausible, specific candidate answer to that exact gap, not just generic extra signal.
• Facial affect/engagement recognition — an active research area literally aimed at detecting learner engagement/disengagement from webcam video in real time (OpenFace is the standard open-source toolkit; recent deep-learning classifiers target online-learning settings specifically). SAITO’s own question, being worked on under a different name elsewhere.
• rPPG (remote heart-rate/HRV from webcam skin-color changes) — real and published, but genuinely fragile: the literature itself flags unresolved questions about whether it captures moment-to-moment HRV reliably outside controlled lighting/motion conditions.

No new modality at all:

• Text-behavioral signals — sail-judge already runs over text chat. Response latency, typing duration, and message-length/complexity shifts are already implicit in the existing transcript. No camera, no new consent flow, no change to Chatto’s protocol.
• Multi-session accumulated trace data (STM) — the most already-real candidate of all: sail-3scs (deferred to Phase 3), already using the same facts-table substrate sail-judge runs on today, and literally the “trace-only” fallback the bootstrap plan named when EEG got parked in the first place. It tracks the same modality as the outcome itself (what a learner says, asks, and gets right over many sessions) rather than an indirect physiological correlate — a real advantage. But that closeness cuts both ways: it’s precisely what makes it tempting to treat “the trace pattern moved” as ground truth without checking, which is the same regressional-Goodhart risk flagged throughout this compendium, just harder to spot here since it doesn’t look like a proxy. It still needs the same external check — does the inferred trajectory actually correlate with held-out Zerbe-item performance, per the mini-study design (sail-1uq0) — before it’s trusted as a reward signal.

None of these dodge the validation crux this whole chapter turns on: checking any candidate against real learning gain is the actual work, and that’s learning science’s specialty (the testing-effect/retrieval- practice/desirable-difficulty literature’s whole methodology — pre- registered designs, delayed retention testing days-to-weeks later, real effect sizes — exactly what sail-1uq0’s mini-study already adopts). It’s not that neuroscience has no answer to the same question — there’s real research linking physiological signals to durable memory specifically (pupil response during encoding predicting trial-by-trial long-term memory formation is a genuine, published finding) — but that literature itself reports historically conflicting results and unresolved long-term predictive validity, nowhere near the unified, replicated status the testing-effect literature has, and nearly all of it is done in the same controlled, discrete-trial paradigm already flagged above — never yet bridged to anything like an open dialogue.

Open questions

• Which of the candidates above, if any, is worth prototyping first — and is any of it worth building before there’s a concrete reason (one of these collaborators wanting to try something, or Chatto growing video capability for an unrelated reason)?
• For the STM/trace-data candidate specifically: what would actually catch “the trace pattern moved but real learning didn’t” before it ever reached a live reward — is a held-out Zerbe-item check sufficient, or does it need something more?
• Does a discrete-trial ERP framing fit a live Socratic dialogue’s actual timing at all, given BrainWaves’ own trials run 0.5–2s and a real conversational intervention doesn’t have a fixed short duration?
• How many repeated firings of the same arm would a clean averaged ERP actually need, and does SAIL’s current live conversation volume get anywhere close to that (78 trials per condition, in BrainWaves’ own simpler two-category example)?
• What regulatory/consent machinery would this need beyond what a K-12 program already has — adult participants, a different institution, a different IRB context entirely?
• If a distinguishable per-arm ERP were found, what’s the actual path from “distinguishable on average, offline” to “usable as a live reward” — is that even the right target, or does this only ever make sense as a slow validation check against the existing cheap proxies (sail-wqs3’s Goodhart table), rather than a reward source in its own right?

Sources

BrainWaves App Designs (design deck, November 2018) — references/brainwaves-app-designs-2018.pdf. Tools named in the speculative section (WebGazer.js, Jeeliz Pupillometry, OpenFace) and the rPPG/pupillometry-and-memory research cited there are public, external work — not SAIL’s own, linked by name rather than vendored.

HTP Projective-Analysis Toy Example

v2026.07.1 reviewed by Torsten 2026-07-18, as good as this can be at this stage. Genuinely new territory; still needs real scoping with this collaborator before it's a design, and says so throughout on purpose. Tracked as sail-lspf.

This chapter doubles as the first real capture of this thread, not a synthesis of already-written material — no prior bean or draft covered it before this compendium.

Honest framing

This collaborator wasn’t part of the AIDLE proposal, but offered help back then and has now asked again about joint work. Same honesty standard as the EEG/fMRI chapter: nothing here is scoped yet, let alone running.

The toy example

The idea discussed: House-Tree-Person (HTP) data — studies, images — mined as a candidate source for grounding sail-judge’s arm design, the same way learning-science literature grounds the current arms (sail-wqs3) and Wood/Bruner/Ross grounds the discourse-driver role. The simple intuition behind it: arms informed by projective analysis (Hammer’s HTP framework), trigger and reward informed by EEG — deliberately the same physiological-grounding idea as the previous chapter, not a second, unrelated one.

What HTP actually is, said plainly

The House-Tree-Person technique (Buck’s original test, John Hammer’s later handbook) is a decades-old projective psychological assessment: a subject draws a house, a tree, and a person, and a clinician interprets features of the drawings — size, placement, line quality, omissions, distortions — as projections of personality or emotional state. Worth naming honestly, the same way this compendium names Goodhart risk in cheap reward proxies (sail-wqs3’s taxonomy) rather than presenting a method as if it were obviously sound: HTP’s psychometric validity is genuinely disputed in mainstream clinical psychology, in the same family of long-standing criticism as the Rorschach — inter-rater reliability and predictive validity have been real, recurring points of contention, not a settled matter. That matters directly here, not just as a caveat: an arm scored against an HTP-derived rubric risks encoding a contested clinical framework’s assumptions as validated ground truth, which is structurally the same failure this compendium already flags for cheap regex heuristics and LLM-judges elsewhere (bandit-arms-gat chapter) — a plausible-looking proxy standing in for a real construct nobody has actually verified it tracks. Any real design here would need to treat HTP-derived signal as another candidate proxy to validate, not a shortcut around validating.

A concrete connection to the previous chapter, not just a juxtaposition

The two neuroscience threads in this compendium aren’t only similar in spirit — there’s an actual, buildable connection between them. HTP’s own task structure (draw a house; draw a tree; draw a person) is naturally three discrete, prompted trials, which maps unusually well onto BrainWaves’ existing Custom-experiment builder (the previous chapter): named stimulus/prompt conditions, a bounded duration per trial, the same Collect → Clean → Analyze pipeline. Rather than inventing new EEG tooling for this idea, the honest first question is whether an HTP-style three-condition drawing task could just be built as another BrainWaves Custom design — reusing real, already-built infrastructure instead of starting from nothing, the same move the rest of this project makes whenever possible (2026-07-11-tech-synthesis-reusable-assets.md’s whole premise). Not attempted; a real, concrete next step if this ever gets picked up.

Before drafting further: this needs actual scoping

The raw note behind this chapter is an intuition, not yet a design. Before any of the above becomes more than a compendium chapter:

• What HTP datasets actually exist and are accessible — de-identified drawing corpora, published coding manuals, anything usable at all — hasn’t been checked. This chapter doesn’t cite one because none has been found yet, not because it wasn’t worth mentioning.
• What “mining HTP data for the SAIL judge” would concretely mean is still open: candidate framings include treating specific HTP coding categories (from Buck’s or Hammer’s manuals) as hypotheses for new arm constructs, the same way generation-effect or retrieval-practice were adopted as constructs — but nothing has been checked against real data, and given the validity concerns above, any such mapping would need real scrutiny before it went anywhere near live code.
• This hasn’t been discussed with this collaborator in any concrete form yet — the toy example is real, the design isn’t. Filing a follow-up scoping bean once there’s an actual conversation to build on is the right next step, not inventing design detail here first.

Open questions

• Does any accessible, appropriately-consented HTP dataset actually exist to check anything against?
• Is HTP’s contested validity a dealbreaker for using it as arm-design grounding at all, or is there a narrower, more defensible slice of it (e.g. purely structural/compositional features, not full clinical interpretation) worth separating out?
• If the BrainWaves-as-drawing-task connection above is real, does it need EEG at all for a first pass, or would it be worth trying the drawing-task/arm-mapping question on its own first, EEG added later?

Skill Trajectory Models

draft outline scaffold — not yet drafted, genuinely new territory. Tracked as sail-d5hv.

Unlike the previous four sections, this one isn’t framed around a single collaborator’s interests — it’s general, cross-cutting groundwork, worth anyone in this compendium reading. What we’ve built so far (sail-judge) is the scaffolding half of SAITO: the live, per-turn asker/discourse- driver bandit. The trajectory half — a model of a learner’s skill changing over many sessions, not just how to intervene in one turn — doesn’t exist yet in any form.

What’s real to ground this in

• sail-3scs — STM v0 on the accumulated trace corpus, currently deferred to Phase 3. Not started: no model, no training run.
• The facts-table substrate sail-judge already runs on (subject/predicate/object/confidence/disputed/source_event) — the same storage STM would need, already shipped for a different purpose.
• The original bootstrap plan’s own framing: STM as the “trace-only” fallback once EEG got parked (2026-07-03-sail-bootstrap-plan.md, Phase 3).
• The circularity caveat already written up in the EEG/fMRI chapter’s speculative section: STM tracks the same modality as the outcome itself, which makes it tempting to treat “the trace pattern moved” as ground truth without checking against real learning gain — worth restating here rather than only in a chapter it’s just a footnote to.

Not yet done

No outline beyond this page exists. This section is a placeholder for a real brainstorming pass, not a summary of one that already happened.

Appendix: The SAITO Closed Learner-Loop

Source: the original AIDLE proposal’s own framing, reused here rather than redrawn. First appeared in ~/txt/aidle/2026-06-03-codip-saito-lab-setup.md (English) and ~/txt/aidle/2026-05-28-saito-uebersicht-de.md (the German version, same figure). Reproduced verbatim as a diagram, not reinterpreted.

flowchart LR
    WHO["<b>WHO</b><br/>is being asked?<br/><i>learner state</i><br/><i>over weeks-to-months</i>"]
    WHEN["<b>WHEN</b><br/>to ask?<br/><i>spacing, retrieval,</i><br/><i>transfer intervals</i>"]
    WHAT["<b>WHAT</b><br/>to ask?<br/><i>productive difficulty,</i><br/><i>scaffolded questions</i>"]
    L((Learner))

    WHO --> WHEN
    WHEN --> WHAT
    WHAT --> L
    L -- "interaction trace +<br/>outcome measurement<br/>weeks later" --> WHO

Fig. 1 — SAITO’s closed learner-loop. The three guiding questions aren’t parallel; they’re sequential and feedback-coupled: learner state (WHO) drives the right moment (WHEN), which produces the next scaffolded question (WHAT); the outcome signal weeks later updates WHO.

What each box actually is, and what SAIL has actually built

The original AIDLE proposal names three implementation components, one per question:

Worth being precise about what this appendix is and isn’t: it’s the original pathway, reused so readers have the same map the AIDLE proposal itself used — not a claim that SAIL has built, or is currently building, all three legs. Right now, exactly one of three is real. The loop’s own feedback arrow — “interaction trace + outcome measurement weeks later” — is itself SAITO’s central bet (measurable capability gain, not immediate helpfulness), and is the thing the pre-registered mini-study (sail-1uq0) exists to actually measure, once it runs.

Appendix: The Artifact Series

draft first cut, 2026-07-18. Tracked as sail-7ie2.

Five self-contained design notes were written about sail-judge’s design as the work happened — not polished afterward, each grounded in a real exchange rather than invented framing (see artifacts/DESIGN.md in this repo for the per-artifact content map: what claim traces to which source). Originally published as private claude.ai Artifacts; reproduced here verbatim as static pages under this domain instead, so a collaborator can actually open them without needing sharing access granted — same content, one fewer moving part.

Start with The Full Map, which links back to the other four:

• The Full Map — the whole planned architecture in one synthesis, spanning grounding sources, the live agent, data/observability, deployment, and one deliberately-parked branch (LLM-as-judge). Backlinks to all four below.
• The Question Mark Problem — the live argument behind the bandit-arms-gat chapter’s design arc and Goodhart table: how the arms went from rhetorical styles to named constructs, and how the per-arm Goodhart classification was actually arrived at.
• Seven Signals — what actually got built and verified afterward: the research-sourced arms, two real bugs found and fixed live, and the parallel judge’s first disagreement.
• Bandits, Recapped — a theoretical companion piece explaining Thompson sampling itself, using sail-judge’s actual arms as the running example rather than an abstract slot machine.
• The GAT-Collapse Question — the discussion-opener the bandit-arms-gat chapter’s “Where GAT shows up” section is drawn from, written to be argued with directly.

Appendix: The Teaching Notebooks

draft first cut, 2026-07-19. Tracked as sail-7ie2.

Julia/Pluto notebooks — small, illustrative, data-grounded teaching artifacts, distinct from both the artifact series (static design notes) and the production implementation itself (sail-judge, which stays in ~/haiku.garden, not reproduced here). One so far:

• Thompson Sampling — a real replay (notebooks/thompson-sampling-teaching.jl) — replays sail-judge’s actual historical fire/reward sequence from a real 2026-07-11 session (notebooks/data/saito-events-2026-07-11.jsonl), not a simulation. Move the round_idx slider and the asker/discourse-driver Beta-posterior plots recompute live, showing each arm’s belief narrow exactly as it did during the real run. Companion, non-interactive version of the same argument: Bandits, Recapped (also linked from the Artifact Series).

Live, not precomputed — this genuinely runs a Julia process (PlutoSliderServer) on trullala, exposed via Tailscale Funnel, rather than a static export or a set of precomputed snapshots (an earlier attempt at the latter didn’t hold up and was dropped). That also means it depends on trullala staying up and funneled — if the link is down, the notebook’s own source is real and runnable locally: notebooks/start-pluto.sh (needs a local Julia install; the notebook’s Project.toml/Manifest.toml pin its exact dependencies).