SAITO reward-signal redesign · 2026-07-11 · filed as sail-wqs3
A map of one evening's conversation — how "the bandit works" turned into "the reward is arguably lying to it," and what that turned up along the way.
Stage 1 — the mechanism works
The SAITO scaffold agent samples from a Beta posterior per prompt arm and picks the highest draw — Thompson sampling, same algorithm family as the graze project. For this problem class the regret bound is already about as good as mathematically possible. It converges fast on whatever it's told to converge on.
hasQuestion = body.includes("?") — saito-scaffold-agent.mjs:341
Cheap, real, already-available — and never actually aimed at anything in particular. It rewards every arm identically for provoking one more question mark, whether or not that has anything to do with learning.
Stage 2 — the fix isn't the math, it's the arms
sail-wqs3 already redefined what the arms are: not tones of voice, but named learning-science mechanisms. Asker (generation effect, retrieval practice, desirable difficulty, metacognitive calibration) and discourse-driver (Wood, Bruner & Ross's 1976 scaffolding functions: marking-critical-features, reduction-in-degrees-of-freedom, direction-maintenance). Naming the arms this precisely is what exposed two things that needed saying before any of it goes to Sandra Hummel for review.
Split
Thread A — what the roles assume
Splitting "teacher" into asker (knowledge-transfer, quizzing) and discourse-driver (process-facilitation, scaffolding) makes explicit what's usually just one person's blended, informal habit. Building it as two addressable roles promotes a personality trait to a professional stance — the instructor-to-mentor shift, made structural rather than left to whoever happens to be teaching.
Thread B — what the reward assumes
A cheap proxy tracks the real goal only under passive observation. Once something optimizes directly for it, the optimizer exploits any path to the proxy — including ones that never touch the real construct. SAITO's bandit, paired with Hummel's pre-registered mini-study KPI, is a rare thing in this literature: a live case with an actual ground truth to check divergence against.
both feed the same reckoning ↓
Convergence — per-arm reckoning
refresher — the law, then the taxonomy
Goodhart's Law, plainly: once a measure becomes the target of optimization, it stops accurately measuring the thing it used to measure (Goodhart, 1975; compressed by Strathern to the version everyone quotes). That's the whole law — one mechanism, no variants, and it doesn't say why or in what shape the failure shows up in a given case.
Manheim & Garrabrant (2018) is the decomposition, not a separate idea: they split that one mechanism into four distinct ways it actually happens — regressional, extremal, causal, adversarial — precise enough to say something different about each arm below, instead of one blanket warning repeated seven times.
Applied to each construct's likeliest cheap proxy:
| Arm | Candidate proxy | Variant | Why |
|---|---|---|---|
| Generation-effectasker | an attempt appears before the answer is given | Regressional | attempt-shaped text mixes real effort with pretense; may share a cause with the real gain rather than produce it |
| Retrieval-practiceasker | the term is recalled, not given | Extremal | holds only while the prompt truly withholds the answer; pushed hard, phrasing drifts toward near-giveaway — recall becomes echo |
| Desirable-difficultyasker | any same-turn ease/fluency signal | Causal † | defined by short-term struggle producing long-term gain — a proxy rewarding ease selects against the point of the arm |
| Metacognitive-calibrationasker | a confidence statement is offered | Regressional→drift | a stated number isn't calibration; once the learner notices it's tracked, they'll produce the phrase reflexively |
| Marking-critical-featuresdiscourse-driver | the flagged term is referenced back | Regressional | verbal echo without an attention shift; same shared-cause risk as generation-effect |
| Reduction-in-degrees-of-freedomdiscourse-driver | the next message narrows in scope | Extremal | valid range is "fewer options, still their move"; pushed hard, the bot just answers for them — over-scaffolding |
| Direction-maintenancediscourse-driver | conversation stays on topic | Causal | continuity can be forced by the bot steering, not caused by genuine sustained motivation |
† no live, same-turn proxy can validate this arm at all — see the exception at right, below.
Validation
Five of the seven arms are regressional or causal: exactly what fitting each candidate proxy against real data (coded transcripts now, Hummel's mini-study later) will catch or clear. Cheap to do before touching the agent's code. Two exceptions don't yield to that alone:
Retrieval-practice and reduction-in-degrees-of-freedom validate fine on data observed so far — the failure only appears once Thompson sampling pushes exploitation into ranges not yet seen. These need a standing range guard, not a one-time fit.
Desirable-difficulty has no honest same-turn signal, by construction. It runs on the delayed mini-study data directly, or gets fixed exposure with no live reward, until that data exists.
Gate
sail-1w41 not sent
Sandra Hummel sanity-checks the construct taxonomy — the mentor/instructor framing, the Goodhart reckoning, all of it — before one line of saito-scaffold-agent.mjs changes. The draft exists (drafts/2026-07-11-email-hummel-synthesis.md); it hasn't gone out yet.
the agent's reward line still reads body.includes("?") — everything above is the design that's waiting on a reply before it replaces it.