sail-judge · 2026-07-12 · sequel to The Question Mark Problem
The line that used to read body.includes("?") for every arm alike now runs seven different checks, each traced to a specific piece of learning-science research — running unattended, tonight, on a box that isn't this laptop.
Recap
hasQuestion = body.includes("?")
The Thompson-sampling mechanism underneath it was already sound — proven regret bounds, converges fast on whatever it's told to converge on. The problem was never the math. It was that "provoked another question mark" isn't a stand-in for anything real, and a proven optimizer will happily converge on it anyway.
Resolution
Every arm now has its own reward function, matched to a specific construct from the learning-science literature — not a rhetorical style, not a generic engagement signal. The proxy for each is deliberately cheap (a word check, a length comparison — no model call), but it's aimed at that construct specifically, not at conversation in general.
| Arm | Research source | What the reward actually checks |
|---|---|---|
| Generation-effectasker | Slamecka & Graf, 1978 | a substantive attempt appears, not a deflecting question |
| Retrieval-practiceasker | Roediger & Karpicke; retrievalpractice.org | a term from earlier history is recalled, absent from the question itself |
| Metacognitive-calibrationasker | judgment-of-learning paradigm | a real confidence expression (percentage or hedge word) is present |
| Desirable-difficultyasker | Bjork | unscored fixed-schedule exposure — no live proxy can validate this one honestly |
| Marking-critical-featuresdiscourse-driver | Wood, Bruner & Ross, 1976 | the next reply references the specific detail just flagged |
| Reduction-in-degrees-of-freedomdiscourse-driver | Wood, Bruner & Ross, 1976 | the next reply is narrower than the recent rolling average |
| Direction-maintenancediscourse-driver | Wood, Bruner & Ross, 1976 | the next reply still connects back to the conversation's original goal |
Verification
Every evaluator passed on paper and in unit fixtures. Running the actual agent against a real room, with real replies, found two genuine bugs neither the design pass nor the fixtures caught:
A word-boundary check placed directly after a % character can never match — % isn't a word character, so the boundary the regex expected doesn't exist there. A bare "maybe 60%" silently failed to reward, with no accompanying hedge word to save it.
/\d{1,3}\s?%|\b(?:percent|confident|...)\b/i
Two separate arms rewarded on a coincidentally shared filler word — "last" (month vs. week), then "earlier" — that had nothing to do with the actual content being checked for. Fixed by excluding a set of generic temporal/discourse words from the overlap check.
Coverage
Thompson sampling exploits winners quickly — left to natural conversation, two arms took the early lead and the bandit correctly kept favoring them, leaving the other four never actually fired live. A separate script forces each arm in turn instead of waiting for luck, checking both the success and the failure path for every one.
Deployment
sail-judge live
Renamed from the scaffold-agent prototype and deployed to a dedicated home-lab box, as a real systemd --user service — restart-always, independent of whether this laptop is even on. Generation runs through a fast remote model instead of local inference, which local hardware here had already ruled out as impractical. A companion timer runs the diagnostic report every 30 minutes on its own.
Still open
A parallel LLM-judge now runs alongside every reward check — never feeding the bandit, purely comparing verdicts. Its first-ever comparison was already a real disagreement: a bare one-word reply had been rewarded as "successfully narrowed" by the cheap heuristic; the judge correctly called that too thin to prove anything. That gap — between what a cheap proxy will accept and what a closer read would — is exactly the raw material an eventual schema-checked scoring layer would need, not a settled question yet.
the mechanism converges on whatever it's told to. what it's told, now, actually comes from somewhere.