sail-judge · 2026-07-14 · fourth in the series — The Question Mark Problem · Seven Signals · Bandits, Recapped

The Full Map

Everything running today, solid-bordered. Everything designed but not built, dashed. One parked research thread, dotted. Not a plan document — a map of exactly where the edges of "built" currently are.

built & running designed, not built parked research mechanism / rigor human / curation

01Grounding — where the numbers come from

Learning-science literature

Slamecka & Graf; Roediger & Karpicke; the JOL confidence paradigm; Wood, Bruner & Ross (1976). Grounds all 7 arms.

sail-wqs3

Sandra Hummel

Construct sanity-check on the taxonomy itself. Draft written, not sent — a real gate, just no longer blocking iteration speed.

sail-1w41

Aaron Steven White

GAT/panproto cooperation — a concrete Goodhart-classification problem to react to, not a vague check-in. Draft written, not sent.

sail-ltu0

The mini-study

Pre-registered, real learner-outcome data. The actual fix for cheap proxies — everything else here is instrumentation until this exists.

Semble connections + riddles.md + tilde.cards

Real, hand-curated connection-making — slow to accumulate. Will ground role/persona choice via an ML distillation step, not a live lookup.

sail-i9j3
grounds ↓

02sail-judge — the live agent (running on trullala)

Role trigger

Now: raw message-count thresholds (messagesSinceQuestion, sinceLastReply) — the dumb piece, unchanged since v1.
Planned: fast signal distilled from Zone 01's real curation data.

sail-i9j3

Thompson-sampling bandit

Beta(α,β) per arm, sample-and-pick-highest. Proven regret bound — already optimal for whatever reward it's given.

REWARD_EVALUATORS

Cheap regex/word-overlap proxies, one per arm, matched to a Goodhart variant. Feed bandit.reward() directly.

Generation backend

Pluggable — Ollama on pad, ScaDS on trullala (faster, fixed the echo problem as a side effect).

The 7 arms

  • generation-effect — asker regressional
  • retrieval-practice — asker extremal
  • metacognitive-calibration — asker regressional→adversarial
  • desirable-difficulty — asker fixed schedule, unscored
  • marking-critical-features — discourse-driver regressional
  • reduction-in-degrees-of-freedom — discourse-driver extremal
  • direction-maintenance — discourse-driver causal

Parallel LLM-judge

Opt-in (SAIL_JUDGE_PARALLEL_JUDGE), comparison-only. First live disagreement caught immediately (trullala, bare "yes" reply).

⊘ no path back into bandit.reward()
logs every fire / reward / judge event ↓

03Data & observability

JSONL event log

Append-only. Source of truth — everything else here is a rebuildable view over it.

sail-judge.duckdb

Independent from drain.duckdb. Migrations (Django-style, JS), derived feature columns, SCRIPT_VERSION + SCHEMA_VERSION + COMMIT_SHA on every row.

Diagnostics (JS + SQL)

Per-arm Goodhart checks, all 4 trend-capable arms now checked for intensifying drift, not just presence. Judge-agreement report.

Observatory layer

Lens-mapped vocabularies over raw signal; shortlists drifting arms instead of a uniform report; hands the shortlist to a human.

sail-oj4n
runs on ↓

04Deployment

pad

Dev/testbed. Ollama backend, ~10–15s per generation.

trullala

Unattended, systemd --user (restart-always), ScaDS backend, sub-2s generation. Independent of pad being on.

↺ closing the loop, not yet built: Observatory's human-reviewed shortlist (03) → informs construct taxonomy revisions → back into Grounding (01). The system correcting itself, once a human has looked.

·Parked, deliberately not in this flow

LLM-as-judge as the reward itself

Would relocate the Goodhart problem into RLHF's reward-model-overoptimization failure mode, not solve it. Connected to AIDLE's own Third Axis (RLHF / RLVR / GAT) framing.

sail-g715