UNOFFICIAL/EXPERIMENTAL. The goal, in one sentence: prove with receipts that Pilot gives better demos than a frontier model on its own. Pilot here means the assembled system: a Claude-class model plus the knowledge (catalog, receipts, doctrine, findings), the skills (verb discipline, change descriptions, quota protocols), the scenarios (live-verified runbooks), and the deterministic rails (parity gates, verifiers, apply gates). "Claude on its own" means the same class of model with plain helm and no substrate.
Three workstreams. Each produces demo material and evidence; none needs new org quota or a cluster.
1. The head-to-head benchmark lane (the proof instrument)
Same tasks, two arms, scored on outcomes. Without this lane the claim is an argument; with it the claim is a table.
Arms
- Arm A, bare: a fresh agent session with
helm,kubectl, and a kind cluster. No helm-expt checkout, no skills, no catalog, no receipts. - Arm B, Pilot: the same model driving the helm-expt substrate: catalog packages, cub, skills, verifiers, parity gates.
The honest-run protocol matters more than the tasks. Arm A must not be sabotaged: it gets a clean environment, the chart's own docs, and the same task wording. Arm B must not be coached: the task wording is identical, and Pilot finds its own way through the substrate. Both arms run the same model version. Each run's full transcript is kept as the receipt.
Task set (first cut, one per failure class we have receipts for)
| Task | The trap it contains | Bare-arm failure mode we expect |
|---|---|---|
| Install redis, change replicas, upgrade the chart, keep the change | Helm wipes local edits on upgrade | Change lost silently, or manual values-file surgery |
| Install a hook-carrying chart and prove the hook ran | Hooks are invisible in render output | Asserts success without observing the Job |
| Install kube-prometheus-stack no-crds on a bare cluster | CRD ordering | CR apply fails or is applied blind |
| Repoint every image in a three-env fleet to a mirror, prove it landed | No fleet visibility, no record | Grep-and-hope, no proof, one env missed |
| "Give me a standalone redis with metrics" | The 2,392-line values file | Hand-written values or hallucinated YAML, no parity proof |
| A same-field local edit exists; pull an upstream release | Silent merge behavior | Divergence not detected (we only found it via receipts) |
Scoring (per task, both arms)
- Correct: the end state is what the task asked for, verified by inspection, not by the agent's claim.
- Silent failures: count of things wrong that the arm did not report.
- Receipts: what evidence exists afterward that a third party could check.
- Reversibility: can the change be undone from the record alone.
- Time and turns: secondary, reported not scored.
Status
Designed here; not yet run. The fuzz lanes (run-bad-decisions-fuzz, cub-installer fuzz) are the harness precedent. The bare arm needs a clean sandbox definition so it is honestly bare; that is the main build item.
2. The Pilot wiring gap (make the variant demo honestly Pilot-driven)
scripts/pilot-generate-variant.mjs proved the mechanism with the intent frozen as a constant. For a demo to say "Pilot did this," the intent-to- switches step must come from outside at run time:
- The script accepts
--intent "<words>" --switches <file>where the switches file is produced by whatever agent ran the mapping (the Pilot checkout at$HOME/code/confighub-ai-demo, or any agent). The frozen constant becomes the default fixture so the script still runs standalone and in CI. - The receipt records who mapped the intent (
mappedBy), so a demo receipt says "Pilot chose these switches; the gate proved the render." - Status: shipped alongside this plan (
--intent/--switches/--mapped-by).
3. Switch-effect maps across the top charts (demo breadth)
The redis map generalizes: the toggles are auto-extracted from helm show values (top-level <key>.enabled plus the architecture axis when present), rendered flipped, classified by object-set diff. One command per chart, one committed map per chart. Status: shipped alongside this plan (npm run pilot:switch-map -- --chart <name>; registry starts with redis, postgresql, rabbitmq, nginx, grafana). Maps regenerate on chart bumps.
What a "better demo" looks like, concretely
The demo Pilot can give that bare Claude cannot, using only shipped pieces:
- "Show me what
sentinel.enabledactually does" - answered in seconds from the committed map, with the exact object diff, for any mapped chart. - "Give me a standalone redis with metrics" - variant generated from words, parity-gated, receipted, landed as governed config.
- "Upgrade it without losing my change" - the ladder story, recorded as revisions.
- "Prove the hook ran" - the three-consumer OCI receipt.
- "Move every environment off this registry" - the fleet migration walkthrough, run live, with the laggard visible.
Each of these is a receipt-backed set piece; the benchmark lane turns the contrast with the bare arm into numbers.
Task 1 status: RUN (2026-07-05)
Executed live on kind cluster hx-bench-1, both arms, torn down after. Receipt: task1-upgrade-keeps-edit.md. Honest result: both arms can keep the change; plain helm silently reverts it on one forgotten upgrade flag, the substrate keeps it by merge and records why.