kube-prometheus-stack: Serious Chart Review

UNOFFICIAL/EXPERIMENTAL - reviewer-facing snapshot, 2026-07-29. Counts cite generated files; re-check those files before quoting.

kube-prometheus-stack (prometheus-community/kube-prometheus-stack@85.3.3, bases default and no-crds) is the catalog's deliberately hard chart. This page maps what is proved, what is partial, and what remains - so a reviewer can probe the strongest example without mixing proof levels. Compact table: data/serious-chart-reviews/kube-prometheus-stack.csv.

Why This Chart Matters

It concentrates most quirk classes in one install: 10 CRDs, admission webhooks with hook-driven cert patching, cluster RBAC, generated facts, hooks across four phases (pre-install, post-install, pre-upgrade, post-upgrade), large object fanout, dependency-locked subcharts (Grafana, exporters), and real image/security surface. If the model only worked on Redis-class charts, it would prove little; this chart is where the claims earn their keep.

What Current Receipts Prove

• Render parity: 2/2 bases match regular Helm for the pinned chart, values profiles, and capability profile (data/status-dashboard/top20-status.csv, rank 18).
• Local live e2e (default base): pass, with the strict witness recorded as observed and 3/4 witness checks passing on the committed row (data/live-e2e/top20-local-kind.csv; the per-check detail lives in the run's observation receipt).
• Base readiness: default is start-here (all core lanes plus two-cluster parity pass); no-crds is try-with-proof because it still lacks the ConfigHub upload/scan/safe-ops proof lane for this base, even though its live target-fact lanes now pass (data/top20-base-readiness/base-readiness.csv).
• Blast radius, measured not assumed: the crds.enabled base pair (default → no-crds) has a committed re-render diff - 10 predicted, 10 observed object changes, accuracy 1.000 (data/blast-radius-accuracy/). The grafana.adminPassword case is recorded as not-measured-yet - an honest open row, not a claim.

What Live Parity Proves

Both lanes pass for this chart: the selected live Helm-vs-ConfigHub comparison (default and no-crds) and the two-cluster kind parity lane (default and no-crds) - regular Helm in one kind cluster against the cub installer path in another, compared semantically (data/live-kind-parity/summary.csv, data/live-helm-confighub-compare/). Scope: the pinned chart version, the named values profiles, Kubernetes 1.30 kind clusters. This is install-time equivalence on that profile; it is not an any-cluster or any-values claim.

The no-crds row is the useful hard case. It proves more than rendered YAML equivalence: both clusters staged the same compatible Prometheus Operator CRDs, the installer lane also staged the monitoring/kube-prometheus-stack-admission Secret as an explicit target fact, and both sides reached Ready workloads with no semantic object diff other than the allowed installer-created Namespace. The fresh 2026-06-11 two-cluster receipt is runs/live-kind-parity/prometheus-community-kube-prometheus-stack-no-crds/receipt.yaml. The stricter ConfigHub OCI/Argo receipt for the same target-fact contract is runs/live-helm-confighub-compare/prometheus-community-kube-prometheus-stack-no-crds/receipt.yaml: regular Helm, ConfigHub direct apply, and ConfigHub OCI/Argo all reached Ready with no semantic diffs beyond the intentionally created Namespace.

Hook Lifecycle Proof

The chart's admission-webhook certificate Jobs are the hooks. Status in the maintained queue (data/hook-lifecycle/top100-hooks.csv): lifecycle-disposition: lifecycle-observed, receipt_status: observed, with the route hints preserved (preflight-or-presync, postsync check, upgrade action with receipt, ordering and cleanup-policy preservation, webhook readiness observation). This is the strongest hook evidence in the corpus - and it is one chart's hooks observed on one profile, with the standing next action "keep receipt fresh when chart, base, or cluster version changes."

The package's direct script has passed the fresh-install sequence for both bases. The no-crds base also has a staged OCI install-and-upgrade receipt through Argo CD and Flux on separate fresh clusters (summary, receipt). Both controllers installed 85.3.3, moved to the 86.1.0 staged digest, reran the CRD, certificate, workload, and webhook-patch stages, replaced both completed setup Jobs, and passed the runtime checks after upgrade. The receipt does not prove rollback, long-running soak, automatic ConfigHub route selection, or automatic post-success removal of every temporary hook resource. None of this generalizes to another chart's hooks.

CRD / Webhook Lifecycle Proof

• CRD install behavior is covered by the live lanes and the explicit no-crds base; CRD ownership is a recorded user decision.
• CRD/webhook runtime lifecycle observation is now receipted for this chart's own operator and admission webhook through direct apply, Argo CD, and Flux. Related certificate-injection and conversion behavior is also demonstrated on cert-manager and External Secrets (data/lifecycle-observations/cert-manager-eso/).
• CRD upgrade now has three bounded results. The committed render-level delta for the 85.3.3 → 86.1.0 candidate (kps-crd-upgrade-delta-85.3.3-to-86.1.0.yaml, regenerable via node scripts/kps-crd-upgrade-delta.mjs --verify): 6 of 10 CRDs change, every change is an additive schema property path (none removed), and no version entry, served/storage flag, or conversion strategy changes. The live rehearsal receipt (receipt.yaml, regenerable via npm run kps:crd-upgrade-live) applies the 85.3.3 CRDs to a fresh kind API server, applies the 86.1.0 CRDs over them, waits for Established, and confirms a ServiceMonitor server-side dry-run before and after the upgrade. The staged OCI controller receipt then runs the complete no-crds 85.3.3 to 86.1.0 path through Argo CD and Flux and checks all ten CRDs after the upgrade.
• Workload upgrade now has a bounded live receipt (receipt.yaml, regenerable via npm run kps:workload-upgrade-live). It installs kube-prometheus-stack 85.3.3 with the committed default values, waits for workloads to become Ready, upgrades the same Helm release to 86.1.0, waits again, records Helm history, and deletes the kind cluster. That proves the regular Helm upgrade path for the tested values and kind profile. The staged OCI controller receipt separately proves the selected no-crds result through Argo CD and Flux. Neither proves automatic ConfigHub route selection, rollback, soak, private overlays, or production target upgrades.

Production Support State

From data/production-disposition/ the chart is production-review-ready with 7 dispositions accepted and 0 open. From data/production-support-decisions/prometheus-community-kube-prometheus-stack/support-decision.yaml, the default base is supported for one declared target scope: vanilla kind, namespace monitoring, ConfigHub OCI delivery, and Argo CD. The committed receipt retains the historical machine label cub-lk-kind-vanilla; current local examples create this class of target with cub cluster up.

That support decision records a mutable-image exception, scan/security acceptance, lifecycle decision, target-fact decision, and fresh ConfigHub OCI/Argo evidence for the declared scope. Stricter environments may still need a hardened or digest-pinned base.

The no-crds base remains a separate support decision. Its older runtime receipt is valuable because it blocks when target CRDs are missing; the fresh two-cluster and ConfigHub OCI/Argo receipts are valuable because they prove the positive path when compatible external CRDs and the admission Secret are staged. Support for that base still needs a target-scoped production decision, freshness policy, and production target review for how those prerequisites are supplied on the chosen target.

Claims That Must Not Be Made Yet

• "Production-supported for every target" - the supported decision is target-scoped to one declared kind/namespace/OCI/Argo path.
• "ConfigHub upgrades are solved" - the repo has render-level CRD evidence, a live API-server CRD rehearsal, a regular-Helm workload upgrade receipt, and a staged OCI no-crds upgrade through Argo CD and Flux. ConfigHub does not yet select that route automatically, and there is no rollback, soak, private-overlay, or production-target upgrade receipt.
• "Hooks are solved" - this chart's install and one upgrade are observed on named delivery paths. Automatic post-success cleanup of every temporary hook resource remains open. The claim is per-chart, per-profile, and freshness-bounded.
• "Works on any Kubernetes" - every live claim is bounded to the 1.30 capability profile; this chart's CRDs are exactly where profile drift bites.
• "All blast radius is measured" - one value path is measured at accuracy 1.000; the value-source-map backlog (e.g. grafana.adminPassword) is recorded as not measured.

Suggested Next Receipts

In value order: (1) automatic ConfigHub selection of the recorded route; (2) a target-scoped no-crds production-support decision that uses the proven staged OCI path on the chosen production target; (3) rollback and longer-running receipts; (4) automatic post-success cleanup evidence for the temporary hook resources; (5) a hardened or digest-pinned base for stricter environments that should not reuse the public proof scope's mutable-image and scan exceptions.