Helm Upgrade Crash Example

UNOFFICIAL/EXPERIMENTAL

A customer once described a major outage caused by a Helm chart upgrade. The important product claim here is not that ConfigHub makes every upgrade safe. The claim is narrower and more useful:

Helm upgrade becomes a staged, reviewable, rehearsable, gated change with
evidence at each stage, instead of one opaque production mutation.

The Failure Mode

The risky path is familiar:

helm upgrade ...

That single command can hide many different changes:

• CRD schema changes;
• webhook changes;
• immutable selector or StatefulSet changes;
• PVC, storage, or retention changes;
• Service identity and DNS changes;
• RBAC expansion;
• hook behavior;
• generated Secrets or certificates;
• namespace or release-name fanout;
• image changes;
• removed objects.

The problem is not that Helm is bad. The problem is that the upgrade is too easy to treat as one operation when it actually contains several lifecycle decisions.

How The Staged Model Helps

For a serious chart, the ConfigHub/installer path breaks the upgrade into checks a team can inspect.

This does not guarantee that production cannot fail. It makes more failure modes visible before production, and it leaves a receipt trail when something does fail.

Concrete Example: kube-prometheus-stack

The first serious upgrade seed is:

prometheus-community/kube-prometheus-stack
85.3.3 -> 86.1.0
base: default

This chart is useful because it includes CRDs, admission webhooks, generated credentials, dependencies, RBAC, and many rendered monitoring objects.

Current evidence in this repo:

This is already useful. It still does not claim that every production upgrade path is safe.

Example Workflow

The exact space, target, chart, and values depend on the user. The shape is:

# 1. Verify the supported version and candidate version surfaces are current.
npm run refresh:survival:verify

# 2. Compare the committed old and new rendered object sets.
mkdir -p .tmp/upgrade-review
diff -u \
  recipes/prometheus-community/kube-prometheus-stack/85.3.3/revisions/default/r001/rendered/release-objects.yaml \
  recipes/prometheus-community/kube-prometheus-stack/86.1.0/revisions/default/r001/rendered/release-objects.yaml \
  > .tmp/upgrade-review/kps-85.3.3-to-86.1.0.diff || true

# 3. Inspect the committed CRD delta for the serious candidate.
sed -n '1,160p' \
  data/serious-chart-reviews/kps-crd-upgrade-delta-85.3.3-to-86.1.0.yaml

# 4. Check the live CRD upgrade rehearsal receipt.
npm run kps:crd-upgrade-live:verify

# 5. Check the live regular-Helm workload upgrade rehearsal receipt.
npm run kps:workload-upgrade-live:verify

# 6. Check the measured blast-radius cases.
npm run blast-radius:accuracy:verify

# 7. Check current parity and live evidence already committed for the corpus.
npm run kind-parity:verify
npm run live-parity:verify

For a fresh upgrade on a new chart or private values file, the same workflow would be run with new rendered inputs and new receipts. The ConfigHub operation then becomes a reviewed change, not a blind upgrade:

New to cub? Install the cub CLI first. You can pull and render public catalog packages without an account. Commands that save or change ConfigHub data require you to sign in.

# Pseudo-command shape. Use the current cub commands for your space and target.
cub unit diff --space <candidate-space>
cub changeset create --space <candidate-space> <upgrade-review>
cub function vet --space <candidate-space>
cub unit approve --space <candidate-space> --changeset <upgrade-review>
cub unit apply --space <candidate-space> --changeset <upgrade-review> --wait

The point is that the reviewer can see the exact object change, the scan result, the target prerequisites, and the observation receipts before production is declared healthy.

What This Would Have Caught

This model is especially useful for upgrade failures caused by hidden scope:

• "This upgrade changes CRDs" becomes a CRD delta and live API-server rehearsal.
• "This chart needs a hook or lifecycle action" becomes a route, an observation, or a named blocker.
• "This value only looked small" becomes a blast-radius case against the actual object diff.
• "This target is not shaped correctly" becomes a target-fact or target-fit blocker before production.
• "The apply succeeded but the app is broken" becomes a live observation gap, not a false green.

Consul is a useful companion example. The secure-mesh path needs target facts for TLS and secrets, and the proof target needs the right Kubernetes topology. That is not a ConfigHub worker requirement. It is the install contract becoming visible before the install is treated as supported.

What Not To Claim

Do not claim that this repo has solved all Helm upgrades.

Do not claim that the kube-prometheus-stack candidate is a supported replacement for every target. The current evidence is scoped to the recorded version pair, base, values, and kind profile.

Do not claim that render parity alone proves upgrade safety. Render parity is the baseline. Upgrade safety needs lifecycle review, target prerequisites, scan/gate receipts, live before/after observations, and a scoped support decision.

Do not claim that ConfigHub-managed upgrade orchestration and rollback are complete for this example yet. The repo has a render-level CRD delta, a live API-server CRD upgrade rehearsal, and a live regular-Helm workload upgrade rehearsal. ConfigHub-managed upgrade receipt, rollback receipt, soak receipt, private-overlay upgrade receipt, and production-target upgrade receipt remain future work.

Where This Fits

This is the day-2 version of the seven-stage model:

Use Serious Chart Proof for the kube-prometheus-stack proof path and Upgrade Story Plan for the upgrade lane definition.