# The ConfigHub Workshop in depth This is the technical detail that used to live in the repository README. It is moved here unchanged so the README can stay a short front door. ## The Idea Pre-rendering Helm makes the real deployment object set visible before anyone has to approve, promote, or operate it. In this experiment, AI helps analyze public Helm charts and draft deterministic `cub installer` recipes. The proof pipeline then checks the work mechanically: ```text public Helm chart -> deterministic installer recipe -> deterministic rendered object set -> base variants and optional derived ConfigHub variants -> scans, gates, receipts, and live observations ``` That matters because many Helm problems are not just install-time problems. They show up later as upgrade surprises, hidden value interactions, environment drift, unclear ownership, unreviewed YAML changes, or uncertainty about what actually reached a cluster. The catalog is meant to show a better path: ```text Day 0: install from a known recipe and inspect the exact objects. Day 1: create variants deliberately and compare the rendered results. Day 2: scan, promote, observe, upgrade, and audit with receipts. ``` The first experience should not require full product signup. Public tools can be inserted at any useful OCI boundary: ```text work -> OCI OCI -> work OCI -> work -> OCI ``` Here, `work` means render, inspect, explain, test, scan, compare, or edit the configuration. **Serverless** means that work does not depend on ConfigHub Server. **Anonymous** means it uses no ConfigHub account. A local command or CI job can be both. ConfigHub becomes useful when a team wants to save the objects and their history, create variants, approve changes, promote them, or roll them out to a fleet. The public catalog packages are published to Google Artifact Registry with anonymous read access on the `helm-expt` repository. Full signup becomes useful when the user wants private charts, private overlays, derived ConfigHub variants, production approvals, managed receipts, or fleet operations. In practice: browse the catalog anonymously; pull public package artifacts without a Google registry login; verify receipts locally; create a full ConfigHub account only when you want ConfigHub to manage private or production state. ## Why Helm is good at producing Kubernetes objects. The pain starts when teams need to answer ordinary operational questions: ```text What exactly did this values file produce? Is prod getting the same objects that staging reviewed? Which variant changed, and why? Did we scan the exact objects we are about to deploy? Can we prove what was rendered, checked, uploaded, and observed? ``` ConfigHub is the proof and variant layer around those Helm-generated objects. The goal is not to replace Helm. The goal is to make Helm output reviewable, comparable, scannable, promotable, and auditable. Render parity is necessary, but it is not the whole value. Render parity proves that the `cub installer` path did not silently invent a different Kubernetes object set from regular Helm. The harder user problems start when a chart needs target facts, staged CRDs, admission certificates, provider prerequisites, controller-owned fields, hook-like lifecycle behavior, or fresh live observation. The catalog makes those requirements explicit, stages or routes them, observes the result where evidence exists, and records the boundary of the claim. kube-prometheus-stack is the main example. The `no-crds` base is not just a smaller YAML bundle; it is a contract that compatible CRDs and admission certificate material must already exist or be staged before config-only delivery. That is the point of the model: make the hidden operational contract visible before users trust the install. We use AI to accelerate Helm chart analysis and recipe creation. We use `cub installer` to prove the resulting recipes produce correct, Helm-equivalent, reviewable base variants and derived ConfigHub variants. This is not a single prompt. It is an AI-assisted workflow with explicit decision points and mechanical checks. For the recipe-generation workflow and the "where pieces go" table, see [Introduction To The Harness](../../docs/user/introduction-to-the-harness.md). ## How Values And Overlays Are Supported Helm users already customize charts with values files, `--set` flags, umbrella charts, wrapper charts, and sometimes Kustomize overlays. The rule in this repo is not "stop doing that." The rule is: ```text If a customization changes Helm render inputs, object count, object shape, or lifecycle semantics, make it a reviewed cub installer recipe/package base. If it refines already-rendered ConfigHub Units through approved post-render fields, facts, links, targets, gates, functions, or checks, make it a derived ConfigHub variant using `cub variant create` plus guided review. ``` There are three product questions behind that routing: | Question | Product lane | Typical answer in this repo | | --- | --- | --- | | How many useful configs should be free or out of the box? | public catalog bases | enough reviewed base variants to cover common, high-value install shapes such as default, existing Secret, server-only, ClusterIP, storage, or observability modes | | What can users customize inside ConfigHub after a base is reviewed? | derived ConfigHub variants | environment, region, customer, target, gates, observation policy, target facts, links, and approved post-render field fills | | Which cases are more complex and likely managed or paid? | import, private overlays, and stacks | private values and wrapper charts, GitOps import, fleet variants, full app stacks, private patch SLAs, old-version support, and production receipts | The public catalog should carry enough free base variants that most users do not start from a blank chart analysis. Derived variants should cover many environment and customer cases without rerendering Helm. Full import, private overlay, and stack workflows are still important, but they should be presented as a richer lane over the same proof model rather than as the first tutorial vocabulary. Before OCI delivery, users need to make three separate decisions: | Question | Use | Examples | | --- | --- | --- | | Does this change Helm render inputs, object count, object shape, or lifecycle semantics? | `cub installer` base variant | values files, component on/off, storage, ingress, CRDs, RBAC, Kustomize patches that change object shape | | Does this refine an already-reviewed ConfigHub object set after render? | derived ConfigHub variant | target, environment, region, namespace when represented, labels, gates, observation policy, links, target facts, allowed placeholder or TransformPaths fills | | Is this required before Kubernetes or GitOps can use the artifact? | delivery prerequisite or receipt | existing Secret, StorageClass, IngressClass, CRD owner, GitOps pull secret, OCI digest/signature, hook/lifecycle decision | Examples: | Customization | Route | | --- | --- | | Helm values file that changes storage, ingress, CRDs, RBAC, args, env, object count, topology, or lifecycle behavior | new `cub installer` base / rendered revision | | Replica count over an already-rendered workload field | derived ConfigHub variant or Day-2 operation when the base allows that path; base variant only when it changes topology or chart branches | | Kustomize overlay that changes the install shape | explicit recipe/base overlay with digest and diff | | Namespace, target, labels, approval gates, observation policy | derived ConfigHub variant | | Existing Secret reference already represented in the rendered objects | derived ConfigHub variant with target fact/check | | Approved post-clone function or TransformPaths fill over existing fields | derived ConfigHub variant with mutation receipt | | Wrapper chart plus platform values plus customer overlay values | managed overlay import; usually needs ConfigHub Server | NGINX-style extension slots are common. In the current evidence set, 13/20 top-20 catalog charts and 82/100 top-100 chart facts expose raw manifests, `tpl` snippets, config blocks, sidecars, or add-on slots. In the broader top-500 source scan, 254/500 charts expose raw/extra manifest values and 363/500 use `tpl` or raw/extra manifest values. Leaving those slots empty can stay inside a supported catalog base. Populating them creates a new reviewed install shape and should route back through a `cub installer` base variant with render parity, scans, gates, and receipts. See [Extension Slots](../../docs/user/extension-slots.md) for the user-facing routing guide. For the detailed algorithm, see [Customization Algorithm](../../docs/reference/customization-algorithm.md). For the product tier boundaries, see [Product Support Tiers For Helm Scenarios](../../docs/user/product-support-tiers.md). For the seven-stage lifecycle and render parity boundary, see [Seven-Stage Helm Lifecycle](../../docs/reference/seven-stage-helm-lifecycle.md). For the simple variant creation guide, see [Creating Variants](../../docs/user/creating-variants.md). For wrapper charts and customer overlays, see [Custom Overlays](../../docs/user/custom-overlays.md). For the OCI delivery boundary, see [Choosing Base Variants, Derived Variants, And Delivery Changes](../../docs/user/change-routing-before-oci.md). For the extension-slot coverage table, see [Extension Slot Coverage](../../data/extension-slots/summary.md). For Helm hooks specifically, see [Hook Lifecycle Strategy](../../docs/user/hook-lifecycle-strategy.md). For a single serial order through the user docs, see [User Docs Reading Order](../../docs/user/README.md). If you are asking why this is more than a one-shot Helm upload, start with [Why This Exists](../../docs/user/why-this-exists.md). For the shortest product model, start with [What You Get](../../docs/user/what-you-get.md). ## Command Choice Use the shortest command that answers the question you are asking: | Goal | Command path | | --- | --- | | See what an arbitrary chart and values render, without ConfigHub state. | `cub helm template` | | Store an arbitrary chart, its inputs, and its rendered base in ConfigHub. | `cub helm install` | | Load rendered files or a literal configuration OCI into ConfigHub Units. | `cub variant upload --component --variant ` | | Adopt an existing Argo, Flux, KRM, or rendered-manifest app. | Read or export it from its current controller, then upload the reviewed YAML with `cub variant upload`. | | Use a maintained catalog entry with supported bases, receipts, scans, and live evidence. | `cub installer setup --pull --base ` | | Upload a reviewed rendered base into ConfigHub. | `cub installer upload` | | Create an environment, region, customer, or target variant after upload. | `cub variant create` | The durable catalog path starts at `cub installer`. The future bridge from a direct Helm render/install into a maintained recipe is tracked as `cub installer import helm`; it is not used as a current command in this repo. For the full routing table, see [Choosing Commands](../../docs/user/choosing-commands.md). ## What Is Proven Today > **What a full test run verifies — and what you can expect to be true:** see > [`docs/reference/verification-properties.md`](../../docs/reference/verification-properties.md) for the > acceptance contract — the verified properties, current coverage, and the > honest limitations. The generated status dashboard is the authority for current counts: [data/status-dashboard/summary.md](../../data/status-dashboard/summary.md). The numbers below are the current public snapshot, not a promise that every chart/base row has every lane complete. ```text 20 popular Helm charts have catalog entries. 20/20 top-20 charts have at least one passing local kind live/e2e receipt. 20/20 top-20 charts have chart-level ConfigHub upload, scan, and safe-ops proof receipt sets. 19/20 top-20 charts are production-review-ready by disposition receipt. 1/20 top-20 charts is production-blocked pending pre-review disposition. 1/20 top-20 charts still needs pre-review target-fact preflight disposition. 20/20 top-20 charts have target-scoped support decision artifacts. 17/20 top-20 charts have supported target-scoped proof scopes. 2/20 top-20 charts are superseded deprecated source charts. 0/20 top-20 charts have rejected target-scoped proof scopes. 1/20 top-20 charts remains a draft support decision. 110 charts have recipe/package proof artifacts. 199 chart/base rows have Helm-template versus cub installer render parity. 198/199 chart/base rows currently have the stricter in-ConfigHub proof lane marked pass. 148/199 chart/base rows have local live proof. 135/199 chart/base rows have GitOps/OCI live proof marked pass. The selected live Helm-vs-ConfigHub comparison lane has 198 committed receipts: 135 pass, 53 watch, 10 blocked rows, and 0 ConfigHub/OCI semantic parity defect receipts. The two-cluster kind parity corpus has 178 committed receipts: 121 pass, 10 watch, 47 blocked rows, 153/178 semantic parity pass rows, and 16 semantic parity defects. 11/100 public top-100 source-scan rows contain Helm hooks. 5 maintained hook queue rows have hook route receipts. 5/5 maintained hook queue rows have lifecycle observation receipts. 0/5 maintained hook queue rows remain partial or route-only. 10/10 explicit lifecycle observation rows are recorded in the outcome spine. Separately, cert-manager has 2/2 `startupapicheck` Helm hook lifecycle observations. External Secrets has lifecycle-observation receipts for CRD/webhook/controller-owned runtime behavior, but no Helm hook in the tested bases. Webhook certificate lifecycle receipts cover staged generated Secrets and staged CRDs for selected ingress-nginx, VPA, and kube-prometheus-stack bases. These receipts demonstrate the observation pattern, not universal hook support. Top-500 source/catalog analysis exists as catalog-planning data for future recipe promotion work. ``` Do not hand-copy these counts into another claim. The generated dashboard and outcome summaries are the source of truth whenever new receipts land. The production support decision contract is in [data/production-disposition/support-decision-contract.md](../../data/production-disposition/support-decision-contract.md). For the user-facing version of that boundary, see [Production Support Decisions](../../docs/user/production-support-decisions.md). The `20/20` claims above are chart-level catalog coverage. Per-base and per-lane status is stricter; use `data/status-dashboard/summary.md` and `data/outcome-coverage/base-outcomes.csv` before making a claim about a specific chart/base pair. Use [Chain Of Proof](../../docs/user/chain-of-proof.md) to avoid mixing proof levels: render proof is not live proof, and live proof has a freshness boundary. For the current non-pass live rows, use `data/live-parity-rerun-plan/summary.md`; it separates semantic parity defects from target prerequisites, target-fit rows, runtime watch rows, and lifecycle work. The current selected live comparison proof means: ```text rendered cub installer objects -> kubectl apply to local kind -> rollout/object checks pass -> observation receipt is committed and verified ``` GitOps/OCI runtime verification lives in a separate live lane: ```text ConfigHub OCI -> Argo CD or Flux -> cluster sync -> live observation through cub-scout / controller evidence ``` That lane depends on a live GitOps controller and cluster, so it is documented and exercised outside the pure local `npm run verify` corpus. The strict live Helm-vs-ConfigHub comparison lane currently has 198 committed receipts: 135 pass, 53 watch, and 10 blocked rows. The non-pass rows are routed to target prerequisites, capability profiles, render-input modeling, image retention, runtime readiness, or GitOps controller review before they become broader support claims. The two-cluster kind parity lane has 178 committed receipts overall: 121 pass, 10 watch, and 47 blocked rows. It is stricter about semantic object equality and currently reports 16 semantic parity defects to inspect before turning those rows into stronger live claims. The lane separates semantic parity from target prerequisites, runtime readiness, hooks, storage behavior, and operating policy. See the generated summaries for exact chart/base status: [Runtime/GitOps Wave](../../data/runtime-gitops/summary.md) and [Live Helm-vs-ConfigHub Parity](../../data/live-helm-confighub-compare/summary.md). For the all-base-variant parity lane, see [Two-Cluster Kind Parity](../../data/live-kind-parity/summary.md). For the exact non-pass follow-up queue, see [Live Parity Rerun Plan](../../data/live-parity-rerun-plan/summary.md). Start here: ```text site/index.html Generated static catalog view: command routing, proof counters, top-20 bases, top-100 readiness, top-500 evidence boundary, and extension slots. CATALOG.md The top-level chart catalog: charts first, variants underneath. docs/user/what-you-get.md The compact product model: why this exists, what is proven, and what remains product work. docs/user/outcomes-and-tests.md What the repo promises, which tests prove each promise, and where the CSVs live. docs/user/helm-pain-points.md How common Helm pain points map to helm-expt, ConfigHub, cub installer, and live observation. data/status-dashboard/summary.md One-page current status: top100, top500 evidence, proof lanes, hooks, quirks, GitOps, and live parity. data/status-dashboard/top20-status.csv Compact chart-by-chart status for the top-20 catalog: recommended base, setup command, base readiness, lane counts, feature summary, gaps, and next action. data/top20-base-readiness/start-here.md The shortest list of clean first catalog paths: chart, base, command, and production-support reminder. data/top20-base-readiness/summary.md One row per top-20 base variant: which bases are clean first paths and which need prerequisites, runtime review, or hook lifecycle work. data/top100-readiness/summary.md The curated planning view: current top-100 adoption bucket, strongest evidence, next action, and first work queues. docs/user/top100-status.md The plain-English top-100 answer for Helm users: what is ready to try, what needs target prerequisites, what needs operator review, and where plain Helm is still more direct. data/top100-user-readiness/summary.md The generated chart-by-chart version of that user-readiness view. data/top100-readiness/next80-queues.md The compact operating queue for the 80 proof-grade charts that are not yet catalog-supported: promotion review, limitation review, or user-shaped variant work. data/extension-slots/summary.md NGINX-like extension slots: 13/20 top-20 catalog charts and 82/100 top-100 chart facts surface raw manifests, tpl snippets, config blocks, sidecars, or add-on slots. The top-500 source scan sees raw/extra manifest values in 254/500 charts and tpl or raw/extra manifest values in 363/500 charts. docs/user/current-proof-status.md What is proven now, and which generated summaries are authoritative. docs/user/verification-lanes.md What each proof lane means, and what it does not prove. docs/user/adopting-existing-apps.md How existing Argo, Flux, KRM, rendered-manifest, and live-resource apps enter the ConfigHub model. docs/README.md The documentation map: what each doc family is for. data/live-e2e/summary.md The top-20 live-test status table. data/production-disposition/summary.md What still has to be resolved before each chart can be production-promoted. data/outcome-coverage/summary.md Outcome promises, proving tests, and front-door CSVs for charts, bases, derived variants, and features. data/pain-point-coverage/summary.md Helm pain points, current answers, handoffs, evidence paths, and remaining gaps. data/variant-path-coverage/summary.md Proof status per chart, base variant, derived variant, diff, and operation path. data/edge-recovery/summary.md Graph fragments recovered from catalog-supported recipe artifacts. data/latest-top20-refresh/summary.md Which supported top-20 charts are current and which need new proof paths. ``` ## Pick Your Path ```text I want to check the repo quickly. Run Quick Verify. I want to see what charts and variants exist. Open CATALOG.md. I want the easiest browsable catalog with setup commands. Open site/index.html. I want a short show-and-tell sequence. Read docs/user/tutorial-sequence.md. I want to know which command to use. Read docs/user/choosing-commands.md. I already run apps through Argo, Flux, KRM, or rendered manifests. Read docs/user/adopting-existing-apps.md. I want the user docs in order. Read docs/user/README.md. I want to understand how to create variants. Read docs/user/creating-variants.md. I have wrapper chart, platform, or customer overlay values. Read docs/user/custom-overlays.md. I installed Redis and want proof it matches the catalog. Run Verify Your Install. I want the full ConfigHub walkthrough. Follow Quick Start With Redis. ``` ## Quick Start With Redis Redis is the happy-path demo because it is small, familiar, and still exercises the important proof chain. The local verification path above does not need a ConfigHub account. The upload part of this demo does. Read the runnable script: ```text docs/demo/redis/demo-script.md ``` The demo uses real commands, including: ```sh cub installer setup \ --pull oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/bitnami-redis:25.5.3@sha256:7ad5fa6de0aa9c29df8cd26650893ebae6ad149a7c5ac33a8beedf5b02e2ac33 \ --base reuse-existing-secret \ --work-dir .tmp/demo/redis-reuse-existing-secret \ --non-interactive \ --namespace redis npm run redis:compare npm run verify npm run redis:verify-install:render -- --base reuse-existing-secret --work-dir .tmp/demo/redis-reuse-existing-secret --namespace redis npm run redis:verify-install:confighub -- --base reuse-existing-secret --space ``` The full ConfigHub upload command is in `docs/demo/redis/demo-script.md`; it is longer because it records labels such as component, layer, owner, chart version, variant, and proof. Redis has two catalog variants: ```text default Renders Secret redis/redis from the pinned demo password. cub installer separates that Secret into out/secrets. reuse-existing-secret Renders no Redis Secret. Requires pre-staged Secret redis/redis-existing-secret key redis-password. ``` The key Redis proof files are: ```text recipes/bitnami/redis/25.5.3/CATALOG.md packages/bitnami/redis/25.5.3/installer.yaml data/installer-oci-packages/summary.md recipes/bitnami/redis/25.5.3/revisions/default/r001/rendered/release-objects.yaml recipes/bitnami/redis/25.5.3/revisions/default/r001/receipts/helm-equivalence-receipt.yaml runs/redis-local-kind/latest/observation-receipt.yaml ``` ### Redis Secret Handling Redis has two supported secret models in this catalog. | Variant | Secret model | What ConfigHub records | Cluster requirement | | --- | --- | --- | --- | | `default` | Helm renders `Secret redis/redis` from the pinned demo password in `effective-values.yaml`. `cub installer` separates that Secret into `out/secrets`. | The rendered Secret is part of the proof, but it is not uploaded as a ConfigHub Unit/OCI artifact. Workloads record an expected external reference to `v1/Secret redis/redis`. | For direct local tests, apply `out/secrets` before `out/manifests`. | | `reuse-existing-secret` | Helm renders no Redis Secret. Workloads reference `redis/redis-existing-secret` key `redis-password`. | The recipe records this as an installer `externalRequires` and variant `targetFacts.requiredSecrets` requirement. No secret material is stored. | Pre-stage `Secret redis/redis-existing-secret` with key `redis-password`; the verifier checks it exists. | The point is deliberate: ConfigHub should carry proof, references, labels, target-fact requirements, and receipts. It should not silently become the place where generated Redis credentials are hidden. ## How A Chart Is Organized For every chart, read the catalog page first: ```text CATALOG.md recipes////CATALOG.md ``` The root `CATALOG.md` is the shelf: charts first, variants underneath, with the recommended first variant and installer package OCI ref visible. Then open the per-chart `CATALOG.md` for the chart source, recipe, rendered objects, source package path, receipts, scans, and current support status. The main folders are: ```text recipes/ Chart analysis, variants, rendered objects, receipts, and catalogs. packages/ Executable cub installer packages. docs/demo/ Plain-English per-chart walkthroughs and transcripts. docs/user/introduction-to-the-harness.md How recipes are generated and how chart behavior is placed into recipes, variants, facts, profiles, gates, receipts, or live observations. runs/ Receipts from ConfigHub, local kind, and proof runs. data/live-e2e/ Top-20 local kind live/e2e summary. data/production-disposition/ What is still required before production support. data/latest-top20-refresh/ Latest top-20 version refresh snapshot plus Redis promotion and Kubara overlay work orders. data/top500-catalog-analysis/ Current top-500 analysis and proof index. ``` ## The Pitch ```text You can keep using public Helm charts, but stop approving guesses. ConfigHub gives you named base and derived variants, exact rendered objects, checks, receipts, and a safer path from test to production. ``` Helm gives you charts. ConfigHub gives you managed, reviewable, scannable, promotable base and derived variants from those charts.