_The Coverage Ladder layer of the helm-expt test map._
This file explains when to run each class of test and what kind of claim each test can support. It complements npm-scripts.md, which lists the individual commands.
The goal is not to run every test after every edit. The goal is to keep claims matched to evidence:
static corpus check
-> render parity
-> ConfigHub proof
-> local Kubernetes proof
-> GitOps/OCI proof
-> strict live Helm-vs-ConfigHub parity
-> lifecycle observation
-> production support decisionTest Principles
Use scoped checks while working. Run the full repo gate only before broad merges, public review, or release-style claims.
Do not use npm run verify as a substitute for a live test. It proves committed artifacts and receipts are internally consistent. It does not create fresh live evidence.
Do not use a live test as a substitute for render parity. Live success can hide object drift. Render parity must still prove that regular Helm and cub installer produced the same intended object set under recorded inputs.
Commit live receipts when the claim depends on them. If a live run is not committed and indexed by the matching generated summary, do not claim the lane changed.
Separate semantic parity from target fit. A chart/base can have perfect object parity and still need a CRD, Secret, StorageClass, hook route, controller readiness check, or operating-policy decision.
Coverage Ladder
| Level | Evidence | Claim |
|---|---|---|
| 1 | Recipe, package, source lock, effective values, receipts, catalog map | The chart/base is represented in the corpus. |
| 2 | Helm template versus cub installer setup receipt | The base renders the same intended Kubernetes objects as regular Helm, with documented intentional differences. |
| 3 | ConfigHub upload, scan, and safe-operation receipts | The rendered objects can become ConfigHub Units and participate in ConfigHub review. |
| 4 | Local kind observation receipt | The rendered objects work on the tested Kubernetes target. |
| 5 | ConfigHub OCI plus Argo or Flux receipt | The ConfigHub artifact can be reconciled by a GitOps controller on the tested target. |
| 6 | Strict live parity receipt | Regular Helm and the cub installer path reached the recorded equivalent live outcome. |
| 7 | Lifecycle observation receipt | Controller-owned, hook-like, or post-apply behavior was observed fresh. |
| 8 | Production disposition and target-scoped decision | The chart is accepted for a named production support scope. |
Use the narrowest true claim. For example, render parity is not the same as GitOps live, and production-review-ready is not the same as production-supported.
Current Corpus Shape
Top-20 is the public catalog lane. It has named base variants, broad proof coverage, live evidence, and production-disposition inputs. Exact status still depends on chart, base, and lane.
Top-100 is the maintained proof corpus. Every chart has recipe/package proof and render parity, but most rows still need catalog promotion review, useful variants, or selected live lanes.
Top-500 is source and catalog-planning reconnaissance. It shows where Helm pain appears across popular public charts. It is not an installable catalog.
Authoritative generated views:
| Need | Start with |
|---|---|
| Current aggregate status | data/status-dashboard/summary.md |
| Outcome and lane coverage | data/outcome-coverage/summary.md |
| Top-20 base readiness | data/top20-base-readiness/summary.md |
| Top-100 workability | data/top100-readiness/summary.md |
| Top-500 evidence map | data/top500-catalog-analysis/summary.md |
| Live parity residue | data/live-parity-rerun-plan/summary.md |
| Hook and lifecycle boundary | data/lifecycle-boundary/summary.md |
| Test command details | tests/npm-scripts.md |
When To Run What
| Work | Checks |
|---|---|
| One manual doc edit | npm run docs:verify |
| Command example edit | npm run docs:verify and npm run installer:command-surface:verify |
| Generated data edit | The owner *:verify, then npm run data:index:verify if CSVs were added, removed, renamed, or reclassified |
| Static site edit | npm run site:verify |
| One curated chart proof edit | <chart>:compare, <chart>:verify-proof, <chart>:verify-proof:self-test, <chart>:verify-package |
| Shared proof-kit edit | Representative chart checks for the changed hook, then all migrated top-20 proof/package checks before broad merge |
| Recipe/package graph or target facts | Chart-specific checks, npm run verify:artifact-chain, and npm run installer:target-facts:verify when facts changed |
| Broad proof-model change | Focused checks for each changed surface, then npm run verify |
| Fresh local Kubernetes evidence | The lane runner that writes the receipt, followed by the lane summary and verifier |
| Fresh GitOps/OCI evidence | tests/chart-install-test or tests/chart-install-sweep, followed by the runtime/GitOps summary verifier |
| Fresh strict live parity evidence | npm run kind-parity:run for one chart/base, or the top-20 variant runner for a campaign |
Live Test Work Unit
The atomic live work unit is one chart/base row:
chart
base variant
target cluster or pair of clusters
commands run
rendered object digest
runtime observations
result: pass, watch, blocked, or fail
receipt pathEvery receipt should explain non-pass rows. A watch or blocked row is useful when it separates:
- semantic object parity defect;
- target prerequisite;
- runtime readiness;
- hook or lifecycle behavior;
- storage or operating-policy decision;
- infrastructure/provisioning failure.
Do not collapse those cases into a generic failure.
Strict Helm-vs-Installer Parity
The preferred parity test uses two identical vanilla kind clusters:
New to cub? Install the cub CLI first. You can pull and render public catalog packages without an account. Commands that save or change ConfigHub data require you to sign in.
What this command does. cub installer is a released, open-source plugin for the cub CLI. cub installer setup pulls a catalog package and writes its Kubernetes files locally. It does not apply those files to a cluster; use kubectl, Argo CD, or Flux for delivery. The generated scripts stop before doing any work when the plugin or kustomize is missing.
cluster A: regular Helm install
cluster B: cub installer setup + kubectl apply
compare intended objects and live outcomesThis keeps the parity question separate from ConfigHub delivery machinery. It does not require ConfigHub's cub cluster helper.
Use ConfigHub/OCI live tests for a different question:
Can ConfigHub Units be published or applied through OCI and reconciled by Argo
or Flux on a live target?Both lanes matter. They prove different outcomes.
Run strict live parity serially unless a lock/guard prevents concurrent cleanup from deleting another run's kind clusters. Every kubectl command should pin the intended kubeconfig and context.
Chart Refresh Strategy
Do not replace a supported chart version just because upstream released a new one.
Use this path:
current supported version
-> latest candidate render/proof
-> diff, quirk, and gap review
-> live or lifecycle evidence where needed
-> production-disposition update if needed
-> catalog promotion decisionOlder supported versions can remain valuable for patch, audit, and enterprise support. Keep old versions until the project has an explicit replacement or retirement decision.
Scaling Strategy
Scale by rows, not by narrative claims:
top20 chart/base rows
-> top100 proof rows
-> top500 promotion candidatesParallel runs are appropriate for independent chart/base rows, but each live runner must own its cluster names, kubeconfigs, namespaces, ConfigHub spaces, and cleanup. Receipt aggregation should be mechanical: collect row receipts, regenerate summaries, then run the matching verifiers.
The desired end state is a catalog where every chart/base row has:
- render parity;
- in-ConfigHub proof when the row is presented as a ConfigHub-managed path;
- live proof for the target delivery mode being claimed;
- lifecycle receipts for hooks, CRDs, webhooks, generated data, or controller-owned fields;
- production support decisions only for explicitly scoped targets.