You can work through this Guide two ways, and both use the same commands. Read it top to bottom yourself and run each command as it appears, or hand the whole Guide to an agent with the assistant task below and check its answers against the same retained files. Composing a recipe for the first time needs the aicr CLI and network access to pull component charts; every command below reads only committed files and needs neither.
The Catalog retains four AICR entries for one recipe, eks-h100-training-kubeflow, an EKS, H100, Ubuntu, Kubeflow training platform: a raw v0.14.0 base rendered for Flux, and three Argo CD wrapper bases at v0.14.0, v0.19.0, and v0.20.0. Each retained base resolves from its listing first, for example site/listings/aicr-eks-h100-training-kubeflow-v0-14-0-argocd.json. Read a listing before running anything; it names the recipe, the criteria, the object count, and the flattening verdict for that exact entry.
Jump to the assistant task. Complete the setup first if this is your first Guide.
Set up
Install the pinned Workshop plugin as described in the Adapt setup. That gives you the cub command surface this Guide uses. Every command below runs in the evidence checkout, this repository, against the committed AICR entries under examples/aicr. Composing a recipe from scratch with the aicr CLI needs network access to pull the named component charts; reading the retained recipe, rendered objects, and receipts does not.
Check your agent the same way every time
Predicting the exit code and the one answering field before you run each command is the same rule the skill states from the agent's side in "Say What You Expect Before You Run It", and the three moves in the Helm Guide's agent-checking section apply here without change: make the agent cite full object identities (argoproj.io/v1alpha1|Application|argocd|gpu-operator, not "the GPU operator app"), compare its answer with the retained inventory or receipt the question names, then run that question's gate and confirm it rejects a wrong answer before you trust that it accepts a right one.
One AICR-specific habit sits beside those three. Every retained entry here is an Argo CD wrapper: one root Application and sixteen component Applications that each still name a Helm chart or a local chart path. An agent that reports "AICR rendered the workload" without naming which sixteen sources Argo CD still has to process later has answered a smaller question than the one this Guide asks.
1. What does this recipe compose, and what must already exist?
Someone has a pinned AICR recipe and wants the exact object list and the destination prerequisites before choosing a cluster to point it at.
Why AICR does this. The recipe's criteria (accelerator: h100, intent: training, os: ubuntu, platform: kubeflow, service: eks) select fifteen components and order them; composing the recipe with a deployer turns that selection into exact wrapper objects. Reading the wrapper objects answers what will exist as literal bytes. It does not answer what the destination must already provide, because Argo CD itself, its Application CRD, and the storage and node facts the recipe assumes are never among the objects the wrapper renders.
Run it in the evidence checkout, using the v0.14.0 Argo CD entry.
Predict first. Before you run it, ask your agent what this will print: the exit code, and the one field that answers the question (here, the object count and the namespace cub config check reports as required). Run it. If the result matches, the explanation stands. If it differs, the agent guessed; read the real output before trusting it.
New to cub? Install the cub CLI first. Public catalog packages pull and render anonymously, and you sign in only once a command saves or changes ConfigHub data.
cat examples/aicr/eks-h100-training-kubeflow/argocd-rendered/templates/*.yaml > eks-h100-v0-14-0-argocd.yaml
cub config check ./eks-h100-v0-14-0-argocd.yamlRead the result. Expect exit 0. The command reports 17 objects, all Application (argoproj.io/v1alpha1), and reports argocd under the namespaces that must already exist, because none of the 17 objects is a Namespace object. It reports zero CRDs, because none of the 17 objects is a CustomResourceDefinition; that count says nothing about whether a CRD needs to exist elsewhere; it counts CRD objects present in this file, and none is. The real CRD dependency is that Argo CD's own Application custom resource must already be served before any of these 17 objects means anything, which is exactly why it is recorded separately as a target fact rather than left for the object counter to discover.
Check the agent. Ask it what this composition assumes beyond the 17 objects, then compare its answer with the v0.19.0 route intent, whose target facts apply to this recipe lineage: Argo CD and its Application CRD must exist, the target must be an EKS cluster with H100 nodes running Ubuntu, a gp3 StorageClass must be available, and GPU nodes must carry nvidia.com/gpu.present=true. Then run the gate:
npm run aicr-argocd-example:verifyThe command prints verified AICR Argo CD OCI example (54 source files, 17 Applications, 2 local OCI layouts) for this v0.14.0 entry, checking the source bundle and rendered checksums that back the object count above. Read the generation receipt alongside it: the claim is that AICR generated a portable Argo CD Helm chart and 17 exact Application objects, both OCI artifacts are publicly pullable at their recorded digests, and Argo CD reconciliation and GPU-cluster health have not run.
What this does not prove. Reading 17 wrapper objects establishes nothing about the cluster, and it does not prove the sixteen referenced charts render cleanly. The nested-source table for the newer v0.20.0 entry shows all sixteen local renders captured, eight of them containing CRDs of their own, zero containing Helm hook objects; that table is the closest local answer to "what do the component charts actually need," and it is not claimed for v0.14.0.
Where it goes next. Retain the reviewed object set with the same command the listing itself gives:
cub variant upload --dry-run --component eks-h100-training-kubeflow --variant argocd --space aicr-eks-h100-training-kubeflow-v0-14-0-argocd --unit-annotation workshop.confighub.com/object-set-sha256=sha256:fc03b2950c81db63a4853424fcb30942e9ecf0e94e54f7474439421dcc254801 ./rendered2. How does one AICR version differ from another?
A platform team has retained v0.14.0, v0.19.0, and v0.20.0 of the same recipe and needs the exact object and field that moved between them, not a changelog.
Why AICR does this. Every retained version was composed from the same criteria, so a version bump changes component chart versions, sync-wave placement, or embedded health checks, not the component list. Comparing the exact rendered Applications is how you see which of those three things actually moved for a given upgrade.
Run it in the evidence checkout, using the ConfigHub-exported single-file release for v0.19.0 and v0.20.0.
Predict first. Before you run it, ask your agent what this will print: the exit code, and the one field that answers the question (here, the count of changed Applications and which ones). Run it. If the result matches, the explanation stands. If it differs, the agent guessed; read the real output before trusting it.
cub config diff examples/aicr/eks-h100-training-kubeflow-v0-19-0/confighub-release/aicr-eks-h100-training-kubeflow.yaml examples/aicr/eks-h100-training-kubeflow-v0-20-0/confighub-release/aicr-eks-h100-training-kubeflow.yaml --json --exit-code --out v19-v20-diff.jsonRead the result. Expect exit 1, a finding rather than a failure: 17 objects on each side, 0 added, 0 removed. Every one of the 17 reports changed, because each carries ConfigHub's own recorded confighub.com/origin annotation naming the space and revision it was retained from, and that annotation differs between the v0.19.0 staging export and the v0.20.0 production export. Read past that recorded bookkeeping field to the object's other changed paths, and only 4 of the 17 carry one: aicr-stack (/spec/source/repoURL, /spec/source/targetRevision, /spec/source/helm/valuesObject/repoURL), kubeflow-trainer-post and nodewright-customizations (/spec/source/repoURL, /spec/source/targetRevision), and nvsentinel (/spec/source/targetRevision, from v1.9.0 to v1.20.0). The other 13 Applications are byte-identical apart from the provenance annotation.
Check the agent. Ask it to separate the recorded provenance annotation from the recipe-driven change before it reports a count, then compare its answer with the retained version-diff record, which computes the same v0.19.0-to-v0.20.0 transition from committed bytes: 4 of 17 component Applications changed version, 0 changed sync-wave, and NVSentinel's own health check tightened its timeout from 5m to 90s and added 2 DaemonSet checks. Then run the gate:
npm run aicr-version-diff:verifyThe command reprints verified 5 retained AICR versions; latest transition v0.20.0 -> v1.0.0 changes 8 of 17 component Applications. That printed line covers the newest retained pair, a fourth entry beyond the three this Guide walks. The v0.19.0-to-v0.20.0 numbers above come from the same underlying data/aicr-version-diff/diff.json record, which retains every adjacent transition, not only the latest one. The gate compares retained files and nothing more, so the newest figure says what changed between the two recorded bundles and says nothing about how either one behaves on a cluster.
What this does not prove. A version/wave-unchanged component can still carry a changed values block the classifier does not itemize; the record's own closing note says a full comparison retains objectComparisons with every changed JSON-pointer path for exactly this reason. Neither this comparison nor its gate runs anything against EKS.
Where it goes next. Read the four real changes against the v0.20.0 flattening verdict before promoting; a repoURL or targetRevision change is exactly the kind of edit that verdict's recheck rule names.
3. Does this workload fit the H100 accelerator target?
Someone has this recipe's declared accelerator, h100, and a candidate destination, and needs to know whether the two actually match before delivery.
Why AICR does this differently from a generic GPU match. The Match Guide answers this question for one InferenceService compared against a supplied Node snapshot, using a retained model.yaml and nodes.yaml pair built for that exercise. No equivalent pairing exists in this repository for this recipe's own 17 Applications, so this Guide does not invent one. What is retained instead is the fit question itself, recorded as a named step.
Read the record, in the evidence checkout: the v0.20.0 route intent names a step with the id source-variant-target-match. It asks the platform operator, or an admission policy, to compare what the destination has with what NVIDIA's recipe expects. Its status is recorded-not-run. It requires a "target snapshot, selected source-catalog digest, and an explained match or exception." The selected recipe itself is recorded in the source-catalog record: h100-eks-ubuntu-training-kubeflow, structural status pass, with the overlays applied to reach it listed in order.
Check the agent. Ask it to name the two things a real fit check would need to compare, then confirm it names the recipe's criteria block (accelerator: h100, service: eks, os: ubuntu, platform: kubeflow) on one side and an observed target snapshot on the other, not a generic "does this have a GPU" question. There is no gate to run here, because source-variant-target-match is recorded not-run; running the Match Guide's generic exercise would check a different model against different node facts, not this recipe's own fit question.
What this does not prove. This question stays open in this repository. The recipe names its intended target; nothing here observes a real cluster's node facts and runs the comparison. Treat "the criteria say H100" as the recipe's declared intent, not as a passed match.
Where it goes next. Collect a target snapshot from the destination cluster, then run the fit comparison the step already names before treating this recipe as validated for that cluster. That snapshot-and-compare step is the pending live trial, not a local rung this Guide can complete today.
4. What component order, nested sources, and prerequisites travel with this base?
A GitOps operator has the 17 wrapper objects and needs to know what order they require, which sixteen still name an external source, and what the destination must provide before any of it reconciles.
Why AICR does this. AICR assigns every component Application a deploymentOrder and a sync-wave annotation, but the wrapper objects themselves still name Helm charts or local chart paths for Argo CD to process later. Both facts have to travel with the 17 objects, or a plain unordered apply misses work the recipe already decided.
Run it in the evidence checkout, using the v0.20.0 entry.
Predict first. Before you run it, ask your agent what this will print: the exit code, and the one field that answers the question (here, the number of distinct sync-waves and how many components define one). Run it. If the result matches, the explanation stands. If it differs, the agent guessed; read the real output before trusting it.
grep -c "argocd.argoproj.io/sync-wave" examples/aicr/eks-h100-training-kubeflow-v0-20-0/argocd-rendered/templates/*.yamlRead the result. Expect exit 0, with 16 files reporting exactly one match each and aicr-stack.yaml (the root Application) reporting none; the root has no sync-wave, because it deploys first by definition. The recipe deploymentOrder names the same sixteen components in the order their dependencyRefs require: storage and networking first (aws-ebs-csi-driver, aws-efa), then cert-manager and nfd, then gpu-operator and its dependents, ending with nvsentinel and prometheus-adapter. The v0.20.0 route intent records this as the component-order step, owned by Argo CD, status recorded-not-run, and records a fourth step, downstream-chart-lifecycle, for the CRDs, hooks, certificates, and setup Jobs each of the sixteen referenced charts may carry on its own.
Check the agent. Ask it to name which of the sixteen components render their own CRDs, then compare its answer with the nested-sources table: cert-manager (6 CRDs), gpu-operator (2), kai-scheduler (6), kubeflow-trainer (4), nfd (4), nodewright-operator (2), nvidia-dra-driver-gpu (2), and prometheus-operator-crds (10), for 8 of the 16 in total, with 0 Helm hook objects found in any of the sixteen. There is no separate gate for this question beyond the route-intent record itself; the v0.20.0 generation receipt is the retained source for the exact command that produced these bytes.
What this does not prove. A local render capturing a CRD is not the same as watching that CRD reach Established, and "no Helm hook objects found" in this rendering pass does not certify that no future chart version adds one. aicr-node-inspection and aicr-configuration-validation, the last two steps the same record names, are recorded available-not-run-for-this-entry and not-run.
Where it goes next. Resolve argocd-prerequisite and component-order for the destination you have chosen before relying on Argo CD reconciliation, then work through downstream-chart-lifecycle chart by chart using each referenced chart's own flattening decision, the same rule the v0.20.0 flattening verdict states.
Give an assistant the whole AICR path
Start a fresh session with this checkout prepared and ordinary tool approvals:
Walk the four questions in docs/user/workshop-aicr-guide.md in order. For
each one, run the given command, save its output to its own file, and report
the actual exit code. Name object identities in full, including API version,
kind, namespace, and name. For question 1, compare your answer with
examples/aicr/eks-h100-training-kubeflow-v0-19-0/route-intent.yaml and run
npm run aicr-argocd-example:verify. For question 2, separate the recorded
confighub.com/origin annotation from the recipe-driven change before you
report a changed-object count, compare with data/aicr-version-diff/summary.md,
and run npm run aicr-version-diff:verify, reporting the exact printed line.
For question 3, read
examples/aicr/eks-h100-training-kubeflow-v0-20-0/route-intent.yaml and report
the source-variant-target-match route's status exactly as recorded; do not
report a match result that was not run. For
question 4, read the recipe's deploymentOrder and the nested-sources table
and report which components render their own CRDs. Do not run the aicr CLI
or any cub command beyond the ones this Guide names, and do not contact a
cluster, ConfigHub, or a registry. State every boundary each question lists
as not proven, and mark anything you did not run as not checked rather than
passing.Read the report against this Guide rather than accepting it. An agent that reports question 3 as "matched" or "fits" has invented a result the record does not contain; the honest answer is that the fit check is recorded and not run.
Know what this Guide does not settle
Every answer above comes from reading composed wrapper objects and retained comparisons. That is a local result. It is not a ConfigHub record, a delivered change, or a live-cluster proof, whatever the object count or exit code says.
All four retained entries carry a flatten-with-routes verdict: the exact wrapper objects can be kept literally, but the component order, the nested Helm sources, the accelerator-target fit, and the CRD and hook work inside those sixteen sources are lifecycle work that has to travel beside them. Resolving it once does not resolve it again after the recipe version, the destination, or the delivery runtime changes; each flattening verdict names its own recheck triggers.
The object counts and digests cited throughout this Guide are the committed Catalog evidence for these entries: the v0.14.0 Argo CD BaseVariantRecord, the v0.19.0 record, and the v0.20.0 record. Collecting a real target snapshot, running the accelerator-fit comparison, and watching Argo CD reconcile these objects and the sixteen charts they reference on an actual EKS cluster are the pending trial this Guide does not run. Missing coverage stays marked as not checked, here and everywhere else.