One AICR v0.20.0 recipe for EKS H100 training

What it is. AICR recipe aicr-eks-h100-training-kubeflow v0.20.0, as 17 Argo CD Applications.

Install it. Apply its Applications on a cluster that runs Argo CD. The five steps below give the commands.

OCI bundle. oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:0.20.0

Needs first. Compare observed destination facts with the intended NVIDIA recipe. Make the Argo CD Application API and controller available. The plan below lists 4 more.

UNOFFICIAL/EXPERIMENTAL. This entry keeps NVIDIA AICR v0.20.0 beside the v0.14.0, v0.18.0, and v0.19.0 entries. It uses the same EKS, H100, Ubuntu, training, and Kubeflow choices, so you can see what the new release changes without replacing the earlier configurations.

Why use this entry?

The two public OCI artifacts have different jobs:

Artifact	What it contains	Digest
Source package	The AICR-generated app-of-apps Helm chart.	`sha256:2eedf6b36ea2cfab828245e38f72b7ed344915b6695c6a56d92a744d25992431`
Literal configuration	The 17 exact Argo CD Application objects, one YAML layer per object.	`sha256:4779ed69b9858791379a93704bb92ac11ec3b72e987b34921e293dd695415be8`

Run this repository check to pull both artifacts without registry credentials and compare every pulled blob with the retained copy:

npm run aicr-v0200:public-verify

The public OCI receipt records the exact references, digests, and anonymous pulls.

Recipe, base and variant

Three words name three different points. A recipe is chosen before flattening. A base is the flattened result. A variant is a copy of the base with changes for one environment.

Word	Meaning here	Current v0.20.0 status
Recipe	NVIDIA's curated combination of service, accelerator, operating system, workload, and platform: `h100-eks-ubuntu-training-kubeflow`.	Selected and retained.
Base	The checked starting configuration: source and intent, 17 exact Applications, lifecycle plan, field policy, digests, and receipts.	Retained in the Catalog, published as OCI, and stored in the live ConfigHub Catalog organization.
Variant	A reviewed change made from the base for development, staging, production, a region, or a customer.	Development, staging, and production variants now contain one promoted Grafana Secret change.

The source record keeps three counts separate: 103 overlay files in the retained source tree, 102 entries in the embedded catalog, and 45 resolved leaves. The five choices select one of those leaves. NVIDIA curates the built-in recipes. Other Catalog providers can add target-specific configurations. A snapshot diff alone cannot decide what a particular target ought to contain.

The selection is now an imported record rather than metadata inferred by the ConfigHub Workshop generator:

Field	Retained value
Provider	NVIDIA, acting as the source-catalog curator
Source catalog	NVIDIA AICR built-in catalog `v0.20.0`
Catalog content digest	`sha256:676f2d59eacd79ae1b72e5cbe00216b577def1da412dbdabb032f317a62dc1d8`
Selected recipe	`h100-eks-ubuntu-training-kubeflow`
Selection dimensions	`service=eks`, `accelerator=h100`, `os=ubuntu`, `intent=training`, `platform=kubeflow`

The generated base-variant record copies those fields from the source-catalog record. The ConfigHub upload receipt carries the same provider, catalog digest, selected recipe, and dimensions beside the exact-object comparison. This keeps the provider's source choice separate from the ConfigHub base and the later development, staging, and production changes. Provider-linked evidence is not treated as ConfigHub delivery or runtime evidence.

What the source produces

AICR resolves the five criteria into a recipe with 15 ordered components. It then generates an Argo CD app-of-apps Helm chart. Helm renders that wrapper chart into 17 exact Application objects.

The digest index binds the selected inputs, source package, rendered Applications, and OCI manifests under one platform digest. The generation receipt records the commands and source version. The source verification checks the release checksum, signed binary provenance, recipe-catalog signature, and SBOM attestation.

Rendering and lifecycle work

The wrapper is only a partial flattening boundary. The 17 Application objects are exact. Sixteen of them point to Helm charts or local chart sources that a controller processes later.

The Catalog now materializes those 16 sources separately. Each record binds the fetched chart archive or local chart tree, the retained values, and the rendered object set with SHA-256 digests. Together they produce 409 local Kubernetes objects. Eight components contain 36 CRDs. No Helm hook object appears in this selected render.

These are local renders. They show what each source produces with the selected values. They do not show that Argo CD or Flux applied the objects, that the CRDs became ready, or that a workload ran on H100 hardware.

The route intent records the lifecycle work that may be needed: ordering, CRDs, prerequisites, and AICR health checks. The destination-specific records then say who performs that work for Argo CD and Flux. Neither record says that the work ran.

The Flux bundle builds locally into 29 controller objects. Its NVSentinel HelmRelease uses CreateReplace for CRDs because the component record declares NVSentinel as their sole owner. The setting is not copied to components that share CRDs. A live Flux CRD upgrade still needs a real Git source and target.

The field-policy assessment separates source-controlled choices from fields that may become reviewed ConfigHub changes.

Stored, changed, and promoted in ConfigHub

The ConfigHub base stores the 17 Application objects and a README as separate Units. Development, staging, and production are derived from that base.

The development change removes the literal Grafana administrator password and uses the existing Secret monitoring/aicr-grafana-admin instead. A dry run named the affected kube-prometheus-stack Application without changing stored data. One ChangeOrder then moved that change through staging and production. No other Application changed.

Every environment requires approval before release. An attempted unapproved release was blocked. After the production configuration and README were approved, ConfigHub published a release OCI with this manifest digest:

sha256:2576c38241454d44998c4ad26615552e9b04ed745e06724f4213c1c39a8e47d1

The release was pulled back by that digest. Its configuration matches the promoted production object set and its README matches the approved production README. Promotion moved a reviewed change between ConfigHub environments; release publication made the resulting OCI available for delivery.

This proves the ConfigHub records, change, promotion, approval, OCI publication, digest pull, and file comparison. It does not prove that Argo CD or Flux reconciled the release, that EKS accepted the lifecycle work, or that an H100 workload ran.

What changed from v0.19.0?

The computed comparison records four source-version changes and no ordering change:

the wrapper moves from 0.19.0 to 0.20.0;
kubeflow-trainer-post and nodewright-customizations move to 0.20.0;
NVSentinel moves from v1.9.0 to v1.20.0.

NVSentinel's health check now checks two driver-labelled DaemonSets as well as the labeler Deployment and pods. Its overall timeout changes from five minutes to 90 seconds, so a stalled DaemonSet reports the useful failure sooner. The optional zero-desired cases remain excluded. This is a source comparison; the health check has not run on EKS in this entry.

Current status

Stage	Result	Record
Select the AICR recipe	Complete. The five criteria resolve to one recipe that NVIDIA curates.	Source record
Verify the upstream source	Complete. Checksums, binary provenance, recipe-catalog signature, and SBOM attestation pass.	Source verification
Generate the wrapper and exact Applications	Complete. The retained result contains 17 Applications.	Generation receipt
Publish source and literal configuration OCI	Complete. Both artifacts pull anonymously and match the retained bytes.	Public OCI receipt
Materialize the 16 nested sources	Complete locally. The records bind 16 source artifacts and values sets to 409 objects, including 36 CRDs.	Nested source inventory
Resolve lifecycle work for Argo CD and Flux	Plans recorded. Both remain blocked until a destination, controller, and required credentials are available.	Route resolution summary
Retain a ConfigHub base and create derived variants	Complete. The base, development, staging, and production Spaces retain exact configuration and their own README Units.	ConfigHub base receipt and promotion receipt
Check approval, promote, and publish a release OCI	Complete. One named change reached production; an unapproved release was blocked; the approved release pulled back by exact digest and matched.	Required-approval check and release OCI receipt
Deliver through Argo CD or Flux	Not run for v0.20.0.	Requires destination-specific receipts.
Run on EKS and H100	Not run. No training or NIM request, observation, or rollback is claimed.	Tracked in issue #1581.

The configuration work is now retained through an approved production release. The next step is destination evidence: reconcile the release through Argo CD on EKS, check its CRD and prerequisite handling, run an H100 workload, record live state, and test exact rollback. Flux and small-fleet evidence follow that run.

Use this entry in ConfigHub

Every Catalog entry page carries these five steps in this order. Each step says what the Catalog has run for this entry, and a step with a missing precondition gives no command.

Steps 1 and 2 need no account. Install the cub CLI before any cub command. Step 2 uses cub config diff, which cub plugin install confighub/cub-workshop@v0.6.61 adds. A live run on 2026-10-08 used these cub variant commands on other configuration, with no Target. It is not a run for this entry.

What is fixed, and what you supply

Fixed when this entry was built. These went into the build, so the objects already carry them. To change one, build the entry again from its source.

accelerator is h100
intent is training
os is ubuntu
platform is kubeflow
service is eks
acceleratedNodeSelector is nvidia.com/gpu.present=true
repoURL is oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow
storageClass is gp3
workloadSelector is app.kubernetes.io/part-of=training
deployer is argocd-helm

Yours to supply or check. The destination must have these before the objects are applied. The words in italics say what the Catalog knows about each.

Argo CD and its Application CRD must exist before the retained wrapper objects can be applied. Declared. Not checked on any destination.
The destination must match the selected EKS, H100, Ubuntu, and Kubeflow training recipe. Declared. Not checked on any destination.
A gp3 StorageClass must be available for the selected storage input. Declared. Not checked on any destination.
GPU nodes must carry nvidia.com/gpu.present=true for the selected placement. Declared. Not checked on any destination.
EFA, RDMA, driver, and nvidia_peermem requirements must be checked against the actual node and network role. Declared. Not checked on any destination.

This entry's plan, and how to test it

These are the steps that plain objects cannot carry. ConfigHub stores them and does not carry them out. The words in italics say what the Catalog knows about each.

Once the destination is known. compare observed destination facts with the intended NVIDIA recipe Who: selected destination actor. Written down. Not run by the Catalog.
Once the destination is known. make the Argo CD Application API and controller available Who: selected destination actor. Written down. Not run by the Catalog.
Once the destination is known. reconcile component Applications in AICR dependency order Who: selected destination actor. Written down. Not run by the Catalog.
Once the destination is known. handle CRDs, hooks, certificates, operators, and setup Jobs for every nested source Who: selected destination actor. Recorded as materialized-awaits-destination-execution.
Once the destination is known. collect and compare node facts without requiring a selected recipe Who: selected destination actor. Recorded as available-not-run-for-this-entry.
Once the destination is known. run checks whose prerequisites are satisfied by the selected recipe Who: selected destination actor. Not run.

Read what a plan is, and why some entries have one.

1. Get the exact objects Run for this entry

The Catalog rendered this entry and retains its 17 Argo CD Applications as files that one inventory lists.

# make the directories the inventory names
$ mkdir -p aicr-eks-h100-training-kubeflow-v0-20-0-argocd/templates

# fetch the inventory, then every file it lists
$ curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-20-0-argocd/checksums.txt https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-20-0/argocd-rendered/checksums.txt

$ cut -c67- aicr-eks-h100-training-kubeflow-v0-20-0-argocd/checksums.txt | xargs -I{} curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-20-0-argocd/{} https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-20-0/argocd-rendered/{}

# check every file against the inventory
$ (cd aicr-eks-h100-training-kubeflow-v0-20-0-argocd && shasum -a 256 -c checksums.txt)

# check the inventory against the listing digest
$ shasum -a 256 aicr-eks-h100-training-kubeflow-v0-20-0-argocd/checksums.txt
f2a4f9e60e872bf2266a8dc89a239e704b0dd5f77a643c3bdf9561a088de99cc  aicr-eks-h100-training-kubeflow-v0-20-0-argocd/checksums.txt

# join the files into one for step 2
$ cat aicr-eks-h100-training-kubeflow-v0-20-0-argocd/templates/*.yaml > aicr-eks-h100-training-kubeflow-v0-20-0-argocd.yaml

2. Compare with another version or base Not run for this entry

The listing records no comparison for this entry. The commands run on the retained files, with no account. The Catalog holds 4 other versions, v0.14.0, v0.18.0, v0.19.0 and v1.0.0.

# fetch the argocd base at version v0.19.0
$ mkdir -p aicr-eks-h100-training-kubeflow-v0-19-0-argocd/templates

$ curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-19-0-argocd/checksums.txt https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-19-0/argocd-rendered/checksums.txt

$ cut -c67- aicr-eks-h100-training-kubeflow-v0-19-0-argocd/checksums.txt | xargs -I{} curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-19-0-argocd/{} https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-19-0/argocd-rendered/{}

# join them into one file
$ cat aicr-eks-h100-training-kubeflow-v0-19-0-argocd/templates/*.yaml > aicr-eks-h100-training-kubeflow-v0-19-0-argocd.yaml

# list what changed from v0.19.0 to v0.20.0
$ cub config diff aicr-eks-h100-training-kubeflow-v0-19-0-argocd.yaml aicr-eks-h100-training-kubeflow-v0-20-0-argocd.yaml --summary

3. Upload it as a variant Run for this entry

The Catalog saved this entry in ConfigHub. The commands use its published literal configuration bundle. Read the receipt.

# sign in; steps 3 to 5 write to your ConfigHub organization
$ cub auth login

# preview the upload of the published bundle; nothing is written
$ cub variant upload --dry-run --component eks-h100-training-kubeflow --variant argocd --space aicr-eks-h100-training-kubeflow-v0-20-0-argocd oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:0.20.0

# upload the bundle as a base Space of Units
$ cub variant upload --component eks-h100-training-kubeflow --variant argocd --space aicr-eks-h100-training-kubeflow-v0-20-0-argocd oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:0.20.0

4. Deploy it Partly run for this entry

A delivery is partly recorded for this entry, and no run has passed. Argo CD is partly recorded, Flux does not apply, and kubectl has no record. The commands name the Target workshop/target, which cub cluster up --name workshop creates on a local kind cluster with Argo CD. Use your own Target for a real destination. The listing records 5 inputs that the destination must supply, so read them before you deploy. Read the receipt.

# clone the base onto a Target as a dev variant
$ cub variant create dev aicr-eks-h100-training-kubeflow-v0-20-0-argocd --target workshop/target --space-pattern "template:aicr-eks-h100-training-kubeflow-v0-20-0-argocd-dev"

# publish the Release that Argo CD or Flux pulls
$ cub release publish aicr-eks-h100-training-kubeflow-v0-20-0-argocd-dev

5. Promote a change Run for this entry

A promotion ran for this entry. It followed base to development to staging to production. Promotion carries a change in the base to the dev variant. The base changes when you edit a Unit or upload a newer version into the same Space. Read the receipt.

# preview what the dev variant would take from the base
$ cub variant promote aicr-eks-h100-training-kubeflow-v0-20-0-argocd-dev --dry-run -o mutations

# promote once you have read the preview
$ cub variant promote aicr-eks-h100-training-kubeflow-v0-20-0-argocd-dev --change-desc "Pull the reviewed base forward"

For agentsThese steps are the nextSteps array of aicr-eks-h100-training-kubeflow-v0-20-0-argocd.json. Each state was read from flattened.retainedObjects, flattened.inventory, source.ociRef, oci.bundles, oci.runtimes, lifecycle.promotion, lifecycle.coverage and variants.known in the same listing. A command marked needsAccount there contacts ConfigHub. A step's notes hold its cautions, carries lists the CRDs the base brings, and liveWalk links the dated log that ran its commands.

Technical notes and disclaimers

This entry is a set of Argo CD Applications, so a test needs Argo CD. The Catalog has no test script for it.