Skip to content
Browse Stacks

Deep Dives

AI: Claude and Codex patterns

ConfigHub: Deploy and manage

Stacks
Configs
Apps
Plugins
Guides
Docs

One AICR v1.0.0 recipe for EKS H100 training

Use it when you want to read what NVIDIA's first 1.0 release produces for an EKS H100 training platform. You can inspect all 17 generated Argo CD Applications in this repository without a ConfigHub account, an EKS cluster, or a GPU. It is one of the NVIDIA AICR pages. You can also view this as a markdown file.

Use this entry in ConfigHub in five steps, from fetching its exact objects to promoting a change.

What it is. AICR recipe aicr-eks-h100-training-kubeflow v1.0.0, as 17 Argo CD Applications.

Install it. Apply its Applications on a cluster that runs Argo CD. The five steps below give the commands.

OCI bundle. oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:1.0.0

Needs first. Compare observed destination facts with the intended NVIDIA recipe. Make the Argo CD Application API and controller available. The plan below lists 7 more.

UNOFFICIAL/EXPERIMENTAL. This entry keeps NVIDIA AICR v1.0.0 beside the v0.14.0, v0.18.0, v0.19.0, and v0.20.0 entries. It uses the same EKS, H100, Ubuntu, training, and Kubeflow choices, so you can see what the v1.0.0 release changes without replacing the earlier configurations.

Why use this entry?

Less has been done for v1.0.0 than for v0.20.0. The entry is generated, kept in this repository, and checked against NVIDIA's signed release. Its source package and its literal configuration are published as two public OCI artifacts. Its base is stored in ConfigHub as 17 Units. The status table says what has run.

The two public OCI artifacts have different jobs.

ArtifactWhat it containsDigest
Source packageThe AICR-generated app-of-apps Helm chart.sha256:b187253b95bfe5a30d28128b94ed5edeb39f4e08ad8b409d25ebe3c9982eebb3
Literal configurationThe 17 exact Argo CD Application objects, one YAML layer per object.sha256:631dba7ef35a676eafa1590f11149c4bae02867af046dba36dc1dcfd5c9edc19

Run this repository check to pull both artifacts without registry credentials and compare every pulled blob with the retained copy.

npm run aicr-v100:public-verify

The public OCI receipt records the exact references, digests, and anonymous pulls.

This entry in the model's five words

The model page describes every entry as Pack, Recipe, Bundle, Variant, and Deployment.

WordFor the v1.0.0 training entry
Source catalogNVIDIA's AICR built-in catalog, version v1.0.0.
Data packNone. The entry uses NVIDIA's catalog as it is.
RecipeFive criteria, service=eks, accelerator=h100, os=ubuntu, intent=training, and platform=kubeflow, with the build inputs listed below.
Generatoraicr recipe, then aicr bundle with the argocd-helm deployer, then helm template.
Bundle17 exact Argo CD Application objects, retained here with a plan of nine steps and published as oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:1.0.0.
BaseStored in ConfigHub as one Space with 17 Units, one for each Application.
VariantNone yet. No copy of the base has been made for an environment.
DeploymentNone. Nothing has been delivered or run.

What the source produces

AICR reads nine overlays and turns the five criteria into one recipe with 16 components, in order. It makes one Helm chart from 15 of them. It leaves out dra-node-labeler, because no DRA eviction node label was supplied. Helm renders that chart into 17 Application objects, and those are what this entry keeps.

The entry is only partly flattened. Sixteen of the 17 Application objects still name a Helm chart. Argo CD renders those charts later, on the cluster.

  • The generation receipt records the commands, the inputs, and the source version.
  • The digest index gives one digest that covers the inputs, the source package, and the rendered Applications. If any of them changes, the digest changes.
  • The source record names NVIDIA as the provider, the catalog version, and the selected recipe.
  • The source verification checks the release checksum, the signed binary provenance, the recipe-catalog signature, and the SBOM attestation.

Run these two repository checks to repeat the verification without Docker or network access:

npm run aicr-training-v100:verify
npm run aicr-provenance-v100:verify

The inputs fixed when the entry was built

InputValue
Storage classgp3
Accelerated node selectornvidia.com/gpu.present=true
Workload selectorapp.kubernetes.io/part-of=training
System node selectornodeGroup=system-worker. This is a placeholder. Change it to the label on your own cluster's system node group.
Deployerargocd-helm

A change to one of these inputs is made before the bundle, by generating the entry again. A change for one environment is made after the bundle, as a variant in ConfigHub.

The plan

Not all source configuration can be completely flattened. The configuration steps taken after flattening are called the plan. This entry's plan has nine steps, and none has been carried out.

StepWhat it asks
Match the targetCompare what the destination has with what NVIDIA's recipe expects.
Argo CD firstMake the Argo CD Application API and controller available.
Component orderReconcile the component Applications in AICR's dependency order.
The charts insideHandle CRDs, hooks, certificates, operators, and setup Jobs for every nested source. The nested sources are not materialized for v1.0.0.
Upgrade over an earlier installationFollow the manual steps AICR lists before applying this bundle over an earlier one.
System node selectorReplace the placeholder with the label on the destination cluster's system node group.
DRA plugin evictionDecide whether the DRA kubelet plugin is evicted before a GPU driver container restarts.
Node inspectionCollect and compare node facts with AICR's own tool.
Configuration validationRun the AICR checks whose prerequisites the recipe satisfies.

The route intent is the record behind this table. The field-policy assessment says which fields NVIDIA's source controls and which you may change in ConfigHub.

What changed from v0.20.0?

The computed comparison records these differences. The Catalog retains no v0.21.0 entry, so the step between is not compared.

v0.20.0v1.0.0
Components in the recipe1516
Argo CD Applications1717
Distinct sync-waves55

Eight components pull a different chart version, and none moved to another sync-wave.

Componentv0.20.0v1.0.0
gpu-operator26.3.326.7.1
kai-schedulerv0.14.1v0.16.9
nodewright-operatorv0.17.1v0.19.0
nvidia-dra-driver-gpu0.4.10.5.0
nvsentinelv1.20.0v1.25.0
aicr-stack, kubeflow-trainer-post, nodewright-customizations0.20.01.0.0

The recipe's own format also moves, from aicr.run/v1alpha2 to aicr.run/v1. NVSentinel's health check now covers three DaemonSets, where it covered two. The full comparison finds 15 changed Application objects. This is a source comparison. No health check has run on EKS in this entry.

Current status

StageResultRecord
Select the AICR recipeComplete. The five criteria resolve to one recipe that NVIDIA curates.Source record
Verify the upstream sourceComplete. Checksums, binary provenance, recipe-catalog signature, and SBOM attestation pass.Source verification
Generate the wrapper and exact ApplicationsComplete. The retained result contains 17 Applications.Generation receipt
Publish source and literal configuration OCIComplete. Both artifacts were pulled without registry credentials at their recorded digests.Public OCI receipt
Materialize the 16 nested sourcesNot done for v1.0.0.The plan's fourth step.
Store the base in ConfigHubComplete. One Space holds 17 Units, and every stored object matches the retained file.Upload receipt
Make variants for development, staging and productionNot run.None.
Check approval, promote, and publish a release OCINot run.None.
Deliver through Argo CD or FluxNot run.None.
Run on EKS and H100Not run. No training request, observation, or rollback is claimed.Tracked in issue #1581.

The next step for this entry is one the v0.20.0 entry has already taken, which is to make a variant of the base for each environment.

Use this entry in ConfigHub

Every Catalog entry page carries these five steps in this order. Each step says what the Catalog has run for this entry, and a step with a missing precondition gives no command.

Steps 1 and 2 need no account. Install the cub CLI before any cub command. Step 2 uses cub config diff, which cub plugin install confighub/cub-workshop@v0.6.61 adds. A live run on 2026-10-08 used these cub variant commands on other configuration, with no Target. It is not a run for this entry.

What is fixed, and what you supply

Fixed when this entry was built. These went into the build, so the objects already carry them. To change one, build the entry again from its source.

  • accelerator is h100
  • intent is training
  • os is ubuntu
  • platform is kubeflow
  • service is eks
  • acceleratedNodeSelector is nvidia.com/gpu.present=true
  • repoURL is oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow
  • storageClass is gp3
  • systemNodeSelector is nodeGroup=system-worker (placeholder, change it to match your cluster)
  • workloadSelector is app.kubernetes.io/part-of=training
  • deployer is argocd-helm

Yours to supply or check. The destination must have these before the objects are applied. The words in italics say what the Catalog knows about each.

  • Argo CD and its Application CRD must exist before the retained wrapper objects can be applied. Declared. Not checked on any destination.
  • The destination must match the selected EKS, H100, Ubuntu, and Kubeflow training recipe. Declared. Not checked on any destination.
  • A gp3 StorageClass must be available for the selected storage input. Declared. Not checked on any destination.
  • Nodes that run system components must carry the system node selector label. This bundle carries the placeholder nodeGroup=system-worker, and the destination's own label replaces it. Declared. Not checked on any destination.
  • GPU nodes must carry nvidia.com/gpu.present=true before the node-tuning object is applied. Declared. Not checked on any destination.
  • The GPU driver, EFA device plugin, and node tuning must suit the actual node and network role. Declared. Not checked on any destination.
  • systemNodeSelector=nodeGroup=system-worker is a placeholder, confirmed as one on 2026-10-07. Change it to the label on your own cluster's system node group. It is written into spec.source.helm.values of 12 rendered Applications at 24 field paths. A different value changes the bundle bytes and every digest in this entry, so the entry must be regenerated with the new value or a variant must be and more. The listing gives the rest. A placeholder. Change it to match your cluster.
This entry's plan, and how to test it

These are the steps that plain objects cannot carry. ConfigHub stores them and does not carry them out. The words in italics say what the Catalog knows about each.

  • Once the destination is known. compare observed destination facts with the intended NVIDIA recipe Who: selected destination actor. Written down. Not run by the Catalog.
  • Once the destination is known. make the Argo CD Application API and controller available Who: selected destination actor. Written down. Not run by the Catalog.
  • Once the destination is known. reconcile component Applications in AICR dependency order Who: selected destination actor. Written down. Not run by the Catalog.
  • Once the destination is known. handle CRDs, hooks, certificates, operators, and setup Jobs for every nested source Who: selected destination actor. Recorded as not-yet-materialized-for-v1.0.0.
  • Around an install. follow the manual steps AICR lists before applying this bundle over an earlier installation Who: selected destination actor. Written down. Not run by the Catalog.
  • Once the destination is known. replace the placeholder system node selector nodeGroup=system-worker with the label on the destination cluster's system node group Who: selected destination actor. Written down. Not run by the Catalog.
  • Once the destination is known. decide whether the DRA kubelet plugin is evicted before a GPU driver container restarts Who: selected destination actor. Written down. Not run by the Catalog.
  • Once the destination is known. collect and compare node facts Who: selected destination actor. Recorded as available-not-run-for-this-entry.

The listing records 9 steps. This list shows the first 8.

Read what a plan is, and why some entries have one.

1. Get the exact objects Run for this entry

The Catalog rendered this entry and retains its 17 Argo CD Applications as files that one inventory lists.

# make the directories the inventory names
$ mkdir -p aicr-eks-h100-training-kubeflow-v1-0-0-argocd/templates

# fetch the inventory, then every file it lists
$ curl -fsSL -o aicr-eks-h100-training-kubeflow-v1-0-0-argocd/checksums.txt https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v1-0-0/argocd-rendered/checksums.txt

$ cut -c67- aicr-eks-h100-training-kubeflow-v1-0-0-argocd/checksums.txt | xargs -I{} curl -fsSL -o aicr-eks-h100-training-kubeflow-v1-0-0-argocd/{} https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v1-0-0/argocd-rendered/{}

# check every file against the inventory
$ (cd aicr-eks-h100-training-kubeflow-v1-0-0-argocd && shasum -a 256 -c checksums.txt)

# check the inventory against the listing digest
$ shasum -a 256 aicr-eks-h100-training-kubeflow-v1-0-0-argocd/checksums.txt
45baec5d12b562cec93aa771d1f631f19e8b6c68ab79fd7a8f6314dedff9c47e  aicr-eks-h100-training-kubeflow-v1-0-0-argocd/checksums.txt

# join the files into one for step 2
$ cat aicr-eks-h100-training-kubeflow-v1-0-0-argocd/templates/*.yaml > aicr-eks-h100-training-kubeflow-v1-0-0-argocd.yaml

2. Compare with another version or base Not run for this entry

The listing records no comparison for this entry. The commands run on the retained files, with no account. The Catalog holds 4 other versions, v0.14.0, v0.18.0, v0.19.0 and v0.20.0.

# fetch the argocd base at version v0.20.0
$ mkdir -p aicr-eks-h100-training-kubeflow-v0-20-0-argocd/templates

$ curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-20-0-argocd/checksums.txt https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-20-0/argocd-rendered/checksums.txt

$ cut -c67- aicr-eks-h100-training-kubeflow-v0-20-0-argocd/checksums.txt | xargs -I{} curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-20-0-argocd/{} https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-20-0/argocd-rendered/{}

# join them into one file
$ cat aicr-eks-h100-training-kubeflow-v0-20-0-argocd/templates/*.yaml > aicr-eks-h100-training-kubeflow-v0-20-0-argocd.yaml

# list what changed from v0.20.0 to v1.0.0
$ cub config diff aicr-eks-h100-training-kubeflow-v0-20-0-argocd.yaml aicr-eks-h100-training-kubeflow-v1-0-0-argocd.yaml --summary

3. Upload it as a variant Run for this entry

The Catalog saved this entry in ConfigHub. The commands use its published literal configuration bundle. Read the receipt.

# sign in; steps 3 to 5 write to your ConfigHub organization
$ cub auth login

# preview the upload of the published bundle; nothing is written
$ cub variant upload --dry-run --component eks-h100-training-kubeflow --variant argocd --space aicr-eks-h100-training-kubeflow-v1-0-0-argocd oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:1.0.0

# upload the bundle as a base Space of Units
$ cub variant upload --component eks-h100-training-kubeflow --variant argocd --space aicr-eks-h100-training-kubeflow-v1-0-0-argocd oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:1.0.0

4. Deploy it Partly run for this entry

A delivery is partly recorded for this entry, and no run has passed. Argo CD is partly recorded, Flux does not apply, and kubectl has no record. The commands name the Target workshop/target, which cub cluster up --name workshop creates on a local kind cluster with Argo CD. Use your own Target for a real destination. The listing records 7 inputs that the destination must supply, so read them before you deploy. Read the record that states this.

# clone the base onto a Target as a dev variant
$ cub variant create dev aicr-eks-h100-training-kubeflow-v1-0-0-argocd --target workshop/target --space-pattern "template:aicr-eks-h100-training-kubeflow-v1-0-0-argocd-dev"

# publish the Release that Argo CD or Flux pulls
$ cub release publish aicr-eks-h100-training-kubeflow-v1-0-0-argocd-dev

5. Promote a change Not run for this entry

No promotion is recorded for this entry. Promotion carries a change in the base to the dev variant. The base changes when you edit a Unit or upload a newer version into the same Space.

# preview what the dev variant would take from the base
$ cub variant promote aicr-eks-h100-training-kubeflow-v1-0-0-argocd-dev --dry-run -o mutations

# promote once you have read the preview
$ cub variant promote aicr-eks-h100-training-kubeflow-v1-0-0-argocd-dev --change-desc "Pull the reviewed base forward"

Technical notes and disclaimers

  • This entry is a set of Argo CD Applications, so a test needs Argo CD. The Catalog has no test script for it.

Generated from the committed markdown file docs/demo/aicr/eks-h100-training-kubeflow-v1-0-0.md. The source file is the authoritative version.