AICR v0.19.0: retain the source, objects, and install plan
An AI platform configuration has more than one useful form. The AICR recipe records what the user selected. The generated Helm chart is the source package Argo CD can process. The rendered Argo CD Applications are the exact configuration ConfigHub can retain and compare. The lifecycle record says what must happen around normal Kubernetes apply. It is one of the NVIDIA AICR pages. You can also view this as a markdown file.
Needs first. Make the Argo CD Application API and controller available. Reconcile component Applications in AICR dependency order. The plan below lists 2 more.
UNOFFICIAL/EXPERIMENTAL. This entry keeps NVIDIA AICR v0.19.0 beside the v0.14.0 and v0.18.0 entries. It uses the same EKS, H100, Ubuntu, training, and Kubeflow choices so the versions can be compared without replacing either earlier result.
npm run aicr-provenance-v0190:verify
npm run aicr-training-v0190:verify
npm run aicr-training-v0190:verify-artifacts
Why this entry exists
Keeping those records together answers different questions without pretending that one YAML directory answers all of them.
What was selected
AICR v0.19.0 resolved one native AICR recipe from these choices:
Choice
Value
Service
EKS
Accelerator
H100
Operating system
Ubuntu
Workload
Training
Platform
Kubeflow
The recipe contains 15 components and their dependency order. This use of recipe is specific to AICR. It is not our general name for an OCI package, plain YAML, or a ConfigHub revision.
How the configuration is processed
Step
What happens for this entry
Select
AICR resolves the five choices into its native recipe.
Materialize
AICR generates an Argo CD app-of-apps Helm chart. Helm renders that chart into 17 Applications.
Flatten
The 17 Applications are retained as exact files and as a literal configuration OCI.
Route lifecycle work
The route record assigns the Argo CD prerequisite, component ordering, downstream chart work, and AICR health checks.
Protect fields
The field-policy record says which choices belong to the source and which may become later ConfigHub changes.
Retain
Checksums, two local OCI layouts, one platform digest, and the source receipts bind the retained files.
This is a partial flattening boundary. The 17 Applications are exact, but 16 of them still point to Helm charts or local chart paths. Argo CD renders those sources later. This entry therefore does not contain every Kubernetes object that will eventually run on the target cluster.
The nested-source report records a local render for all 16 component sources. Eight rendered outputs contain CRDs. This closes the source-discovery gap, but it does not make the nested charts safe to flatten or prove that they ran on EKS.
What changed since v0.18.0
The computed comparison shows the same component set, dependency order, Application count, and sync-wave shape. The root package version moved to 0.19.0. The local kubeflow-trainer-post and nodewright-customizations sources also moved to 0.19.0. The other chart source versions stayed the same.
The health-check text changed even though the selected platform shape did not. That is why the catalog retains the source recipe and receipts as well as the rendered Applications: two releases can produce the same top-level shape while changing the checks and source material around it.
Which fields AICR controls
AICR v0.19.0 adds named gpuStack profiles to its AKS and GKE families. Those profiles lock related GPU-driver settings so a later override cannot create a mixed ownership model. The retained EKS recipe does not declare that profile; supplying it is refused before generation.
For this EKS entry, GPU-driver ownership is therefore still a target decision. We record that result instead of copying the AKS rule onto EKS. A later ConfigHub change should preserve a source-owned field only when the exact source version and selected profile say it is locally changeable.
Source and artifact checks
The release archive matched NVIDIA's checksum list. The v0.19.0 CLI binary and recipe-catalog signatures both verified against NVIDIA's exact release workflow with networking disabled. A wrong signer identity was refused.
The retained source chart OCI contains the AICR-generated Helm chart. The literal configuration OCI contains the 17 exact Application files as separate YAML layers. A fresh Helm render of the retained source chart produces the same Kubernetes data as those files.
Complete. The base, development, and staging Spaces exist. Development changes one Grafana field to use an existing Secret, and staging receives that exact reviewed change.
Complete. ConfigHub first refused an unapproved release. After both current staging Units were approved, the current retained release published successfully. The receipt records its release number and digests.
Complete. The registry resolved the exact manifest digest, and the pulled release matched all 17 promoted Applications after removing only ConfigHub's provenance annotation.
Recorded, but blocked. The resolution binds the staging object digest to EKS, H100, and Argo CD and names the added Grafana Secret, CRDs, ordering, and checks. It still needs destination and runtime receipts.
Not run. There is no v0.19.0 Argo-on-EKS, H100 training, NIM request, observation, or rollback receipt.
This remains the next hardware-backed test.
Deliver through Flux or a small fleet
Not run for v0.19.0.
These remain separate follow-up tests.
What remains
The next complete run needs an EKS destination with H100 nodes, NGC access, the required Secret and storage inputs, and Argo CD. It should deliver the recorded ConfigHub release digest, run the nested chart routes in order, make one real training or NIM request, record observations, and roll back to the previous exact release. Flux delivery and a small Sveltos fleet come after that single-cluster result.
Use this entry in ConfigHub
Every Catalog entry page carries these five steps in this order. Each step says what the Catalog has run for this entry, and a step with a missing precondition gives no command.
Steps 1 and 2 need no account. Install the cub CLI before any cub command. Step 2 uses cub config diff, which cub plugin install confighub/cub-workshop@v0.6.61 adds. A live run on 2026-10-08 used these cub variant commands on other configuration, with no Target. It is not a run for this entry.
What is fixed, and what you supply
Fixed when this entry was built. These went into the build, so the objects already carry them. To change one, build the entry again from its source.
accelerator is h100
intent is training
os is ubuntu
platform is kubeflow
service is eks
acceleratedNodeSelector is nvidia.com/gpu.present=true
repoURL is oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt
storageClass is gp3
workloadSelector is app.kubernetes.io/part-of=training
deployer is argocd-helm
Yours to supply or check. The destination must have these before the objects are applied. The words in italics say what the Catalog knows about each.
Argo CD and its Application CRD must exist before these objects can be applied. Declared. Not checked on any destination.
The target must be an EKS cluster with H100 nodes running Ubuntu. Declared. Not checked on any destination.
A gp3 StorageClass must be available for the selected storage input. Declared. Not checked on any destination.
GPU nodes must carry nvidia.com/gpu.present=true for the selected node placement. Declared. Not checked on any destination.
This entry's plan, and how to test it
These are the steps that plain objects cannot carry. ConfigHub stores them and does not carry them out. The words in italics say what the Catalog knows about each.
Once the destination is known. make the Argo CD Application API and controller available Who: selected destination actor. Recorded as recorded-not-run-for-v0.19.0.
Once the destination is known. reconcile component Applications in AICR dependency order Who: selected destination actor. Recorded as recorded-not-run-for-v0.19.0.
Once the destination is known. handle hooks, CRDs, certificates, and setup Jobs in each referenced chart Who: selected destination actor. Recorded as nested-sources-rendered-routes-not-resolved.
Once the destination is known. run the deployment and conformance checks carried by the AICR recipe Who: selected destination actor. Not run.
The Catalog rendered this entry and retains its 17 Argo CD Applications as files that one inventory lists.
# make the directories the inventory names$ mkdir -p aicr-eks-h100-training-kubeflow-v0-19-0-argocd/templates
# fetch the inventory, then every file it lists$ curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-19-0-argocd/checksums.txt https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-19-0/argocd-rendered/checksums.txt
$ cut -c67- aicr-eks-h100-training-kubeflow-v0-19-0-argocd/checksums.txt | xargs -I{} curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-19-0-argocd/{} https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-19-0/argocd-rendered/{}
# check every file against the inventory$ (cd aicr-eks-h100-training-kubeflow-v0-19-0-argocd && shasum -a 256 -c checksums.txt)
# check the inventory against the listing digest$ shasum -a 256 aicr-eks-h100-training-kubeflow-v0-19-0-argocd/checksums.txt
0825f8ba4888da84810184289cb1b0e0313b11f8f7414a4e91bae37fff5b7b94 aicr-eks-h100-training-kubeflow-v0-19-0-argocd/checksums.txt
# join the files into one for step 2$ cat aicr-eks-h100-training-kubeflow-v0-19-0-argocd/templates/*.yaml > aicr-eks-h100-training-kubeflow-v0-19-0-argocd.yaml
2. Compare with another version or base Not run for this entry
The listing records no comparison for this entry. The commands run on the retained files, with no account. The Catalog holds 4 other versions, v0.14.0, v0.18.0, v0.20.0 and v1.0.0.
# fetch the argocd base at version v0.18.0$ mkdir -p aicr-eks-h100-training-kubeflow-v0-18-0-argocd/templates
$ curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-18-0-argocd/checksums.txt https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-18-0/argocd-rendered/checksums.txt
$ cut -c67- aicr-eks-h100-training-kubeflow-v0-18-0-argocd/checksums.txt | xargs -I{} curl -fsSL -o aicr-eks-h100-training-kubeflow-v0-18-0-argocd/{} https://raw.githubusercontent.com/confighub/helm-expt/main/examples/aicr/eks-h100-training-kubeflow-v0-18-0/argocd-rendered/{}
# join them into one file$ cat aicr-eks-h100-training-kubeflow-v0-18-0-argocd/templates/*.yaml > aicr-eks-h100-training-kubeflow-v0-18-0-argocd.yaml
# list what changed from v0.18.0 to v0.19.0$ cub config diff aicr-eks-h100-training-kubeflow-v0-18-0-argocd.yaml aicr-eks-h100-training-kubeflow-v0-19-0-argocd.yaml --summary
3. Upload it as a variant Run for this entry
The Catalog saved this entry in ConfigHub. The commands use its published literal configuration bundle. Read the receipt.
# sign in; steps 3 to 5 write to your ConfigHub organization$ cub auth login
# preview the upload of the published bundle; nothing is written$ cub variant upload --dry-run --component eks-h100-training-kubeflow --variant argocd --space aicr-eks-h100-training-kubeflow-v0-19-0-argocd oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:0.19.0
# upload the bundle as a base Space of Units$ cub variant upload --component eks-h100-training-kubeflow --variant argocd --space aicr-eks-h100-training-kubeflow-v0-19-0-argocd oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:0.19.0
4. Deploy it Partly run for this entry
A delivery is partly recorded for this entry, and no run has passed. Argo CD is partly recorded, Flux does not apply, and kubectl has no record. The commands name the Target workshop/target, which cub cluster up --name workshop creates on a local kind cluster with Argo CD. Use your own Target for a real destination. The listing records 4 inputs that the destination must supply, so read them before you deploy. Read the receipt.
# clone the base onto a Target as a dev variant$ cub variant create dev aicr-eks-h100-training-kubeflow-v0-19-0-argocd --target workshop/target --space-pattern "template:aicr-eks-h100-training-kubeflow-v0-19-0-argocd-dev"
# publish the Release that Argo CD or Flux pulls$ cub release publish aicr-eks-h100-training-kubeflow-v0-19-0-argocd-dev
5. Promote a change Run for this entry
A promotion ran for this entry. It followed base to development to staging. Promotion carries a change in the base to the dev variant. The base changes when you edit a Unit or upload a newer version into the same Space. Read the receipt.
# preview what the dev variant would take from the base$ cub variant promote aicr-eks-h100-training-kubeflow-v0-19-0-argocd-dev --dry-run -o mutations
# promote once you have read the preview$ cub variant promote aicr-eks-h100-training-kubeflow-v0-19-0-argocd-dev --change-desc "Pull the reviewed base forward"
Technical notes and disclaimers
This entry is a set of Argo CD Applications, so a test needs Argo CD. The Catalog has no test script for it.