AICR EKS H100 training example

Generated at: 2026-07-30T12:38:02.000Z UTC · source: committed helm-expt evidence for this rendered repository document.

This example asks NVIDIA AICR v0.14.0 for an EKS platform with H100 accelerators, Ubuntu, Kubeflow, and a training workload. AICR selects 15 versioned components and puts them in dependency order. The result includes storage, networking, certificates, GPU support, monitoring, scheduling, and Kubeflow training.

ConfigHub does not replace AICR. AICR decides what belongs in the platform package. ConfigHub records that decision as configuration a team can inspect, compare, approve, and assign to clusters. Development, staging, production, or cluster-class changes can then be kept as named variants instead of rebuilding the package by hand.

What problem this solves

The recipe pins most of the platform before installation. Four choices shape this example: the StorageClass, the selector for accelerated nodes, the selector for training workloads, and the source the delivery controller reads. The first three are fixed when the bundle is generated. The fourth must point at the published bundle.

Keeping the recipe, those four choices, and the generated files together answers two practical questions later: what did we ask AICR to build, and what did each cluster actually receive?

What is committed

aicr.yaml is the AICRConfig that describes the recipe and bundle commands.
recipe.yaml is the generated component recipe.
flux-bundle/README.md describes AICR's original Git-oriented Flux resources.
flux-oci-bundle/kustomization.yaml is the entry point for the Flux resources AICR generated for OCI delivery; its checksums.txt records every generated file.
local-oci-manifest.json is the exact manifest from the temporary local OCI registry.
generation-receipt.yaml records the release commit, binary checksums, criteria, options, commands, local OCI digest, Flux requirements, and current proof boundary.
argocd-helm-bundle/Chart.yaml is AICR's portable Argo CD source chart.
argocd-rendered/checksums.txt records the 17 concrete Argo CD Application objects rendered from that chart.
argocd-oci-receipt.yaml joins the source chart, rendered objects, OCI layouts, and public references.
public-oci-receipt.yaml records the public digests and anonymous pull checks for both artifacts.
confighub-upload-receipt.yaml records the ConfigHub Space, Unit, source digest, policy, and exact-object comparison from the live upload.
apply-policy-receipt.yaml records what happened when the unapproved AICR Unit was submitted for a dry-run apply.
promotion-readiness-receipt.yaml records the persistent base, development, and staging Spaces and their exact configuration change.

The original Git-oriented bundle still contains https://github.com/YOUR_ORG/YOUR_REPO.git. It is kept so the limitation is visible; it is not deployable.

The OCI-oriented bundle has no repository placeholder. AICR replaces its two local charts with ArtifactGenerator resources, and their HelmRelease objects refer to ExternalArtifact outputs. The generated checksums.txt originally contained the absolute output directory. The committed copy changes only those path prefixes to paths relative to the bundle root, so the checksum list is portable and does not expose a workstation path.

Rebuild the example

Install AICR v0.14.0, then run these commands from the example directory:

aicr recipe \
  --service eks \
  --accelerator h100 \
  --os ubuntu \
  --intent training \
  --platform kubeflow \
  --output recipe.yaml

aicr bundle \
  --recipe recipe.yaml \
  --deployer flux \
  --output flux-bundle \
  --storage-class gp3 \
  --accelerated-node-selector nvidia.com/gpu.present=true \
  --workload-selector app.kubernetes.io/part-of=training

That command reproduces the Git-oriented bundle. For a usable Git bundle, add --repo https://github.com/ORG/REPOSITORY.git, regenerate it, and review the resulting GitRepository and HelmRelease objects before upload.

The OCI-oriented run used the same recipe and choices:

aicr bundle \
  --recipe recipe.yaml \
  --deployer flux \
  --output oci://REGISTRY/aicr-eks-h100-training-kubeflow:v0.14.0 \
  --image-refs oci-image-ref.txt \
  --storage-class gp3 \
  --accelerated-node-selector nvidia.com/gpu.present=true \
  --workload-selector app.kubernetes.io/part-of=training

For a local HTTP registry, add --plain-http. The recorded local manifest digest is sha256:b8089d6400833531c5d0d91177e70cf48f2b86896b0214c2b3318916ac4d29ba. This proves local packaging and pullability. It does not prove a public registry push or a working GPU cluster.

Run npm run aicr-example:verify to check both bundles. The verifier checks the criteria, command options, release and binary checksums, every generated file, the local OCI manifest, the two ArtifactGenerator resources, and the matching ExternalArtifact references.

What Flux needs for the OCI bundle

AICR's OCI form requires Flux v2.7 or newer, the source-watcher controller, and ExternalArtifact=true on helm-controller. The target also needs an OCIRepository named aicr-bundle in flux-system that points at the published AICR artifact. Those are part of the operating configuration, so the receipt records them instead of leaving them as setup knowledge.

The Argo CD path

AICR can also generate an argocd-helm bundle. This output is a Helm chart that creates one parent Argo CD Application and 16 component Application objects. The components use sync waves 0 through 15, so storage, certificates, GPU support, monitoring, scheduling, and training are presented to Argo CD in the intended order.

Two OCI artifacts have different jobs:

Artifact	What it contains	Who uses it
AICR Argo source package	The generated Helm chart, values, local chart files, and Argo CD templates	Helm and Argo CD
Rendered Argo configuration	The 17 exact `Application` objects, one YAML layer per object	ConfigHub

Both artifacts are public in Google Artifact Registry:

oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd:0.14.0
oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:0.14.0

The source package digest is sha256:1120c9a17b23f2c885121034d55445f2d3948c95c23756d02fa2209e4baf8c2e. The 17-object configuration digest is sha256:dcf7feeeeaece04cb5d55cbc1106862172b3ae77718154252b39db1ad8957010. Both passed anonymous pull checks. The small OCI layouts are also committed, so the verifier can compare the public artifacts with the exact source and rendered files.

Run npm run aicr-argocd-example:verify to check the source package, the 17 Applications, both OCI manifests and layouts, every file checksum, the source references, and the complete set of sync waves. Run npm run aicr-argocd-example:public-verify to pull both public artifacts without registry credentials and check their recorded digests.

The ConfigHub base variant

The live demo imports the public literal configuration OCI:

New to cub? Install the cub CLI first. You can pull and render public catalog packages without an account. Commands that save or change ConfigHub data require you to sign in.

cub variant upload \
  --component aicr-eks-h100-training-kubeflow \
  --variant v0-14-0-argocd \
  --space aicr-eks-h100-training-kubeflow-v0-14-0-argocd \
  --granularity minimal \
  oci://europe-west1-docker.pkg.dev/nth-fort-499605-q5/helm-expt/aicr-eks-h100-training-kubeflow-argocd-config:0.14.0

ConfigHub pulled digest sha256:dcf7feeeeaece04cb5d55cbc1106862172b3ae77718154252b39db1ad8957010, created one base Space, and stored all 17 Application objects in one Unit. A verifier compared every live object with the committed AICR output and confirmed sync waves 0 through 15.

The upload also names two requirements that must already exist: the argocd Namespace and the default Argo CD AppProject. The live Space records both in its README.

cub variant upload does not run AICR or render charts. It imports exact YAML and records the source reference and resolved digest on the Space. ConfigHub can then show changes to those files and manage derived variants for environments or cluster classes.

OCI in, ConfigHub, OCI out

A separate live test checked the next boundary. It imported the same 17-Application literal configuration OCI into a temporary ConfigHub base, published a ConfigHub release OCI, pulled that release back, and compared every Application with the AICR input.

All 17 Applications matched. ConfigHub added its confighub.com/origin annotation to each object so the released configuration can be traced back to its ConfigHub record. The input and output OCI digests are different because they are different artifacts; the object comparison proves that the Kubernetes configuration stayed the same.

Read the plain-English result or the live receipt. The test used a temporary local input registry and a throwaway cluster only to obtain a legitimate ConfigHub release target and scoped OCI credential. It did not apply the Applications, reconcile the AICR stack, create an EKS cluster, or run a GPU workload.

The public pull does not need a Google login. Argo CD delivery and live GPU-cluster reconciliation have not run. The example proves generation, public OCI packaging, exact ConfigHub upload, policy attachment, and promotion, but it does not claim a working GPU platform.

The uploaded base requires approval because it changes cluster-wide GPU, monitoring, and training-platform configuration. That rule applies before production too. A live dry-run of the exact 17-Application Unit was rejected by the required-approval gate. The Unit had no target attached, its revision and data hash stayed unchanged, and nothing was sent to Kubernetes. Run npm run aicr-argocd-example:hub-policy-check while authenticated to repeat that check.

Change it in development, then promote it

The generated AICR configuration includes the example Grafana setting adminPassword: admin inside the kube-prometheus-stack Application. ConfigHub can keep the AICR output as the base, while development and staging use an existing Secret named aicr-grafana-admin.

The persistent helm-catalog demo follows four steps:

Import the exact 17-Application OCI bundle as a base variant.
Create development from the base and staging from development.
Preview and apply the Grafana Secret change in development. Only Application argocd/kube-prometheus-stack changed.
Preview the staging promotion, confirm that the preview changed nothing, then run the promotion and confirm that staging matched the reviewed development config.

The persistent promotion receipt records the public OCI digests, three live Spaces, exact changed Application, dry-run results, promotion revision, policy checks, and required approval. Each Space has one short README that explains what to inspect.

The configuration Unit is promoted by itself so development and staging can keep different README text. This is the same upstream upgrade operation used by a bulk variant promotion, scoped to the one Unit that contains the 17 Applications.

Run the live check while authenticated:

CUB_CONTEXT=<context> npm run aicr-argocd-example:promotion-verify

A separate scratch proof records an earlier temporary run of the same configuration change. Neither proof started a Kubernetes cluster, so neither claims Argo CD delivery, application health, or GPU workload health.