--- # Source: cluster-readiness-engine/crds/nvcre.nvidia.com_bandwidthmeasurements.yaml --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.20.0 name: bandwidthmeasurements.nvcre.nvidia.com spec: group: nvcre.nvidia.com names: kind: BandwidthMeasurement listKind: BandwidthMeasurementList plural: bandwidthmeasurements singular: bandwidthmeasurement scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: BandwidthMeasurement is the Schema for the bandwidthmeasurements API properties: apiVersion: description: |- APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: description: |- Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object spec: description: spec defines the desired state of BandwidthMeasurement properties: jobRef: description: jobRef is the reference to the NVCRE Job for which bandwidth should be measured. properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string required: - kind - name type: object x-kubernetes-map-type: atomic logProfileRef: description: |- logProfileRef is the name of the cluster-scoped LogProfile to use for log parsing. The LogProfile defines a bandwidthResult regex pattern for extracting NCCL bandwidth data. minLength: 1 type: string sampleInterval: description: |- sampleInterval is how often to sample pod logs while the Job is running. Default: 60s. type: string testType: description: |- testType identifies the NCCL collective operation being measured (e.g., "all_reduce", "alltoall"). Used as the "nccl_test" Prometheus label to distinguish results on dashboards. type: string required: - logProfileRef type: object status: description: status defines the observed state of BandwidthMeasurement properties: completionTime: description: completionTime is the time when the measurement completed (when the referenced Job reached a terminal state). format: date-time type: string conditions: description: |- conditions represent the current state of the BandwidthMeasurement resource. Condition types: - "Measuring": the measurement is in progress (referenced Job is running) - "Complete": the measurement has finished (referenced Job succeeded or failed) The status of each condition is one of True, False, or Unknown. items: description: Condition contains details for one aspect of the current state of this API Resource. properties: lastTransitionTime: description: |- lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. format: date-time type: string message: description: |- message is a human readable message indicating details about the transition. This may be an empty string. maxLength: 32768 type: string observedGeneration: description: |- observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. format: int64 minimum: 0 type: integer reason: description: |- reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty. maxLength: 1024 minLength: 1 pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ type: string status: description: status of the condition, one of True, False, Unknown. enum: - "True" - "False" - Unknown type: string type: description: type of condition in CamelCase or in foo.example.com/CamelCase. maxLength: 316 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ type: string required: - lastTransitionTime - message - reason - status - type type: object type: array x-kubernetes-list-map-keys: - type x-kubernetes-list-type: map results: description: |- results contains per-message-size average bandwidth measurements. Each entry represents the running average of all observed measurements for that size. items: description: BandwidthResult stores the average bandwidth for a single message size. properties: algBW: description: algBW is the average algorithmic bandwidth in GB/s. type: string busBW: description: busBW is the average bus bandwidth in GB/s. type: string samples: description: samples is the number of measurements averaged for this size. type: integer sizeBytes: description: sizeBytes is the message size in bytes. format: int64 type: integer required: - algBW - busBW - samples - sizeBytes type: object type: array startTime: description: startTime is the time when the measurement started (when the referenced Job began running). format: date-time type: string type: object required: - spec type: object served: true storage: true subresources: status: {} --- # Source: cluster-readiness-engine/crds/nvcre.nvidia.com_certifications.yaml --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.20.0 name: certifications.nvcre.nvidia.com spec: group: nvcre.nvidia.com names: kind: Certification listKind: CertificationList plural: certifications singular: certification scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: Certification is the Schema for the certifications API properties: apiVersion: description: |- APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: description: |- Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object spec: description: |- spec defines the desired state of Certification. The entire spec is immutable after creation: the controller never updates an active Workflow from an edited parent, so accepting edits would either silently ignore them or apply them only to later categories. To run with different inputs, delete the Certification and create a new one. properties: categories: description: |- Categories are the list of certificate categories required for the Target. The MaxItems bound keeps the CEL rules nested under each category (the resources request/limit comparison) within the API server's validation cost budget; the catalog defines far fewer entries than 64. items: properties: domain: description: Domain is the high level Domain that the certificate belongs to like training, inference etc type: string options: description: |- options provides per-category configuration overrides. Fields set here take precedence over their CertificationSpec counterparts. properties: enableCheckpoint: description: |- enableCheckpoint enables checkpointing for training workloads. When true, provisions a PVC for checkpoint storage, adds checkpoint config (pvcName/maxRestarts), and enables --save/--load in training scripts. When false (default), uses emptyDir and disables save/load. Non-training entries ignore this field. type: boolean enableMNNVL: description: |- enableMNNVL enables Multi-Node NVLink (NCCL_MNNVL_ENABLE=1) for training and communication workloads. Defaults to false (NCCL_MNNVL_ENABLE=0). Enable this when running on platforms with multi-node NVLink connectivity (e.g., GB300 NVL72). Can be overridden per-category via categories[].options.enableMNNVL. type: boolean exitDurationMins: description: |- exitDurationMins sets the training duration in minutes. Maps to EXIT_DURATION_MINS env var. Default: 30. format: int32 minimum: 1 type: integer gpusPerNode: description: |- gpusPerNode optionally overrides the number of GPUs per node used by catalog workloads. If not specified, the controller derives the default from the GPU architecture in target.nodeSelector (e.g., 4 for GB200/GB300, 8 for H100). Use this field when your hardware has a non-standard GPU count per node. format: int32 minimum: 1 type: integer image: description: |- image overrides the workload container image for catalog workloads. It replaces the trainer image in the rendered job template, the image of the primary workload container of every replicated job in the resolved TrainingRuntime dependencies, and the image of every init container in those pods whose image exactly equals the primary's pre-override image (the workload-derived inits such as fix-ssh-permissions and megatron-clone); init containers with a distinct image (such as GCP's tcpxo-daemon) and any additional containers keep their catalog images. Applied after catalog and platform overrides resolve, so it also replaces an image a platform override selects. Beware: on AWS EFA platforms (H100, GB200) the platform overrides land the workers on an nccl-tests image that ships the aws-ofi-nccl (EFA) plugin; setting image replaces that image, and the operator then owns the EFA OFI plugin being present in the replacement. maxLength: 512 minLength: 1 type: string imagePullSecrets: description: |- imagePullSecrets is an optional list of references to secrets for pulling container images used by catalog workloads. If not specified, the cluster's default image pull configuration (e.g., ServiceAccount) is used. items: description: |- LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic type: array maxBytes: description: |- maxBytes sets the maximum message size for NCCL tests (e.g., "16G", "32G"). Maps to the NCCL perf test `-e` flag. Default: "16G" (GB200/GB300 override: "32G"). pattern: ^[0-9]+(K|M|G|T)?$ type: string maxConcurrent: description: |- maxConcurrent limits the number of simultaneously running jobs within a Workflow. Useful for diagnose-mode parallel screening to avoid fabric saturation. 0 means unlimited. Default: 0. format: int32 type: integer maxRestarts: description: |- maxRestarts sets the maximum number of checkpoint-based restarts for training workloads. Maps to checkpoint.maxRestarts in the Job spec. Only used when enableCheckpoint is true. Default: catalog-defined. format: int32 minimum: 0 type: integer maxSteps: description: |- maxSteps sets the maximum training steps. Maps to trainer.max_steps in the NeMo config. Default: 50. format: int32 minimum: -1 type: integer measurementTimeout: description: |- measurementTimeout is the maximum time to wait after a Job succeeds for measurement data (bandwidth, goodput, etc.) before failing threshold validation. Accepts Go duration strings (e.g., "5m", "10m", "30m"). When not set, defaults to "5m". pattern: ^([0-9]+(h|m|s|ms))+$ type: string minGroupSize: description: |- minGroupSize sets the smallest group size at which diagnose's internal bisection stops splitting. Groups at this size that fail become suspects. Default: 2. format: int32 minimum: 1 type: integer mlnxPerNode: description: |- mlnxPerNode overrides the auto-detected Mellanox NIC count per node. Used by platforms with InfiniBand or RoCE networking (Azure, OCI, TogetherAI). If not specified, derived from GPU architecture and platform via the catalog's gpu-defaults.yaml (e.g., 8 for most architectures, 2 for OCI L40s). format: int32 minimum: 0 type: integer nicResourceName: description: |- nicResourceName sets the Kubernetes extended resource name of the RDMA NIC devices requested on workload containers for on-prem GB200/GB300 targets (e.g., "rdma/ib", "nvidia.com/mlnxnics"); the name depends on the RDMA device plugin the site runs, and the per-container count comes from mlnxPerNode. When unset, the controller auto-detects the name on on-prem GB200/GB300 targets: a single candidate resource (rdma/* or nvidia.com/mlnxnics) allocatable at the resolved mlnxPerNode count on every target node is requested; zero or several qualifying candidates inject nothing and emit a NICResourceDetection event explaining what was found. Offline nvcrectl render (without --dry-run) has no cluster to inspect and uses only this field. The value must be a fully qualified extended resource name: a DNS-subdomain domain, a slash, and a name segment of at most 63 characters; the reserved kubernetes.io and k8s.io domains (including their subdomains) are rejected, matching what Kubernetes accepts as an extended resource. maxLength: 253 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/[a-zA-Z0-9]([-A-Za-z0-9_.]{0,61}[a-zA-Z0-9])?$ type: string x-kubernetes-validations: - message: nicResourceName must not use the reserved kubernetes.io or k8s.io domains rule: '!self.contains(''kubernetes.io/'') && !self.startsWith(''k8s.io/'') && !self.contains(''.k8s.io/'')' nodesPerJob: description: |- nodesPerJob is the number of nodes per job for multi-node workloads. When nil at both global and per-category level, the controller auto-selects: - Entries with per-node-count configs (training): largest config <= matching nodes. - All other entries: all matching nodes. When set, clamped to min(nodesPerJob, matchingNodes). format: int32 minimum: 1 type: integer numCycles: description: |- numCycles sets the number of run cycles for NCCL tests. Each cycle runs numIterations iterations and prints results separately. Maps to the NCCL perf test `-N` flag. Default: 10. format: int32 minimum: 1 type: integer numIterations: description: |- numIterations sets the number of timed iterations per message size for NCCL tests. Maps to the NCCL perf test `-n` flag. Default: 100. format: int32 minimum: 1 type: integer repeatCount: description: |- repeatCount sets the number of orchestration iterations for the Workflow. Each iteration runs all groups and collects results. Multiple iterations allow repeated testing for intermittent failures. Default: 1. format: int32 minimum: 1 type: integer resources: description: |- resources overrides the CPU and memory resources of training workload containers. Training entries default to DGX-class sizing (limits: cpu "128" / memory "800Gi"; requests: cpu "64" / memory "500Gi") — set this on smaller nodes so training pods can schedule. Each of the four values independently falls back to its default when unset. GPU count is controlled by gpusPerNode. Non-training entries ignore this field. Platform-specific catalog overrides (for example AWS with H100 GPUs) may supersede these values. properties: limits: description: limits overrides the container resource limits. properties: cpu: anyOf: - type: integer - type: string description: cpu is the CPU quantity (e.g., "6", "1500m"). maxLength: 32 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - message: cpu must be a non-negative quantity rule: quantity(string(self)).compareTo(quantity("0")) >= 0 memory: anyOf: - type: integer - type: string description: memory is the memory quantity (e.g., "48Gi"). maxLength: 32 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - message: memory must be a non-negative quantity rule: quantity(string(self)).compareTo(quantity("0")) >= 0 type: object requests: description: requests overrides the container resource requests. properties: cpu: anyOf: - type: integer - type: string description: cpu is the CPU quantity (e.g., "6", "1500m"). maxLength: 32 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - message: cpu must be a non-negative quantity rule: quantity(string(self)).compareTo(quantity("0")) >= 0 memory: anyOf: - type: integer - type: string description: memory is the memory quantity (e.g., "48Gi"). maxLength: 32 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - message: memory must be a non-negative quantity rule: quantity(string(self)).compareTo(quantity("0")) >= 0 type: object type: object x-kubernetes-validations: - message: requests.cpu must not exceed limits.cpu rule: '!has(self.requests) || !has(self.limits) || !has(self.requests.cpu) || !has(self.limits.cpu) || quantity(string(self.requests.cpu)).compareTo(quantity(string(self.limits.cpu))) <= 0' - message: requests.memory must not exceed limits.memory rule: '!has(self.requests) || !has(self.limits) || !has(self.requests.memory) || !has(self.limits.memory) || quantity(string(self.requests.memory)).compareTo(quantity(string(self.limits.memory))) <= 0' saveInterval: description: |- saveInterval sets the checkpoint save frequency in training steps. NeMo 6: maps to --save-interval; NeMo 4: maps to every_n_train_steps. Only used when enableCheckpoint is true. Default: 250. format: int32 minimum: 1 type: integer saveRetainInterval: description: |- saveRetainInterval retains checkpoints at multiples of this value, deleting intermediate checkpoints (NeMo 6 only, --save-retain-interval). The most recent checkpoint is always kept regardless. Only used when enableCheckpoint is true. Default: 1000. format: int32 minimum: 1 type: integer saveTopK: description: |- saveTopK keeps only the top K checkpoints by monitored metric (NeMo 4 only, save_top_k). Older checkpoints that aren't the best are deleted to save storage. Only used when enableCheckpoint is true. Default: 1. format: int32 minimum: 1 type: integer sourceRepo: description: |- sourceRepo is the Git repository for the source checkout this category clones at pod start. Each catalog entry defines what its source is and its default upstream; point this at an internal mirror for air-gapped or restricted-egress clusters. Entries that clone no source ignore it. The URL must use an authenticated remote scheme (https:// or ssh://). http:// and git:// are intentionally rejected: the cloned source is executed by the workload, and those transports are unauthenticated. scp-style git@host:path syntax and file:// are intentionally rejected: the contract is a remote git mirror. Non-TLS mirrors and local source belong on the pre-seeded checkpoint PVC path or in the workload image instead. maxLength: 2048 pattern: ^(https|ssh)://[A-Za-z0-9._~:/@%+-]+$ type: string storageClassName: description: |- storageClassName is the StorageClass to use for PersistentVolumeClaim dependencies created by catalog entries. If not specified, catalog entries that require PVCs will use the cluster's default StorageClass. type: string storageSize: description: |- storageSize sets the PVC size for checkpoint storage (e.g., "10Ti", "500Gi"). Must be a valid Kubernetes resource quantity. Only used when enableCheckpoint is true. Default: "10Ti". pattern: ^([0-9]+(\.[0-9]+)?)(Ki|Mi|Gi|Ti|Pi|Ei)?$ type: string testScale: description: |- testScale controls the orchestration strategy for NCCL communication tests. Supported values: - "intra-node": each node tested independently (nodesPerJob=1) - "intra-rack": topology-partitioned per nvidia.com/gpu.clique - "diagnose": adaptive fault isolation (topology-aware hierarchical group testing) - "full-scale": all nodes in a single group (default) Non-communication entries ignore this field. enum: - intra-node - intra-rack - diagnose - full-scale type: string thresholds: additionalProperties: type: string description: |- thresholds defines performance thresholds as CEL expressions. Keys are metric names (e.g., "busBandwidthGBps", "goodputRatio"). Values are CEL expressions using a `value` variable (float64). Example: {"busBandwidthGBps": "value >= 900", "avgStepTimeSec": "value <= 3.0"} type: object timeoutPerJob: description: |- timeoutPerJob is the maximum time to wait for each job to complete. Accepts Go duration strings (e.g., "1h", "30m", "2h30m"). When not set, defaults to "1h" (or "15m" for diagnose test scale). pattern: ^([0-9]+(h|m|s|ms))+$ type: string workloadMetadata: description: |- workloadMetadata sets labels on the workload object each category's Job creates (today a Kubeflow TrainJob). Use it to place the workload in a Kueue local queue with kueue.x-k8s.io/queue-name, or for any other integration keyed on the submitted object's labels. Because CertificationSpec inlines CategoryOptions, this one field is read both globally at spec.workloadMetadata and per category at spec.categories[].options.workloadMetadata. Unlike the map and slice options next to it, the two levels merge per key rather than replacing wholesale: a category can change or add a key but cannot remove a global one. Set keys per category instead of globally when categories need genuinely different maps. These labels are not pod labels. When spec.gangScheduler is set, its resolved queue label is added to every category; a category that supplies that same key with a different value fails. properties: labels: additionalProperties: description: |- WorkloadLabelValue is a Kubernetes label value in a WorkloadMetadata labels map. It exists as a named type only so controller-gen emits the length and pattern constraints under the map's additionalProperties: there is no marker that attaches value constraints to a map[string]string. maxLength: 63 pattern: ^$|^[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$ type: string description: |- labels are merged onto the generated workload object's metadata.labels. Keys must be valid Kubernetes label keys: an optional DNS-subdomain prefix of at most 253 characters followed by "/", then a name of at most 63 characters. Values follow Kubernetes label-value syntax and may be empty; an empty value is a real label, not a deletion. The controller-owned keys "app.kubernetes.io/managed-by" and anything under the "nvcre.nvidia.com/" prefix are rejected rather than silently overwritten — they identify controller ownership and Job association. At most 32 entries. That bound is an enforced API limit, chosen as the product decision for how many workload labels one object may carry. (A finite maxProperties is separately required to bound the static cost of the CEL rules below, but the cost estimator accepts far larger values, so it does not pick this number.) The key grammar uses [.] rather than \. because the marker parser collapses the escape and CEL then rejects \. as an invalid escape in a string literal. The two are equivalent to the regex engine. maxProperties: 32 type: object x-kubernetes-validations: - message: labels must not set the controller-owned keys app.kubernetes.io/managed-by or any key under the nvcre.nvidia.com/ prefix rule: self.all(k, k != 'app.kubernetes.io/managed-by' && !k.startsWith('nvcre.nvidia.com/')) - message: 'each label key must be a valid Kubernetes label key: an optional DNS-subdomain prefix followed by ''/'', then a name of at most 63 characters beginning and ending with an alphanumeric character' rule: self.all(k, k.matches('^([a-z0-9]([-a-z0-9]*[a-z0-9])?([.][a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$')) - message: each label key prefix must be at most 253 characters rule: 'self.all(k, k.contains(''/'') ? k.split(''/'')[0].size() <= 253 : true)' type: object type: object variant: description: Variant is the lower level type such as nemotron, deepseek, nccl etc. type: string required: - domain - variant type: object maxItems: 64 minItems: 1 type: array enableCheckpoint: description: |- enableCheckpoint enables checkpointing for training workloads. When true, provisions a PVC for checkpoint storage, adds checkpoint config (pvcName/maxRestarts), and enables --save/--load in training scripts. When false (default), uses emptyDir and disables save/load. Non-training entries ignore this field. type: boolean enableMNNVL: description: |- enableMNNVL enables Multi-Node NVLink (NCCL_MNNVL_ENABLE=1) for training and communication workloads. Defaults to false (NCCL_MNNVL_ENABLE=0). Enable this when running on platforms with multi-node NVLink connectivity (e.g., GB300 NVL72). Can be overridden per-category via categories[].options.enableMNNVL. type: boolean exitDurationMins: description: |- exitDurationMins sets the training duration in minutes. Maps to EXIT_DURATION_MINS env var. Default: 30. format: int32 minimum: 1 type: integer gangScheduler: description: |- gangScheduler opts every category's workload pods into a gang-aware scheduler such as KAI Scheduler. When set, schedulerName is injected into every pod template of every category's resolved TrainingRuntime and the queue label is applied, so the scheduler holds all pods until the full gang can be placed. Applied after catalog and platform overrides resolve, so it also replaces the scheduler a catalog entry hardcodes. properties: queue: description: |- queue is the scheduler queue to submit the workload to. Defaults to "default-queue" if not specified. When non-empty, must be a valid Kubernetes label value: at most 63 characters, beginning and ending with an alphanumeric character, and containing only alphanumerics, hyphens, underscores, or dots. maxLength: 63 pattern: ^$|^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$ type: string queueLabelKey: description: |- queueLabelKey is the label key the queue name is written under, for schedulers that read a different label than KAI Scheduler does. Defaults to "kai.scheduler/queue" if not specified. Set it to "runai/queue" (with schedulerName "runai-scheduler") on clusters running the NVIDIA Run:ai platform. When non-empty, must be a valid Kubernetes label key (qualified name): an optional DNS-subdomain prefix of at most 253 characters followed by "/", then a name of at most 63 characters beginning and ending with an alphanumeric character and containing only alphanumerics, hyphens, underscores, or dots. maxLength: 317 pattern: ^$|^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$ type: string x-kubernetes-validations: - message: queueLabelKey prefix must be at most 253 characters rule: 'self.contains(''/'') ? self.split(''/'')[0].size() <= 253 : true' - message: queueLabelKey must not use the controller-owned key app.kubernetes.io/managed-by or any key under the nvcre.nvidia.com/ prefix rule: self != 'app.kubernetes.io/managed-by' && !self.startsWith('nvcre.nvidia.com/') schedulerName: description: schedulerName is the name of the gang-aware scheduler to use (e.g. "kai-scheduler"). minLength: 1 type: string required: - schedulerName type: object gpusPerNode: description: |- gpusPerNode optionally overrides the number of GPUs per node used by catalog workloads. If not specified, the controller derives the default from the GPU architecture in target.nodeSelector (e.g., 4 for GB200/GB300, 8 for H100). Use this field when your hardware has a non-standard GPU count per node. format: int32 minimum: 1 type: integer image: description: |- image overrides the workload container image for catalog workloads. It replaces the trainer image in the rendered job template, the image of the primary workload container of every replicated job in the resolved TrainingRuntime dependencies, and the image of every init container in those pods whose image exactly equals the primary's pre-override image (the workload-derived inits such as fix-ssh-permissions and megatron-clone); init containers with a distinct image (such as GCP's tcpxo-daemon) and any additional containers keep their catalog images. Applied after catalog and platform overrides resolve, so it also replaces an image a platform override selects. Beware: on AWS EFA platforms (H100, GB200) the platform overrides land the workers on an nccl-tests image that ships the aws-ofi-nccl (EFA) plugin; setting image replaces that image, and the operator then owns the EFA OFI plugin being present in the replacement. maxLength: 512 minLength: 1 type: string imagePullSecrets: description: |- imagePullSecrets is an optional list of references to secrets for pulling container images used by catalog workloads. If not specified, the cluster's default image pull configuration (e.g., ServiceAccount) is used. items: description: |- LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic type: array maxBytes: description: |- maxBytes sets the maximum message size for NCCL tests (e.g., "16G", "32G"). Maps to the NCCL perf test `-e` flag. Default: "16G" (GB200/GB300 override: "32G"). pattern: ^[0-9]+(K|M|G|T)?$ type: string maxConcurrent: description: |- maxConcurrent limits the number of simultaneously running jobs within a Workflow. Useful for diagnose-mode parallel screening to avoid fabric saturation. 0 means unlimited. Default: 0. format: int32 type: integer maxRestarts: description: |- maxRestarts sets the maximum number of checkpoint-based restarts for training workloads. Maps to checkpoint.maxRestarts in the Job spec. Only used when enableCheckpoint is true. Default: catalog-defined. format: int32 minimum: 0 type: integer maxSteps: description: |- maxSteps sets the maximum training steps. Maps to trainer.max_steps in the NeMo config. Default: 50. format: int32 minimum: -1 type: integer measurementTimeout: description: |- measurementTimeout is the maximum time to wait after a Job succeeds for measurement data (bandwidth, goodput, etc.) before failing threshold validation. Accepts Go duration strings (e.g., "5m", "10m", "30m"). When not set, defaults to "5m". pattern: ^([0-9]+(h|m|s|ms))+$ type: string minGroupSize: description: |- minGroupSize sets the smallest group size at which diagnose's internal bisection stops splitting. Groups at this size that fail become suspects. Default: 2. format: int32 minimum: 1 type: integer mlnxPerNode: description: |- mlnxPerNode overrides the auto-detected Mellanox NIC count per node. Used by platforms with InfiniBand or RoCE networking (Azure, OCI, TogetherAI). If not specified, derived from GPU architecture and platform via the catalog's gpu-defaults.yaml (e.g., 8 for most architectures, 2 for OCI L40s). format: int32 minimum: 0 type: integer nicResourceName: description: |- nicResourceName sets the Kubernetes extended resource name of the RDMA NIC devices requested on workload containers for on-prem GB200/GB300 targets (e.g., "rdma/ib", "nvidia.com/mlnxnics"); the name depends on the RDMA device plugin the site runs, and the per-container count comes from mlnxPerNode. When unset, the controller auto-detects the name on on-prem GB200/GB300 targets: a single candidate resource (rdma/* or nvidia.com/mlnxnics) allocatable at the resolved mlnxPerNode count on every target node is requested; zero or several qualifying candidates inject nothing and emit a NICResourceDetection event explaining what was found. Offline nvcrectl render (without --dry-run) has no cluster to inspect and uses only this field. The value must be a fully qualified extended resource name: a DNS-subdomain domain, a slash, and a name segment of at most 63 characters; the reserved kubernetes.io and k8s.io domains (including their subdomains) are rejected, matching what Kubernetes accepts as an extended resource. maxLength: 253 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/[a-zA-Z0-9]([-A-Za-z0-9_.]{0,61}[a-zA-Z0-9])?$ type: string x-kubernetes-validations: - message: nicResourceName must not use the reserved kubernetes.io or k8s.io domains rule: '!self.contains(''kubernetes.io/'') && !self.startsWith(''k8s.io/'') && !self.contains(''.k8s.io/'')' nodesPerJob: description: |- nodesPerJob is the number of nodes per job for multi-node workloads. When nil at both global and per-category level, the controller auto-selects: - Entries with per-node-count configs (training): largest config <= matching nodes. - All other entries: all matching nodes. When set, clamped to min(nodesPerJob, matchingNodes). format: int32 minimum: 1 type: integer numCycles: description: |- numCycles sets the number of run cycles for NCCL tests. Each cycle runs numIterations iterations and prints results separately. Maps to the NCCL perf test `-N` flag. Default: 10. format: int32 minimum: 1 type: integer numIterations: description: |- numIterations sets the number of timed iterations per message size for NCCL tests. Maps to the NCCL perf test `-n` flag. Default: 100. format: int32 minimum: 1 type: integer repeatCount: description: |- repeatCount sets the number of orchestration iterations for the Workflow. Each iteration runs all groups and collects results. Multiple iterations allow repeated testing for intermittent failures. Default: 1. format: int32 minimum: 1 type: integer resources: description: |- resources overrides the CPU and memory resources of training workload containers. Training entries default to DGX-class sizing (limits: cpu "128" / memory "800Gi"; requests: cpu "64" / memory "500Gi") — set this on smaller nodes so training pods can schedule. Each of the four values independently falls back to its default when unset. GPU count is controlled by gpusPerNode. Non-training entries ignore this field. Platform-specific catalog overrides (for example AWS with H100 GPUs) may supersede these values. properties: limits: description: limits overrides the container resource limits. properties: cpu: anyOf: - type: integer - type: string description: cpu is the CPU quantity (e.g., "6", "1500m"). maxLength: 32 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - message: cpu must be a non-negative quantity rule: quantity(string(self)).compareTo(quantity("0")) >= 0 memory: anyOf: - type: integer - type: string description: memory is the memory quantity (e.g., "48Gi"). maxLength: 32 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - message: memory must be a non-negative quantity rule: quantity(string(self)).compareTo(quantity("0")) >= 0 type: object requests: description: requests overrides the container resource requests. properties: cpu: anyOf: - type: integer - type: string description: cpu is the CPU quantity (e.g., "6", "1500m"). maxLength: 32 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - message: cpu must be a non-negative quantity rule: quantity(string(self)).compareTo(quantity("0")) >= 0 memory: anyOf: - type: integer - type: string description: memory is the memory quantity (e.g., "48Gi"). maxLength: 32 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - message: memory must be a non-negative quantity rule: quantity(string(self)).compareTo(quantity("0")) >= 0 type: object type: object x-kubernetes-validations: - message: requests.cpu must not exceed limits.cpu rule: '!has(self.requests) || !has(self.limits) || !has(self.requests.cpu) || !has(self.limits.cpu) || quantity(string(self.requests.cpu)).compareTo(quantity(string(self.limits.cpu))) <= 0' - message: requests.memory must not exceed limits.memory rule: '!has(self.requests) || !has(self.limits) || !has(self.requests.memory) || !has(self.limits.memory) || quantity(string(self.requests.memory)).compareTo(quantity(string(self.limits.memory))) <= 0' saveInterval: description: |- saveInterval sets the checkpoint save frequency in training steps. NeMo 6: maps to --save-interval; NeMo 4: maps to every_n_train_steps. Only used when enableCheckpoint is true. Default: 250. format: int32 minimum: 1 type: integer saveRetainInterval: description: |- saveRetainInterval retains checkpoints at multiples of this value, deleting intermediate checkpoints (NeMo 6 only, --save-retain-interval). The most recent checkpoint is always kept regardless. Only used when enableCheckpoint is true. Default: 1000. format: int32 minimum: 1 type: integer saveTopK: description: |- saveTopK keeps only the top K checkpoints by monitored metric (NeMo 4 only, save_top_k). Older checkpoints that aren't the best are deleted to save storage. Only used when enableCheckpoint is true. Default: 1. format: int32 minimum: 1 type: integer sourceRepo: description: |- sourceRepo is the Git repository for the source checkout this category clones at pod start. Each catalog entry defines what its source is and its default upstream; point this at an internal mirror for air-gapped or restricted-egress clusters. Entries that clone no source ignore it. The URL must use an authenticated remote scheme (https:// or ssh://). http:// and git:// are intentionally rejected: the cloned source is executed by the workload, and those transports are unauthenticated. scp-style git@host:path syntax and file:// are intentionally rejected: the contract is a remote git mirror. Non-TLS mirrors and local source belong on the pre-seeded checkpoint PVC path or in the workload image instead. maxLength: 2048 pattern: ^(https|ssh)://[A-Za-z0-9._~:/@%+-]+$ type: string storageClassName: description: |- storageClassName is the StorageClass to use for PersistentVolumeClaim dependencies created by catalog entries. If not specified, catalog entries that require PVCs will use the cluster's default StorageClass. type: string storageSize: description: |- storageSize sets the PVC size for checkpoint storage (e.g., "10Ti", "500Gi"). Must be a valid Kubernetes resource quantity. Only used when enableCheckpoint is true. Default: "10Ti". pattern: ^([0-9]+(\.[0-9]+)?)(Ki|Mi|Gi|Ti|Pi|Ei)?$ type: string target: description: target specifies which nodes to include in the orchestration. properties: matchExpressions: description: |- matchExpressions is a list of node selector requirements. Supports In, NotIn, Exists, DoesNotExist operators. All expressions must match (AND with nodeSelector). items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array nodeNames: description: |- nodeNames explicitly lists nodes to include. Nodes must also match nodeSelector and matchExpressions, if set. items: type: string type: array nodeSelector: additionalProperties: type: string description: nodeSelector selects nodes by label key-value pairs. type: object taintSelectors: description: |- taintSelectors selects nodes that have ALL matching taints. A node must have every taint in this list to be included. The controller auto-injects matching tolerations onto the workload pods so they can schedule onto the selected tainted nodes. Setting this field is the supported way to run training workloads (e.g. TrainJob) on tainted nodes. When taintSelectors is set, the explicit tolerations replace the blanket Operator: Exists toleration that MPI workloads (NCCL tests) otherwise receive — declared selectors always win. items: description: TaintSelector selects nodes by taint. properties: effect: description: |- effect is the taint effect to match (NoSchedule, PreferNoSchedule, NoExecute). If empty, matches any effect. type: string key: description: key is the taint key to match. minLength: 1 type: string value: description: value is the taint value to match. If empty, matches any value. type: string required: - key type: object type: array type: object testScale: description: |- testScale controls the orchestration strategy for NCCL communication tests. Supported values: - "intra-node": each node tested independently (nodesPerJob=1) - "intra-rack": topology-partitioned per nvidia.com/gpu.clique - "diagnose": adaptive fault isolation (topology-aware hierarchical group testing) - "full-scale": all nodes in a single group (default) Non-communication entries ignore this field. enum: - intra-node - intra-rack - diagnose - full-scale type: string thresholds: additionalProperties: type: string description: |- thresholds defines performance thresholds as CEL expressions. Keys are metric names (e.g., "busBandwidthGBps", "goodputRatio"). Values are CEL expressions using a `value` variable (float64). Example: {"busBandwidthGBps": "value >= 900", "avgStepTimeSec": "value <= 3.0"} type: object timeoutPerJob: description: |- timeoutPerJob is the maximum time to wait for each job to complete. Accepts Go duration strings (e.g., "1h", "30m", "2h30m"). When not set, defaults to "1h" (or "15m" for diagnose test scale). pattern: ^([0-9]+(h|m|s|ms))+$ type: string workloadMetadata: description: |- workloadMetadata sets labels on the workload object each category's Job creates (today a Kubeflow TrainJob). Use it to place the workload in a Kueue local queue with kueue.x-k8s.io/queue-name, or for any other integration keyed on the submitted object's labels. Because CertificationSpec inlines CategoryOptions, this one field is read both globally at spec.workloadMetadata and per category at spec.categories[].options.workloadMetadata. Unlike the map and slice options next to it, the two levels merge per key rather than replacing wholesale: a category can change or add a key but cannot remove a global one. Set keys per category instead of globally when categories need genuinely different maps. These labels are not pod labels. When spec.gangScheduler is set, its resolved queue label is added to every category; a category that supplies that same key with a different value fails. properties: labels: additionalProperties: description: |- WorkloadLabelValue is a Kubernetes label value in a WorkloadMetadata labels map. It exists as a named type only so controller-gen emits the length and pattern constraints under the map's additionalProperties: there is no marker that attaches value constraints to a map[string]string. maxLength: 63 pattern: ^$|^[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$ type: string description: |- labels are merged onto the generated workload object's metadata.labels. Keys must be valid Kubernetes label keys: an optional DNS-subdomain prefix of at most 253 characters followed by "/", then a name of at most 63 characters. Values follow Kubernetes label-value syntax and may be empty; an empty value is a real label, not a deletion. The controller-owned keys "app.kubernetes.io/managed-by" and anything under the "nvcre.nvidia.com/" prefix are rejected rather than silently overwritten — they identify controller ownership and Job association. At most 32 entries. That bound is an enforced API limit, chosen as the product decision for how many workload labels one object may carry. (A finite maxProperties is separately required to bound the static cost of the CEL rules below, but the cost estimator accepts far larger values, so it does not pick this number.) The key grammar uses [.] rather than \. because the marker parser collapses the escape and CEL then rejects \. as an invalid escape in a string literal. The two are equivalent to the regex engine. maxProperties: 32 type: object x-kubernetes-validations: - message: labels must not set the controller-owned keys app.kubernetes.io/managed-by or any key under the nvcre.nvidia.com/ prefix rule: self.all(k, k != 'app.kubernetes.io/managed-by' && !k.startsWith('nvcre.nvidia.com/')) - message: 'each label key must be a valid Kubernetes label key: an optional DNS-subdomain prefix followed by ''/'', then a name of at most 63 characters beginning and ending with an alphanumeric character' rule: self.all(k, k.matches('^([a-z0-9]([-a-z0-9]*[a-z0-9])?([.][a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$')) - message: each label key prefix must be at most 253 characters rule: 'self.all(k, k.contains(''/'') ? k.split(''/'')[0].size() <= 253 : true)' type: object required: - categories - target type: object x-kubernetes-validations: - message: spec is immutable after creation rule: self == oldSelf status: description: status defines the observed state of Certification properties: categoryStatuses: description: categoryStatuses tracks the status of each certification category. items: description: CertificationCategoryStatus tracks the status of a single certification category. properties: domain: description: domain is the category domain. type: string failedNodesRef: description: |- failedNodesRef references the ConfigMap holding this category's failed-nodes list (name, reason, message). Copied from the Workflow. properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string required: - kind - name type: object x-kubernetes-map-type: atomic status: description: |- status is the current status of this category. Values: "Pending", "InProgress", "Succeeded", "Failed" type: string succeededNodesRef: description: |- succeededNodesRef references the ConfigMap holding this category's succeeded-nodes list (comma-separated node names). Copied from the Workflow. properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string required: - kind - name type: object x-kubernetes-map-type: atomic variant: description: variant is the category variant. type: string workflowRef: description: workflowRef references the Workflow created for this category. properties: name: description: name is the name of the Workflow resource. type: string namespace: description: namespace is the namespace of the Workflow resource. type: string required: - name type: object required: - domain - status - variant type: object type: array conditions: description: |- conditions represent the current state of the Certification resource. Condition types: - "InProgress": the Certification is currently running - "Succeeded": the Certification completed successfully - "Failed": the Certification has failed The status of each condition is one of True, False, or Unknown. items: description: Condition contains details for one aspect of the current state of this API Resource. properties: lastTransitionTime: description: |- lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. format: date-time type: string message: description: |- message is a human readable message indicating details about the transition. This may be an empty string. maxLength: 32768 type: string observedGeneration: description: |- observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. format: int64 minimum: 0 type: integer reason: description: |- reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty. maxLength: 1024 minLength: 1 pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ type: string status: description: status of the condition, one of True, False, Unknown. enum: - "True" - "False" - Unknown type: string type: description: type of condition in CamelCase or in foo.example.com/CamelCase. maxLength: 316 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ type: string required: - lastTransitionTime - message - reason - status - type type: object type: array x-kubernetes-list-map-keys: - type x-kubernetes-list-type: map type: object required: - spec type: object served: true storage: true subresources: status: {} --- # Source: cluster-readiness-engine/crds/nvcre.nvidia.com_goodputmeasurements.yaml --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.20.0 name: goodputmeasurements.nvcre.nvidia.com spec: group: nvcre.nvidia.com names: kind: GoodputMeasurement listKind: GoodputMeasurementList plural: goodputmeasurements singular: goodputmeasurement scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: GoodputMeasurement is the Schema for the goodputmeasurements API properties: apiVersion: description: |- APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: description: |- Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object spec: description: spec defines the desired state of GoodputMeasurement properties: jobRef: description: jobRef is the reference to the NVCRE Job for which goodput should be calculated. properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string required: - kind - name type: object x-kubernetes-map-type: atomic logProfileRef: description: |- logProfileRef is the name of the cluster-scoped LogProfile to use for log parsing. The LogProfile defines regex patterns for extracting training metrics from pod logs. minLength: 1 type: string sampleInterval: description: |- sampleInterval is how often to sample pod logs while the Job is running. Default: 60s. type: string required: - logProfileRef type: object status: description: status defines the observed state of GoodputMeasurement properties: applicationStartTime: description: |- applicationStartTime is the timestamp of the first application framework log marker. Persisted so startup resume time can be computed even if the log line scrolls out of the tail window. format: date-time type: string applicationStopTime: description: |- applicationStopTime is the timestamp of the last application log line observed. Used as the interruption time for goodput calculations — more accurate than container termination time or wall clock for stalls and crashes. format: date-time type: string avgStepTimeSec: description: |- avgStepTimeSec is the average time per training step in seconds (excluding warmup). Computed from observed step timings or consecutive step timestamps. type: string avgTFLOPSPerGPU: description: avgTFLOPSPerGPU is the average TFLOPS per GPU observed from training step logs (including warmup). type: string checkpointSaveTimeSec: description: checkpointSaveTimeSec is cumulative checkpoint save overhead in seconds (sum of t_save). type: string completionTime: description: completionTime is the time when the measurement completed (when the referenced Job reached a terminal state). format: date-time type: string conditions: description: |- conditions represent the current state of the GoodputMeasurement resource. Condition types: - "Measuring": the measurement is in progress (referenced Job is running) - "Complete": the measurement has finished (referenced Job succeeded or failed) The status of each condition is one of True, False, or Unknown. items: description: Condition contains details for one aspect of the current state of this API Resource. properties: lastTransitionTime: description: |- lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. format: date-time type: string message: description: |- message is a human readable message indicating details about the transition. This may be an empty string. maxLength: 32768 type: string observedGeneration: description: |- observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. format: int64 minimum: 0 type: integer reason: description: |- reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty. maxLength: 1024 minLength: 1 pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ type: string status: description: status of the condition, one of True, False, Unknown. enum: - "True" - "False" - Unknown type: string type: description: type of condition in CamelCase or in foo.example.com/CamelCase. maxLength: 316 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ type: string required: - lastTransitionTime - message - reason - status - type type: object type: array x-kubernetes-list-map-keys: - type x-kubernetes-list-type: map currentStep: description: currentStep is the latest training step observed from logs. type: integer highestStep: description: highestStep is the highest training step ever reached (may be > currentStep after restart). type: integer interruptionCount: description: interruptionCount is the number of interruptions detected. type: integer lastCheckpointStep: description: lastCheckpointStep is the step number of the last saved checkpoint. type: integer lastCheckpointTime: description: |- lastCheckpointTime is the timestamp of the last saved checkpoint. Persisted so lost work time (t_ch) can be computed after controller restarts. format: date-time type: string lastNonWarmupStep: description: |- lastNonWarmupStep is the highest step whose timing has been accumulated into nonWarmupTimeSec. Used as a watermark to avoid double-counting steps across reconcile cycles. type: integer lastStepTimestamp: description: |- lastStepTimestamp is the timestamp of the most recently observed training step. Used by the Job controller for stall detection. format: date-time type: string logInterval: description: |- logInterval is the training framework's log interval from the LogProfile. Persisted so the Job controller can scale the stall detection threshold by the number of iterations between consecutive logged steps. type: integer lostWorkTimeSec: description: lostWorkTimeSec is cumulative lost work in seconds (sum of t_ch across all interruptions). type: string nonWarmupTimeSec: description: nonWarmupTimeSec is the cumulative time spent on non-warmup steps in seconds (across all runs). type: string pendingInterruption: description: |- pendingInterruption stores an in-progress interruption event that hasn't been completed by a job restart yet. Persisted so state survives controller restarts. properties: checkpointStep: description: checkpointStep is the step number of the last saved checkpoint before interruption. type: integer lastStep: description: lastStep is the last training step observed before interruption. type: integer reason: description: reason is the reason for the interruption. type: string tCh: description: tCh is the lost work time in seconds (time between last checkpoint and interruption). type: string tCheckpoint: description: tCheckpoint is the time of the last checkpoint before the interruption. format: date-time type: string tInterrupt: description: tInterrupt is the time the job was interrupted (workload failure time). format: date-time type: string type: object priorNonWarmupTimeSec: description: priorNonWarmupTimeSec is the accumulated non-warmup time from completed runs (before the current run). type: string priorWarmupTimeSec: description: priorWarmupTimeSec is the accumulated warmup time from completed runs (before the current run). type: string rescheduleTimeSec: description: rescheduleTimeSec is cumulative scheduling delay in seconds (sum of t_re). type: string result: description: |- result is the current runtime goodput ratio (0.0 to 1.0) as a string. Formula: (t_w - t_ch - t_rm - t_re - t_save) / (t_w - t_re) where t_w=trainingTime, t_ch=lostWorkTime, t_rm=resumeTime, t_re=rescheduleTime, t_save=checkpointSaveTime type: string resumeTimeSec: description: 'resumeTimeSec is cumulative resume time in seconds (sum of t_rm: checkpoint load + resume).' type: string startTime: description: startTime is the time when the measurement started (when the referenced Job began running). format: date-time type: string trainingTimeSec: description: trainingTimeSec is the total training wall-clock time in seconds (t_w). type: string warmupBaseStep: description: |- warmupBaseStep is the GlobalStep of the first training step after the most recent run boundary (applicationStart or checkpointRestore). Steps within warmupSteps of this base are warmup. Persisted so warmup detection survives across SinceTime-based incremental log reads where applicationStart is no longer in the read window. type: integer warmupTimeSec: description: warmupTimeSec is the cumulative time spent on warmup steps in seconds (across all runs). type: string type: object required: - spec type: object served: true storage: true subresources: status: {} --- # Source: cluster-readiness-engine/crds/nvcre.nvidia.com_jobs.yaml --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.20.0 name: jobs.nvcre.nvidia.com spec: group: nvcre.nvidia.com names: kind: Job listKind: JobList plural: jobs singular: job scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: Job is the Schema for the jobs API properties: apiVersion: description: |- APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: description: |- Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object spec: description: spec defines the desired state of Job properties: bandwidthMeasurement: description: |- bandwidthMeasurement configures automatic creation of a BandwidthMeasurement child resource that tracks NCCL bandwidth metrics by parsing pod logs. When absent, no measurement is created (suitable for non-NCCL jobs). This field is immutable, including its presence: an object created without it cannot add it later. properties: logProfileRef: description: logProfileRef is the name of the cluster-scoped LogProfile to use for log parsing. minLength: 1 type: string sampleInterval: description: |- sampleInterval is how often to sample pod logs while the Job is running. Default: 60s. type: string testType: description: |- testType identifies the NCCL collective operation (e.g., "all_reduce", "alltoall"). Propagated to the BandwidthMeasurement spec for Prometheus metric labeling. type: string required: - logProfileRef type: object x-kubernetes-validations: - message: bandwidthMeasurement is immutable rule: self == oldSelf checkpoint: description: |- checkpoint configures checkpoint-based restart for the workload. When set, the Job controller restarts the workload on failure if a checkpoint was logged by an associated GoodputMeasurement. The user must define the PVC volume and mounts in the workload spec directly. properties: maxRestarts: description: |- maxRestarts is the maximum number of times the Job will restart on failure when a checkpoint exists. Default: 0 (no restarts). format: int32 type: integer pvcName: description: |- pvcName is the name of the PersistentVolumeClaim used for checkpoint storage. Must match a PVC in the Workflow's dependencies. minLength: 1 type: string required: - pvcName type: object goodputMeasurement: description: |- goodputMeasurement configures automatic creation of a GoodputMeasurement child resource that tracks training goodput metrics by parsing pod logs. When absent, no measurement is created (suitable for non-training jobs). Manually-created GoodputMeasurements continue to work via the existing List-based lookup. This field is immutable, including its presence: an object created without it cannot add it later. properties: logProfileRef: description: logProfileRef is the name of the cluster-scoped LogProfile to use for log parsing. minLength: 1 type: string sampleInterval: description: |- sampleInterval is how often to sample pod logs while the Job is running. Default: 60s. type: string required: - logProfileRef type: object x-kubernetes-validations: - message: goodputMeasurement is immutable rule: self == oldSelf measurementTimeout: description: |- measurementTimeout is the maximum time to wait after the Job succeeds for measurement data before failing threshold validation. Propagated from the Workflow's ValidationSpec by the Workflow controller during Job creation. When nil, the Job controller uses its default (5m). type: string nodeHealthMonitor: description: |- nodeHealthMonitor configures hardware failure detection for nodes running this job's pods. When a failure is detected, the job will be marked with the HardwareFailed condition. This field is immutable, including its presence: an object created without it cannot add it later. properties: cel: description: |- cel defines a CEL expression evaluated against Node objects. The expression must return a boolean. When it evaluates to true, the node is considered to have a hardware failure. The full corev1.Node object is available via the 'node' variable: - node.metadata: ObjectMeta (name, labels, annotations, etc.) - node.spec: NodeSpec (taints, unschedulable, podCIDR, etc.) - node.status: NodeStatus (conditions, addresses, capacity, allocatable, nodeInfo, etc.) Examples: - "node.spec.taints.exists(t, t.key == 'nvidia.com/gpu-unhealthy')" - "'gpud.nvidia.com/unhealthy' in node.metadata.labels" - "node.status.conditions.exists(c, c.type == 'GPUHealthy' && c.status == 'False')" - "node.spec.unschedulable == true" properties: expression: description: |- expression is the CEL expression to evaluate against Node objects. Must return a boolean value. When true, the node is considered unhealthy. maxLength: 2048 minLength: 1 type: string required: - expression type: object type: object x-kubernetes-validations: - message: nodeHealthMonitor is immutable rule: self == oldSelf stallMultiplier: description: |- stallMultiplier configures stall detection for the workload. When set, the Job controller compares the time since the last training step against stallMultiplier * avgStepTime (from the associated GoodputMeasurement). If the elapsed time exceeds this product, the workload is marked as Failed with reason "WorkloadStalled". Requires a GoodputMeasurement for this Job; if none exists, stall detection is skipped. Example: a value of 10 means the job is stalled if no step has been logged for 10x the average step duration. format: int32 minimum: 1 type: integer startupStallTimeoutSeconds: description: |- startupStallTimeoutSeconds is the maximum time (in seconds) to wait after the application framework starts for the first training step. If no step is observed within this timeout, the workload is marked Failed with reason "WorkloadStalled". Only active when stallMultiplier is set. Default: 1200 (20 minutes). format: int32 minimum: 1 type: integer thresholds: additionalProperties: type: string description: |- thresholds maps metric names to CEL expressions for performance validation. Propagated from WorkflowSpec.Validation.Performance.Thresholds by the Workflow controller during Job creation. The Job controller evaluates them after workload success and sets ValidationFailed if any threshold is violated. Keys: "busBandwidthGBps", "goodputRatio", "avgTFLOPsPerGPU", "avgStepTimeSec", etc. Values: CEL expressions using `value` variable, e.g. "value >= 900" type: object workload: description: |- workload defines the workload to run. The workload is created as a child resource of the Job. maxProperties: 1 minProperties: 1 properties: trainJob: description: trainJob configures a Kubeflow TrainJob workload. properties: activeDeadlineSeconds: description: |- activeDeadlineSeconds specifies the duration in seconds relative to the TrainJob start time (which resets on resume from suspension) that the TrainJob may be active before the system tries to terminate it. Value must be a positive integer. Once reached, all running Pods are terminated and the TrainJob status becomes Failed with reason: DeadlineExceeded. format: int64 minimum: 1 type: integer x-kubernetes-validations: - message: field is immutable rule: self == oldSelf initializer: description: initializer defines the configuration of the initializer. properties: dataset: description: dataset defines the configuration for the dataset initialization and pre-processing. properties: env: description: |- env is the list of environment variables to set in the dataset initializer container. These values will be merged with the TrainingRuntime's dataset initializer environments. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map secretRef: description: |- secretRef is the reference to the secret with credentials to download dataset. Secret must be created in the TrainJob's namespace. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic storageUri: description: storageUri is the URI for the dataset provider. maxLength: 2048 type: string type: object model: description: model defines the configuration for the pre-trained model initialization properties: env: description: |- env is the list of environment variables to set in the model initializer container. These values will be merged with the TrainingRuntime's model initializer environments. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map secretRef: description: |- secretRef is the reference to the secret with credentials to download model. Secret must be created in the TrainJob's namespace. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic storageUri: description: storageUri is the URI for the model provider. maxLength: 2048 type: string type: object type: object x-kubernetes-validations: - message: field is immutable rule: self == oldSelf managedBy: default: trainer.kubeflow.org/trainjob-controller description: |- managedBy is used to indicate the controller or entity that manages a TrainJob. The value must be either an empty, `trainer.kubeflow.org/trainjob-controller` or `kueue.x-k8s.io/multikueue`. The built-in TrainJob controller reconciles TrainJob which don't have this field at all or the field value is the reserved string `trainer.kubeflow.org/trainjob-controller`, but delegates reconciling TrainJobs with a 'kueue.x-k8s.io/multikueue' to the Kueue. The field is immutable. maxLength: 253 type: string x-kubernetes-validations: - message: ManagedBy must be trainer.kubeflow.org/trainjob-controller or kueue.x-k8s.io/multikueue if set rule: self in ['trainer.kubeflow.org/trainjob-controller', 'kueue.x-k8s.io/multikueue'] - message: field is immutable rule: self == oldSelf runtimePatches: description: |- runtimePatches defines custom patches applied to the TrainJob's Runtime. Patches are keyed by manager to provide clear ownership and avoid conflicts between controllers. items: description: |- RuntimePatch represents a custom patch applied to the TrainJob's training runtime template. Patches are keyed by manager to provide clear ownership and avoid conflicts between controllers. properties: manager: description: |- manager indicates who owns this patch entry. It can be set by the user, external controllers, or admission webhooks to track ownership and avoid conflicts. For example, Kueue sets this field to "kueue.x-k8s.io/manager". maxLength: 253 minLength: 1 type: string x-kubernetes-validations: - message: field is immutable rule: self == oldSelf time: description: |- time is the timestamp of when this patch was last written. Set by the Trainer admission webhook on each create or update. Used for observability only, not used as a list map key. format: date-time type: string trainingRuntimeSpec: description: trainingRuntimeSpec defines allowed patches for ClusterTrainingRuntime or TrainingRuntime-based jobs. properties: template: description: template patches the JobSet template. properties: metadata: description: |- metadata patches the JobSet object metadata. Only labels and annotations are allowed. type: object spec: description: spec patches the JobSet spec with restricted fields. properties: replicatedJobs: description: replicatedJobs defines per-job patches, keyed by job name. items: description: ReplicatedJobPatch defines patches for a specific replicated job within the JobSet. properties: name: description: name is the name of the replicated job to patch. maxLength: 253 minLength: 1 type: string template: description: template patches the Job template for this replicated job. properties: metadata: description: |- metadata patches the Job template metadata. Only labels and annotations are allowed. type: object spec: description: spec patches the Job spec with restricted fields. properties: template: description: template patches the Pod template for this Job. properties: metadata: description: |- metadata patches the Pod template metadata. Only labels and annotations are allowed. type: object spec: description: spec patches the Pod spec with the fields managers are permitted to set. properties: affinity: description: affinity patches the Pod's scheduling affinity. properties: nodeAffinity: description: Describes node affinity scheduling rules for the pod. properties: preferredDuringSchedulingIgnoredDuringExecution: description: |- The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node matches the corresponding matchExpressions; the node(s) with the highest sum are the most preferred. items: description: |- An empty preferred scheduling term matches all objects with implicit weight 0 (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op). properties: preference: description: A node selector term, associated with the corresponding weight. properties: matchExpressions: description: A list of node selector requirements by node's labels. items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchFields: description: A list of node selector requirements by node's fields. items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic type: object x-kubernetes-map-type: atomic weight: description: Weight associated with matching the corresponding nodeSelectorTerm, in the range 1-100. format: int32 type: integer required: - preference - weight type: object type: array x-kubernetes-list-type: atomic requiredDuringSchedulingIgnoredDuringExecution: description: |- If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to an update), the system may or may not try to eventually evict the pod from its node. properties: nodeSelectorTerms: description: Required. A list of node selector terms. The terms are ORed. items: description: |- A null or empty node selector term matches no objects. The requirements of them are ANDed. The TopologySelectorTerm type implements a subset of the NodeSelectorTerm. properties: matchExpressions: description: A list of node selector requirements by node's labels. items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchFields: description: A list of node selector requirements by node's fields. items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic type: object x-kubernetes-map-type: atomic type: array x-kubernetes-list-type: atomic required: - nodeSelectorTerms type: object x-kubernetes-map-type: atomic type: object podAffinity: description: Describes pod affinity scheduling rules (e.g. co-locate this pod in the same node, zone, etc. as some other pod(s)). properties: preferredDuringSchedulingIgnoredDuringExecution: description: |- The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. items: description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s) properties: podAffinityTerm: description: Required. A pod affinity term, associated with the corresponding weight. properties: labelSelector: description: |- A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic matchLabelKeys: description: |- MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic mismatchLabelKeys: description: |- MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic namespaceSelector: description: |- A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: description: |- namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array x-kubernetes-list-type: atomic topologyKey: description: |- This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. type: string required: - topologyKey type: object weight: description: |- weight associated with matching the corresponding podAffinityTerm, in the range 1-100. format: int32 type: integer required: - podAffinityTerm - weight type: object type: array x-kubernetes-list-type: atomic requiredDuringSchedulingIgnoredDuringExecution: description: |- If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied. items: description: |- Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key matches that of any node on which a pod of the set of pods is running properties: labelSelector: description: |- A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic matchLabelKeys: description: |- MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic mismatchLabelKeys: description: |- MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic namespaceSelector: description: |- A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: description: |- namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array x-kubernetes-list-type: atomic topologyKey: description: |- This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. type: string required: - topologyKey type: object type: array x-kubernetes-list-type: atomic type: object podAntiAffinity: description: Describes pod anti-affinity scheduling rules (e.g. avoid putting this pod in the same node, zone, etc. as some other pod(s)). properties: preferredDuringSchedulingIgnoredDuringExecution: description: |- The scheduler will prefer to schedule pods to nodes that satisfy the anti-affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling anti-affinity expressions, etc.), compute a sum by iterating through the elements of this field and subtracting "weight" from the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. items: description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s) properties: podAffinityTerm: description: Required. A pod affinity term, associated with the corresponding weight. properties: labelSelector: description: |- A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic matchLabelKeys: description: |- MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic mismatchLabelKeys: description: |- MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic namespaceSelector: description: |- A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: description: |- namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array x-kubernetes-list-type: atomic topologyKey: description: |- This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. type: string required: - topologyKey type: object weight: description: |- weight associated with matching the corresponding podAffinityTerm, in the range 1-100. format: int32 type: integer required: - podAffinityTerm - weight type: object type: array x-kubernetes-list-type: atomic requiredDuringSchedulingIgnoredDuringExecution: description: |- If the anti-affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the anti-affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied. items: description: |- Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key matches that of any node on which a pod of the set of pods is running properties: labelSelector: description: |- A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic matchLabelKeys: description: |- MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic mismatchLabelKeys: description: |- MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic namespaceSelector: description: |- A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: description: |- namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array x-kubernetes-list-type: atomic topologyKey: description: |- This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. type: string required: - topologyKey type: object type: array x-kubernetes-list-type: atomic type: object type: object containers: description: containers patches specific containers in the target job templates. items: description: ContainerPatch represents parameters that can be patched using PodSpecPatch. properties: env: description: |- env is the list of environment variables to set in the container. These values will be merged with the Runtime's environments. These values can't be set for container with the name: `node`, `dataset-initializer`, or `model-initializer`. For those containers the envs can only be set via Trainer or Initializer APIs. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map name: description: name for the container. Runtime must have this container. maxLength: 253 minLength: 1 type: string securityContext: description: |- securityContext patches the container's security context. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ properties: allowPrivilegeEscalation: description: |- AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged 2) has CAP_SYS_ADMIN Note that this field cannot be set when spec.os.name is windows. type: boolean appArmorProfile: description: |- appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". type: string type: description: |- type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. type: string required: - type type: object capabilities: description: |- The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. properties: add: description: Added capabilities items: description: Capability represent POSIX capabilities type type: string type: array x-kubernetes-list-type: atomic drop: description: Removed capabilities items: description: Capability represent POSIX capabilities type type: string type: array x-kubernetes-list-type: atomic type: object privileged: description: |- Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. type: boolean procMount: description: |- procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. type: string readOnlyRootFilesystem: description: |- Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. type: boolean runAsGroup: description: |- The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer runAsNonRoot: description: |- Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: boolean runAsUser: description: |- The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer seLinuxOptions: description: |- The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. properties: level: description: Level is SELinux level label that applies to the container. type: string role: description: Role is a SELinux role label that applies to the container. type: string type: description: Type is a SELinux type label that applies to the container. type: string user: description: User is a SELinux user label that applies to the container. type: string type: object seccompProfile: description: |- The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. type: string type: description: |- type indicates which kind of seccomp profile will be applied. Valid options are: Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. type: string required: - type type: object windowsOptions: description: |- The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. properties: gmsaCredentialSpec: description: |- GMSACredentialSpec is where the GMSA admission webhook (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. type: string gmsaCredentialSpecName: description: GMSACredentialSpecName is the name of the GMSA credential spec to use. type: string hostProcess: description: |- HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. type: boolean runAsUserName: description: |- The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: string type: object type: object volumeMounts: description: volumeMounts are the volumes to mount into the container's filesystem. items: description: VolumeMount describes a mounting of a Volume within a container. properties: mountPath: description: |- Path within the container at which the volume should be mounted. Must not contain ':'. type: string mountPropagation: description: |- mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. When RecursiveReadOnly is set to IfPossible or to Enabled, MountPropagation must be None or unspecified (which defaults to None). type: string name: description: This must match the Name of a Volume. type: string readOnly: description: |- Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. type: boolean recursiveReadOnly: description: |- RecursiveReadOnly specifies whether read-only mounts should be handled recursively. If ReadOnly is false, this field has no meaning and must be unspecified. If ReadOnly is true, and this field is set to Disabled, the mount is not made recursively read-only. If this field is set to IfPossible, the mount is made recursively read-only, if it is supported by the container runtime. If this field is set to Enabled, the mount is made recursively read-only if it is supported by the container runtime, otherwise the pod will not be started and an error will be generated to indicate the reason. If this field is set to IfPossible or Enabled, MountPropagation must be set to None (or be unspecified, which defaults to None). If this field is not specified, it is treated as an equivalent of Disabled. type: string subPath: description: |- Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). type: string subPathExpr: description: |- Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. type: string required: - mountPath - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map required: - name type: object maxItems: 64 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map x-kubernetes-validations: - message: field is immutable rule: self == oldSelf imagePullSecrets: description: imagePullSecrets patches the image pull secrets for the Pods in the target job templates. items: description: |- LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic maxItems: 64 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map x-kubernetes-validations: - message: field is immutable rule: self == oldSelf initContainers: description: initContainers patches the init containers in the target job templates. items: description: ContainerPatch represents parameters that can be patched using PodSpecPatch. properties: env: description: |- env is the list of environment variables to set in the container. These values will be merged with the Runtime's environments. These values can't be set for container with the name: `node`, `dataset-initializer`, or `model-initializer`. For those containers the envs can only be set via Trainer or Initializer APIs. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map name: description: name for the container. Runtime must have this container. maxLength: 253 minLength: 1 type: string securityContext: description: |- securityContext patches the container's security context. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ properties: allowPrivilegeEscalation: description: |- AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged 2) has CAP_SYS_ADMIN Note that this field cannot be set when spec.os.name is windows. type: boolean appArmorProfile: description: |- appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". type: string type: description: |- type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. type: string required: - type type: object capabilities: description: |- The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. properties: add: description: Added capabilities items: description: Capability represent POSIX capabilities type type: string type: array x-kubernetes-list-type: atomic drop: description: Removed capabilities items: description: Capability represent POSIX capabilities type type: string type: array x-kubernetes-list-type: atomic type: object privileged: description: |- Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. type: boolean procMount: description: |- procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. type: string readOnlyRootFilesystem: description: |- Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. type: boolean runAsGroup: description: |- The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer runAsNonRoot: description: |- Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: boolean runAsUser: description: |- The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer seLinuxOptions: description: |- The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. properties: level: description: Level is SELinux level label that applies to the container. type: string role: description: Role is a SELinux role label that applies to the container. type: string type: description: Type is a SELinux type label that applies to the container. type: string user: description: User is a SELinux user label that applies to the container. type: string type: object seccompProfile: description: |- The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. type: string type: description: |- type indicates which kind of seccomp profile will be applied. Valid options are: Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. type: string required: - type type: object windowsOptions: description: |- The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. properties: gmsaCredentialSpec: description: |- GMSACredentialSpec is where the GMSA admission webhook (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. type: string gmsaCredentialSpecName: description: GMSACredentialSpecName is the name of the GMSA credential spec to use. type: string hostProcess: description: |- HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. type: boolean runAsUserName: description: |- The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: string type: object type: object volumeMounts: description: volumeMounts are the volumes to mount into the container's filesystem. items: description: VolumeMount describes a mounting of a Volume within a container. properties: mountPath: description: |- Path within the container at which the volume should be mounted. Must not contain ':'. type: string mountPropagation: description: |- mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. When RecursiveReadOnly is set to IfPossible or to Enabled, MountPropagation must be None or unspecified (which defaults to None). type: string name: description: This must match the Name of a Volume. type: string readOnly: description: |- Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. type: boolean recursiveReadOnly: description: |- RecursiveReadOnly specifies whether read-only mounts should be handled recursively. If ReadOnly is false, this field has no meaning and must be unspecified. If ReadOnly is true, and this field is set to Disabled, the mount is not made recursively read-only. If this field is set to IfPossible, the mount is made recursively read-only, if it is supported by the container runtime. If this field is set to Enabled, the mount is made recursively read-only if it is supported by the container runtime, otherwise the pod will not be started and an error will be generated to indicate the reason. If this field is set to IfPossible or Enabled, MountPropagation must be set to None (or be unspecified, which defaults to None). If this field is not specified, it is treated as an equivalent of Disabled. type: string subPath: description: |- Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). type: string subPathExpr: description: |- Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. type: string required: - mountPath - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map required: - name type: object maxItems: 64 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map x-kubernetes-validations: - message: field is immutable rule: self == oldSelf nodeSelector: additionalProperties: type: string description: nodeSelector patches the node selector to place Pods on specific nodes. type: object schedulingGates: description: |- schedulingGates patches the scheduling gates for the Pods in the target job templates. More info: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/ items: description: PodSchedulingGate is associated to a Pod to guard its scheduling. properties: name: description: |- Name of the scheduling gate. Each scheduling gate must have a unique name field. type: string required: - name type: object maxItems: 16 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map securityContext: description: |- securityContext patches the Pod's security context. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ properties: appArmorProfile: description: |- appArmorProfile is the AppArmor options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". type: string type: description: |- type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. type: string required: - type type: object fsGroup: description: |- A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod: 1. The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw---- If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer fsGroupChangePolicy: description: |- fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows. type: string runAsGroup: description: |- The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer runAsNonRoot: description: |- Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: boolean runAsUser: description: |- The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer seLinuxChangePolicy: description: |- seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. Valid values are "MountOption" and "Recursive". "Recursive" means relabeling of all files on all Pod volumes by the container runtime. This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node. "MountOption" mounts all eligible Pod volumes with `-o context` mount option. This requires all Pods that share the same volume to use the same SELinux label. It is not possible to share the same volume among privileged and unprivileged Pods. Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their CSIDriver instance. Other volumes are always re-labelled recursively. "MountOption" value is allowed only when SELinuxMount feature gate is enabled. If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes and "Recursive" for all other volumes. This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers. All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. Note that this field cannot be set when spec.os.name is windows. type: string seLinuxOptions: description: |- The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. properties: level: description: Level is SELinux level label that applies to the container. type: string role: description: Role is a SELinux role label that applies to the container. type: string type: description: Type is a SELinux type label that applies to the container. type: string user: description: User is a SELinux user label that applies to the container. type: string type: object seccompProfile: description: |- The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. type: string type: description: |- type indicates which kind of seccomp profile will be applied. Valid options are: Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. type: string required: - type type: object supplementalGroups: description: |- A list of groups applied to the first process run in each container, in addition to the container's primary GID and fsGroup (if specified). If the SupplementalGroupsPolicy feature is enabled, the supplementalGroupsPolicy field determines whether these are in addition to or instead of any group memberships defined in the container image. If unspecified, no additional groups are added, though group memberships defined in the container image may still be used, depending on the supplementalGroupsPolicy field. Note that this field cannot be set when spec.os.name is windows. items: format: int64 type: integer type: array x-kubernetes-list-type: atomic supplementalGroupsPolicy: description: |- Defines how supplemental groups of the first container processes are calculated. Valid values are "Merge" and "Strict". If not specified, "Merge" is used. (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled and the container runtime must implement support for this feature. Note that this field cannot be set when spec.os.name is windows. type: string sysctls: description: |- Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows. items: description: Sysctl defines a kernel parameter to be set properties: name: description: Name of a property to set type: string value: description: Value of a property to set type: string required: - name - value type: object type: array x-kubernetes-list-type: atomic windowsOptions: description: |- The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. properties: gmsaCredentialSpec: description: |- GMSACredentialSpec is where the GMSA admission webhook (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. type: string gmsaCredentialSpecName: description: GMSACredentialSpecName is the name of the GMSA credential spec to use. type: string hostProcess: description: |- HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. type: boolean runAsUserName: description: |- The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: string type: object type: object x-kubernetes-validations: - message: field is immutable rule: self == oldSelf serviceAccountName: description: serviceAccountName patches the service account for the Pods in the target job templates. maxLength: 253 type: string x-kubernetes-validations: - message: field is immutable rule: self == oldSelf tolerations: description: tolerations patches the Pod's tolerations. items: description: |- The pod this Toleration is attached to tolerates any taint that matches the triple using the matching operator . properties: effect: description: |- Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. type: string key: description: |- Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys. type: string operator: description: |- Operator represents a key's relationship to the value. Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category. Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). type: string tolerationSeconds: description: |- TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system. format: int64 type: integer value: description: |- Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string. type: string type: object maxItems: 128 type: array x-kubernetes-list-type: atomic volumes: description: volumes patches the Pod's volumes. items: description: Volume represents a named volume in a pod that may be accessed by any container in the pod. properties: awsElasticBlockStore: description: |- awsElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Deprecated: AWSElasticBlockStore is deprecated. All operations for the in-tree awsElasticBlockStore type are redirected to the ebs.csi.aws.com CSI driver. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore properties: fsType: description: |- fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore type: string partition: description: |- partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty). format: int32 type: integer readOnly: description: |- readOnly value true will force the readOnly setting in VolumeMounts. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore type: boolean volumeID: description: |- volumeID is unique ID of the persistent disk resource in AWS (Amazon EBS volume). More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore type: string required: - volumeID type: object azureDisk: description: |- azureDisk represents an Azure Data Disk mount on the host and bind mount to the pod. Deprecated: AzureDisk is deprecated. All operations for the in-tree azureDisk type are redirected to the disk.csi.azure.com CSI driver. properties: cachingMode: description: 'cachingMode is the Host Caching mode: None, Read Only, Read Write.' type: string diskName: description: diskName is the Name of the data disk in the blob storage type: string diskURI: description: diskURI is the URI of data disk in the blob storage type: string fsType: default: ext4 description: |- fsType is Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string kind: description: 'kind expected values are Shared: multiple blob disks per storage account Dedicated: single blob disk per storage account Managed: azure managed data disk (only in managed availability set). defaults to shared' type: string readOnly: default: false description: |- readOnly Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean required: - diskName - diskURI type: object azureFile: description: |- azureFile represents an Azure File Service mount on the host and bind mount to the pod. Deprecated: AzureFile is deprecated. All operations for the in-tree azureFile type are redirected to the file.csi.azure.com CSI driver. properties: readOnly: description: |- readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean secretName: description: secretName is the name of secret that contains Azure Storage Account Name and Key type: string shareName: description: shareName is the azure share Name type: string required: - secretName - shareName type: object cephfs: description: |- cephFS represents a Ceph FS mount on the host that shares a pod's lifetime. Deprecated: CephFS is deprecated and the in-tree cephfs type is no longer supported. properties: monitors: description: |- monitors is Required: Monitors is a collection of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it items: type: string type: array x-kubernetes-list-type: atomic path: description: 'path is Optional: Used as the mounted root, rather than the full Ceph tree, default is /' type: string readOnly: description: |- readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it type: boolean secretFile: description: |- secretFile is Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it type: string secretRef: description: |- secretRef is Optional: SecretRef is reference to the authentication secret for User, default is empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic user: description: |- user is optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it type: string required: - monitors type: object cinder: description: |- cinder represents a cinder volume attached and mounted on kubelets host machine. Deprecated: Cinder is deprecated. All operations for the in-tree cinder type are redirected to the cinder.csi.openstack.org CSI driver. More info: https://examples.k8s.io/mysql-cinder-pd/README.md properties: fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md type: string readOnly: description: |- readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md type: boolean secretRef: description: |- secretRef is optional: points to a secret object containing parameters used to connect to OpenStack. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic volumeID: description: |- volumeID used to identify the volume in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md type: string required: - volumeID type: object configMap: description: configMap represents a configMap that should populate this volume properties: defaultMode: description: |- defaultMode is optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer items: description: |- items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. items: description: Maps a string key to a path within a volume. properties: key: description: key is the key to project. type: string mode: description: |- mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: |- path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. type: string required: - key - path type: object type: array x-kubernetes-list-type: atomic name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: optional specify whether the ConfigMap or its keys must be defined type: boolean type: object x-kubernetes-map-type: atomic csi: description: csi (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers. properties: driver: description: |- driver is the name of the CSI driver that handles this volume. Consult with your admin for the correct name as registered in the cluster. type: string fsType: description: |- fsType to mount. Ex. "ext4", "xfs", "ntfs". If not provided, the empty value is passed to the associated CSI driver which will determine the default filesystem to apply. type: string nodePublishSecretRef: description: |- nodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secret references are passed. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic readOnly: description: |- readOnly specifies a read-only configuration for the volume. Defaults to false (read/write). type: boolean volumeAttributes: additionalProperties: type: string description: |- volumeAttributes stores driver-specific properties that are passed to the CSI driver. Consult your driver's documentation for supported values. type: object required: - driver type: object downwardAPI: description: downwardAPI represents downward API about the pod that should populate this volume properties: defaultMode: description: |- Optional: mode bits to use on created files by default. Must be a Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer items: description: Items is a list of downward API volume file items: description: DownwardAPIVolumeFile represents information to create the file containing the pod field properties: fieldRef: description: 'Required: Selects a field of the pod: only annotations, labels, name, namespace and uid are supported.' properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic mode: description: |- Optional: mode bits used to set permissions on this file, must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: 'Required: Path is the relative path name of the file to be created. Must not be absolute or contain the ''..'' path. Must be utf-8 encoded. The first item of the relative path must not start with ''..''' type: string resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, requests.cpu and requests.memory) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic required: - path type: object type: array x-kubernetes-list-type: atomic type: object emptyDir: description: |- emptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir properties: medium: description: |- medium represents what type of storage medium should back this directory. The default is "" which means to use the node's default medium. Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir type: string sizeLimit: anyOf: - type: integer - type: string description: |- sizeLimit is the total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object ephemeral: description: |- ephemeral represents a volume that is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed. Use this if: a) the volume is only needed while the pod runs, b) features of normal volumes like restoring from snapshot or capacity tracking are needed, c) the storage driver is specified through a storage class, and d) the storage driver supports dynamic volume provisioning through a PersistentVolumeClaim (see EphemeralVolumeSource for more information on the connection between this volume type and PersistentVolumeClaim). Use PersistentVolumeClaim or one of the vendor-specific APIs for volumes that persist for longer than the lifecycle of an individual pod. Use CSI for light-weight local ephemeral volumes if the CSI driver is meant to be used that way - see the documentation of the driver for more information. A pod can use both types of ephemeral volumes and persistent volumes at the same time. properties: volumeClaimTemplate: description: |- Will be used to create a stand-alone PVC to provision the volume. The pod in which this EphemeralVolumeSource is embedded will be the owner of the PVC, i.e. the PVC will be deleted together with the pod. The name of the PVC will be `-` where `` is the name from the `PodSpec.Volumes` array entry. Pod validation will reject the pod if the concatenated name is not valid for a PVC (for example, too long). An existing PVC with that name that is not owned by the pod will *not* be used for the pod to avoid using an unrelated volume by mistake. Starting the pod is then blocked until the unrelated PVC is removed. If such a pre-created PVC is meant to be used by the pod, the PVC has to updated with an owner reference to the pod once the pod exists. Normally this should not be necessary, but it may be useful when manually reconstructing a broken cluster. This field is read-only and no changes will be made by Kubernetes to the PVC after it has been created. Required, must not be nil. properties: metadata: description: |- May contain labels and annotations that will be copied into the PVC when creating it. No other fields are allowed and will be rejected during validation. type: object spec: description: |- The specification for the PersistentVolumeClaim. The entire content is copied unchanged into the PVC that gets created from this template. The same fields as in a PersistentVolumeClaim are also valid here. properties: accessModes: description: |- accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1 items: type: string type: array x-kubernetes-list-type: atomic dataSource: description: |- dataSource field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource. properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string required: - kind - name type: object x-kubernetes-map-type: atomic dataSourceRef: description: |- dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the dataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, when namespace isn't specified in dataSourceRef, both fields (dataSource and dataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. When namespace is specified in dataSourceRef, dataSource isn't set to the same value and must be empty. There are three important differences between dataSource and dataSourceRef: * While dataSource only allows two specific types of objects, dataSourceRef allows any non-core object, as well as PersistentVolumeClaim objects. * While dataSource ignores disallowed values (dropping them), dataSourceRef preserves all values, and generates an error if a disallowed value is specified. * While dataSource only allows local objects, dataSourceRef allows objects in any namespaces. (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled. (Alpha) Using the namespace field of dataSourceRef requires the CrossNamespaceVolumeDataSource feature gate to be enabled. properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string namespace: description: |- Namespace is the namespace of resource being referenced Note that when a namespace is specified, a gateway.networking.k8s.io/ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. (Alpha) This field requires the CrossNamespaceVolumeDataSource feature gate to be enabled. type: string required: - kind - name type: object resources: description: |- resources represents the minimum resources the volume should have. Users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources properties: limits: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object requests: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object type: object selector: description: selector is a label query over volumes to consider for binding. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic storageClassName: description: |- storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 type: string volumeAttributesClassName: description: |- volumeAttributesClassName may be used to set the VolumeAttributesClass used by this claim. If specified, the CSI driver will create or update the volume with the attributes defined in the corresponding VolumeAttributesClass. This has a different purpose than storageClassName, it can be changed after the claim is created. An empty string or nil value indicates that no VolumeAttributesClass will be applied to the claim. If the claim enters an Infeasible error state, this field can be reset to its previous value (including nil) to cancel the modification. If the resource referred to by volumeAttributesClass does not exist, this PersistentVolumeClaim will be set to a Pending state, as reflected by the modifyVolumeStatus field, until such as a resource exists. More info: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/ type: string volumeMode: description: |- volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec. type: string volumeName: description: volumeName is the binding reference to the PersistentVolume backing this claim. type: string type: object required: - spec type: object type: object fc: description: fc represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod. properties: fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string lun: description: 'lun is Optional: FC target lun number' format: int32 type: integer readOnly: description: |- readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean targetWWNs: description: 'targetWWNs is Optional: FC target worldwide names (WWNs)' items: type: string type: array x-kubernetes-list-type: atomic wwids: description: |- wwids Optional: FC volume world wide identifiers (wwids) Either wwids or combination of targetWWNs and lun must be set, but not both simultaneously. items: type: string type: array x-kubernetes-list-type: atomic type: object flexVolume: description: |- flexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. Deprecated: FlexVolume is deprecated. Consider using a CSIDriver instead. properties: driver: description: driver is the name of the driver to use for this volume. type: string fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". The default filesystem depends on FlexVolume script. type: string options: additionalProperties: type: string description: 'options is Optional: this field holds extra command options if any.' type: object readOnly: description: |- readOnly is Optional: defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean secretRef: description: |- secretRef is Optional: secretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic required: - driver type: object flocker: description: |- flocker represents a Flocker volume attached to a kubelet's host machine. This depends on the Flocker control service being running. Deprecated: Flocker is deprecated and the in-tree flocker type is no longer supported. properties: datasetName: description: |- datasetName is Name of the dataset stored as metadata -> name on the dataset for Flocker should be considered as deprecated type: string datasetUUID: description: datasetUUID is the UUID of the dataset. This is unique identifier of a Flocker dataset type: string type: object gcePersistentDisk: description: |- gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Deprecated: GCEPersistentDisk is deprecated. All operations for the in-tree gcePersistentDisk type are redirected to the pd.csi.storage.gke.io CSI driver. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk properties: fsType: description: |- fsType is filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk type: string partition: description: |- partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty). More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk format: int32 type: integer pdName: description: |- pdName is unique name of the PD resource in GCE. Used to identify the disk in GCE. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk type: string readOnly: description: |- readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk type: boolean required: - pdName type: object gitRepo: description: |- gitRepo represents a git repository at a particular revision. Deprecated: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. properties: directory: description: |- directory is the target directory name. Must not contain or start with '..'. If '.' is supplied, the volume directory will be the git repository. Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name. type: string repository: description: repository is the URL type: string revision: description: revision is the commit hash for the specified revision. type: string required: - repository type: object glusterfs: description: |- glusterfs represents a Glusterfs mount on the host that shares a pod's lifetime. Deprecated: Glusterfs is deprecated and the in-tree glusterfs type is no longer supported. properties: endpoints: description: endpoints is the endpoint name that details Glusterfs topology. type: string path: description: |- path is the Glusterfs volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod type: string readOnly: description: |- readOnly here will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod type: boolean required: - endpoints - path type: object hostPath: description: |- hostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath properties: path: description: |- path of the directory on the host. If the path is a symlink, it will follow the link to the real path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath type: string type: description: |- type for HostPath Volume Defaults to "" More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath type: string required: - path type: object image: description: |- image represents an OCI object (a container image or artifact) pulled and mounted on the kubelet's host machine. The volume is resolved at pod startup depending on which PullPolicy value is provided: - Always: the kubelet always attempts to pull the reference. Container creation will fail If the pull fails. - Never: the kubelet never pulls the reference and only uses a local image or artifact. Container creation will fail if the reference isn't present. - IfNotPresent: the kubelet pulls if the reference isn't already present on disk. Container creation will fail if the reference isn't present and the pull fails. The volume gets re-resolved if the pod gets deleted and recreated, which means that new remote content will become available on pod recreation. A failure to resolve or pull the image during pod startup will block containers from starting and may add significant latency. Failures will be retried using normal volume backoff and will be reported on the pod reason and message. The types of objects that may be mounted by this volume are defined by the container runtime implementation on a host machine and at minimum must include all valid types supported by the container image field. The OCI object gets mounted in a single directory (spec.containers[*].volumeMounts.mountPath) by merging the manifest layers in the same way as for container images. The volume will be mounted read-only (ro). Sub path mounts for containers are not supported (spec.containers[*].volumeMounts.subpath) before 1.33. The field spec.securityContext.fsGroupChangePolicy has no effect on this volume type. properties: pullPolicy: description: |- Policy for pulling OCI objects. Possible values are: Always: the kubelet always attempts to pull the reference. Container creation will fail If the pull fails. Never: the kubelet never pulls the reference and only uses a local image or artifact. Container creation will fail if the reference isn't present. IfNotPresent: the kubelet pulls if the reference isn't already present on disk. Container creation will fail if the reference isn't present and the pull fails. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. type: string reference: description: |- Required: Image or artifact reference to be used. Behaves in the same way as pod.spec.containers[*].image. Pull secrets will be assembled in the same way as for the container image by looking up node credentials, SA image pull secrets, and pod spec image pull secrets. More info: https://kubernetes.io/docs/concepts/containers/images This field is optional to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets. type: string type: object iscsi: description: |- iscsi represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes/#iscsi properties: chapAuthDiscovery: description: chapAuthDiscovery defines whether support iSCSI Discovery CHAP authentication type: boolean chapAuthSession: description: chapAuthSession defines whether support iSCSI Session CHAP authentication type: boolean fsType: description: |- fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi type: string initiatorName: description: |- initiatorName is the custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterface simultaneously, new iSCSI interface : will be created for the connection. type: string iqn: description: iqn is the target iSCSI Qualified Name. type: string iscsiInterface: default: default description: |- iscsiInterface is the interface Name that uses an iSCSI transport. Defaults to 'default' (tcp). type: string lun: description: lun represents iSCSI Target Lun number. format: int32 type: integer portals: description: |- portals is the iSCSI Target Portal List. The portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260). items: type: string type: array x-kubernetes-list-type: atomic readOnly: description: |- readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. type: boolean secretRef: description: secretRef is the CHAP Secret for iSCSI target and initiator authentication properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic targetPortal: description: |- targetPortal is iSCSI Target Portal. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260). type: string required: - iqn - lun - targetPortal type: object name: description: |- name of the volume. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string nfs: description: |- nfs represents an NFS mount on the host that shares a pod's lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs properties: path: description: |- path that is exported by the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs type: string readOnly: description: |- readOnly here will force the NFS export to be mounted with read-only permissions. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs type: boolean server: description: |- server is the hostname or IP address of the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs type: string required: - path - server type: object persistentVolumeClaim: description: |- persistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims properties: claimName: description: |- claimName is the name of a PersistentVolumeClaim in the same namespace as the pod using this volume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims type: string readOnly: description: |- readOnly Will force the ReadOnly setting in VolumeMounts. Default false. type: boolean required: - claimName type: object photonPersistentDisk: description: |- photonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine. Deprecated: PhotonPersistentDisk is deprecated and the in-tree photonPersistentDisk type is no longer supported. properties: fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string pdID: description: pdID is the ID that identifies Photon Controller persistent disk type: string required: - pdID type: object portworxVolume: description: |- portworxVolume represents a portworx volume attached and mounted on kubelets host machine. Deprecated: PortworxVolume is deprecated. All operations for the in-tree portworxVolume type are redirected to the pxd.portworx.com CSI driver. properties: fsType: description: |- fSType represents the filesystem type to mount Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs". Implicitly inferred to be "ext4" if unspecified. type: string readOnly: description: |- readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean volumeID: description: volumeID uniquely identifies a Portworx volume type: string required: - volumeID type: object projected: description: projected items for all in one resources secrets, configmaps, and downward API properties: defaultMode: description: |- defaultMode are the mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer sources: description: |- sources is the list of volume projections. Each entry in this list handles one source. items: description: |- Projection that may be projected along with other supported volume types. Exactly one of these fields must be set. properties: clusterTrustBundle: description: |- ClusterTrustBundle allows a pod to access the `.spec.trustBundle` field of ClusterTrustBundle objects in an auto-updating file. Alpha, gated by the ClusterTrustBundleProjection feature gate. ClusterTrustBundle objects can either be selected by name, or by the combination of signer name and a label selector. Kubelet performs aggressive normalization of the PEM contents written into the pod filesystem. Esoteric PEM features such as inter-block comments and block headers are stripped. Certificates are deduplicated. The ordering of certificates within the file is arbitrary, and Kubelet may change the order over time. properties: labelSelector: description: |- Select all ClusterTrustBundles that match this label selector. Only has effect if signerName is set. Mutually-exclusive with name. If unset, interpreted as "match nothing". If set but empty, interpreted as "match everything". properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic name: description: |- Select a single ClusterTrustBundle by object name. Mutually-exclusive with signerName and labelSelector. type: string optional: description: |- If true, don't block pod startup if the referenced ClusterTrustBundle(s) aren't available. If using name, then the named ClusterTrustBundle is allowed not to exist. If using signerName, then the combination of signerName and labelSelector is allowed to match zero ClusterTrustBundles. type: boolean path: description: Relative path from the volume root to write the bundle. type: string signerName: description: |- Select all ClusterTrustBundles that match this signer name. Mutually-exclusive with name. The contents of all selected ClusterTrustBundles will be unified and deduplicated. type: string required: - path type: object configMap: description: configMap information about the configMap data to project properties: items: description: |- items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. items: description: Maps a string key to a path within a volume. properties: key: description: key is the key to project. type: string mode: description: |- mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: |- path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. type: string required: - key - path type: object type: array x-kubernetes-list-type: atomic name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: optional specify whether the ConfigMap or its keys must be defined type: boolean type: object x-kubernetes-map-type: atomic downwardAPI: description: downwardAPI information about the downwardAPI data to project properties: items: description: Items is a list of DownwardAPIVolume file items: description: DownwardAPIVolumeFile represents information to create the file containing the pod field properties: fieldRef: description: 'Required: Selects a field of the pod: only annotations, labels, name, namespace and uid are supported.' properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic mode: description: |- Optional: mode bits used to set permissions on this file, must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: 'Required: Path is the relative path name of the file to be created. Must not be absolute or contain the ''..'' path. Must be utf-8 encoded. The first item of the relative path must not start with ''..''' type: string resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, requests.cpu and requests.memory) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic required: - path type: object type: array x-kubernetes-list-type: atomic type: object podCertificate: description: |- Projects an auto-rotating credential bundle (private key and certificate chain) that the pod can use either as a TLS client or server. Kubelet generates a private key and uses it to send a PodCertificateRequest to the named signer. Once the signer approves the request and issues a certificate chain, Kubelet writes the key and certificate chain to the pod filesystem. The pod does not start until certificates have been issued for each podCertificate projected volume source in its spec. Kubelet will begin trying to rotate the certificate at the time indicated by the signer using the PodCertificateRequest.Status.BeginRefreshAt timestamp. Kubelet can write a single file, indicated by the credentialBundlePath field, or separate files, indicated by the keyPath and certificateChainPath fields. The credential bundle is a single file in PEM format. The first PEM entry is the private key (in PKCS#8 format), and the remaining PEM entries are the certificate chain issued by the signer (typically, signers will return their certificate chain in leaf-to-root order). Prefer using the credential bundle format, since your application code can read it atomically. If you use keyPath and certificateChainPath, your application must make two separate file reads. If these coincide with a certificate rotation, it is possible that the private key and leaf certificate you read may not correspond to each other. Your application will need to check for this condition, and re-read until they are consistent. The named signer controls chooses the format of the certificate it issues; consult the signer implementation's documentation to learn how to use the certificates it issues. properties: certificateChainPath: description: |- Write the certificate chain at this path in the projected volume. Most applications should use credentialBundlePath. When using keyPath and certificateChainPath, your application needs to check that the key and leaf certificate are consistent, because it is possible to read the files mid-rotation. type: string credentialBundlePath: description: |- Write the credential bundle at this path in the projected volume. The credential bundle is a single file that contains multiple PEM blocks. The first PEM block is a PRIVATE KEY block, containing a PKCS#8 private key. The remaining blocks are CERTIFICATE blocks, containing the issued certificate chain from the signer (leaf and any intermediates). Using credentialBundlePath lets your Pod's application code make a single atomic read that retrieves a consistent key and certificate chain. If you project them to separate files, your application code will need to additionally check that the leaf certificate was issued to the key. type: string keyPath: description: |- Write the key at this path in the projected volume. Most applications should use credentialBundlePath. When using keyPath and certificateChainPath, your application needs to check that the key and leaf certificate are consistent, because it is possible to read the files mid-rotation. type: string keyType: description: |- The type of keypair Kubelet will generate for the pod. Valid values are "RSA3072", "RSA4096", "ECDSAP256", "ECDSAP384", "ECDSAP521", and "ED25519". type: string maxExpirationSeconds: description: |- maxExpirationSeconds is the maximum lifetime permitted for the certificate. Kubelet copies this value verbatim into the PodCertificateRequests it generates for this projection. If omitted, kube-apiserver will set it to 86400(24 hours). kube-apiserver will reject values shorter than 3600 (1 hour). The maximum allowable value is 7862400 (91 days). The signer implementation is then free to issue a certificate with any lifetime *shorter* than MaxExpirationSeconds, but no shorter than 3600 seconds (1 hour). This constraint is enforced by kube-apiserver. `kubernetes.io` signers will never issue certificates with a lifetime longer than 24 hours. format: int32 type: integer signerName: description: Kubelet's generated CSRs will be addressed to this signer. type: string userAnnotations: additionalProperties: type: string description: |- userAnnotations allow pod authors to pass additional information to the signer implementation. Kubernetes does not restrict or validate this metadata in any way. These values are copied verbatim into the `spec.unverifiedUserAnnotations` field of the PodCertificateRequest objects that Kubelet creates. Entries are subject to the same validation as object metadata annotations, with the addition that all keys must be domain-prefixed. No restrictions are placed on values, except an overall size limitation on the entire field. Signers should document the keys and values they support. Signers should deny requests that contain keys they do not recognize. type: object required: - keyType - signerName type: object secret: description: secret information about the secret data to project properties: items: description: |- items if unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. items: description: Maps a string key to a path within a volume. properties: key: description: key is the key to project. type: string mode: description: |- mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: |- path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. type: string required: - key - path type: object type: array x-kubernetes-list-type: atomic name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: optional field specify whether the Secret or its key must be defined type: boolean type: object x-kubernetes-map-type: atomic serviceAccountToken: description: serviceAccountToken is information about the serviceAccountToken data to project properties: audience: description: |- audience is the intended audience of the token. A recipient of a token must identify itself with an identifier specified in the audience of the token, and otherwise should reject the token. The audience defaults to the identifier of the apiserver. type: string expirationSeconds: description: |- expirationSeconds is the requested duration of validity of the service account token. As the token approaches expiration, the kubelet volume plugin will proactively rotate the service account token. The kubelet will start trying to rotate the token if the token is older than 80 percent of its time to live or if the token is older than 24 hours.Defaults to 1 hour and must be at least 10 minutes. format: int64 type: integer path: description: |- path is the path relative to the mount point of the file to project the token into. type: string required: - path type: object type: object type: array x-kubernetes-list-type: atomic type: object quobyte: description: |- quobyte represents a Quobyte mount on the host that shares a pod's lifetime. Deprecated: Quobyte is deprecated and the in-tree quobyte type is no longer supported. properties: group: description: |- group to map volume access to Default is no group type: string readOnly: description: |- readOnly here will force the Quobyte volume to be mounted with read-only permissions. Defaults to false. type: boolean registry: description: |- registry represents a single or multiple Quobyte Registry services specified as a string as host:port pair (multiple entries are separated with commas) which acts as the central registry for volumes type: string tenant: description: |- tenant owning the given Quobyte volume in the Backend Used with dynamically provisioned Quobyte volumes, value is set by the plugin type: string user: description: |- user to map volume access to Defaults to serivceaccount user type: string volume: description: volume is a string that references an already created Quobyte volume by name. type: string required: - registry - volume type: object rbd: description: |- rbd represents a Rados Block Device mount on the host that shares a pod's lifetime. Deprecated: RBD is deprecated and the in-tree rbd type is no longer supported. properties: fsType: description: |- fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd type: string image: description: |- image is the rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: string keyring: default: /etc/ceph/keyring description: |- keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: string monitors: description: |- monitors is a collection of Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it items: type: string type: array x-kubernetes-list-type: atomic pool: default: rbd description: |- pool is the rados pool name. Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: string readOnly: description: |- readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: boolean secretRef: description: |- secretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic user: default: admin description: |- user is the rados user name. Default is admin. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: string required: - image - monitors type: object scaleIO: description: |- scaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes. Deprecated: ScaleIO is deprecated and the in-tree scaleIO type is no longer supported. properties: fsType: default: xfs description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Default is "xfs". type: string gateway: description: gateway is the host address of the ScaleIO API Gateway. type: string protectionDomain: description: protectionDomain is the name of the ScaleIO Protection Domain for the configured storage. type: string readOnly: description: |- readOnly Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean secretRef: description: |- secretRef references to the secret for ScaleIO user and other sensitive information. If this is not provided, Login operation will fail. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic sslEnabled: description: sslEnabled Flag enable/disable SSL communication with Gateway, default false type: boolean storageMode: default: ThinProvisioned description: |- storageMode indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is ThinProvisioned. type: string storagePool: description: storagePool is the ScaleIO Storage Pool associated with the protection domain. type: string system: description: system is the name of the storage system as configured in ScaleIO. type: string volumeName: description: |- volumeName is the name of a volume already created in the ScaleIO system that is associated with this volume source. type: string required: - gateway - secretRef - system type: object secret: description: |- secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret properties: defaultMode: description: |- defaultMode is Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer items: description: |- items If unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. items: description: Maps a string key to a path within a volume. properties: key: description: key is the key to project. type: string mode: description: |- mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: |- path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. type: string required: - key - path type: object type: array x-kubernetes-list-type: atomic optional: description: optional field specify whether the Secret or its keys must be defined type: boolean secretName: description: |- secretName is the name of the secret in the pod's namespace to use. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret type: string type: object storageos: description: |- storageOS represents a StorageOS volume attached and mounted on Kubernetes nodes. Deprecated: StorageOS is deprecated and the in-tree storageos type is no longer supported. properties: fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string readOnly: description: |- readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean secretRef: description: |- secretRef specifies the secret to use for obtaining the StorageOS API credentials. If not specified, default values will be attempted. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic volumeName: description: |- volumeName is the human-readable name of the StorageOS volume. Volume names are only unique within a namespace. type: string volumeNamespace: description: |- volumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created. type: string type: object vsphereVolume: description: |- vsphereVolume represents a vSphere volume attached and mounted on kubelets host machine. Deprecated: VsphereVolume is deprecated. All operations for the in-tree vsphereVolume type are redirected to the csi.vsphere.vmware.com CSI driver. properties: fsType: description: |- fsType is filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string storagePolicyID: description: storagePolicyID is the storage Policy Based Management (SPBM) profile ID associated with the StoragePolicyName. type: string storagePolicyName: description: storagePolicyName is the storage Policy Based Management (SPBM) profile name. type: string volumePath: description: volumePath is the path that identifies vSphere volume vmdk type: string required: - volumePath type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map x-kubernetes-validations: - message: field is immutable rule: self == oldSelf type: object type: object type: object type: object required: - name type: object maxItems: 100 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map type: object type: object type: object required: - manager type: object maxItems: 16 type: array x-kubernetes-list-map-keys: - manager x-kubernetes-list-type: map runtimeRef: description: runtimeRef is the reference to the training runtime. properties: apiGroup: default: trainer.kubeflow.org description: |- apiGroup of the runtime being referenced. Defaults to `trainer.kubeflow.org`. maxLength: 253 type: string kind: default: ClusterTrainingRuntime description: |- kind of the runtime being referenced. Defaults to ClusterTrainingRuntime. maxLength: 253 type: string name: description: |- name of the runtime being referenced. When namespaced-scoped TrainingRuntime is used, the TrainJob must have the same namespace as the deployed runtime. maxLength: 253 minLength: 1 type: string required: - name type: object x-kubernetes-validations: - message: field is immutable rule: self == oldSelf suspend: default: false description: suspend defines whether to suspend the running TrainJob. type: boolean trainer: description: trainer defines the configuration of the trainer. properties: args: description: args for the entrypoint for the training container. items: maxLength: 65536 type: string maxItems: 128 type: array x-kubernetes-list-type: atomic command: description: command for the entrypoint of the training container. items: maxLength: 65536 type: string maxItems: 128 type: array x-kubernetes-list-type: atomic env: description: |- env is the list of environment variables to set in the training container. These values will be merged with the TrainingRuntime's trainer environments. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map image: description: image is the container image for the training container. maxLength: 500 type: string numNodes: description: numNodes is the number of training nodes. format: int32 type: integer numProcPerNode: description: |- numProcPerNode is the number of processes/workers/slots on every training node. For the MPI runtime only int value can be set to represent number of slots per node. For the Torch runtime the value defaults to `auto` and can be overridden with an int. format: int32 type: integer resourcesPerNode: description: resourcesPerNode defines the compute resources for each training node. properties: claims: description: |- Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container. This field depends on the DynamicResourceAllocation feature gate. This field is immutable. It can only be set for containers. items: description: ResourceClaim references one entry in PodSpec.ResourceClaims. properties: name: description: |- Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. type: string request: description: |- Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. type: string required: - name type: object type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map limits: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object requests: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object type: object type: object x-kubernetes-validations: - message: field is immutable rule: self == oldSelf required: - runtimeRef type: object type: object x-kubernetes-validations: - message: workload is immutable rule: self == oldSelf workloadMetadata: description: |- workloadMetadata sets labels on the workload object created from spec.workload. This is the canonical boundary for workload-object labels: WorkloadRun and Certification both resolve their own workloadMetadata into this field, and a hand-authored Workflow writes spec.jobTemplate.spec.workloadMetadata directly. Ordinary Job labels (Workflow spec.jobTemplate.metadata.labels) are not copied here. Labels are executable policy, so opting a workload into an admission controller or a queue has to be written down explicitly. Immutable in both presence and value, so a checkpoint restart recreates the workload with the labels it was admitted with. properties: labels: additionalProperties: description: |- WorkloadLabelValue is a Kubernetes label value in a WorkloadMetadata labels map. It exists as a named type only so controller-gen emits the length and pattern constraints under the map's additionalProperties: there is no marker that attaches value constraints to a map[string]string. maxLength: 63 pattern: ^$|^[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$ type: string description: |- labels are merged onto the generated workload object's metadata.labels. Keys must be valid Kubernetes label keys: an optional DNS-subdomain prefix of at most 253 characters followed by "/", then a name of at most 63 characters. Values follow Kubernetes label-value syntax and may be empty; an empty value is a real label, not a deletion. The controller-owned keys "app.kubernetes.io/managed-by" and anything under the "nvcre.nvidia.com/" prefix are rejected rather than silently overwritten — they identify controller ownership and Job association. At most 32 entries. That bound is an enforced API limit, chosen as the product decision for how many workload labels one object may carry. (A finite maxProperties is separately required to bound the static cost of the CEL rules below, but the cost estimator accepts far larger values, so it does not pick this number.) The key grammar uses [.] rather than \. because the marker parser collapses the escape and CEL then rejects \. as an invalid escape in a string literal. The two are equivalent to the regex engine. maxProperties: 32 type: object x-kubernetes-validations: - message: labels must not set the controller-owned keys app.kubernetes.io/managed-by or any key under the nvcre.nvidia.com/ prefix rule: self.all(k, k != 'app.kubernetes.io/managed-by' && !k.startsWith('nvcre.nvidia.com/')) - message: 'each label key must be a valid Kubernetes label key: an optional DNS-subdomain prefix followed by ''/'', then a name of at most 63 characters beginning and ending with an alphanumeric character' rule: self.all(k, k.matches('^([a-z0-9]([-a-z0-9]*[a-z0-9])?([.][a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$')) - message: each label key prefix must be at most 253 characters rule: 'self.all(k, k.contains(''/'') ? k.split(''/'')[0].size() <= 253 : true)' type: object x-kubernetes-validations: - message: workloadMetadata is immutable rule: self == oldSelf required: - workload type: object x-kubernetes-validations: - message: workloadMetadata cannot be added or removed after creation rule: has(self.workloadMetadata) == has(oldSelf.workloadMetadata) - message: nodeHealthMonitor cannot be added or removed after creation rule: has(self.nodeHealthMonitor) == has(oldSelf.nodeHealthMonitor) - message: goodputMeasurement cannot be added or removed after creation rule: has(self.goodputMeasurement) == has(oldSelf.goodputMeasurement) - message: bandwidthMeasurement cannot be added or removed after creation rule: has(self.bandwidthMeasurement) == has(oldSelf.bandwidthMeasurement) status: description: status defines the observed state of Job properties: conditions: description: |- conditions represent the current state of the Job. Only one of the following conditions can be True at any given time: - "InProgress": the Job is currently running - "Succeeded": the Job has completed successfully - "Failed": the Job has failed - "HardwareFailed": a hardware failure was detected on a node The condition message contains details about the current state for debugging. items: description: Condition contains details for one aspect of the current state of this API Resource. properties: lastTransitionTime: description: |- lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. format: date-time type: string message: description: |- message is a human readable message indicating details about the transition. This may be an empty string. maxLength: 32768 type: string observedGeneration: description: |- observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. format: int64 minimum: 0 type: integer reason: description: |- reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty. maxLength: 1024 minLength: 1 pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ type: string status: description: status of the condition, one of True, False, Unknown. enum: - "True" - "False" - Unknown type: string type: description: type of condition in CamelCase or in foo.example.com/CamelCase. maxLength: 316 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ type: string required: - lastTransitionTime - message - reason - status - type type: object type: array x-kubernetes-list-map-keys: - type x-kubernetes-list-type: map failedNodes: description: |- failedNodes lists nodes that failed during this Job, each with the reason for failure. Populated when the Job fails, a hardware failure is detected, or a performance threshold is violated. A node may appear multiple times with different reasons; entries are keyed by the (name, reason) pair. items: properties: message: description: message is the detailed failure message for this node, matching the originating Job condition message. type: string name: description: name is the Kubernetes node name. type: string reason: description: |- reason is the failure type for this node, matching the originating Job condition reason. - HardwareFailureDetected: a node health check (CEL) flagged the node. - ThresholdViolation: a performance threshold (bandwidth/goodput) was not met. - WorkloadFailed: the workload exited non-zero, stalled, or otherwise failed. enum: - HardwareFailureDetected - ThresholdViolation - WorkloadFailed type: string required: - name - reason type: object type: array x-kubernetes-list-map-keys: - name - reason x-kubernetes-list-type: map failureLog: description: |- failureLog captures the tail of pod logs from the most recent workload failure. Populated when the Job transitions to Failed. Only the pod that caused the failure (earliest non-zero exit code) is captured. Overwritten on each retry. Always set on failure: when no pod was available to read, reason and tail record why rather than leaving the field unset. properties: exitCode: description: exitCode is the container's exit code. format: int32 type: integer nodeName: description: nodeName is the node the failed pod was running on. type: string podName: description: podName is the name of the pod that failed. type: string reason: description: reason is the termination reason (e.g., "OOMKilled", "Error"). type: string tail: description: |- tail is the end of the failed container's logs, up to 32 KB. When no pod was available to read, this explains why instead. type: string required: - exitCode - nodeName - podName type: object restartCount: description: restartCount tracks the number of times the workload has been restarted from checkpoint. format: int32 type: integer workloadRef: description: |- workloadRef references the workload resource created by this Job. Populated after the workload is created. Used to track workload status. properties: apiVersion: description: apiVersion is the API version of the workload resource. type: string kind: description: kind is the kind of the workload resource. type: string name: description: name is the name of the workload resource. type: string namespace: description: |- namespace is the namespace of the workload resource. If not specified, defaults to the Job's namespace. type: string required: - apiVersion - kind - name type: object workloadStartTime: description: |- workloadStartTime is when the workload was first observed running rather than pending. A workload held by an admission controller (for example a TrainJob suspended by Kueue until quota is available) is pending, and its queued time does not count against timeoutPerJob or stall detection: timeoutPerJob is measured from this timestamp, and the stall clock never starts before it. Cleared on checkpoint restart so the replacement workload gets a fresh budget. format: date-time type: string type: object required: - spec type: object served: true storage: true subresources: status: {} --- # Source: cluster-readiness-engine/crds/nvcre.nvidia.com_logprofiles.yaml --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.20.0 name: logprofiles.nvcre.nvidia.com spec: group: nvcre.nvidia.com names: kind: LogProfile listKind: LogProfileList plural: logprofiles singular: logprofile scope: Cluster versions: - name: v1alpha1 schema: openAPIV3Schema: description: |- LogProfile is the Schema for the logprofiles API. A cluster-scoped resource that defines regex patterns for parsing training logs of a specific framework (e.g., Nemotron 4/NeMo, Nemotron 6/Megatron). properties: apiVersion: description: |- APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: description: |- Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object spec: description: spec defines the log parsing configuration properties: containerName: description: |- containerName is the container to read logs from. If empty, the controller auto-detects the container name from the workload. type: string logInterval: description: |- logInterval is the training framework's log interval (e.g., 10 for --log-interval 10). When set, training steps whose GlobalStep is not a multiple of this value are filtered out. This removes inflated "first after boundary" log lines that include startup or restart overhead. Also used as the default step delta for the first step in a window when computing total training time. type: integer patterns: description: patterns defines regex patterns for extracting training events from log lines. properties: applicationStart: description: |- applicationStart matches a line indicating the application has started. Well-known captures: timestamp properties: example: description: |- example is a sample log line that the regex should match. Serves as documentation and is validated when the LogProfile is loaded. type: string regex: description: |- regex is the regular expression with Go named capture groups. Uses (?P...) syntax to name extracted fields. Well-known capture names used by the goodput calculator: Common: timestamp Training: globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTime Checkpoint: step, path, saveDuration The controller parses named captures into typed values: int: globalStep, iteration, epoch, step float64: stepTiming, loss, tflops, elapsedTime, saveDuration string: path time: timestamp (parsed using LogProfile.spec.timestamp.layout) minLength: 1 type: string units: additionalProperties: type: string description: |- units specifies the unit for duration/timing captures. The controller normalizes all durations to seconds internally. Supported values: "s" (seconds), "ms" (milliseconds), "us" (microseconds). Only needed for duration-type captures: stepTiming, elapsedTime, saveDuration. If a duration capture is not listed here, it defaults to "s" (seconds). Example: {"elapsedTime": "ms"} means the captured value is in milliseconds. type: object required: - regex type: object bandwidthResult: description: |- bandwidthResult matches NCCL bandwidth test result lines. Used by the BandwidthMeasurement controller (not the GoodputMeasurement controller). Well-known captures: size (int, bytes), algBW (float, GB/s), busBW (float, GB/s) properties: example: description: |- example is a sample log line that the regex should match. Serves as documentation and is validated when the LogProfile is loaded. type: string regex: description: |- regex is the regular expression with Go named capture groups. Uses (?P...) syntax to name extracted fields. Well-known capture names used by the goodput calculator: Common: timestamp Training: globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTime Checkpoint: step, path, saveDuration The controller parses named captures into typed values: int: globalStep, iteration, epoch, step float64: stepTiming, loss, tflops, elapsedTime, saveDuration string: path time: timestamp (parsed using LogProfile.spec.timestamp.layout) minLength: 1 type: string units: additionalProperties: type: string description: |- units specifies the unit for duration/timing captures. The controller normalizes all durations to seconds internally. Supported values: "s" (seconds), "ms" (milliseconds), "us" (microseconds). Only needed for duration-type captures: stepTiming, elapsedTime, saveDuration. If a duration capture is not listed here, it defaults to "s" (seconds). Example: {"elapsedTime": "ms"} means the captured value is in milliseconds. type: object required: - regex type: object checkpointDone: description: |- checkpointDone matches checkpoint save completion lines. Well-known captures: timestamp, step, path properties: example: description: |- example is a sample log line that the regex should match. Serves as documentation and is validated when the LogProfile is loaded. type: string regex: description: |- regex is the regular expression with Go named capture groups. Uses (?P...) syntax to name extracted fields. Well-known capture names used by the goodput calculator: Common: timestamp Training: globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTime Checkpoint: step, path, saveDuration The controller parses named captures into typed values: int: globalStep, iteration, epoch, step float64: stepTiming, loss, tflops, elapsedTime, saveDuration string: path time: timestamp (parsed using LogProfile.spec.timestamp.layout) minLength: 1 type: string units: additionalProperties: type: string description: |- units specifies the unit for duration/timing captures. The controller normalizes all durations to seconds internally. Supported values: "s" (seconds), "ms" (milliseconds), "us" (microseconds). Only needed for duration-type captures: stepTiming, elapsedTime, saveDuration. If a duration capture is not listed here, it defaults to "s" (seconds). Example: {"elapsedTime": "ms"} means the captured value is in milliseconds. type: object required: - regex type: object checkpointLoaded: description: |- checkpointLoaded matches checkpoint load completed lines. Well-known captures: timestamp, path, step properties: example: description: |- example is a sample log line that the regex should match. Serves as documentation and is validated when the LogProfile is loaded. type: string regex: description: |- regex is the regular expression with Go named capture groups. Uses (?P...) syntax to name extracted fields. Well-known capture names used by the goodput calculator: Common: timestamp Training: globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTime Checkpoint: step, path, saveDuration The controller parses named captures into typed values: int: globalStep, iteration, epoch, step float64: stepTiming, loss, tflops, elapsedTime, saveDuration string: path time: timestamp (parsed using LogProfile.spec.timestamp.layout) minLength: 1 type: string units: additionalProperties: type: string description: |- units specifies the unit for duration/timing captures. The controller normalizes all durations to seconds internally. Supported values: "s" (seconds), "ms" (milliseconds), "us" (microseconds). Only needed for duration-type captures: stepTiming, elapsedTime, saveDuration. If a duration capture is not listed here, it defaults to "s" (seconds). Example: {"elapsedTime": "ms"} means the captured value is in milliseconds. type: object required: - regex type: object checkpointRestore: description: |- checkpointRestore matches checkpoint restore (load start) lines. Well-known captures: timestamp, path, step properties: example: description: |- example is a sample log line that the regex should match. Serves as documentation and is validated when the LogProfile is loaded. type: string regex: description: |- regex is the regular expression with Go named capture groups. Uses (?P...) syntax to name extracted fields. Well-known capture names used by the goodput calculator: Common: timestamp Training: globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTime Checkpoint: step, path, saveDuration The controller parses named captures into typed values: int: globalStep, iteration, epoch, step float64: stepTiming, loss, tflops, elapsedTime, saveDuration string: path time: timestamp (parsed using LogProfile.spec.timestamp.layout) minLength: 1 type: string units: additionalProperties: type: string description: |- units specifies the unit for duration/timing captures. The controller normalizes all durations to seconds internally. Supported values: "s" (seconds), "ms" (milliseconds), "us" (microseconds). Only needed for duration-type captures: stepTiming, elapsedTime, saveDuration. If a duration capture is not listed here, it defaults to "s" (seconds). Example: {"elapsedTime": "ms"} means the captured value is in milliseconds. type: object required: - regex type: object checkpointSave: description: |- checkpointSave matches checkpoint save start lines. Well-known captures: timestamp, step, path, saveDuration properties: example: description: |- example is a sample log line that the regex should match. Serves as documentation and is validated when the LogProfile is loaded. type: string regex: description: |- regex is the regular expression with Go named capture groups. Uses (?P...) syntax to name extracted fields. Well-known capture names used by the goodput calculator: Common: timestamp Training: globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTime Checkpoint: step, path, saveDuration The controller parses named captures into typed values: int: globalStep, iteration, epoch, step float64: stepTiming, loss, tflops, elapsedTime, saveDuration string: path time: timestamp (parsed using LogProfile.spec.timestamp.layout) minLength: 1 type: string units: additionalProperties: type: string description: |- units specifies the unit for duration/timing captures. The controller normalizes all durations to seconds internally. Supported values: "s" (seconds), "ms" (milliseconds), "us" (microseconds). Only needed for duration-type captures: stepTiming, elapsedTime, saveDuration. If a duration capture is not listed here, it defaults to "s" (seconds). Example: {"elapsedTime": "ms"} means the captured value is in milliseconds. type: object required: - regex type: object trainingStep: description: |- trainingStep matches training iteration log lines. Well-known captures: timestamp, globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTime properties: example: description: |- example is a sample log line that the regex should match. Serves as documentation and is validated when the LogProfile is loaded. type: string regex: description: |- regex is the regular expression with Go named capture groups. Uses (?P...) syntax to name extracted fields. Well-known capture names used by the goodput calculator: Common: timestamp Training: globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTime Checkpoint: step, path, saveDuration The controller parses named captures into typed values: int: globalStep, iteration, epoch, step float64: stepTiming, loss, tflops, elapsedTime, saveDuration string: path time: timestamp (parsed using LogProfile.spec.timestamp.layout) minLength: 1 type: string units: additionalProperties: type: string description: |- units specifies the unit for duration/timing captures. The controller normalizes all durations to seconds internally. Supported values: "s" (seconds), "ms" (milliseconds), "us" (microseconds). Only needed for duration-type captures: stepTiming, elapsedTime, saveDuration. If a duration capture is not listed here, it defaults to "s" (seconds). Example: {"elapsedTime": "ms"} means the captured value is in milliseconds. type: object required: - regex type: object warmupStep: description: |- warmupStep matches lines that indicate a warmup/startup iteration. When a training step line also matches this pattern, the step is flagged as warmup and excluded from average step time computation. Well-known captures: none required (presence of match is sufficient). properties: example: description: |- example is a sample log line that the regex should match. Serves as documentation and is validated when the LogProfile is loaded. type: string regex: description: |- regex is the regular expression with Go named capture groups. Uses (?P...) syntax to name extracted fields. Well-known capture names used by the goodput calculator: Common: timestamp Training: globalStep, iteration, epoch, stepTiming, loss, tflops, elapsedTime Checkpoint: step, path, saveDuration The controller parses named captures into typed values: int: globalStep, iteration, epoch, step float64: stepTiming, loss, tflops, elapsedTime, saveDuration string: path time: timestamp (parsed using LogProfile.spec.timestamp.layout) minLength: 1 type: string units: additionalProperties: type: string description: |- units specifies the unit for duration/timing captures. The controller normalizes all durations to seconds internally. Supported values: "s" (seconds), "ms" (milliseconds), "us" (microseconds). Only needed for duration-type captures: stepTiming, elapsedTime, saveDuration. If a duration capture is not listed here, it defaults to "s" (seconds). Example: {"elapsedTime": "ms"} means the captured value is in milliseconds. type: object required: - regex type: object type: object timestamp: description: timestamp configures how to parse timestamps captured by event patterns. properties: layout: description: |- layout is the Go time layout string for parsing (?P...) captures. Examples: "2006-01-02T15:04:05.999999999Z" (NeMo/RFC3339), "2006-01-02 15:04:05.999999" (Megatron bracketed) minLength: 1 type: string required: - layout type: object warmupSteps: description: |- warmupSteps is the number of initial training steps per run to flag as warmup. Steps after applicationStart or checkpointRestore up to this count are marked as warmup and excluded from average step time computation. type: integer workerStrategy: description: |- workerStrategy defines how to read logs from multi-worker jobs. Default: single worker (read from worker-0 only). properties: lifecyclePod: default: first description: |- lifecyclePod specifies which worker has lifecycle data (checkpoints, app start). "first" (worker-0) or "last" (highest index). Default: "first". Only used when type is "Multi". enum: - first - last type: string replicatedJobName: default: trainer description: |- replicatedJobName selects which JobSet replicatedJob to read logs from. For MPI workloads, set to "launcher" (output goes to the launcher pod). For torch workloads, use the default "trainer". Only applies to TrainJob workloads that use JobSet. type: string trainingStepPod: default: first description: |- trainingStepPod specifies which worker has training step data (iterations, metrics). "first" or "last". Default: "first". Only used when type is "Multi". enum: - first - last type: string type: description: |- type determines the log reading strategy. "Single": read all data from worker-0. "Multi": read lifecycle data from one pod, training steps from another. enum: - Single - Multi type: string required: - type type: object required: - patterns - timestamp type: object required: - spec type: object served: true storage: true --- # Source: cluster-readiness-engine/crds/nvcre.nvidia.com_workflows.yaml --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.20.0 name: workflows.nvcre.nvidia.com spec: group: nvcre.nvidia.com names: kind: Workflow listKind: WorkflowList plural: workflows singular: workflow scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: Workflow is the Schema for the workflows API properties: apiVersion: description: |- APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: description: |- Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object spec: description: spec defines the desired state of Workflow properties: dependencies: description: |- dependencies are resources required for the workload (ComputeDomain, etc.) Resources are conditionally created based on "when" clauses. items: description: DependencySpec defines a resource that must be created before the workload. properties: when: description: |- when defines conditions for creating this dependency. If not specified, the dependency is always created. properties: config: description: config matches against values in validation.performance.config. x-kubernetes-preserve-unknown-fields: true expression: description: |- expression is a CEL expression that evaluates to a boolean. Available variables: gpuArchitecture, platform, workloadKind, topology, config Example: "gpuArchitecture == 'gb200' && platform == 'aws'" type: string gpuArchitecture: description: gpuArchitecture matches against detected GPU architecture. properties: equals: description: equals checks for exact match. type: string in: description: in checks if value is in the list. items: type: string type: array notIn: description: notIn checks if value is not in the list. items: type: string type: array type: object platform: description: platform matches against detected cloud platform. properties: equals: description: equals checks for exact match. type: string in: description: in checks if value is in the list. items: type: string type: array notIn: description: notIn checks if value is not in the list. items: type: string type: array type: object topology: description: topology matches against cluster topology characteristics. properties: domainCount: description: domainCount matches against the number of topology domains (racks, zones, etc.). properties: equals: description: equals checks for exact match. type: integer greaterThan: description: greaterThan checks if value is greater than threshold. type: integer lessThan: description: lessThan checks if value is less than threshold. type: integer type: object mode: description: |- mode matches against topology mode. Examples: "single-rack", "multi-rack", "multi-zone" type: string type: object workloadKind: description: |- workloadKind matches against the workload kind. Example: "TrainJob" type: string type: object type: object x-kubernetes-embedded-resource: true x-kubernetes-preserve-unknown-fields: true type: array gangScheduler: description: |- gangScheduler records the gang-scheduling intent of the WorkloadRun or Certification that generated this Workflow, with the queue and queue label key already resolved to their effective values. It is a consistency contract, not a second place to configure gang scheduling. The Workflow controller uses it after overrides resolve to check that every effective queue label and pod scheduler name in the Job template and the referenced TrainingRuntime dependency still agrees with what the owner asked for, and to restore the workload-object queue label if an override removed it. A conflicting runtime override fails the Workflow rather than being silently repaired. It sits outside jobTemplate, dependencies and orchestration, so no override can reach it, and it is immutable in both presence and value so the contract holds across every group and iteration. A rendered Workflow submitted on its own therefore enforces the same checks without needing its original WorkloadRun. properties: queue: description: |- queue is the scheduler queue to submit the workload to. Defaults to "default-queue" if not specified. When non-empty, must be a valid Kubernetes label value: at most 63 characters, beginning and ending with an alphanumeric character, and containing only alphanumerics, hyphens, underscores, or dots. maxLength: 63 pattern: ^$|^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$ type: string queueLabelKey: description: |- queueLabelKey is the label key the queue name is written under, for schedulers that read a different label than KAI Scheduler does. Defaults to "kai.scheduler/queue" if not specified. Set it to "runai/queue" (with schedulerName "runai-scheduler") on clusters running the NVIDIA Run:ai platform. When non-empty, must be a valid Kubernetes label key (qualified name): an optional DNS-subdomain prefix of at most 253 characters followed by "/", then a name of at most 63 characters beginning and ending with an alphanumeric character and containing only alphanumerics, hyphens, underscores, or dots. maxLength: 317 pattern: ^$|^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$ type: string x-kubernetes-validations: - message: queueLabelKey prefix must be at most 253 characters rule: 'self.contains(''/'') ? self.split(''/'')[0].size() <= 253 : true' - message: queueLabelKey must not use the controller-owned key app.kubernetes.io/managed-by or any key under the nvcre.nvidia.com/ prefix rule: self != 'app.kubernetes.io/managed-by' && !self.startsWith('nvcre.nvidia.com/') schedulerName: description: schedulerName is the name of the gang-aware scheduler to use (e.g. "kai-scheduler"). minLength: 1 type: string required: - schedulerName type: object x-kubernetes-validations: - message: gangScheduler is immutable rule: self == oldSelf jobTemplate: description: jobTemplate defines the Job template used to create Jobs for each execution group. properties: metadata: description: metadata is standard object metadata applied to created Jobs. type: object spec: description: spec is the Job specification used as a template for each execution group. properties: bandwidthMeasurement: description: |- bandwidthMeasurement configures automatic creation of a BandwidthMeasurement child resource that tracks NCCL bandwidth metrics by parsing pod logs. When absent, no measurement is created (suitable for non-NCCL jobs). This field is immutable, including its presence: an object created without it cannot add it later. properties: logProfileRef: description: logProfileRef is the name of the cluster-scoped LogProfile to use for log parsing. minLength: 1 type: string sampleInterval: description: |- sampleInterval is how often to sample pod logs while the Job is running. Default: 60s. type: string testType: description: |- testType identifies the NCCL collective operation (e.g., "all_reduce", "alltoall"). Propagated to the BandwidthMeasurement spec for Prometheus metric labeling. type: string required: - logProfileRef type: object x-kubernetes-validations: - message: bandwidthMeasurement is immutable rule: self == oldSelf checkpoint: description: |- checkpoint configures checkpoint-based restart for the workload. When set, the Job controller restarts the workload on failure if a checkpoint was logged by an associated GoodputMeasurement. The user must define the PVC volume and mounts in the workload spec directly. properties: maxRestarts: description: |- maxRestarts is the maximum number of times the Job will restart on failure when a checkpoint exists. Default: 0 (no restarts). format: int32 type: integer pvcName: description: |- pvcName is the name of the PersistentVolumeClaim used for checkpoint storage. Must match a PVC in the Workflow's dependencies. minLength: 1 type: string required: - pvcName type: object goodputMeasurement: description: |- goodputMeasurement configures automatic creation of a GoodputMeasurement child resource that tracks training goodput metrics by parsing pod logs. When absent, no measurement is created (suitable for non-training jobs). Manually-created GoodputMeasurements continue to work via the existing List-based lookup. This field is immutable, including its presence: an object created without it cannot add it later. properties: logProfileRef: description: logProfileRef is the name of the cluster-scoped LogProfile to use for log parsing. minLength: 1 type: string sampleInterval: description: |- sampleInterval is how often to sample pod logs while the Job is running. Default: 60s. type: string required: - logProfileRef type: object x-kubernetes-validations: - message: goodputMeasurement is immutable rule: self == oldSelf measurementTimeout: description: |- measurementTimeout is the maximum time to wait after the Job succeeds for measurement data before failing threshold validation. Propagated from the Workflow's ValidationSpec by the Workflow controller during Job creation. When nil, the Job controller uses its default (5m). type: string nodeHealthMonitor: description: |- nodeHealthMonitor configures hardware failure detection for nodes running this job's pods. When a failure is detected, the job will be marked with the HardwareFailed condition. This field is immutable, including its presence: an object created without it cannot add it later. properties: cel: description: |- cel defines a CEL expression evaluated against Node objects. The expression must return a boolean. When it evaluates to true, the node is considered to have a hardware failure. The full corev1.Node object is available via the 'node' variable: - node.metadata: ObjectMeta (name, labels, annotations, etc.) - node.spec: NodeSpec (taints, unschedulable, podCIDR, etc.) - node.status: NodeStatus (conditions, addresses, capacity, allocatable, nodeInfo, etc.) Examples: - "node.spec.taints.exists(t, t.key == 'nvidia.com/gpu-unhealthy')" - "'gpud.nvidia.com/unhealthy' in node.metadata.labels" - "node.status.conditions.exists(c, c.type == 'GPUHealthy' && c.status == 'False')" - "node.spec.unschedulable == true" properties: expression: description: |- expression is the CEL expression to evaluate against Node objects. Must return a boolean value. When true, the node is considered unhealthy. maxLength: 2048 minLength: 1 type: string required: - expression type: object type: object x-kubernetes-validations: - message: nodeHealthMonitor is immutable rule: self == oldSelf stallMultiplier: description: |- stallMultiplier configures stall detection for the workload. When set, the Job controller compares the time since the last training step against stallMultiplier * avgStepTime (from the associated GoodputMeasurement). If the elapsed time exceeds this product, the workload is marked as Failed with reason "WorkloadStalled". Requires a GoodputMeasurement for this Job; if none exists, stall detection is skipped. Example: a value of 10 means the job is stalled if no step has been logged for 10x the average step duration. format: int32 minimum: 1 type: integer startupStallTimeoutSeconds: description: |- startupStallTimeoutSeconds is the maximum time (in seconds) to wait after the application framework starts for the first training step. If no step is observed within this timeout, the workload is marked Failed with reason "WorkloadStalled". Only active when stallMultiplier is set. Default: 1200 (20 minutes). format: int32 minimum: 1 type: integer thresholds: additionalProperties: type: string description: |- thresholds maps metric names to CEL expressions for performance validation. Propagated from WorkflowSpec.Validation.Performance.Thresholds by the Workflow controller during Job creation. The Job controller evaluates them after workload success and sets ValidationFailed if any threshold is violated. Keys: "busBandwidthGBps", "goodputRatio", "avgTFLOPsPerGPU", "avgStepTimeSec", etc. Values: CEL expressions using `value` variable, e.g. "value >= 900" type: object workload: description: |- workload defines the workload to run. The workload is created as a child resource of the Job. maxProperties: 1 minProperties: 1 properties: trainJob: description: trainJob configures a Kubeflow TrainJob workload. properties: activeDeadlineSeconds: description: |- activeDeadlineSeconds specifies the duration in seconds relative to the TrainJob start time (which resets on resume from suspension) that the TrainJob may be active before the system tries to terminate it. Value must be a positive integer. Once reached, all running Pods are terminated and the TrainJob status becomes Failed with reason: DeadlineExceeded. format: int64 minimum: 1 type: integer x-kubernetes-validations: - message: field is immutable rule: self == oldSelf initializer: description: initializer defines the configuration of the initializer. properties: dataset: description: dataset defines the configuration for the dataset initialization and pre-processing. properties: env: description: |- env is the list of environment variables to set in the dataset initializer container. These values will be merged with the TrainingRuntime's dataset initializer environments. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map secretRef: description: |- secretRef is the reference to the secret with credentials to download dataset. Secret must be created in the TrainJob's namespace. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic storageUri: description: storageUri is the URI for the dataset provider. maxLength: 2048 type: string type: object model: description: model defines the configuration for the pre-trained model initialization properties: env: description: |- env is the list of environment variables to set in the model initializer container. These values will be merged with the TrainingRuntime's model initializer environments. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map secretRef: description: |- secretRef is the reference to the secret with credentials to download model. Secret must be created in the TrainJob's namespace. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic storageUri: description: storageUri is the URI for the model provider. maxLength: 2048 type: string type: object type: object x-kubernetes-validations: - message: field is immutable rule: self == oldSelf managedBy: default: trainer.kubeflow.org/trainjob-controller description: |- managedBy is used to indicate the controller or entity that manages a TrainJob. The value must be either an empty, `trainer.kubeflow.org/trainjob-controller` or `kueue.x-k8s.io/multikueue`. The built-in TrainJob controller reconciles TrainJob which don't have this field at all or the field value is the reserved string `trainer.kubeflow.org/trainjob-controller`, but delegates reconciling TrainJobs with a 'kueue.x-k8s.io/multikueue' to the Kueue. The field is immutable. maxLength: 253 type: string x-kubernetes-validations: - message: ManagedBy must be trainer.kubeflow.org/trainjob-controller or kueue.x-k8s.io/multikueue if set rule: self in ['trainer.kubeflow.org/trainjob-controller', 'kueue.x-k8s.io/multikueue'] - message: field is immutable rule: self == oldSelf runtimePatches: description: |- runtimePatches defines custom patches applied to the TrainJob's Runtime. Patches are keyed by manager to provide clear ownership and avoid conflicts between controllers. items: description: |- RuntimePatch represents a custom patch applied to the TrainJob's training runtime template. Patches are keyed by manager to provide clear ownership and avoid conflicts between controllers. properties: manager: description: |- manager indicates who owns this patch entry. It can be set by the user, external controllers, or admission webhooks to track ownership and avoid conflicts. For example, Kueue sets this field to "kueue.x-k8s.io/manager". maxLength: 253 minLength: 1 type: string x-kubernetes-validations: - message: field is immutable rule: self == oldSelf time: description: |- time is the timestamp of when this patch was last written. Set by the Trainer admission webhook on each create or update. Used for observability only, not used as a list map key. format: date-time type: string trainingRuntimeSpec: description: trainingRuntimeSpec defines allowed patches for ClusterTrainingRuntime or TrainingRuntime-based jobs. properties: template: description: template patches the JobSet template. properties: metadata: description: |- metadata patches the JobSet object metadata. Only labels and annotations are allowed. type: object spec: description: spec patches the JobSet spec with restricted fields. properties: replicatedJobs: description: replicatedJobs defines per-job patches, keyed by job name. items: description: ReplicatedJobPatch defines patches for a specific replicated job within the JobSet. properties: name: description: name is the name of the replicated job to patch. maxLength: 253 minLength: 1 type: string template: description: template patches the Job template for this replicated job. properties: metadata: description: |- metadata patches the Job template metadata. Only labels and annotations are allowed. type: object spec: description: spec patches the Job spec with restricted fields. properties: template: description: template patches the Pod template for this Job. properties: metadata: description: |- metadata patches the Pod template metadata. Only labels and annotations are allowed. type: object spec: description: spec patches the Pod spec with the fields managers are permitted to set. properties: affinity: description: affinity patches the Pod's scheduling affinity. properties: nodeAffinity: description: Describes node affinity scheduling rules for the pod. properties: preferredDuringSchedulingIgnoredDuringExecution: description: |- The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node matches the corresponding matchExpressions; the node(s) with the highest sum are the most preferred. items: description: |- An empty preferred scheduling term matches all objects with implicit weight 0 (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op). properties: preference: description: A node selector term, associated with the corresponding weight. properties: matchExpressions: description: A list of node selector requirements by node's labels. items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchFields: description: A list of node selector requirements by node's fields. items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic type: object x-kubernetes-map-type: atomic weight: description: Weight associated with matching the corresponding nodeSelectorTerm, in the range 1-100. format: int32 type: integer required: - preference - weight type: object type: array x-kubernetes-list-type: atomic requiredDuringSchedulingIgnoredDuringExecution: description: |- If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to an update), the system may or may not try to eventually evict the pod from its node. properties: nodeSelectorTerms: description: Required. A list of node selector terms. The terms are ORed. items: description: |- A null or empty node selector term matches no objects. The requirements of them are ANDed. The TopologySelectorTerm type implements a subset of the NodeSelectorTerm. properties: matchExpressions: description: A list of node selector requirements by node's labels. items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchFields: description: A list of node selector requirements by node's fields. items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic type: object x-kubernetes-map-type: atomic type: array x-kubernetes-list-type: atomic required: - nodeSelectorTerms type: object x-kubernetes-map-type: atomic type: object podAffinity: description: Describes pod affinity scheduling rules (e.g. co-locate this pod in the same node, zone, etc. as some other pod(s)). properties: preferredDuringSchedulingIgnoredDuringExecution: description: |- The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. items: description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s) properties: podAffinityTerm: description: Required. A pod affinity term, associated with the corresponding weight. properties: labelSelector: description: |- A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic matchLabelKeys: description: |- MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic mismatchLabelKeys: description: |- MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic namespaceSelector: description: |- A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: description: |- namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array x-kubernetes-list-type: atomic topologyKey: description: |- This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. type: string required: - topologyKey type: object weight: description: |- weight associated with matching the corresponding podAffinityTerm, in the range 1-100. format: int32 type: integer required: - podAffinityTerm - weight type: object type: array x-kubernetes-list-type: atomic requiredDuringSchedulingIgnoredDuringExecution: description: |- If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied. items: description: |- Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key matches that of any node on which a pod of the set of pods is running properties: labelSelector: description: |- A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic matchLabelKeys: description: |- MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic mismatchLabelKeys: description: |- MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic namespaceSelector: description: |- A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: description: |- namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array x-kubernetes-list-type: atomic topologyKey: description: |- This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. type: string required: - topologyKey type: object type: array x-kubernetes-list-type: atomic type: object podAntiAffinity: description: Describes pod anti-affinity scheduling rules (e.g. avoid putting this pod in the same node, zone, etc. as some other pod(s)). properties: preferredDuringSchedulingIgnoredDuringExecution: description: |- The scheduler will prefer to schedule pods to nodes that satisfy the anti-affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling anti-affinity expressions, etc.), compute a sum by iterating through the elements of this field and subtracting "weight" from the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. items: description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s) properties: podAffinityTerm: description: Required. A pod affinity term, associated with the corresponding weight. properties: labelSelector: description: |- A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic matchLabelKeys: description: |- MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic mismatchLabelKeys: description: |- MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic namespaceSelector: description: |- A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: description: |- namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array x-kubernetes-list-type: atomic topologyKey: description: |- This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. type: string required: - topologyKey type: object weight: description: |- weight associated with matching the corresponding podAffinityTerm, in the range 1-100. format: int32 type: integer required: - podAffinityTerm - weight type: object type: array x-kubernetes-list-type: atomic requiredDuringSchedulingIgnoredDuringExecution: description: |- If the anti-affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the anti-affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied. items: description: |- Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key matches that of any node on which a pod of the set of pods is running properties: labelSelector: description: |- A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic matchLabelKeys: description: |- MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic mismatchLabelKeys: description: |- MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. items: type: string type: array x-kubernetes-list-type: atomic namespaceSelector: description: |- A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: description: |- namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array x-kubernetes-list-type: atomic topologyKey: description: |- This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. type: string required: - topologyKey type: object type: array x-kubernetes-list-type: atomic type: object type: object containers: description: containers patches specific containers in the target job templates. items: description: ContainerPatch represents parameters that can be patched using PodSpecPatch. properties: env: description: |- env is the list of environment variables to set in the container. These values will be merged with the Runtime's environments. These values can't be set for container with the name: `node`, `dataset-initializer`, or `model-initializer`. For those containers the envs can only be set via Trainer or Initializer APIs. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map name: description: name for the container. Runtime must have this container. maxLength: 253 minLength: 1 type: string securityContext: description: |- securityContext patches the container's security context. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ properties: allowPrivilegeEscalation: description: |- AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged 2) has CAP_SYS_ADMIN Note that this field cannot be set when spec.os.name is windows. type: boolean appArmorProfile: description: |- appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". type: string type: description: |- type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. type: string required: - type type: object capabilities: description: |- The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. properties: add: description: Added capabilities items: description: Capability represent POSIX capabilities type type: string type: array x-kubernetes-list-type: atomic drop: description: Removed capabilities items: description: Capability represent POSIX capabilities type type: string type: array x-kubernetes-list-type: atomic type: object privileged: description: |- Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. type: boolean procMount: description: |- procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. type: string readOnlyRootFilesystem: description: |- Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. type: boolean runAsGroup: description: |- The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer runAsNonRoot: description: |- Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: boolean runAsUser: description: |- The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer seLinuxOptions: description: |- The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. properties: level: description: Level is SELinux level label that applies to the container. type: string role: description: Role is a SELinux role label that applies to the container. type: string type: description: Type is a SELinux type label that applies to the container. type: string user: description: User is a SELinux user label that applies to the container. type: string type: object seccompProfile: description: |- The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. type: string type: description: |- type indicates which kind of seccomp profile will be applied. Valid options are: Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. type: string required: - type type: object windowsOptions: description: |- The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. properties: gmsaCredentialSpec: description: |- GMSACredentialSpec is where the GMSA admission webhook (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. type: string gmsaCredentialSpecName: description: GMSACredentialSpecName is the name of the GMSA credential spec to use. type: string hostProcess: description: |- HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. type: boolean runAsUserName: description: |- The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: string type: object type: object volumeMounts: description: volumeMounts are the volumes to mount into the container's filesystem. items: description: VolumeMount describes a mounting of a Volume within a container. properties: mountPath: description: |- Path within the container at which the volume should be mounted. Must not contain ':'. type: string mountPropagation: description: |- mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. When RecursiveReadOnly is set to IfPossible or to Enabled, MountPropagation must be None or unspecified (which defaults to None). type: string name: description: This must match the Name of a Volume. type: string readOnly: description: |- Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. type: boolean recursiveReadOnly: description: |- RecursiveReadOnly specifies whether read-only mounts should be handled recursively. If ReadOnly is false, this field has no meaning and must be unspecified. If ReadOnly is true, and this field is set to Disabled, the mount is not made recursively read-only. If this field is set to IfPossible, the mount is made recursively read-only, if it is supported by the container runtime. If this field is set to Enabled, the mount is made recursively read-only if it is supported by the container runtime, otherwise the pod will not be started and an error will be generated to indicate the reason. If this field is set to IfPossible or Enabled, MountPropagation must be set to None (or be unspecified, which defaults to None). If this field is not specified, it is treated as an equivalent of Disabled. type: string subPath: description: |- Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). type: string subPathExpr: description: |- Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. type: string required: - mountPath - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map required: - name type: object maxItems: 64 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map x-kubernetes-validations: - message: field is immutable rule: self == oldSelf imagePullSecrets: description: imagePullSecrets patches the image pull secrets for the Pods in the target job templates. items: description: |- LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic maxItems: 64 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map x-kubernetes-validations: - message: field is immutable rule: self == oldSelf initContainers: description: initContainers patches the init containers in the target job templates. items: description: ContainerPatch represents parameters that can be patched using PodSpecPatch. properties: env: description: |- env is the list of environment variables to set in the container. These values will be merged with the Runtime's environments. These values can't be set for container with the name: `node`, `dataset-initializer`, or `model-initializer`. For those containers the envs can only be set via Trainer or Initializer APIs. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map name: description: name for the container. Runtime must have this container. maxLength: 253 minLength: 1 type: string securityContext: description: |- securityContext patches the container's security context. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ properties: allowPrivilegeEscalation: description: |- AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged 2) has CAP_SYS_ADMIN Note that this field cannot be set when spec.os.name is windows. type: boolean appArmorProfile: description: |- appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". type: string type: description: |- type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. type: string required: - type type: object capabilities: description: |- The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. properties: add: description: Added capabilities items: description: Capability represent POSIX capabilities type type: string type: array x-kubernetes-list-type: atomic drop: description: Removed capabilities items: description: Capability represent POSIX capabilities type type: string type: array x-kubernetes-list-type: atomic type: object privileged: description: |- Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. type: boolean procMount: description: |- procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. type: string readOnlyRootFilesystem: description: |- Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. type: boolean runAsGroup: description: |- The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer runAsNonRoot: description: |- Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: boolean runAsUser: description: |- The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer seLinuxOptions: description: |- The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. properties: level: description: Level is SELinux level label that applies to the container. type: string role: description: Role is a SELinux role label that applies to the container. type: string type: description: Type is a SELinux type label that applies to the container. type: string user: description: User is a SELinux user label that applies to the container. type: string type: object seccompProfile: description: |- The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. type: string type: description: |- type indicates which kind of seccomp profile will be applied. Valid options are: Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. type: string required: - type type: object windowsOptions: description: |- The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. properties: gmsaCredentialSpec: description: |- GMSACredentialSpec is where the GMSA admission webhook (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. type: string gmsaCredentialSpecName: description: GMSACredentialSpecName is the name of the GMSA credential spec to use. type: string hostProcess: description: |- HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. type: boolean runAsUserName: description: |- The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: string type: object type: object volumeMounts: description: volumeMounts are the volumes to mount into the container's filesystem. items: description: VolumeMount describes a mounting of a Volume within a container. properties: mountPath: description: |- Path within the container at which the volume should be mounted. Must not contain ':'. type: string mountPropagation: description: |- mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. When RecursiveReadOnly is set to IfPossible or to Enabled, MountPropagation must be None or unspecified (which defaults to None). type: string name: description: This must match the Name of a Volume. type: string readOnly: description: |- Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. type: boolean recursiveReadOnly: description: |- RecursiveReadOnly specifies whether read-only mounts should be handled recursively. If ReadOnly is false, this field has no meaning and must be unspecified. If ReadOnly is true, and this field is set to Disabled, the mount is not made recursively read-only. If this field is set to IfPossible, the mount is made recursively read-only, if it is supported by the container runtime. If this field is set to Enabled, the mount is made recursively read-only if it is supported by the container runtime, otherwise the pod will not be started and an error will be generated to indicate the reason. If this field is set to IfPossible or Enabled, MountPropagation must be set to None (or be unspecified, which defaults to None). If this field is not specified, it is treated as an equivalent of Disabled. type: string subPath: description: |- Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). type: string subPathExpr: description: |- Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. type: string required: - mountPath - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map required: - name type: object maxItems: 64 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map x-kubernetes-validations: - message: field is immutable rule: self == oldSelf nodeSelector: additionalProperties: type: string description: nodeSelector patches the node selector to place Pods on specific nodes. type: object schedulingGates: description: |- schedulingGates patches the scheduling gates for the Pods in the target job templates. More info: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/ items: description: PodSchedulingGate is associated to a Pod to guard its scheduling. properties: name: description: |- Name of the scheduling gate. Each scheduling gate must have a unique name field. type: string required: - name type: object maxItems: 16 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map securityContext: description: |- securityContext patches the Pod's security context. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ properties: appArmorProfile: description: |- appArmorProfile is the AppArmor options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". type: string type: description: |- type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. type: string required: - type type: object fsGroup: description: |- A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod: 1. The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw---- If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer fsGroupChangePolicy: description: |- fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows. type: string runAsGroup: description: |- The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer runAsNonRoot: description: |- Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: boolean runAsUser: description: |- The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer seLinuxChangePolicy: description: |- seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. Valid values are "MountOption" and "Recursive". "Recursive" means relabeling of all files on all Pod volumes by the container runtime. This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node. "MountOption" mounts all eligible Pod volumes with `-o context` mount option. This requires all Pods that share the same volume to use the same SELinux label. It is not possible to share the same volume among privileged and unprivileged Pods. Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their CSIDriver instance. Other volumes are always re-labelled recursively. "MountOption" value is allowed only when SELinuxMount feature gate is enabled. If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes and "Recursive" for all other volumes. This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers. All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. Note that this field cannot be set when spec.os.name is windows. type: string seLinuxOptions: description: |- The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. properties: level: description: Level is SELinux level label that applies to the container. type: string role: description: Role is a SELinux role label that applies to the container. type: string type: description: Type is a SELinux type label that applies to the container. type: string user: description: User is a SELinux user label that applies to the container. type: string type: object seccompProfile: description: |- The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: description: |- localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. type: string type: description: |- type indicates which kind of seccomp profile will be applied. Valid options are: Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. type: string required: - type type: object supplementalGroups: description: |- A list of groups applied to the first process run in each container, in addition to the container's primary GID and fsGroup (if specified). If the SupplementalGroupsPolicy feature is enabled, the supplementalGroupsPolicy field determines whether these are in addition to or instead of any group memberships defined in the container image. If unspecified, no additional groups are added, though group memberships defined in the container image may still be used, depending on the supplementalGroupsPolicy field. Note that this field cannot be set when spec.os.name is windows. items: format: int64 type: integer type: array x-kubernetes-list-type: atomic supplementalGroupsPolicy: description: |- Defines how supplemental groups of the first container processes are calculated. Valid values are "Merge" and "Strict". If not specified, "Merge" is used. (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled and the container runtime must implement support for this feature. Note that this field cannot be set when spec.os.name is windows. type: string sysctls: description: |- Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows. items: description: Sysctl defines a kernel parameter to be set properties: name: description: Name of a property to set type: string value: description: Value of a property to set type: string required: - name - value type: object type: array x-kubernetes-list-type: atomic windowsOptions: description: |- The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. properties: gmsaCredentialSpec: description: |- GMSACredentialSpec is where the GMSA admission webhook (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. type: string gmsaCredentialSpecName: description: GMSACredentialSpecName is the name of the GMSA credential spec to use. type: string hostProcess: description: |- HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. type: boolean runAsUserName: description: |- The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. type: string type: object type: object x-kubernetes-validations: - message: field is immutable rule: self == oldSelf serviceAccountName: description: serviceAccountName patches the service account for the Pods in the target job templates. maxLength: 253 type: string x-kubernetes-validations: - message: field is immutable rule: self == oldSelf tolerations: description: tolerations patches the Pod's tolerations. items: description: |- The pod this Toleration is attached to tolerates any taint that matches the triple using the matching operator . properties: effect: description: |- Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. type: string key: description: |- Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys. type: string operator: description: |- Operator represents a key's relationship to the value. Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category. Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). type: string tolerationSeconds: description: |- TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system. format: int64 type: integer value: description: |- Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string. type: string type: object maxItems: 128 type: array x-kubernetes-list-type: atomic volumes: description: volumes patches the Pod's volumes. items: description: Volume represents a named volume in a pod that may be accessed by any container in the pod. properties: awsElasticBlockStore: description: |- awsElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Deprecated: AWSElasticBlockStore is deprecated. All operations for the in-tree awsElasticBlockStore type are redirected to the ebs.csi.aws.com CSI driver. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore properties: fsType: description: |- fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore type: string partition: description: |- partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty). format: int32 type: integer readOnly: description: |- readOnly value true will force the readOnly setting in VolumeMounts. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore type: boolean volumeID: description: |- volumeID is unique ID of the persistent disk resource in AWS (Amazon EBS volume). More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore type: string required: - volumeID type: object azureDisk: description: |- azureDisk represents an Azure Data Disk mount on the host and bind mount to the pod. Deprecated: AzureDisk is deprecated. All operations for the in-tree azureDisk type are redirected to the disk.csi.azure.com CSI driver. properties: cachingMode: description: 'cachingMode is the Host Caching mode: None, Read Only, Read Write.' type: string diskName: description: diskName is the Name of the data disk in the blob storage type: string diskURI: description: diskURI is the URI of data disk in the blob storage type: string fsType: default: ext4 description: |- fsType is Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string kind: description: 'kind expected values are Shared: multiple blob disks per storage account Dedicated: single blob disk per storage account Managed: azure managed data disk (only in managed availability set). defaults to shared' type: string readOnly: default: false description: |- readOnly Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean required: - diskName - diskURI type: object azureFile: description: |- azureFile represents an Azure File Service mount on the host and bind mount to the pod. Deprecated: AzureFile is deprecated. All operations for the in-tree azureFile type are redirected to the file.csi.azure.com CSI driver. properties: readOnly: description: |- readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean secretName: description: secretName is the name of secret that contains Azure Storage Account Name and Key type: string shareName: description: shareName is the azure share Name type: string required: - secretName - shareName type: object cephfs: description: |- cephFS represents a Ceph FS mount on the host that shares a pod's lifetime. Deprecated: CephFS is deprecated and the in-tree cephfs type is no longer supported. properties: monitors: description: |- monitors is Required: Monitors is a collection of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it items: type: string type: array x-kubernetes-list-type: atomic path: description: 'path is Optional: Used as the mounted root, rather than the full Ceph tree, default is /' type: string readOnly: description: |- readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it type: boolean secretFile: description: |- secretFile is Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it type: string secretRef: description: |- secretRef is Optional: SecretRef is reference to the authentication secret for User, default is empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic user: description: |- user is optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it type: string required: - monitors type: object cinder: description: |- cinder represents a cinder volume attached and mounted on kubelets host machine. Deprecated: Cinder is deprecated. All operations for the in-tree cinder type are redirected to the cinder.csi.openstack.org CSI driver. More info: https://examples.k8s.io/mysql-cinder-pd/README.md properties: fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md type: string readOnly: description: |- readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md type: boolean secretRef: description: |- secretRef is optional: points to a secret object containing parameters used to connect to OpenStack. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic volumeID: description: |- volumeID used to identify the volume in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md type: string required: - volumeID type: object configMap: description: configMap represents a configMap that should populate this volume properties: defaultMode: description: |- defaultMode is optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer items: description: |- items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. items: description: Maps a string key to a path within a volume. properties: key: description: key is the key to project. type: string mode: description: |- mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: |- path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. type: string required: - key - path type: object type: array x-kubernetes-list-type: atomic name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: optional specify whether the ConfigMap or its keys must be defined type: boolean type: object x-kubernetes-map-type: atomic csi: description: csi (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers. properties: driver: description: |- driver is the name of the CSI driver that handles this volume. Consult with your admin for the correct name as registered in the cluster. type: string fsType: description: |- fsType to mount. Ex. "ext4", "xfs", "ntfs". If not provided, the empty value is passed to the associated CSI driver which will determine the default filesystem to apply. type: string nodePublishSecretRef: description: |- nodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secret references are passed. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic readOnly: description: |- readOnly specifies a read-only configuration for the volume. Defaults to false (read/write). type: boolean volumeAttributes: additionalProperties: type: string description: |- volumeAttributes stores driver-specific properties that are passed to the CSI driver. Consult your driver's documentation for supported values. type: object required: - driver type: object downwardAPI: description: downwardAPI represents downward API about the pod that should populate this volume properties: defaultMode: description: |- Optional: mode bits to use on created files by default. Must be a Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer items: description: Items is a list of downward API volume file items: description: DownwardAPIVolumeFile represents information to create the file containing the pod field properties: fieldRef: description: 'Required: Selects a field of the pod: only annotations, labels, name, namespace and uid are supported.' properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic mode: description: |- Optional: mode bits used to set permissions on this file, must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: 'Required: Path is the relative path name of the file to be created. Must not be absolute or contain the ''..'' path. Must be utf-8 encoded. The first item of the relative path must not start with ''..''' type: string resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, requests.cpu and requests.memory) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic required: - path type: object type: array x-kubernetes-list-type: atomic type: object emptyDir: description: |- emptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir properties: medium: description: |- medium represents what type of storage medium should back this directory. The default is "" which means to use the node's default medium. Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir type: string sizeLimit: anyOf: - type: integer - type: string description: |- sizeLimit is the total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object ephemeral: description: |- ephemeral represents a volume that is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed. Use this if: a) the volume is only needed while the pod runs, b) features of normal volumes like restoring from snapshot or capacity tracking are needed, c) the storage driver is specified through a storage class, and d) the storage driver supports dynamic volume provisioning through a PersistentVolumeClaim (see EphemeralVolumeSource for more information on the connection between this volume type and PersistentVolumeClaim). Use PersistentVolumeClaim or one of the vendor-specific APIs for volumes that persist for longer than the lifecycle of an individual pod. Use CSI for light-weight local ephemeral volumes if the CSI driver is meant to be used that way - see the documentation of the driver for more information. A pod can use both types of ephemeral volumes and persistent volumes at the same time. properties: volumeClaimTemplate: description: |- Will be used to create a stand-alone PVC to provision the volume. The pod in which this EphemeralVolumeSource is embedded will be the owner of the PVC, i.e. the PVC will be deleted together with the pod. The name of the PVC will be `-` where `` is the name from the `PodSpec.Volumes` array entry. Pod validation will reject the pod if the concatenated name is not valid for a PVC (for example, too long). An existing PVC with that name that is not owned by the pod will *not* be used for the pod to avoid using an unrelated volume by mistake. Starting the pod is then blocked until the unrelated PVC is removed. If such a pre-created PVC is meant to be used by the pod, the PVC has to updated with an owner reference to the pod once the pod exists. Normally this should not be necessary, but it may be useful when manually reconstructing a broken cluster. This field is read-only and no changes will be made by Kubernetes to the PVC after it has been created. Required, must not be nil. properties: metadata: description: |- May contain labels and annotations that will be copied into the PVC when creating it. No other fields are allowed and will be rejected during validation. type: object spec: description: |- The specification for the PersistentVolumeClaim. The entire content is copied unchanged into the PVC that gets created from this template. The same fields as in a PersistentVolumeClaim are also valid here. properties: accessModes: description: |- accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1 items: type: string type: array x-kubernetes-list-type: atomic dataSource: description: |- dataSource field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource. properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string required: - kind - name type: object x-kubernetes-map-type: atomic dataSourceRef: description: |- dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the dataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, when namespace isn't specified in dataSourceRef, both fields (dataSource and dataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. When namespace is specified in dataSourceRef, dataSource isn't set to the same value and must be empty. There are three important differences between dataSource and dataSourceRef: * While dataSource only allows two specific types of objects, dataSourceRef allows any non-core object, as well as PersistentVolumeClaim objects. * While dataSource ignores disallowed values (dropping them), dataSourceRef preserves all values, and generates an error if a disallowed value is specified. * While dataSource only allows local objects, dataSourceRef allows objects in any namespaces. (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled. (Alpha) Using the namespace field of dataSourceRef requires the CrossNamespaceVolumeDataSource feature gate to be enabled. properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string namespace: description: |- Namespace is the namespace of resource being referenced Note that when a namespace is specified, a gateway.networking.k8s.io/ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. (Alpha) This field requires the CrossNamespaceVolumeDataSource feature gate to be enabled. type: string required: - kind - name type: object resources: description: |- resources represents the minimum resources the volume should have. Users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources properties: limits: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object requests: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object type: object selector: description: selector is a label query over volumes to consider for binding. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic storageClassName: description: |- storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 type: string volumeAttributesClassName: description: |- volumeAttributesClassName may be used to set the VolumeAttributesClass used by this claim. If specified, the CSI driver will create or update the volume with the attributes defined in the corresponding VolumeAttributesClass. This has a different purpose than storageClassName, it can be changed after the claim is created. An empty string or nil value indicates that no VolumeAttributesClass will be applied to the claim. If the claim enters an Infeasible error state, this field can be reset to its previous value (including nil) to cancel the modification. If the resource referred to by volumeAttributesClass does not exist, this PersistentVolumeClaim will be set to a Pending state, as reflected by the modifyVolumeStatus field, until such as a resource exists. More info: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/ type: string volumeMode: description: |- volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec. type: string volumeName: description: volumeName is the binding reference to the PersistentVolume backing this claim. type: string type: object required: - spec type: object type: object fc: description: fc represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod. properties: fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string lun: description: 'lun is Optional: FC target lun number' format: int32 type: integer readOnly: description: |- readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean targetWWNs: description: 'targetWWNs is Optional: FC target worldwide names (WWNs)' items: type: string type: array x-kubernetes-list-type: atomic wwids: description: |- wwids Optional: FC volume world wide identifiers (wwids) Either wwids or combination of targetWWNs and lun must be set, but not both simultaneously. items: type: string type: array x-kubernetes-list-type: atomic type: object flexVolume: description: |- flexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. Deprecated: FlexVolume is deprecated. Consider using a CSIDriver instead. properties: driver: description: driver is the name of the driver to use for this volume. type: string fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". The default filesystem depends on FlexVolume script. type: string options: additionalProperties: type: string description: 'options is Optional: this field holds extra command options if any.' type: object readOnly: description: |- readOnly is Optional: defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean secretRef: description: |- secretRef is Optional: secretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic required: - driver type: object flocker: description: |- flocker represents a Flocker volume attached to a kubelet's host machine. This depends on the Flocker control service being running. Deprecated: Flocker is deprecated and the in-tree flocker type is no longer supported. properties: datasetName: description: |- datasetName is Name of the dataset stored as metadata -> name on the dataset for Flocker should be considered as deprecated type: string datasetUUID: description: datasetUUID is the UUID of the dataset. This is unique identifier of a Flocker dataset type: string type: object gcePersistentDisk: description: |- gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Deprecated: GCEPersistentDisk is deprecated. All operations for the in-tree gcePersistentDisk type are redirected to the pd.csi.storage.gke.io CSI driver. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk properties: fsType: description: |- fsType is filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk type: string partition: description: |- partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty). More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk format: int32 type: integer pdName: description: |- pdName is unique name of the PD resource in GCE. Used to identify the disk in GCE. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk type: string readOnly: description: |- readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk type: boolean required: - pdName type: object gitRepo: description: |- gitRepo represents a git repository at a particular revision. Deprecated: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. properties: directory: description: |- directory is the target directory name. Must not contain or start with '..'. If '.' is supplied, the volume directory will be the git repository. Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name. type: string repository: description: repository is the URL type: string revision: description: revision is the commit hash for the specified revision. type: string required: - repository type: object glusterfs: description: |- glusterfs represents a Glusterfs mount on the host that shares a pod's lifetime. Deprecated: Glusterfs is deprecated and the in-tree glusterfs type is no longer supported. properties: endpoints: description: endpoints is the endpoint name that details Glusterfs topology. type: string path: description: |- path is the Glusterfs volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod type: string readOnly: description: |- readOnly here will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod type: boolean required: - endpoints - path type: object hostPath: description: |- hostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath properties: path: description: |- path of the directory on the host. If the path is a symlink, it will follow the link to the real path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath type: string type: description: |- type for HostPath Volume Defaults to "" More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath type: string required: - path type: object image: description: |- image represents an OCI object (a container image or artifact) pulled and mounted on the kubelet's host machine. The volume is resolved at pod startup depending on which PullPolicy value is provided: - Always: the kubelet always attempts to pull the reference. Container creation will fail If the pull fails. - Never: the kubelet never pulls the reference and only uses a local image or artifact. Container creation will fail if the reference isn't present. - IfNotPresent: the kubelet pulls if the reference isn't already present on disk. Container creation will fail if the reference isn't present and the pull fails. The volume gets re-resolved if the pod gets deleted and recreated, which means that new remote content will become available on pod recreation. A failure to resolve or pull the image during pod startup will block containers from starting and may add significant latency. Failures will be retried using normal volume backoff and will be reported on the pod reason and message. The types of objects that may be mounted by this volume are defined by the container runtime implementation on a host machine and at minimum must include all valid types supported by the container image field. The OCI object gets mounted in a single directory (spec.containers[*].volumeMounts.mountPath) by merging the manifest layers in the same way as for container images. The volume will be mounted read-only (ro). Sub path mounts for containers are not supported (spec.containers[*].volumeMounts.subpath) before 1.33. The field spec.securityContext.fsGroupChangePolicy has no effect on this volume type. properties: pullPolicy: description: |- Policy for pulling OCI objects. Possible values are: Always: the kubelet always attempts to pull the reference. Container creation will fail If the pull fails. Never: the kubelet never pulls the reference and only uses a local image or artifact. Container creation will fail if the reference isn't present. IfNotPresent: the kubelet pulls if the reference isn't already present on disk. Container creation will fail if the reference isn't present and the pull fails. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. type: string reference: description: |- Required: Image or artifact reference to be used. Behaves in the same way as pod.spec.containers[*].image. Pull secrets will be assembled in the same way as for the container image by looking up node credentials, SA image pull secrets, and pod spec image pull secrets. More info: https://kubernetes.io/docs/concepts/containers/images This field is optional to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets. type: string type: object iscsi: description: |- iscsi represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes/#iscsi properties: chapAuthDiscovery: description: chapAuthDiscovery defines whether support iSCSI Discovery CHAP authentication type: boolean chapAuthSession: description: chapAuthSession defines whether support iSCSI Session CHAP authentication type: boolean fsType: description: |- fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi type: string initiatorName: description: |- initiatorName is the custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterface simultaneously, new iSCSI interface : will be created for the connection. type: string iqn: description: iqn is the target iSCSI Qualified Name. type: string iscsiInterface: default: default description: |- iscsiInterface is the interface Name that uses an iSCSI transport. Defaults to 'default' (tcp). type: string lun: description: lun represents iSCSI Target Lun number. format: int32 type: integer portals: description: |- portals is the iSCSI Target Portal List. The portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260). items: type: string type: array x-kubernetes-list-type: atomic readOnly: description: |- readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. type: boolean secretRef: description: secretRef is the CHAP Secret for iSCSI target and initiator authentication properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic targetPortal: description: |- targetPortal is iSCSI Target Portal. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260). type: string required: - iqn - lun - targetPortal type: object name: description: |- name of the volume. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string nfs: description: |- nfs represents an NFS mount on the host that shares a pod's lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs properties: path: description: |- path that is exported by the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs type: string readOnly: description: |- readOnly here will force the NFS export to be mounted with read-only permissions. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs type: boolean server: description: |- server is the hostname or IP address of the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs type: string required: - path - server type: object persistentVolumeClaim: description: |- persistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims properties: claimName: description: |- claimName is the name of a PersistentVolumeClaim in the same namespace as the pod using this volume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims type: string readOnly: description: |- readOnly Will force the ReadOnly setting in VolumeMounts. Default false. type: boolean required: - claimName type: object photonPersistentDisk: description: |- photonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine. Deprecated: PhotonPersistentDisk is deprecated and the in-tree photonPersistentDisk type is no longer supported. properties: fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string pdID: description: pdID is the ID that identifies Photon Controller persistent disk type: string required: - pdID type: object portworxVolume: description: |- portworxVolume represents a portworx volume attached and mounted on kubelets host machine. Deprecated: PortworxVolume is deprecated. All operations for the in-tree portworxVolume type are redirected to the pxd.portworx.com CSI driver. properties: fsType: description: |- fSType represents the filesystem type to mount Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs". Implicitly inferred to be "ext4" if unspecified. type: string readOnly: description: |- readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean volumeID: description: volumeID uniquely identifies a Portworx volume type: string required: - volumeID type: object projected: description: projected items for all in one resources secrets, configmaps, and downward API properties: defaultMode: description: |- defaultMode are the mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer sources: description: |- sources is the list of volume projections. Each entry in this list handles one source. items: description: |- Projection that may be projected along with other supported volume types. Exactly one of these fields must be set. properties: clusterTrustBundle: description: |- ClusterTrustBundle allows a pod to access the `.spec.trustBundle` field of ClusterTrustBundle objects in an auto-updating file. Alpha, gated by the ClusterTrustBundleProjection feature gate. ClusterTrustBundle objects can either be selected by name, or by the combination of signer name and a label selector. Kubelet performs aggressive normalization of the PEM contents written into the pod filesystem. Esoteric PEM features such as inter-block comments and block headers are stripped. Certificates are deduplicated. The ordering of certificates within the file is arbitrary, and Kubelet may change the order over time. properties: labelSelector: description: |- Select all ClusterTrustBundles that match this label selector. Only has effect if signerName is set. Mutually-exclusive with name. If unset, interpreted as "match nothing". If set but empty, interpreted as "match everything". properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic name: description: |- Select a single ClusterTrustBundle by object name. Mutually-exclusive with signerName and labelSelector. type: string optional: description: |- If true, don't block pod startup if the referenced ClusterTrustBundle(s) aren't available. If using name, then the named ClusterTrustBundle is allowed not to exist. If using signerName, then the combination of signerName and labelSelector is allowed to match zero ClusterTrustBundles. type: boolean path: description: Relative path from the volume root to write the bundle. type: string signerName: description: |- Select all ClusterTrustBundles that match this signer name. Mutually-exclusive with name. The contents of all selected ClusterTrustBundles will be unified and deduplicated. type: string required: - path type: object configMap: description: configMap information about the configMap data to project properties: items: description: |- items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. items: description: Maps a string key to a path within a volume. properties: key: description: key is the key to project. type: string mode: description: |- mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: |- path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. type: string required: - key - path type: object type: array x-kubernetes-list-type: atomic name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: optional specify whether the ConfigMap or its keys must be defined type: boolean type: object x-kubernetes-map-type: atomic downwardAPI: description: downwardAPI information about the downwardAPI data to project properties: items: description: Items is a list of DownwardAPIVolume file items: description: DownwardAPIVolumeFile represents information to create the file containing the pod field properties: fieldRef: description: 'Required: Selects a field of the pod: only annotations, labels, name, namespace and uid are supported.' properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic mode: description: |- Optional: mode bits used to set permissions on this file, must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: 'Required: Path is the relative path name of the file to be created. Must not be absolute or contain the ''..'' path. Must be utf-8 encoded. The first item of the relative path must not start with ''..''' type: string resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, requests.cpu and requests.memory) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic required: - path type: object type: array x-kubernetes-list-type: atomic type: object podCertificate: description: |- Projects an auto-rotating credential bundle (private key and certificate chain) that the pod can use either as a TLS client or server. Kubelet generates a private key and uses it to send a PodCertificateRequest to the named signer. Once the signer approves the request and issues a certificate chain, Kubelet writes the key and certificate chain to the pod filesystem. The pod does not start until certificates have been issued for each podCertificate projected volume source in its spec. Kubelet will begin trying to rotate the certificate at the time indicated by the signer using the PodCertificateRequest.Status.BeginRefreshAt timestamp. Kubelet can write a single file, indicated by the credentialBundlePath field, or separate files, indicated by the keyPath and certificateChainPath fields. The credential bundle is a single file in PEM format. The first PEM entry is the private key (in PKCS#8 format), and the remaining PEM entries are the certificate chain issued by the signer (typically, signers will return their certificate chain in leaf-to-root order). Prefer using the credential bundle format, since your application code can read it atomically. If you use keyPath and certificateChainPath, your application must make two separate file reads. If these coincide with a certificate rotation, it is possible that the private key and leaf certificate you read may not correspond to each other. Your application will need to check for this condition, and re-read until they are consistent. The named signer controls chooses the format of the certificate it issues; consult the signer implementation's documentation to learn how to use the certificates it issues. properties: certificateChainPath: description: |- Write the certificate chain at this path in the projected volume. Most applications should use credentialBundlePath. When using keyPath and certificateChainPath, your application needs to check that the key and leaf certificate are consistent, because it is possible to read the files mid-rotation. type: string credentialBundlePath: description: |- Write the credential bundle at this path in the projected volume. The credential bundle is a single file that contains multiple PEM blocks. The first PEM block is a PRIVATE KEY block, containing a PKCS#8 private key. The remaining blocks are CERTIFICATE blocks, containing the issued certificate chain from the signer (leaf and any intermediates). Using credentialBundlePath lets your Pod's application code make a single atomic read that retrieves a consistent key and certificate chain. If you project them to separate files, your application code will need to additionally check that the leaf certificate was issued to the key. type: string keyPath: description: |- Write the key at this path in the projected volume. Most applications should use credentialBundlePath. When using keyPath and certificateChainPath, your application needs to check that the key and leaf certificate are consistent, because it is possible to read the files mid-rotation. type: string keyType: description: |- The type of keypair Kubelet will generate for the pod. Valid values are "RSA3072", "RSA4096", "ECDSAP256", "ECDSAP384", "ECDSAP521", and "ED25519". type: string maxExpirationSeconds: description: |- maxExpirationSeconds is the maximum lifetime permitted for the certificate. Kubelet copies this value verbatim into the PodCertificateRequests it generates for this projection. If omitted, kube-apiserver will set it to 86400(24 hours). kube-apiserver will reject values shorter than 3600 (1 hour). The maximum allowable value is 7862400 (91 days). The signer implementation is then free to issue a certificate with any lifetime *shorter* than MaxExpirationSeconds, but no shorter than 3600 seconds (1 hour). This constraint is enforced by kube-apiserver. `kubernetes.io` signers will never issue certificates with a lifetime longer than 24 hours. format: int32 type: integer signerName: description: Kubelet's generated CSRs will be addressed to this signer. type: string userAnnotations: additionalProperties: type: string description: |- userAnnotations allow pod authors to pass additional information to the signer implementation. Kubernetes does not restrict or validate this metadata in any way. These values are copied verbatim into the `spec.unverifiedUserAnnotations` field of the PodCertificateRequest objects that Kubelet creates. Entries are subject to the same validation as object metadata annotations, with the addition that all keys must be domain-prefixed. No restrictions are placed on values, except an overall size limitation on the entire field. Signers should document the keys and values they support. Signers should deny requests that contain keys they do not recognize. type: object required: - keyType - signerName type: object secret: description: secret information about the secret data to project properties: items: description: |- items if unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. items: description: Maps a string key to a path within a volume. properties: key: description: key is the key to project. type: string mode: description: |- mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: |- path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. type: string required: - key - path type: object type: array x-kubernetes-list-type: atomic name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: optional field specify whether the Secret or its key must be defined type: boolean type: object x-kubernetes-map-type: atomic serviceAccountToken: description: serviceAccountToken is information about the serviceAccountToken data to project properties: audience: description: |- audience is the intended audience of the token. A recipient of a token must identify itself with an identifier specified in the audience of the token, and otherwise should reject the token. The audience defaults to the identifier of the apiserver. type: string expirationSeconds: description: |- expirationSeconds is the requested duration of validity of the service account token. As the token approaches expiration, the kubelet volume plugin will proactively rotate the service account token. The kubelet will start trying to rotate the token if the token is older than 80 percent of its time to live or if the token is older than 24 hours.Defaults to 1 hour and must be at least 10 minutes. format: int64 type: integer path: description: |- path is the path relative to the mount point of the file to project the token into. type: string required: - path type: object type: object type: array x-kubernetes-list-type: atomic type: object quobyte: description: |- quobyte represents a Quobyte mount on the host that shares a pod's lifetime. Deprecated: Quobyte is deprecated and the in-tree quobyte type is no longer supported. properties: group: description: |- group to map volume access to Default is no group type: string readOnly: description: |- readOnly here will force the Quobyte volume to be mounted with read-only permissions. Defaults to false. type: boolean registry: description: |- registry represents a single or multiple Quobyte Registry services specified as a string as host:port pair (multiple entries are separated with commas) which acts as the central registry for volumes type: string tenant: description: |- tenant owning the given Quobyte volume in the Backend Used with dynamically provisioned Quobyte volumes, value is set by the plugin type: string user: description: |- user to map volume access to Defaults to serivceaccount user type: string volume: description: volume is a string that references an already created Quobyte volume by name. type: string required: - registry - volume type: object rbd: description: |- rbd represents a Rados Block Device mount on the host that shares a pod's lifetime. Deprecated: RBD is deprecated and the in-tree rbd type is no longer supported. properties: fsType: description: |- fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd type: string image: description: |- image is the rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: string keyring: default: /etc/ceph/keyring description: |- keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: string monitors: description: |- monitors is a collection of Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it items: type: string type: array x-kubernetes-list-type: atomic pool: default: rbd description: |- pool is the rados pool name. Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: string readOnly: description: |- readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: boolean secretRef: description: |- secretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic user: default: admin description: |- user is the rados user name. Default is admin. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it type: string required: - image - monitors type: object scaleIO: description: |- scaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes. Deprecated: ScaleIO is deprecated and the in-tree scaleIO type is no longer supported. properties: fsType: default: xfs description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Default is "xfs". type: string gateway: description: gateway is the host address of the ScaleIO API Gateway. type: string protectionDomain: description: protectionDomain is the name of the ScaleIO Protection Domain for the configured storage. type: string readOnly: description: |- readOnly Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean secretRef: description: |- secretRef references to the secret for ScaleIO user and other sensitive information. If this is not provided, Login operation will fail. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic sslEnabled: description: sslEnabled Flag enable/disable SSL communication with Gateway, default false type: boolean storageMode: default: ThinProvisioned description: |- storageMode indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is ThinProvisioned. type: string storagePool: description: storagePool is the ScaleIO Storage Pool associated with the protection domain. type: string system: description: system is the name of the storage system as configured in ScaleIO. type: string volumeName: description: |- volumeName is the name of a volume already created in the ScaleIO system that is associated with this volume source. type: string required: - gateway - secretRef - system type: object secret: description: |- secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret properties: defaultMode: description: |- defaultMode is Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer items: description: |- items If unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. items: description: Maps a string key to a path within a volume. properties: key: description: key is the key to project. type: string mode: description: |- mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. format: int32 type: integer path: description: |- path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. type: string required: - key - path type: object type: array x-kubernetes-list-type: atomic optional: description: optional field specify whether the Secret or its keys must be defined type: boolean secretName: description: |- secretName is the name of the secret in the pod's namespace to use. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret type: string type: object storageos: description: |- storageOS represents a StorageOS volume attached and mounted on Kubernetes nodes. Deprecated: StorageOS is deprecated and the in-tree storageos type is no longer supported. properties: fsType: description: |- fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string readOnly: description: |- readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. type: boolean secretRef: description: |- secretRef specifies the secret to use for obtaining the StorageOS API credentials. If not specified, default values will be attempted. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic volumeName: description: |- volumeName is the human-readable name of the StorageOS volume. Volume names are only unique within a namespace. type: string volumeNamespace: description: |- volumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created. type: string type: object vsphereVolume: description: |- vsphereVolume represents a vSphere volume attached and mounted on kubelets host machine. Deprecated: VsphereVolume is deprecated. All operations for the in-tree vsphereVolume type are redirected to the csi.vsphere.vmware.com CSI driver. properties: fsType: description: |- fsType is filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. type: string storagePolicyID: description: storagePolicyID is the storage Policy Based Management (SPBM) profile ID associated with the StoragePolicyName. type: string storagePolicyName: description: storagePolicyName is the storage Policy Based Management (SPBM) profile name. type: string volumePath: description: volumePath is the path that identifies vSphere volume vmdk type: string required: - volumePath type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map x-kubernetes-validations: - message: field is immutable rule: self == oldSelf type: object type: object type: object type: object required: - name type: object maxItems: 100 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map type: object type: object type: object required: - manager type: object maxItems: 16 type: array x-kubernetes-list-map-keys: - manager x-kubernetes-list-type: map runtimeRef: description: runtimeRef is the reference to the training runtime. properties: apiGroup: default: trainer.kubeflow.org description: |- apiGroup of the runtime being referenced. Defaults to `trainer.kubeflow.org`. maxLength: 253 type: string kind: default: ClusterTrainingRuntime description: |- kind of the runtime being referenced. Defaults to ClusterTrainingRuntime. maxLength: 253 type: string name: description: |- name of the runtime being referenced. When namespaced-scoped TrainingRuntime is used, the TrainJob must have the same namespace as the deployed runtime. maxLength: 253 minLength: 1 type: string required: - name type: object x-kubernetes-validations: - message: field is immutable rule: self == oldSelf suspend: default: false description: suspend defines whether to suspend the running TrainJob. type: boolean trainer: description: trainer defines the configuration of the trainer. properties: args: description: args for the entrypoint for the training container. items: maxLength: 65536 type: string maxItems: 128 type: array x-kubernetes-list-type: atomic command: description: command for the entrypoint of the training container. items: maxLength: 65536 type: string maxItems: 128 type: array x-kubernetes-list-type: atomic env: description: |- env is the list of environment variables to set in the training container. These values will be merged with the TrainingRuntime's trainer environments. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object maxItems: 128 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map image: description: image is the container image for the training container. maxLength: 500 type: string numNodes: description: numNodes is the number of training nodes. format: int32 type: integer numProcPerNode: description: |- numProcPerNode is the number of processes/workers/slots on every training node. For the MPI runtime only int value can be set to represent number of slots per node. For the Torch runtime the value defaults to `auto` and can be overridden with an int. format: int32 type: integer resourcesPerNode: description: resourcesPerNode defines the compute resources for each training node. properties: claims: description: |- Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container. This field depends on the DynamicResourceAllocation feature gate. This field is immutable. It can only be set for containers. items: description: ResourceClaim references one entry in PodSpec.ResourceClaims. properties: name: description: |- Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. type: string request: description: |- Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. type: string required: - name type: object type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map limits: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object requests: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object type: object type: object x-kubernetes-validations: - message: field is immutable rule: self == oldSelf required: - runtimeRef type: object type: object x-kubernetes-validations: - message: workload is immutable rule: self == oldSelf workloadMetadata: description: |- workloadMetadata sets labels on the workload object created from spec.workload. This is the canonical boundary for workload-object labels: WorkloadRun and Certification both resolve their own workloadMetadata into this field, and a hand-authored Workflow writes spec.jobTemplate.spec.workloadMetadata directly. Ordinary Job labels (Workflow spec.jobTemplate.metadata.labels) are not copied here. Labels are executable policy, so opting a workload into an admission controller or a queue has to be written down explicitly. Immutable in both presence and value, so a checkpoint restart recreates the workload with the labels it was admitted with. properties: labels: additionalProperties: description: |- WorkloadLabelValue is a Kubernetes label value in a WorkloadMetadata labels map. It exists as a named type only so controller-gen emits the length and pattern constraints under the map's additionalProperties: there is no marker that attaches value constraints to a map[string]string. maxLength: 63 pattern: ^$|^[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$ type: string description: |- labels are merged onto the generated workload object's metadata.labels. Keys must be valid Kubernetes label keys: an optional DNS-subdomain prefix of at most 253 characters followed by "/", then a name of at most 63 characters. Values follow Kubernetes label-value syntax and may be empty; an empty value is a real label, not a deletion. The controller-owned keys "app.kubernetes.io/managed-by" and anything under the "nvcre.nvidia.com/" prefix are rejected rather than silently overwritten — they identify controller ownership and Job association. At most 32 entries. That bound is an enforced API limit, chosen as the product decision for how many workload labels one object may carry. (A finite maxProperties is separately required to bound the static cost of the CEL rules below, but the cost estimator accepts far larger values, so it does not pick this number.) The key grammar uses [.] rather than \. because the marker parser collapses the escape and CEL then rejects \. as an invalid escape in a string literal. The two are equivalent to the regex engine. maxProperties: 32 type: object x-kubernetes-validations: - message: labels must not set the controller-owned keys app.kubernetes.io/managed-by or any key under the nvcre.nvidia.com/ prefix rule: self.all(k, k != 'app.kubernetes.io/managed-by' && !k.startsWith('nvcre.nvidia.com/')) - message: 'each label key must be a valid Kubernetes label key: an optional DNS-subdomain prefix followed by ''/'', then a name of at most 63 characters beginning and ending with an alphanumeric character' rule: self.all(k, k.matches('^([a-z0-9]([-a-z0-9]*[a-z0-9])?([.][a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$')) - message: each label key prefix must be at most 253 characters rule: 'self.all(k, k.contains(''/'') ? k.split(''/'')[0].size() <= 253 : true)' type: object x-kubernetes-validations: - message: workloadMetadata is immutable rule: self == oldSelf required: - workload type: object x-kubernetes-validations: - message: workloadMetadata cannot be added or removed after creation rule: has(self.workloadMetadata) == has(oldSelf.workloadMetadata) - message: nodeHealthMonitor cannot be added or removed after creation rule: has(self.nodeHealthMonitor) == has(oldSelf.nodeHealthMonitor) - message: goodputMeasurement cannot be added or removed after creation rule: has(self.goodputMeasurement) == has(oldSelf.goodputMeasurement) - message: bandwidthMeasurement cannot be added or removed after creation rule: has(self.bandwidthMeasurement) == has(oldSelf.bandwidthMeasurement) required: - spec type: object namespace: description: |- namespace is the target namespace for Jobs and dependencies created by this Workflow. If not specified, the controller auto-generates one. The controller creates the namespace if it doesn't exist and sets an owner reference so it's cleaned up when the Workflow is deleted. type: string orchestration: description: orchestration defines topology-aware orchestration strategy. properties: diagnose: description: |- diagnose enables adaptive fault isolation using topology-aware hierarchical group testing. Stage 1a screens each topology domain in parallel, Stage 1b tests inter-domain fabric, Stage 2 bisects failed groups, Stage 3 confirms suspects. Identifies all faulty nodes in O(d log N) tests. properties: minGroupSize: default: 2 description: |- minGroupSize is the smallest group at which bisection stops. Nodes in failed groups of this size become suspects for confirmation. minimum: 1 type: integer topologyKey: description: |- topologyKey is the node label key for topology-aware screening. When set, Stage 1 groups nodes by this label (e.g., one group per clique). When empty, nodes are grouped into disjoint chunks of ceil(sqrt(N)). Example: "nvidia.com/gpu.clique" type: string type: object execution: description: execution defines how jobs are scheduled across groups. properties: maxConcurrent: default: 0 description: |- maxConcurrent is the maximum number of jobs to run concurrently. 0 means unlimited. Default is 0. minimum: 0 type: integer retryFailedGroups: default: 0 description: |- retryFailedGroups is the number of times to retry a failed group's job. After the workload controller exhausts its own restarts (backoffLimit), the orchestration re-creates the entire job up to this many times. 0 means no retries. Default is 0. minimum: 0 type: integer timeoutPerJob: description: timeoutPerJob is the maximum time to wait for each job to complete. type: string type: object iterations: default: 1 description: |- iterations is the number of times to run all jobs (one full pass = one iteration). Default is 1. minimum: 1 type: integer target: description: |- target specifies which nodes to include in the orchestration. When nil, all nodes in the cluster are targeted. properties: matchExpressions: description: |- matchExpressions is a list of node selector requirements. Supports In, NotIn, Exists, DoesNotExist operators. All expressions must match (AND with nodeSelector). items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array nodeNames: description: |- nodeNames explicitly lists nodes to include. Nodes must also match nodeSelector and matchExpressions, if set. items: type: string type: array nodeSelector: additionalProperties: type: string description: nodeSelector selects nodes by label key-value pairs. type: object taintSelectors: description: |- taintSelectors selects nodes that have ALL matching taints. A node must have every taint in this list to be included. The controller auto-injects matching tolerations onto the workload pods so they can schedule onto the selected tainted nodes. Setting this field is the supported way to run training workloads (e.g. TrainJob) on tainted nodes. When taintSelectors is set, the explicit tolerations replace the blanket Operator: Exists toleration that MPI workloads (NCCL tests) otherwise receive — declared selectors always win. items: description: TaintSelector selects nodes by taint. properties: effect: description: |- effect is the taint effect to match (NoSchedule, PreferNoSchedule, NoExecute). If empty, matches any effect. type: string key: description: key is the taint key to match. minLength: 1 type: string value: description: value is the taint value to match. If empty, matches any value. type: string required: - key type: object type: array type: object topology: description: |- topology configures rack/zone-aware placement. When set, jobs are packed into the minimum number of topology domains. When nil, nodes are grouped by simple chunking (no topology awareness). properties: strictDomain: description: |- strictDomain creates one group per topology domain at its natural size. Groups never cross domain boundaries. Set by testScale=intra-rack. type: boolean topologyKey: description: |- topologyKey is the node label key that identifies the topology domain. Examples: "nvidia.com/gpu.clique", "topology.kubernetes.io/zone" Nodes with the same label value are in the same domain (rack/zone). minLength: 1 type: string required: - topologyKey type: object type: object overrides: description: |- overrides defines conditional patches to the base spec. Overrides are evaluated in order; all matching overrides are applied sequentially. items: description: OverrideSpec defines a conditional patch to the base WorkflowSpec. properties: dependencies: description: dependencies are additional dependencies appended when this override matches. items: description: DependencySpec defines a resource that must be created before the workload. properties: when: description: |- when defines conditions for creating this dependency. If not specified, the dependency is always created. properties: config: description: config matches against values in validation.performance.config. x-kubernetes-preserve-unknown-fields: true expression: description: |- expression is a CEL expression that evaluates to a boolean. Available variables: gpuArchitecture, platform, workloadKind, topology, config Example: "gpuArchitecture == 'gb200' && platform == 'aws'" type: string gpuArchitecture: description: gpuArchitecture matches against detected GPU architecture. properties: equals: description: equals checks for exact match. type: string in: description: in checks if value is in the list. items: type: string type: array notIn: description: notIn checks if value is not in the list. items: type: string type: array type: object platform: description: platform matches against detected cloud platform. properties: equals: description: equals checks for exact match. type: string in: description: in checks if value is in the list. items: type: string type: array notIn: description: notIn checks if value is not in the list. items: type: string type: array type: object topology: description: topology matches against cluster topology characteristics. properties: domainCount: description: domainCount matches against the number of topology domains (racks, zones, etc.). properties: equals: description: equals checks for exact match. type: integer greaterThan: description: greaterThan checks if value is greater than threshold. type: integer lessThan: description: lessThan checks if value is less than threshold. type: integer type: object mode: description: |- mode matches against topology mode. Examples: "single-rack", "multi-rack", "multi-zone" type: string type: object workloadKind: description: |- workloadKind matches against the workload kind. Example: "TrainJob" type: string type: object type: object x-kubernetes-embedded-resource: true x-kubernetes-preserve-unknown-fields: true type: array jobTemplate: description: |- jobTemplate is a strategic merge patch applied to the base jobTemplate. Maps merge recursively, named arrays (e.g., env vars) merge by key, unnamed arrays (e.g., args) replace entirely. Stored as raw JSON to avoid CRD validation cost budget issues. x-kubernetes-preserve-unknown-fields: true jobTemplatePatch: description: |- jobTemplatePatch is an RFC 6902 JSON Patch applied to the base jobTemplate. Use for precise operations: remove specific env vars, add at index, test a value before patching (preconditions). Applied AFTER the jobTemplate strategic merge patch. x-kubernetes-preserve-unknown-fields: true orchestration: description: |- orchestration overrides orchestration settings (target, topology, execution, iterations). Non-nil fields replace the base value; nil fields are preserved. properties: execution: description: execution overrides how jobs are scheduled across groups. properties: maxConcurrent: default: 0 description: |- maxConcurrent is the maximum number of jobs to run concurrently. 0 means unlimited. Default is 0. minimum: 0 type: integer retryFailedGroups: default: 0 description: |- retryFailedGroups is the number of times to retry a failed group's job. After the workload controller exhausts its own restarts (backoffLimit), the orchestration re-creates the entire job up to this many times. 0 means no retries. Default is 0. minimum: 0 type: integer timeoutPerJob: description: timeoutPerJob is the maximum time to wait for each job to complete. type: string type: object iterations: description: iterations overrides the number of times to run all jobs. type: integer target: description: target overrides which nodes to include in the orchestration. properties: matchExpressions: description: |- matchExpressions is a list of node selector requirements. Supports In, NotIn, Exists, DoesNotExist operators. All expressions must match (AND with nodeSelector). items: description: |- A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: description: |- Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: description: |- An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array nodeNames: description: |- nodeNames explicitly lists nodes to include. Nodes must also match nodeSelector and matchExpressions, if set. items: type: string type: array nodeSelector: additionalProperties: type: string description: nodeSelector selects nodes by label key-value pairs. type: object taintSelectors: description: |- taintSelectors selects nodes that have ALL matching taints. A node must have every taint in this list to be included. The controller auto-injects matching tolerations onto the workload pods so they can schedule onto the selected tainted nodes. Setting this field is the supported way to run training workloads (e.g. TrainJob) on tainted nodes. When taintSelectors is set, the explicit tolerations replace the blanket Operator: Exists toleration that MPI workloads (NCCL tests) otherwise receive — declared selectors always win. items: description: TaintSelector selects nodes by taint. properties: effect: description: |- effect is the taint effect to match (NoSchedule, PreferNoSchedule, NoExecute). If empty, matches any effect. type: string key: description: key is the taint key to match. minLength: 1 type: string value: description: value is the taint value to match. If empty, matches any value. type: string required: - key type: object type: array type: object topology: description: topology overrides rack/zone-aware placement configuration. properties: strictDomain: description: |- strictDomain creates one group per topology domain at its natural size. Groups never cross domain boundaries. Set by testScale=intra-rack. type: boolean topologyKey: description: |- topologyKey is the node label key that identifies the topology domain. Examples: "nvidia.com/gpu.clique", "topology.kubernetes.io/zone" Nodes with the same label value are in the same domain (rack/zone). minLength: 1 type: string required: - topologyKey type: object type: object when: description: when defines conditions for applying this override. properties: config: description: config matches against values in validation.performance.config. x-kubernetes-preserve-unknown-fields: true expression: description: |- expression is a CEL expression that evaluates to a boolean. Available variables: gpuArchitecture, platform, workloadKind, topology, config Example: "gpuArchitecture == 'gb200' && platform == 'aws'" type: string gpuArchitecture: description: gpuArchitecture matches against detected GPU architecture. properties: equals: description: equals checks for exact match. type: string in: description: in checks if value is in the list. items: type: string type: array notIn: description: notIn checks if value is not in the list. items: type: string type: array type: object platform: description: platform matches against detected cloud platform. properties: equals: description: equals checks for exact match. type: string in: description: in checks if value is in the list. items: type: string type: array notIn: description: notIn checks if value is not in the list. items: type: string type: array type: object topology: description: topology matches against cluster topology characteristics. properties: domainCount: description: domainCount matches against the number of topology domains (racks, zones, etc.). properties: equals: description: equals checks for exact match. type: integer greaterThan: description: greaterThan checks if value is greater than threshold. type: integer lessThan: description: lessThan checks if value is less than threshold. type: integer type: object mode: description: |- mode matches against topology mode. Examples: "single-rack", "multi-rack", "multi-zone" type: string type: object workloadKind: description: |- workloadKind matches against the workload kind. Example: "TrainJob" type: string type: object required: - when type: object type: array validation: description: validation defines performance validation and metrics collection. properties: performance: description: performance configures performance validation and metrics collection. properties: config: description: config contains plugin-specific configuration parameters. x-kubernetes-preserve-unknown-fields: true enabled: default: false description: enabled determines if performance validation is active. type: boolean measurementTimeout: description: |- measurementTimeout is the maximum time to wait after a Job succeeds for measurement data before failing threshold validation. Propagated to each Job's spec during creation. type: string plugin: description: |- plugin is the name of the benchmark plugin to use. Examples: "nemo-training", "nccl-tests" type: string thresholds: description: |- thresholds defines minimum acceptable performance thresholds. When set, the Workflow controller propagates these to each Job's measurement configs during creation. The Job controller evaluates them after the workload succeeds and sets the ValidationFailed condition if any threshold is not met. properties: thresholds: additionalProperties: type: string description: thresholds maps metric names to CEL expressions. type: object type: object tracking: description: tracking defines what results to track. properties: perDomain: default: false description: |- perDomain tracks aggregated results per topology domain. Domains are defined by the topology key in orchestration. For example, if using "nvidia.com/gpu.clique", results are aggregated per unique clique value. type: boolean perNode: default: false description: perNode tracks results for each individual node. type: boolean type: object required: - enabled type: object type: object required: - jobTemplate - orchestration type: object x-kubernetes-validations: - message: gangScheduler cannot be added or removed after creation rule: has(self.gangScheduler) == has(oldSelf.gangScheduler) status: description: status defines the observed state of Workflow properties: conditions: description: |- conditions represent the current state of the Workflow resource. Condition types: - "InProgress": the Workflow is currently running - "Succeeded": the Workflow completed successfully - "Failed": the Workflow has failed The status of each condition is one of True, False, or Unknown. items: description: Condition contains details for one aspect of the current state of this API Resource. properties: lastTransitionTime: description: |- lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. format: date-time type: string message: description: |- message is a human readable message indicating details about the transition. This may be an empty string. maxLength: 32768 type: string observedGeneration: description: |- observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. format: int64 minimum: 0 type: integer reason: description: |- reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty. maxLength: 1024 minLength: 1 pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ type: string status: description: status of the condition, one of True, False, Unknown. enum: - "True" - "False" - Unknown type: string type: description: type of condition in CamelCase or in foo.example.com/CamelCase. maxLength: 316 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ type: string required: - lastTransitionTime - message - reason - status - type type: object type: array x-kubernetes-list-map-keys: - type x-kubernetes-list-type: map dependencyRefs: description: dependencyRefs lists the dependency resources created by this Workflow. items: description: DependencyResourceRef tracks a dependency resource created by a Workflow. properties: apiVersion: description: apiVersion of the created resource. type: string groupName: description: groupName identifies the group this dependency was created for (scope=job). type: string iteration: description: iteration identifies the iteration this dependency was created for (scope=job). type: integer kind: description: kind of the created resource. type: string name: description: name of the created resource. type: string namespace: description: namespace of the created resource. Empty for cluster-scoped resources. type: string scope: description: scope is the lifecycle scope of this dependency (workflow, job). type: string required: - apiVersion - kind - name type: object type: array domainResults: additionalProperties: description: DomainResult contains aggregated metrics for a topology domain. properties: averageTflops: description: |- averageTFLOPs is the average TFLOPs across all nodes in the domain as a string. Example: "1500.5" type: string nodeCount: description: nodeCount is the number of nodes in this domain. type: integer status: description: |- status is the overall status for this domain. Examples: "Passed", "Failed", "Running" type: string required: - nodeCount - status type: object description: |- domainResults contains aggregated metrics grouped by topology domain values. Key is the domain value (e.g., "clique-0", "rack-1", "zone-a"). Only populated when spec.validation.performance.tracking.perDomain is true. type: object failedNodesRef: description: |- failedNodesRef references the ConfigMap holding this Workflow's failed-nodes list (name, reason, message). It is written incrementally as Jobs fail. properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string required: - kind - name type: object x-kubernetes-map-type: atomic namespace: description: namespace is the resolved namespace where Jobs and dependencies are created. type: string nodeResults: additionalProperties: description: NodeResult contains performance metrics for a single node. properties: domain: description: |- domain is the topology domain this node belongs to. For example, if using "nvidia.com/gpu.clique", this is the clique value. type: string group: description: group is the group name that ran on this node. type: string iterations: description: iterations contains metrics for each iteration on this node. items: description: IterationMetrics contains performance metrics for one iteration. properties: duration: description: |- duration is the iteration duration in seconds as a string. Example: "3600.5" type: string iterationNumber: description: iterationNumber is the iteration index (1-based). type: integer memoryUsedGB: description: |- memoryUsedGB is the peak GPU memory usage in GB as a string. Example: "640.25" type: string status: description: |- status is the status of this iteration. Examples: "Succeeded", "Failed" type: string tflops: description: |- tflops is the measured TFLOPs performance as a string. Example: "1500.5" type: string required: - iterationNumber - status type: object type: array status: description: |- status is the validation status for this node. Examples: "Passed", "Failed", "Running" type: string required: - domain - group - status type: object description: |- nodeResults contains per-node performance metrics. Only populated when spec.validation.performance.tracking.perNode is true. type: object orchestration: description: orchestration contains the state of orchestration execution. properties: appliedOverrides: description: |- appliedOverrides records which overrides from spec.overrides[] were applied after platform and GPU architecture detection completed. items: description: AppliedOverride records that a specific override was applied during reconciliation. properties: index: description: index is the zero-based position of the override in spec.overrides[]. type: integer noOp: description: |- noOp is true when the override matched but produced no changes. This usually indicates a misconfigured override with wrong field paths. type: boolean patches: description: |- patches describes what the override modifies. Example: "jobTemplate, 2 dependencies" type: string when: description: |- when is a human-readable summary of the conditions that matched. Example: "gpuArchitecture=h100, platform=aws" type: string required: - index - patches - when type: object type: array completedIterations: description: completedIterations is the number of iterations fully completed. type: integer currentIteration: description: currentIteration is the iteration currently in progress (1-based). type: integer detectedGPUArchitecture: description: |- detectedGPUArchitecture is the GPU architecture detected from node labels. Examples: "h100", "gb200", "a100", "l40s", "unknown" type: string detectedPlatform: description: |- detectedPlatform is the cloud platform detected from node labels. Used by WhenSpec for conditional dependency creation. Examples: "aws", "gcp", "onprem", "unknown" type: string diagnose: description: diagnose tracks adaptive fault isolation algorithm progress. properties: crossBoundaryState: description: |- crossBoundaryState tracks in-progress cross-boundary probing. Nil when not in cross-boundary stage. properties: originStage: description: |- originStage records which diagnose stage triggered the cross-boundary probing, used to determine MNNVL settings for probe jobs. type: string pendingProbes: description: pendingProbes are boundary investigations still in progress. items: description: |- CrossBoundaryProbe tracks one cross-boundary investigation between two halves of a bisection group that both passed individually. properties: domain: description: domain is the topology domain being investigated. type: string halfA: description: halfA is the first half of the bisection that passed. items: type: string type: array halfB: description: halfB is the second half of the bisection that passed. items: type: string type: array probeRound: description: probeRound is the current probe round within this investigation. type: integer required: - domain - halfA - halfB - probeRound type: object type: array required: - originStage type: object healthyNodes: description: healthyNodes are nodes confirmed healthy at any stage (accumulated). items: type: string type: array x-kubernetes-list-type: set infrastructureFaults: description: |- infrastructureFaults records detected infrastructure-level faults where bisection both-halves-pass but the full group fails. Each entry identifies the node groups whose interconnecting infrastructure is suspected faulty. items: description: |- InfrastructureFault records a detected infrastructure-level fault between two node groups in the same topology domain. The fault is in the interconnect (NVSwitch trunk, spine switch, cable) rather than in individual nodes. properties: domain: description: domain is the topology domain where the fault was detected. type: string groupA: description: groupA is one set of nodes on one side of the faulty boundary. items: type: string type: array groupB: description: groupB is the other set of nodes. items: type: string type: array stage: description: stage is the diagnose stage where the fault was detected. type: string required: - domain - groupA - groupB - stage type: object type: array noNVLScreeningResults: additionalProperties: description: DomainScreeningResult records the outcome of screening one topology domain. properties: nodes: description: nodes in this domain. items: type: string type: array passed: description: passed is true if the screening test succeeded. type: boolean required: - nodes - passed type: object description: |- noNVLScreeningResults preserves per-domain screening outcomes from the intra-screening-no-nvl stage (MNNVL disabled). Same structure as screeningResults. type: object noNVLSuspectNodes: description: |- noNVLSuspectNodes are suspect nodes that originated from the intra-screening-no-nvl stage. Confirmation tests for these nodes run with NCCL_MNNVL_ENABLE=0 to isolate fabric issues. items: type: string type: array x-kubernetes-list-type: set representativeNodes: description: |- representativeNodes are one healthy node per topology domain, selected after intra-domain screening for inter-domain fabric testing. items: type: string type: array round: description: round is the total number of completed rounds across all stages. type: integer screeningResults: additionalProperties: description: DomainScreeningResult records the outcome of screening one topology domain. properties: nodes: description: nodes in this domain. items: type: string type: array passed: description: passed is true if the screening test succeeded. type: boolean required: - nodes - passed type: object description: |- screeningResults preserves per-domain screening outcomes across stages. Key is the domain name (e.g., clique UUID), value is the domain's screening result. type: object stage: description: |- stage is the current stage of the diagnose algorithm. Values: "intra-screening", "inter-screening", "bisection", "confirmation" type: string suspectNodes: description: |- suspectNodes are nodes in minGroupSize groups that failed bisection, pending confirmation in Stage 3. items: type: string type: array x-kubernetes-list-type: set required: - round - stage type: object excludedNodes: description: |- excludedNodes lists nodes that matched the target but were dropped before scheduling, and exclusionReason says why. A Workflow can succeed while excluding nodes, so these record what was left untested. Causes: the node was cordoned, the target set has more than one GPU architecture and the run continued on the primary architecture alone, or the node reports fewer allocatable GPUs than the workload requests per node. items: type: string type: array exclusionReason: description: exclusionReason explains why excludedNodes were dropped. type: string groups: description: Group phases are reset to Pending at the start of each new iteration. items: description: GroupStatus tracks the state of a scheduling group's job within the current iteration. properties: completionTime: description: completionTime when this group's job finished in the current iteration. format: date-time type: string domainNodeCounts: additionalProperties: type: integer description: |- domainNodeCounts records the number of nodes per topology domain in this group, keyed by domain value. Populated at partition time so reports can show per-clique node counts without fetching nodes. type: object domains: description: domains lists topology domain values this group spans. items: type: string type: array jobRef: description: jobRef references the current job for this group. properties: apiVersion: description: apiVersion is the API version of the workload resource. type: string kind: description: kind is the kind of the workload resource. type: string name: description: name is the name of the workload resource. type: string namespace: description: |- namespace is the namespace of the workload resource. If not specified, defaults to the Job's namespace. type: string required: - apiVersion - kind - name type: object name: description: name is a unique identifier (e.g., "group-0", "group-1-overflow"). type: string nodes: description: nodes in this group. items: type: string type: array overflow: description: |- overflow indicates this group contains borrowed nodes and must wait for the jobs using those nodes to complete before launching. type: boolean phase: description: |- phase is the current phase of this group's job in the current iteration. Reset to Pending at the start of each iteration. enum: - Pending - Running - Succeeded - Failed type: string retries: description: |- retries is the number of times the current iteration's job has been retried. Reset to 0 at the start of each iteration. type: integer startTime: description: startTime when this group's job started in the current iteration. format: date-time type: string required: - name - nodes - phase type: object type: array iterationHistory: description: |- iterationHistory records per-group outcomes for each completed iteration. Appended when an iteration completes and more iterations remain. The final iteration's results stay in Groups (not duplicated here). items: description: IterationResult records the outcome of a completed iteration. properties: groups: description: groups contains the outcome of each group in this iteration. items: description: GroupIterationResult records one group's outcome for a single iteration. properties: completionTime: description: completionTime when the group's job finished. format: date-time type: string jobName: description: |- jobName is the Job that ran for this group in this iteration. Empty if the Job was deleted before the snapshot. type: string name: description: name is the group identifier (matches GroupStatus.Name). type: string phase: description: phase is the terminal phase (Succeeded or Failed). type: string startTime: description: startTime when the group's job started. format: date-time type: string required: - name - phase type: object type: array iteration: description: iteration is the 1-based iteration number. type: integer required: - groups - iteration type: object type: array nodesPerJob: description: nodesPerJob auto-detected from workload template. type: integer totalGroups: description: totalGroups created. type: integer totalNodes: description: totalNodes discovered from target. type: integer required: - nodesPerJob - totalGroups - totalNodes type: object succeededNodesRef: description: |- succeededNodesRef references the ConfigMap holding this Workflow's succeeded-nodes list (comma-separated node names). It is written on terminal success and on terminal failure (capturing the nodes that passed). properties: apiGroup: description: |- APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. type: string kind: description: Kind is the type of resource being referenced type: string name: description: Name is the name of resource being referenced type: string required: - kind - name type: object x-kubernetes-map-type: atomic type: object required: - spec type: object served: true storage: true subresources: status: {} --- # Source: cluster-readiness-engine/crds/nvcre.nvidia.com_workloadruns.yaml --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.20.0 name: workloadruns.nvcre.nvidia.com spec: group: nvcre.nvidia.com names: kind: WorkloadRun listKind: WorkloadRunList plural: workloadruns singular: workloadrun scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: |- WorkloadRun is the Schema for the workloadruns API. It provides a simplified interface for running training, MPI, or arbitrary workloads on GPU clusters with automatic CSP and GPU architecture adaptation. properties: apiVersion: description: |- APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: description: |- Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object spec: description: |- spec defines the desired state of WorkloadRun. The entire spec is immutable after creation: once status.workflowRef is set the controller only mirrors the existing Workflow and never rebuilds it, so accepting edits would silently ignore them. To run with different inputs, delete the WorkloadRun and create a new one. properties: bandwidthMeasurement: description: |- bandwidthMeasurement enables NCCL bandwidth tracking. When set, a BandwidthMeasurement resource is auto-created for each Job. properties: logProfileRef: description: logProfileRef is the name of the cluster-scoped LogProfile to use for log parsing. minLength: 1 type: string sampleInterval: description: |- sampleInterval is how often to sample pod logs while the Job is running. Default: 60s. type: string testType: description: |- testType identifies the NCCL collective operation (e.g., "all_reduce", "alltoall"). Propagated to the BandwidthMeasurement spec for Prometheus metric labeling. type: string required: - logProfileRef type: object checkpoint: description: |- checkpoint configures checkpoint storage. When set, provisions a PVC and enables checkpoint-based restart. properties: maxRestarts: description: |- maxRestarts is how many times to restart from checkpoint on failure. Default: 0. format: int32 minimum: 0 type: integer storageClassName: description: storageClassName is the StorageClass to use. Defaults to cluster default. type: string storageSize: description: storageSize is the PVC size (e.g., "10Ti", "500Gi"). pattern: ^([0-9]+(\.[0-9]+)?)(Ki|Mi|Gi|Ti|Pi|Ei)?$ type: string required: - storageSize type: object config: description: |- config provides model configuration data. When inline is specified, a ConfigMap is created and mounted at /config. When configMapRef is specified, the referenced ConfigMap is mounted at /config. properties: configMapRef: description: |- configMapRef references an existing ConfigMap by name. Mounted at /config. properties: name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string type: object x-kubernetes-map-type: atomic inline: additionalProperties: type: string description: |- inline specifies config data as key-value pairs (filename -> content). Creates a ConfigMap and mounts it at /config. type: object type: object x-kubernetes-validations: - message: inline and configMapRef are mutually exclusive rule: '!(has(self.inline) && has(self.configMapRef))' enableMNNVL: description: |- enableMNNVL explicitly controls Multi-Node NVLink (NCCL_MNNVL_ENABLE). If nil, auto-detected from GPU architecture (enabled for GB200/GB300). type: boolean env: description: |- env specifies additional environment variables for the workload container. These are merged with auto-detected NCCL/platform env vars. User-specified values take precedence over auto-detected values. items: description: EnvVar represents an environment variable present in a Container. properties: name: description: |- Name of the environment variable. May consist of any printable ASCII characters except '='. type: string value: description: |- Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot be used if value is not empty. properties: configMapKeyRef: description: Selects a key of a ConfigMap. properties: key: description: The key to select. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the ConfigMap or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic fieldRef: description: |- Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is written in terms of, defaults to "v1". type: string fieldPath: description: Path of the field to select in the specified API version. type: string required: - fieldPath type: object x-kubernetes-map-type: atomic fileKeyRef: description: |- FileKeyRef selects a key of the env file. Requires the EnvFiles feature gate to be enabled. properties: key: description: |- The key within the env file. An invalid key will prevent the pod from starting. The keys defined within a source may consist of any printable ASCII characters except '='. During Alpha stage of the EnvFiles feature gate, the key size is limited to 128 characters. type: string optional: default: false description: |- Specify whether the file or its key must be defined. If the file or key does not exist, then the env var is not published. If optional is set to true and the specified key does not exist, the environment variable will not be set in the Pod's containers. If optional is set to false and the specified key does not exist, an error will be returned during Pod creation. type: boolean path: description: |- The path within the volume from which to select the file. Must be relative and may not contain the '..' path or start with '..'. type: string volumeName: description: The name of the volume mount containing the env file. type: string required: - key - path - volumeName type: object x-kubernetes-map-type: atomic resourceFieldRef: description: |- Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, optional for env vars' type: string divisor: anyOf: - type: integer - type: string description: Specifies the output format of the exposed resources, defaults to "1" pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string required: - resource type: object x-kubernetes-map-type: atomic secretKeyRef: description: Selects a key of a secret in the pod's namespace properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: default: "" description: |- Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object x-kubernetes-map-type: atomic type: object required: - name type: object type: array framework: description: |- framework specifies how to run the workload. Exactly one of torch, mpi, or exec must be set. maxProperties: 1 minProperties: 1 properties: exec: description: |- exec runs an arbitrary command directly. Auto-generates a simple TrainingRuntime with a single replicatedJob. properties: args: description: args are arguments to the command. items: type: string type: array command: description: command is the entrypoint array (e.g., ["/bin/bash", "/config/train.sh"]). items: type: string minItems: 1 type: array required: - command type: object mpi: description: |- mpi runs MPI-based workloads (e.g., NCCL tests). Auto-generates an MPI TrainingRuntime with launcher+worker pattern, SSH auth, IPC_LOCK capability, and readiness probes. properties: args: description: args are arguments to the MPI binary (e.g., ["-b", "8", "-e", "32G"]). items: type: string type: array binary: description: binary is the MPI binary to run (e.g., "/usr/local/bin/all_reduce_perf_mpi"). minLength: 1 type: string mpiArgs: description: |- mpiArgs are extra arguments to mpirun itself (e.g., ["--allow-run-as-root"]). Standard args (--mca plm_rsh_args, -x NCCL vars) are auto-injected. items: type: string type: array mpiImplementation: default: OpenMPI description: |- mpiImplementation is the MPI implementation to use. Default: "OpenMPI". type: string mpirunPath: description: mpirunPath is the path to the mpirun binary in the container image. minLength: 1 type: string required: - binary - mpirunPath type: object torch: description: |- torch runs distributed training via torchrun. Auto-generates a PyTorch TrainingRuntime with torch mlPolicy, numProcPerNode, GPU resources, and shared memory. properties: args: description: args are additional arguments passed to the script or module. items: type: string type: array module: description: |- module is the Python module to run (torchrun -m ). Mutually exclusive with script. minLength: 1 type: string script: description: |- script is the path to run (torchrun