This example runs a small public Qwen model through vLLM's OpenAI-compatible API. It is the inexpensive runtime check for the larger EKS inference configuration. The EKS configuration uses a quantized 7B model on an NVIDIA L4; this starter uses the same service and client shape with a 0.5B model on an ARM64 CPU.
The choices are pinned:
- vLLM CPU image:
vllm/vllm-openai-cpu:v0.27.1-arm64@sha256:e6745d7ba6610f637c6f22fc06cd730342e50245b6c46767235600483adfbbde - model:
Qwen/Qwen2.5-0.5B-Instruct - model revision:
7ae557604adf67be50417f59c2c2f167def9a775 - maximum context: 512 tokens
- vLLM memory reservation: 20% of the node memory, which stays below this example's 10 GiB container limit
Run it on Kubernetes
Use an ARM64 cluster with at least 4 CPU cores and 10 GiB available memory. The first run downloads the vLLM image and model.
kubectl apply -f namespace.yaml
kubectl apply -f vllm-qwen.yaml
kubectl apply -f chat.yaml
kubectl rollout status -n inference deployment/vllm-qwen --timeout=10m
./request.shThe request asks the model what two plus two is. A successful result must include a non-empty answer from Qwen/Qwen2.5-0.5B-Instruct; Deployment readiness alone is not counted as inference.
Keep it in ConfigHub
The live proof starts from the checked inference-workloads OCI base, creates a derived CPU variant, and replaces only the chat and vllm-qwen Units with the files here. ConfigHub publishes the result as release OCI, Argo CD pulls that exact manifest digest, and the proof runs the model request from the chat pod.
Read source-and-intent.yaml for the exact source, selection, runtime choices, and proof boundary.
Limits
This is a functional CPU test, not a performance test. It does not prove NVIDIA GPU readiness, the 7B AWQ model, EKS, Karpenter, production capacity, or response quality.