Skip to main content

GPU Operators

Deploy NVIDIA operator​

The NVIDIA operator allows administrators of Kubernetes clusters to manage GPUs just like CPUs. It includes everything needed for pods to be able to operate GPUs.

Host OS requirements​

To expose the GPU to the pod correctly, the NVIDIA kernel drivers and the libnvidia-ml library must be correctly installed in the host OS (Operating System). The NVIDIA Operator can automatically install drivers and libraries on some operating systems; check the NVIDIA documentation for information on supported operating system releases.

Starting with GPU Operator v26.3.x, the operator can also manage driver and library installation on any operating system, provided the OS vendor or administrator supplies a compatible driver container image.

Installation of the NVIDIA components on your host OS is out of the scope of this document; reference the NVIDIA documentation for instructions.

Checks for pre-installed NVIDIA drivers/libraries

The following three commands should return a correct output if the kernel driver was correctly installed:

  1. lsmod | grep nvidia

    Returns a list of nvidia kernel modules, for example:

    nvidia_uvm           2129920  0
    nvidia_drm            131072  0
    nvidia_modeset       1572864  1 nvidia_drm
    video                  77824  1 nvidia_modeset
    nvidia               9965568  2 nvidia_uvm,nvidia_modeset
    ecc                    45056  1 nvidia

  2. cat /proc/driver/nvidia/version

    returns the NVRM and GCC version of the driver. For example:

    NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  555.42.06  Release Build  (abuild@host)  Thu Jul 11 12:00:00 UTC 2024
    GCC version:  gcc version 7.5.0 (SUSE Linux)

  3. find /usr/ -iname libnvidia-ml.so

    returns a path to the libnvidia-ml.so library. For example:

    /usr/lib64/libnvidia-ml.so

    This library is used by Kubernetes components to interact with the kernel driver.

Operator installation​

Once the OS is ready and RKE2 is running, install the GPU Operator with the following yaml manifest:

There are two installation options available.

If drivers and libraries are pre-installed or you are using a supported operating system by nvidia, please use the following manifest

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  version: v26.7.2
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    toolkit:
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock

If your operating system vendor supplies a compatible driver image, you can use the driver value field to point to it. For example, in SLES 16.0, you can use the following manifest:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  version: v26.7.2
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    toolkit:
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock
    driver:
      repository: registry.suse.com/third-party/nvidia
      usePrecompiled: true
      version: 595 # This depends on the nvidia driver that works with your GPU architecture
Minimum containerd version

Starting with v26.7.0, the GPU Operator requires containerd 2.0 or later.

Dynamic Resource Allocation (DRA)

v26.7.x also supports installing with a GPUCluster custom resource instead of ClusterPolicy, enabling GPU allocation via native Kubernetes DRA. See the "DRA (Experimental)" tab.

note

The checks below apply to the ClusterPolicy-based (device-plugin) tabs above. If you installed with GPUCluster/DRA instead, see the verification steps in that tab instead — allocatable GPUs aren't exposed as the nvidia.com/gpu extended resource in DRA mode.

After a few minutes, you can make the following checks to verify that everything worked as expected:

  1. Assuming the drivers and libnvidia-ml.so library were previously installed, check if the operator detects them correctly:

    kubectl get node $NODENAME -o jsonpath='{.metadata.labels}' |  grep "nvidia.com"

    You should see labels specifying driver and GPU (e.g. nvidia.com/gpu.machine or nvidia.com/cuda.driver.major)

  2. Check if the gpu was added (by nvidia-device-plugin-daemonset) as an allocatable resource in the node:

    kubectl get node $NODENAME -o jsonpath='{.status.allocatable}'

    You should see "nvidia.com/gpu": followed by the number of gpus in the node

  3. Check that the container runtime binary exists (it gets installed by the nvidia-container-toolkit-daemonset):

    ls /usr/local/nvidia/toolkit/nvidia-container-runtime

  4. (Only if not using NRI) Verify if containerd config was updated to include the nvidia container runtime:

    grep nvidia /var/lib/rancher/rke2/agent/etc/containerd/config.toml

  5. Run a pod to verify that the GPU resource can successfully be scheduled on a pod and the pod can detect it

    apiVersion: v1
    kind: Pod
    metadata:
      name: nbody-gpu-benchmark
      namespace: default
    spec:
      restartPolicy: OnFailure
      containers:
      - name: cuda-container
        image: nvcr.io/nvidia/k8s/cuda-sample:nbody
        args: ["nbody", "-gpu", "-benchmark"]
        resources:
          limits:
            nvidia.com/gpu: 1