Git Product home page Git Product logo

k8s-device-plugin's Introduction

NVIDIA device plugin for Kubernetes

Table of Contents

About

The NVIDIA device plugin for Kubernetes is a Daemonset that allows you to automatically:

  • Expose the number of GPUs on each nodes of your cluster
  • Keep track of the health of your GPUs
  • Run GPU enabled containers in your Kubernetes cluster.

This repository contains NVIDIA's official implementation of the Kubernetes device plugin.

Prerequisites

The list of prerequisites for running the NVIDIA device plugin is described below:

  • NVIDIA drivers ~= 384.81
  • nvidia-docker version > 2.0 (see how to install and it's prerequisites)
  • docker configured with nvidia as the default runtime.
  • Kubernetes version >= 1.10

Quick Start

Preparing your GPU Nodes

The following steps need to be executed on all your GPU nodes. This README assumes that the NVIDIA drivers and nvidia-docker have been installed.

Note that you need to install the nvidia-docker2 package and not the nvidia-container-toolkit. This is because the new --gpus options hasn't reached kubernetes yet. Example:

# Add the package repositories
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

$ sudo apt-get update && sudo apt-get install -y nvidia-docker2
$ sudo systemctl restart docker

You will need to enable the nvidia runtime as your default runtime on your node. We will be editing the docker daemon config file which is usually present at /etc/docker/daemon.json:

{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

if runtimes is not already present, head to the install page of nvidia-docker

Enabling GPU Support in Kubernetes

Once you have enabled this option on all the GPU nodes you wish to use, you can then enable GPU support in your cluster by deploying the following Daemonset:

$ kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta5/nvidia-device-plugin.yml

Running GPU Jobs

NVIDIA GPUs can now be consumed via container level resource requirements using the resource name nvidia.com/gpu:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
    - name: cuda-container
      image: nvidia/cuda:9.0-devel
      resources:
        limits:
          nvidia.com/gpu: 2 # requesting 2 GPUs
    - name: digits-container
      image: nvidia/digits:6.0
      resources:
        limits:
          nvidia.com/gpu: 2 # requesting 2 GPUs

WARNING: if you don't request GPUs when using the device plugin with NVIDIA images all the GPUs on the machine will be exposed inside your container.

Docs

Please note that:

  • the device plugin feature is beta as of Kubernetes v1.11.
  • the NVIDIA device plugin is still considered beta and is missing
    • More comprehensive GPU health checking features
    • GPU cleanup features
    • ...
  • support will only be provided for the official NVIDIA device plugin.

The next sections are focused on building the device plugin and running it.

With Docker

Build

Option 1, pull the prebuilt image from Docker Hub:

$ docker pull nvidia/k8s-device-plugin:1.0.0-beta5

Option 2, build without cloning the repository:

$ docker build -t nvidia/k8s-device-plugin:1.0.0-beta5 https://github.com/NVIDIA/k8s-device-plugin.git#1.0.0-beta5

Option 3, if you want to modify the code:

$ git clone https://github.com/NVIDIA/k8s-device-plugin.git && cd k8s-device-plugin
$ git checkout 1.0.0-beta5
$ docker build -t nvidia/k8s-device-plugin:1.0.0-beta5 .

Run locally

$ docker run --security-opt=no-new-privileges --cap-drop=ALL --network=none -it -v /var/lib/kubelet/device-plugins:/var/lib/kubelet/device-plugins nvidia/k8s-device-plugin:1.0.0-beta5

Deploy as Daemon Set:

$ kubectl create -f nvidia-device-plugin.yml

Without Docker

Build

$ C_INCLUDE_PATH=/usr/local/cuda/include LIBRARY_PATH=/usr/local/cuda/lib64 go build

Run locally

$ ./k8s-device-plugin

Changelog

Version 1.0.0-beta5

  • Add a new plugin.yml variant that is compatible with the CPUManager
  • Change CMD in Dockerfile to ENTRYPOINT
  • Add flag to optionally return list of device nodes in Allocate() call
  • Refactor device plugin to eventually handle multiple resource types
  • Move plugin error retry to event loop so we can exit with a signal
  • Update all vendored dependencies to their latest versions
  • Fix bug that was inadvertently always disabling health checks
  • Update minimal driver version to 384.81

Version 1.0.0-beta4

  • Fixes a bug with a nil pointer dereference around getDevices:CPUAffinity

Version 1.0.0-beta3

  • Manifest is updated for Kubernetes 1.16+ (apps/v1)
  • Adds more logging information

Version 1.0.0-beta2

  • Adds the Topology field for Kubernetes 1.16+

Version 1.0.0-beta1

  • If gRPC throws an error, the device plugin no longer ends up in a non responsive state.

Version 1.0.0-beta

  • Reversioned to SEMVER as device plugins aren't tied to a specific version of kubernetes anymore.

Version 1.11

  • No change.

Version 1.10

  • The device Plugin API is now v1beta1

Version 1.9

  • The device Plugin API changed and is no longer compatible with 1.8
  • Error messages were added

Issues and Contributing

Checkout the Contributing document!

Versioning

Before 1.10 the versioning scheme of the device plugin had to match exactly the version of Kubernetes. After the promotion of device plugins to beta this condition was was no longer required. We quickly noticed that this versioning scheme was very confusing for users as they still expected to see a version of the device plugin for each version of Kubernetes.

We recently decided to reversion to follow a SEMVER scheme. This means that we are currently a beta project (as we depend on the device plugin API which is beta). If you have a version of Kubernetes > 1.10 you can deploy this device plugin.

Upgrading Kubernetes with the device plugin

Upgrading Kubernetes when you have a device plugin deployed doesn't require you to do any, particular changes to your workflow. The API is versioned and is pretty stable (though it is not guaranteed to be non breaking), you can therefore use the 1.0.0-beta3 version starting from kubernetes version 1.10, upgrading kubernetes won't require you to deploy a different version of the device plugin and you will see GPUs re-registering themselves after your node comes back online.

Upgrading the device plugin is a more complex task. It is recommended to drain GPU tasks as we cannot guarantee that GPU tasks will survive a rolling upgrade. However we make best efforts to preserve GPU tasks during an upgrade.

k8s-device-plugin's People

Contributors

carmark avatar dvavili avatar everpeace avatar flx42 avatar guptanswati avatar jjacobelli avatar jwcheong0420 avatar klueska avatar mkumatag avatar nvjmayo avatar rajatchopra avatar renaudwastaken avatar rockrush avatar scorpiocph avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.