Git Product home page Git Product logo

set-up-deep-learning-frameworks-with-gpu-on-google-cloud-platform's Introduction

Set up tensorflow with GPU on Google Cloud Platform

First of all, you should have a google account and sign up Google Cloud Platform (GCP). Note that if you do not need to train your model for some time. You can stop your instance to pause the charging. You can alway start the instance as needed, and everything in the instance will not change.

Create compute engine instance on GCP

First, you shall create a compute engin in the GCP. Click 'Compute Engine' in the menu, then click on 'VM instances'.

Next, you can click ‘Create new instance’ to create new instance. Here, you can name your instance, select server zones and customize your VM. Normally, I select my VM in ‘ us-central1 (lowa)’ (for the purpose of using Nvidia Tesla K80, not all zones have this GPU).

Next, click 'Customise' in the 'Machine type' tab to config your VM. I usually select 2 core CPU, 13 GB memory and 1 NVIDIA Tesla K80.

As for the boot disk, you can select ‘Ubuntu 16.04 LTS’ here with at least 30 GB hard drive (a 100 GB SSD boot disk is recommended, and it will not result in much extra cost). Plus, remenber to allow http/https traffic in the firewall tab.

Last, click 'create' and you are good to go. You can then connect to your VM via SSH.

Set up environment for tenserflow GPU in your VM

Now, after creating the cloud computing VM, you can install nesscary packages to run your program.

Install drivers

Update linux packages

sudo apt-get update

Install dependencies

sudo apt-get install openjdk-8-jdk git python-dev python3-dev python-numpy python3-numpy build-essential python-pip python3-pip python-virtualenv swig python-wheel libcurl3-dev curl

Install Nvidia Drivers

You can searh the suitable driver on [https://developer.nvidia.com/cuda-80-ga2-download-archive]. You can download drivers that you found from the nvidia website (here we use the driver for ubuntu 16.04):

curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb

Then, you should download the key to allow installation:

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

Install downloaded packages:

sudo dpkg -i ./cuda-repo-ubuntu1604_9.0.176-1_amd64.deb

Next, you can install CUDA after finishing the installation of Nvidia driver:

sudo apt-get update 
sudo apt-get install cuda-9-0

Last, you can check if Nvidia driver is correctly installed. Type 'nvidia-smi' in terminal, and you shall see something similar:

yourUserName@instance-1:~$ nvidia-smi
Sun Jul 28 hh:mm:ss 2019  
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   54C    P0    84W / 149W |      0MiB / 11441MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Install cudnn

wget https://s3.amazonaws.com/open-source-william-falcon/cudnn-9.0-linux-x64-v7.3.1.20.tgz
sudo tar -xzvf cudnn-9.0-linux-x64-v7.3.1.20.tgz
sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*

Then, add paths to lib:

export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64"
export CUDA_HOME=/usr/local/cuda
export PATH="$PATH:/usr/local/cuda/bin"
source ~/.bashrc

Install Anaconda

Anaconda provides an easy installation of tensorflow. The download link of following code can be found in the anaconda download page, by right click the downloadable images.

wget https://repo.continuum.io/archive/Anaconda3-2018.12-Linux-x86_64.sh 
bash Anaconda3-2018.12-Linux-x86_64.sh

Add conda to path:

export PATH=~/anaconda3/bin:$PATH

Install tensorflow GPU version via Anaconda

Note that 'tfgpu' in the following code is the name of environment, you can change it to anything as you like.

conda create -n tfgpu tensorflow-gpu
conda activate tfgpu

Tips for use

Every time you connect to the VM, you shall run the tfgpu virtual envirment to use the tensorflow:

conda activate tfgpu

Install PyTorch

You can find the code for installation on [https://pytorch.org/]. We select 'Linux', 'Conda', 'Python 3.7', ''CUDA 9.0 here.

Then you can run the following code to install PyTorch in your terminal (You may also install it in conda virtual environment if you like).

conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

(Optional) other interesting packages that may help you

htop

You can install htop to track your VM usage, it is more readable than 'top':

sudo apt-get install htop

To run htop, type htop' in your ssh terminal.

scikit-learn

Install within conda virtual environment:

conda install scikit-learn

progress

It should be a built-in function in your Python. Nevertheless, sometimes you may not be able to import it. Install within conda virtual environment:

conda install -c conda-forge progress

set-up-deep-learning-frameworks-with-gpu-on-google-cloud-platform's People

Contributors

jiahongchen avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.