Hello,
I'm running into an issue where everything loads, I'm not getting any errors in regard to my installation, but the car will not move (even with the default reward function).
I'm running this on Ubuntu 18.04 and using the installation instructions.
From what I can tell, it doesn't look like the GPU is being utilized even though CUDA is installed and the CUDNN is also installed. I tested tensorflow-gpu and it detects and can run off my CPU.
The PC has a 1080TI and dual Xeon processors. I'll try installing this on another PC with a 2080TI and i5 processor. Though Unfortunately, I'm at a loss.
One other characteristic I noticed was that the robomaker environment seems to fail as well after a period of time. Similar to issue 29
Here is the terminal output from init.sh:
Cloning into 'deepracer'... remote: Enumerating objects: 195, done. remote: Counting objects: 100% (195/195), done. remote: Compressing objects: 100% (159/159), done. remote: Total 1697 (delta 60), reused 141 (delta 36), pack-reused 1502 Receiving objects: 100% (1697/1697), 125.01 MiB | 13.37 MiB/s, done. Resolving deltas: 100% (752/752), done. Checking out files: 100% (803/803), done. Submodule 'deepracer_worlds' (https://github.com/crr0004/deepracer_worlds.git) registered for path 'deepracer_worlds' Submodule 'intel_coach' (https://github.com/NervanaSystems/coach.git) registered for path 'intel_coach' Submodule 'sagemaker-containers' (https://github.com/crr0004/sagemaker-containers.git) registered for path 'sagemaker-containers' Submodule 'sagemaker-python-sdk' (https://github.com/crr0004/sagemaker-python-sdk.git) registered for path 'sagemaker-python-sdk' Submodule 'sagemaker-rl-container' (https://github.com/crr0004/sagemaker-rl-container) registered for path 'sagemaker-rl-container' Submodule 'sagemaker-tensorflow-container' (https://github.com/crr0004/sagemaker-tensorflow-container) registered for path 'sagemaker-tensorflow-container' Cloning into '/home/christian/deepracer-for-dummies/deepracer/deepracer_worlds'... remote: Enumerating objects: 104, done. remote: Counting objects: 100% (104/104), done. remote: Compressing objects: 100% (77/77), done. remote: Total 104 (delta 29), reused 93 (delta 22), pack-reused 0 Receiving objects: 100% (104/104), 8.96 MiB | 12.46 MiB/s, done. Resolving deltas: 100% (29/29), done. Cloning into '/home/christian/deepracer-for-dummies/deepracer/intel_coach'... remote: Enumerating objects: 366, done. remote: Counting objects: 100% (366/366), done. remote: Compressing objects: 100% (206/206), done. remote: Total 8982 (delta 269), reused 236 (delta 160), pack-reused 8616 Receiving objects: 100% (8982/8982), 72.72 MiB | 13.34 MiB/s, done. Resolving deltas: 100% (6148/6148), done. Cloning into '/home/christian/deepracer-for-dummies/deepracer/sagemaker-containers'... remote: Enumerating objects: 1711, done. remote: Total 1711 (delta 0), reused 0 (delta 0), pack-reused 1711 Receiving objects: 100% (1711/1711), 551.66 KiB | 5.16 MiB/s, done. Resolving deltas: 100% (1076/1076), done. Cloning into '/home/christian/deepracer-for-dummies/deepracer/sagemaker-python-sdk'... remote: Enumerating objects: 4690, done. remote: Total 4690 (delta 0), reused 0 (delta 0), pack-reused 4690 Receiving objects: 100% (4690/4690), 52.56 MiB | 13.26 MiB/s, done. Resolving deltas: 100% (3323/3323), done. Cloning into '/home/christian/deepracer-for-dummies/deepracer/sagemaker-rl-container'... remote: Enumerating objects: 9, done. remote: Counting objects: 100% (9/9), done. remote: Compressing objects: 100% (7/7), done. remote: Total 263 (delta 3), reused 7 (delta 2), pack-reused 254 Receiving objects: 100% (263/263), 79.80 KiB | 2.05 MiB/s, done. Resolving deltas: 100% (123/123), done. Cloning into '/home/christian/deepracer-for-dummies/deepracer/sagemaker-tensorflow-container'... remote: Enumerating objects: 1443, done. remote: Total 1443 (delta 0), reused 0 (delta 0), pack-reused 1443 Receiving objects: 100% (1443/1443), 13.29 MiB | 12.19 MiB/s, done. Resolving deltas: 100% (726/726), done. Submodule path 'deepracer_worlds': checked out 'bcf6fa58dd80e7c694323a0e77cfad5fe0e77adf' Submodule path 'intel_coach': checked out '533bb43720311e304ce57a46851c8a100e06e2bf' Submodule path 'sagemaker-containers': checked out 'd11ec7fa61b613d1f95f33d865bf15f27dd96346' Submodule path 'sagemaker-python-sdk': checked out '68101b4f6ecf557d8963dfe59476f8982871d982' Submodule path 'sagemaker-rl-container': checked out '986990bb0425d5529b5af6ab11e34cb634da7c50' Submodule path 'sagemaker-tensorflow-container': checked out '5f5dbb62551d38a63fc3c929ad4ed575376384a2' Cloning into 'aws-deepracer-workshops'... remote: Enumerating objects: 2259, done. remote: Total 2259 (delta 0), reused 0 (delta 0), pack-reused 2259 Receiving objects: 100% (2259/2259), 37.95 MiB | 13.23 MiB/s, done. Resolving deltas: 100% (750/750), done. Branch 'enhance-log-analysis' set up to track remote branch 'enhance-log-analysis' from 'origin'. Switched to a new branch 'enhance-log-analysis' Sending build context to Docker daemon 954.6MB Step 1/19 : FROM python:3.7.3-stretch ---> 34a518642c76 Step 2/19 : RUN apt-get update ---> Using cache ---> 161317e44174 Step 3/19 : RUN apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common ---> Using cache ---> 9f0f56de75ab Step 4/19 : RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - ---> Using cache ---> 2ded7f139edf Step 5/19 : RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" ---> Using cache ---> 9e55cb8dc3a9 Step 6/19 : RUN apt-get update ---> Using cache ---> 69742361cd7e Step 7/19 : RUN apt-get -y install docker-ce ---> Using cache ---> 4dea68525c04 Step 8/19 : RUN mkdir /deepracer ---> Using cache ---> 72f152ed2329 Step 9/19 : RUN mkdir /deepracer/rl_coach ---> Using cache ---> fdf15b00730c Step 10/19 : RUN mkdir /deepracer/sagemaker-python-sdk ---> Using cache ---> 237081995311 Step 11/19 : WORKDIR /deepracer ---> Using cache ---> 48df21a69a9b Step 12/19 : ADD rl_coach rl_coach ---> Using cache ---> c6da8b1247e8 Step 13/19 : ADD sagemaker-python-sdk sagemaker-python-sdk ---> Using cache ---> cc2bfb23e631 Step 14/19 : RUN mkdir /root/.sagemaker ---> Using cache ---> 24887d149302 Step 15/19 : COPY config.yaml /root/.sagemaker/config.yaml ---> Using cache ---> 1aa25d961ae5 Step 16/19 : RUN mkdir /robo ---> Using cache ---> 113fdd14731e Step 17/19 : RUN mkdir /robo/container ---> Using cache ---> dd5df1231d03 Step 18/19 : RUN pip install -U sagemaker-python-sdk/ awscli ipython pandas "urllib3==1.22" "pyyaml==3.13" "python-dateutil==2.8.0" ---> Using cache ---> 6452cfaf974c Step 19/19 : CMD (cd rl_coach; ipython rl_deepracer_coach_robomaker.py) ---> Using cache ---> e6e5b9a9e3b8 Successfully built e6e5b9a9e3b8 Successfully tagged aschu/rl_coach:latest
Here is the output from ./start.sh for training:
` ./start.sh
Creating minio ... done
Creating rl_coach ... done
Creating robomaker ... done
waiting for containers to start up...
attempting to pull up sagemaker logs...
Option “-x” is deprecated and might be removed in a later version of gnome-terminal.
Use “-- ” to terminate the options and put the command line to execute after it.
attempting to open vnc viewer...
Option “-x” is deprecated and might be removed in a later version of gnome-terminal.
Use “-- ” to terminate the options and put the command line to execute after it.
Here is my nvidia-smi output:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104 Driver Version: 410.104 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:03:00.0 On | N/A |
| 0% 49C P5 16W / 250W | 433MiB / 11177MiB | 1% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1808 G /usr/lib/xorg/Xorg 242MiB |
| 0 2000 G /usr/bin/gnome-shell 188MiB |
+-----------------------------------------------------------------------------+
`