GPU deployment unable to get the performance benefits about bmw-tensorflow-inference-api-gpu HOT 1 CLOSED

Hello @srinivaskotamraju.

I would recommend to monitor the GPU utilization before and after running the docker-container to see if the GPU is being used.
For that use the following command: watch -n 0.1 nvidia-smi
This will refresh the GPU utilization data at a rate of 0.1 seconds.

This way you'll get a better insight on whether the GPU is being used and at what rate and also if other processes are using the GPU thus leaving the API with little amount of memory left.

In case you have multiple gpus on the machine, try replacing the gpu number in the NV_GPU variable to the number of the GPU you wish to use to check if the problem persists

from bmw-tensorflow-inference-api-gpu.