Git Product home page Git Product logo

Comments (12)

jkjung-avt avatar jkjung-avt commented on August 13, 2024

You could run the remainder of the script manually. It might still take a long time, though.

$ cd ${HOME}/src/tensorflow-2.3.0
$ bazel build --config=opt \
              --config=cuda \
              --config=noaws \
              --local_cpu_resources=HOST_CPUS*0.25 \
              --local_ram_resources=HOST_RAM*0.5 \
              //tensorflow/tools/pip_package:build_pip_package
$ bazel-bin/tensorflow/tools/pip_package/build_pip_package wheel/tensorflow_pkg

from jetson_nano.

germal avatar germal commented on August 13, 2024

Thanks @jkjung-avt, I try !

from jetson_nano.

germal avatar germal commented on August 13, 2024

Hi @jkjung-avt ,
After many hours the script failed again and always in the same point ,it's odd because the manual import of google.protobuf (exactly 3.8.0) works without problem.
Maybe some artifact from the previous script run was the cause so I restarted from scratch the procedure.
Maybe would be possible for you to upload somewhere the resulting wheel file TF 2.3.0 for Jetson ? Would you be so kind ? Ineed to verify if I can use a software based onTF gpu>=2.2
Thanks
germal

from jetson_nano.

jkjung-avt avatar jkjung-avt commented on August 13, 2024

Please check out the wheel files provided by NVIDIA: https://developer.download.nvidia.com/compute/redist/jp/v44/tensorflow/

And the official installation instructions are here: Installing TensorFlow For Jetson Platform

By the way, based on my experience, the "ModuleNotFoundError: No module named 'google.protobuf'" issue could usually be solved by removing "__init__,py" in the "google" folder. Refer to: jkjung-avt/tensorrt_demos#270

from jetson_nano.

germal avatar germal commented on August 13, 2024

Thank you ! I managed to install 2.3.0 with the wheel provided by Nvidia and the build script is ongoing on another machine.
Now I am facing an OOM error ,that I supposed depending from the model that I am loading .But also decreasing batch size to 1 the problem is still there.
Have you experienced this type of error in Tensorflow 2.x ?
Cheers
germal

from jetson_nano.

jkjung-avt avatar jkjung-avt commented on August 13, 2024

The "ModuleNotFoundError: No module named 'google.protobuf'" is not specific to tensorflow. I think it's due to conflicts in the "__init__.py" between different modules from google. It usually happens when you install "protobuf" along with some other modules by google.

If you search for the error message, you would find a lot of fellow users reporting the same issue. For example, https://stackoverflow.com/questions/31308812/no-module-named-google-protobuf/37249630

from jetson_nano.

germal avatar germal commented on August 13, 2024

Sorry for being imprecise , I was meaning Out of memory error.
I am loading tiny yolo weights and I find impossible that the nano couldn't load them, this is the message

020-11-08 16:11:40.418398: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_: 37363712 memory_limit_: 37363712 available bytes: 0 curr_region_allocation_bytes_: 74727424
2020-11-08 16:11:40.418439: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats:
Limit: 37363712
InUse: 6309376
MaxInUse: 15738368
NumAllocs: 123
MaxAllocSize: 4718592
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0

2020-11-08 16:11:40.418489: W tensorflow/core/common_runtime/bfc_allocator.cc:439] _**_____*****************_________________________
2020-11-08 16:11:40.418567: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at random_op.cc:77 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

I also tried to limit the Tensorflow memory with the following block without luck

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession
config = ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.7
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

Have you some suggestion on this topic ?
Thanks a lot

from jetson_nano.

jkjung-avt avatar jkjung-avt commented on August 13, 2024

Nope. I ran yolov4/yolov3 models with darknet on Jetson Nano. I also converted those yolov3/yolov4 models to TensorRT. They all worked without OOM problem.

Reference:

from jetson_nano.

germal avatar germal commented on August 13, 2024

Thank you @jkjung-avt ,it seems the problem come from the way TF use to access the memory,But for sure something is wrong in my configuration.
I too used yolov3 inference in another application using onlys darknet and performed like a charm !

from jetson_nano.

germal avatar germal commented on August 13, 2024

Hello @jkjung-avt how are you ?
Now I have resolved the memory issue limiting the fraction of memory that TF lock at its bootstrap.
But the yolov2 inference code that I am using ,that works well on the nvidia gpu on my laptop, doesn't detect anything on the nano , the message is bounding box found 0.
I installed the correct cuda 10.0 and cudnn 7 libraries and used all the available TF versions on the nvidia repository, and full or tiny weights but the result is the same , zero detections. I am using a realsense camera for realtime detection of persons.
From your experience on nano and yolov2 is there any parameter I can tune,considering the limited capacity of the nano wrt to realtime detection from camera ? I tried limiting the camera fps but without luck.
Do you know how could I trap a message related to the possible bottleneck in TF to process s stream too fast of data ?
Thanks a lo for you reply
Cheers
germal

from jetson_nano.

jkjung-avt avatar jkjung-avt commented on August 13, 2024

Please consider optimizing and running your model with TensorRT, instead of inferencing with TensorFlow directly. In general, the TensorRT engine runs faster and requires less memory to run. You could refer to my tensorrt_demos repository for examples.

from jetson_nano.

germal avatar germal commented on August 13, 2024

Hello @jkjung-avt thank you for your resources that are definitively a thesaurus.I will try tensort first with your guided examples because in the case of the yolov2 application I am using it is based on this yolov2 https://github.com/experiencor/keras-yolo2 that seems to me has not a straightforward conversion procedure because it is based on two set of weights the frontend and backend. .the problem is there, indeed it runs too slowly on Jetson and detection doesn't happen

from jetson_nano.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.