Comments (7)
If run to train speed of learning is slow down too
from darknet-on-opencl.
The performance of CL is lower than corresponding CUDA implementation on the same device.
@ganyc717 @AndrewSivrit
I want to known that whether the code supports train yolo2 on multiple CL devices?
from darknet-on-opencl.
Hi @Kylin-PHYTIUM
I have checked the bottleneck, the performance blocked at BLAS library, I think nvidia has their own well optimization cuBLAS library which is about 3 times faster than clBLAS.
Currently multi device is not supported, maybe I will add support later.
from darknet-on-opencl.
The same issue happens to me. When I let it run for a while, the speed starts decreasing. Here I plot the times in microseconds over time for some of the required stages. All of them remain stable (img preparation, data copy, result copying etc...) but the prediction grows continuously.
To be exact, the call that grows in time is the call to the prediction of the net from the data, i.e. network_predict(net, X);
.
Any clue?
from darknet-on-opencl.
Just to isolate the problem I observed what happens if I reload the "network *net" object each time I use it. Basically I call net = load_network(wStr1, wStr2, 0);
at the beginning of the loop and free_network(net);
at the end. It can be clearly seen in the following figure that the "slow down" does not happens if the network is reloaded:
Left figure shows that if the net is not reloaded, i.e. it is loaded just at the beginning (which is the logical behaviour), then the prediction time increases. However, right figure shows that if the net is reloaded each time, then no slow down can be observed, but obviously, loading the network takes a couple of seconds, so it is not a solution...
I will keep digging into the problem, I hope this helps to someone else too.
from darknet-on-opencl.
I am not an expert in neither YOLO nor CL, but while debugging a bit the code I found that the code that slows down over time is https://github.com/ganyc717/Darknet-On-OpenCL/blob/master/darknet_cl/src/network.cpp#L781. Particularly, using tiny_yolo_v2, the layer that causes the issue is the layer number 15, which seems to be the region_layer.... The deeper I went was this line
. If I am not wrong, that line queue the call of the kernel, what can cause that continuous increment of time?from darknet-on-opencl.
I modified the code to enable CL_QUEUE_PROFILING_ENABLE. Then I added the following lines to profile the enqueue call:
cl_ulong time_start;
cl_ulong time_end;
clGetEventProfilingInfo(e, CL_PROFILING_COMMAND_START, sizeof(time_start), &time_start, NULL);
clGetEventProfilingInfo(e, CL_PROFILING_COMMAND_END, sizeof(time_end), &time_end, NULL);
double nanoSeconds = time_end-time_start;
printf("OpenCl Execution time is: %0.3f milliseconds \n",nanoSeconds / 1000000.0);
These measures of time remain stable... which confuses me more. It seems that the GPU run it self takes the same time, but when the measure of the cpu call it grows in time:
clock_t t1 = clock();
cl_event e;
cl_int status = clEnqueueNDRangeKernel(*cl->queue, kernel, 3, NULL, global_size, NULL, NULL, NULL, &e);
clock_t t2 = clock();
printf("enqueue : \t %f\n",(float)(t2 - t1) / CLOCKS_PER_SEC);
from darknet-on-opencl.
Related Issues (20)
- Compilation errors HOT 1
- cmake found wrong opencl path HOT 2
- Linking CXX executable darknet issue HOT 1
- opencl execution error, code -11 -11 HOT 5
- Running this on FPGA HOT 6
- GPU to CPU copy speed bottleneck?
- OpenCL execution error, code -50
- compile error: data.cpp
- macOS/Windows support - CMakeLists.txt
- Support for simrdwn project
- how to use it in Android ?
- opencl execution error, code -6 -6 HOT 1
- OpenCL execution Error -13 (CL_MISALIGNED_SUB_BUFFER_OFFSET) on AMD GPU HOT 1
- clEnqueueNDRangeKernel NULL event_wait_list HOT 1
- couldn't open file: cfg/coco.data HOT 1
- opencl execution error, code -54 -54
- macOS on M1: running make command generates 46 warnings and 9 errors HOT 7
- ld: symbol(s) not found for architecture x86_64
- requires CUDA header files installation for AMD GPUs?
- Compile error in 12norm_layer.cpp HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from darknet-on-opencl.