benchmarking miopen
License: BSD 3-Clause "New" or "Revised" License
tsv and log filesgputop: monitor temperature, engine clock, memory clock, fan speedsummarize.sql: summarize layer-wise .tsv benchmarking logs to find the most time consuming layersmake
make benchmark
Take a look into the Makefile or .cpp sources for more details.
Tried miopen-benchmark ubuntu18.04 to give a try.
Building of test went fine but while running the test , got "std::bad_alloc" issue.
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)
Issue is coming from miopen-benchmark's header file when it tries to construct a directory path.
Thread 1 "alexnet" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff2779801 in __GI_abort () at abort.c:79
#2 0x00007ffff2dce8fb in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff2dd4d3a in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff2dd4d95 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff2dd4fe8 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff2dfdf26 in std::__throw_bad_alloc() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x0000000000463ced in __gnu_cxx::new_allocator<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int> >::allocate
(__n=12297829382473034424, this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/ext/new_allocator.h:102
#8 std::allocator_traits<std::allocator<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int> > >::allocate (
__n=12297829382473034424, __a=...) at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/alloc_traits.h:436
#9 std::_Vector_base<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int>, std::allocator<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int> > >::_M_allocate (__n=12297829382473034424, this=<optimized out>)
at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_vector.h:172
#10 std::_Vector_base<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int>, std::allocator<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int> > >::_M_create_storage (__n=12297829382473034424, this=<optimized out>)
at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_vector.h:187
#11 std::_Vector_base<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int>, std::allocator<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int> > >::_Vector_base (__n=12297829382473034424, this=<optimized out>, __a=...)
at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_vector.h:138
#12 std::vector<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int>, std::allocator<std::pair<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int> > >::vector (__n=12297829382473034424, this=<optimized out>, __a=...)
at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/stl_vector.h:284
#13 std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Executor (__begin=..., __end=...,
__results=..., __re=..., __flags=<optimized out>, this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/regex_executor.h:79
#14 std::__detail::__regex_algo_impl<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, char, std::__cxx11::regex_traits<char>, (std::__detail::_RegexExecutorPolicy)0, true> (__s=118 'v', __e=0 '\000', __m=..., __re=..., __flags=(unknown: 0)) at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/regex.tcc:78
#15 0x00000000004589d7 in std::regex_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, char, std::__cxx11::regex_traits<char> > (
__s=<error reading variable: Cannot access memory at address 0x2>, __e=0 '\000', __m=..., __re=..., __flags=(unknown: 0))
at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/regex.h:1995
#16 std::regex_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, char, std::__cxx11::regex_traits<char> > (
__first=<error reading variable: Cannot access memory at address 0x2>, __last=0 '\000', __re=..., __flags=(unknown: 0))
at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/regex.h:2022
#17 std::regex_match<std::char_traits<char>, std::allocator<char>, char, std::__cxx11::regex_traits<char> > (__re=..., __flags=(unknown: 0), __s=...)
at /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/regex.h:2127
#18 ls_dir (dname=..., match=...) at ./miopen.hpp:142
#19 0x000000000045b41f in Device::init_sys_paths (this=0xc8f270) at ./miopen.hpp:175
#20 0x00000000004590c3 in Devices::init_devices () at ./miopen.hpp:289
#21 0x000000000045b2b6 in device_init () at ./miopen.hpp:299
#22 main (argc=2, argv=0x7fffffffdb20) at alexnet.cpp:60
This may be relevant:
https://stackoverflow.com/questions/36106154/how-to-handle-or-avoid-exceptions-from-c11-regex-matching-functions-28-11
Thanks
While running make, I get the error below. The function miopenCreateWithStream is called with 'hipStream_t', an incomplete argument type, when it should of type 'miopenAcceleratorQueue_t'. hipStream_t is typedefed to miopenAcceleratorQueue_t when MIOPEN_BACKEND_HIP is true, but that is not the case here. I tried to set MIOPEN_BACKEND_HIP to true and manually typedef hipStream_t to miopenAcceleratorQuene_t but I still got the same error. Any help would be greatly appreciated.
/opt/rocm/hip/bin/hipcc -g -Wall --amdgpu-target=gfx900 -I/opt/rocm/hip/include -I/opt/rocm/include alexnet.cpp -L/opt/rocm/lib -L/opt/rocm/opencl/lib/x86_64 -lMIOpen -lOpenCL -lmiopengemm -lhipblas-hcc -lrocblas-hcc -o alexnet
In file included from alexnet.cpp:1:
In file included from ./miopen.hpp:4:
/opt/rocm/include/miopen/miopen.h:77:9: error: unknown type name 'hipStream_t'
typedef hipStream_t miopenAcceleratorQueue_t;
^
In file included from alexnet.cpp:1:
./miopen.hpp:335:19: error: no matching function for call to 'miopenCreateWithStream'
CHECK_MIO(miopenCreateWithStream(&h, q));
^~~~~~~~~~~~~~~~~~~~~~
./miopen.hpp:319:30: note: expanded from macro 'CHECK_MIO'
miopenStatus_t miostat = cmd;
^~~
/opt/rocm/include/miopen/miopen.h:119:30: note: candidate function not viable: cannot convert argument of incomplete type 'hipStream_t' (aka 'ihipStream_t *') to 'miopenAcceleratorQueue_t' (aka '_cl_command_queue ') for 2nd argument
MIOPEN_EXPORT miopenStatus_t miopenCreateWithStream(miopenHandle_t handle,
^
2 errors generated.
Died at /opt/rocm/hip/bin/hipcc line 452.
Makefile:18: recipe for target 'alexnet' failed
make: *** [alexnet] Error 2
As per latest release 1.6.4 of rocm driver , library name of rocblas and hipblas needs to be changed from ""-lhipblas-hcc , -lrocblas-hcc" to ""-lhipblas -lrocblas"
Hence, Makefile needs modification as below
LD_FLAGS=-L$(ROCM_PATH)/lib -L$(ROCM_PATH)/opencl/lib/x86_64 -lMIOpen -lOpenCL -lmiopengemm -lhipblas -lrocblas
Besides this, 1 minor modification requires in miopen.hpp as below
#include <hipblas/hipblas.h> to #include <hipblas.h>
Only after above 2 modification, we can able to build the miopen-benchmark tests on rocm release 1.6.4
Upon downloading a fresh, up-to-date copy of the repo, and trying to compile them, I get the following (identical) errors in multi_layers.hpp:
In file included from resnet.cpp:5:
./multi_layers.hpp:277:22: error: invalid operands to binary expression ('int' and 'nullptr_t')
assert(F.get() != nullptr);
./multi_layers.hpp:282:22: error: invalid operands to binary expression ('int' and 'nullptr_t')
assert(F.get() != nullptr);
The simple fix is to change nullptr to NULL, or to overload !=, but it seems strange to me that the comparison to nullptr for F throws an error, while the identical checks for G does not fail (even though G and F are the same type). Not sure if there is a compiler version, include, or library issue?
Matt
Hi, I'm trying to train the dataset using resnet from your benchmark, I found that the initialization of the input is random in your code. And there is no label as input, I'm a little confused. Does this benchmark judge the accuracy of the classification?
@patflick Hi, not sure if you are still maintaining this repo.
I had a question about how Resnet is being timed, particularly what time is accounted for in the "ShortCut" layers. There is a block of layers grouped in ShortCut as per the code but the logs show separate times for layers in the ShortCut + time for ShortCut itself. Example below:
Conv(3x3,pad=1,s=1) (16,64,56,56)->(16,64,56,56)
BatchNorm() (16,64,56,56)->(16,64,56,56)
ReLU() (16,64,56,56)->(16,64,56,56)
Conv(1x1,pad=0,s=1) (16,64,56,56)->(16,256,56,56)
BatchNorm() (16,256,56,56)->(16,256,56,56)
ShortcutF
Can you please explain where is the time spent in ShortCutF in this case? Is this the time for running the last layer + synchronization overheads? Or is it something else?
/cc @soumith
I see that there is a uniform() function in tensor.hpp to initialize a given array to random values. However, it does not seem to ever be called in the code base. Thus, I was wondering how arrays that are allocated in device_alloc() are initialized. Are they not initialized at all? If so, is this the intended behavior?
Matt
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.