slowdown after adding debugging flags about amr-wind HOT 21 CLOSED

@WeiqunZhang set this parameter to OFF during cmake configure

from amr-wind.

@sayerhs @jrood-nrel Could you help me with this?

from amr-wind.

@aarontcopal2 I am not sure if this is an issue with AMR-Wind (or rather how we would address this at AMR-Wind level). Have you tried building RelWithDebInfo builds with GCC/LLVM or Intel (non SYCL/DPC++) compiler to check if it causes this slowdown?

If this occurs only with DPC++ and you absolutely need to use it, then we will have to discuss with Intel compiler devs to understand the issue.

from amr-wind.

@aarontcopal2 We heard back from the Intel compiler team. They have observed this slowdown as well and are working on this. From Intel:

The option -fsycl-device-code-split=per_kernel helps with cutting down the run time JIT compilation time so turning it off is what’s causing most of the regression.

They recommend trying ahead-of-time compilation as a workaround, but it is currently disabled in AMReX. Let me know if you want to attempt this and I'll show you what needs to be done with the AMReX submodule.

from amr-wind.

Sure, you can tell me the steps to enable AOT compilation

from amr-wind.

@sayerhs any updates on this?

from amr-wind.

@sayerhs @jrood-nrel @michaeljbrazell
Could someone help me with this?

from amr-wind.

@aarontcopal2 I apologize for not responding sooner. I am no longer employed at NREL and, therefore, do not have access systems where I can test DPC++ issues. Since this requires some changes at AMReX level (and possibly discussions with Intel devs) I am tagging AMReX devs (@WeiqunZhang @ax3l) here to see if they will be able to provide you with some help.

from amr-wind.

You can try -DCMAKE_CXX_FLAGS="-fsycl-targets=spir64_gen-unknown-unknown-sycldevice -Xs \"-device XXX\"", where XXX is your device. See https://software.intel.com/content/www/us/en/develop/documentation/oneapi-dpcpp-cpp-compiler-dev-guide-and-reference/top/compilation/ahead-of-time-compilation.html.

from amr-wind.

I tried out your suggestion @WeiqunZhang , but I get an error. I have attached the same below:

cmake -Bbuild-DPCPP -DCMAKE_CXX_COMPILER_ID="Clang" -DCMAKE_CXX_COMPILER_VERSION=12.0 -DCMAKE_CXX_STANDARD_COMPUTED_DEFAULT=17 -DCMAKE_CXX_COMPILER=$(which dpcpp) -DCMAKE_C_COMPILER=$(which clang) -DAMR_WIND_ENABLE_MPI=OFF -DAMR_WIND_ENABLE_DPCPP=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DAMReX_DPCPP_SPLIT_KERNEL=FALSE -DCMAKE_CXX_FLAGS="-fsycl-targets=spir64_gen-unknown-unknown-sycldevice -Xs \"-device Gen9\"" ..
/gpfs/jlse-fs0/users/aarontcopal2/code/dpcpp/amr-wind/build/build-DPCPP/amrwind_obj/AMReX_buildInfo.cpp:67:241: error: use of undeclared identifier 'device'
  static const char CXX_flags[] = "-I/gpfs/jlse-fs0/users/aarontcopal2/code/dpcpp/amr-wind -I/gpfs/jlse-fs0/users/aarontcopal2/code/dpcpp/amr-wind/build/build-DPCPP -O2 -g -DNDEBUG -fsycl-targets=spir64_gen-unknown-unknown-sycldevice -Xs "-device Gen9"";
                                                                                                                                                                                                                                                ^
/gpfs/jlse-fs0/users/aarontcopal2/code/dpcpp/amr-wind/build/build-DPCPP/amrwind_obj/AMReX_buildInfo.cpp:67:247: error: expected ';' at end of declaration
  static const char CXX_flags[] = "-I/gpfs/jlse-fs0/users/aarontcopal2/code/dpcpp/amr-wind -I/gpfs/jlse-fs0/users/aarontcopal2/code/dpcpp/amr-wind/build/build-DPCPP -O2 -g -DNDEBUG -fsycl-targets=spir64_gen-unknown-unknown-sycldevice -Xs "-device Gen9"";
                                                                                                                                                                                                                                                      ^
                                                                                                                                                                                                                                                      ;
2 errors generated.
make[2]: *** [amr-wind/CMakeFiles/buildInfoamrwind_obj.dir/build.make:76: amr-wind/CMakeFiles/buildInfoamrwind_obj.dir/__/amrwind_obj/AMReX_buildInfo.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1343: amr-wind/CMakeFiles/buildInfoamrwind_obj.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

from amr-wind.

I guess you can use

-Xs `-device Gen9`

instead of

-Xs \"-device Gen9\"

from amr-wind.

I don't use cmake. But I guess you may also need to add that to the linker flag, CMAKE_EXE_LINKER_FLAGS.

from amr-wind.

@WeiqunZhang Do you mean there should be a line in CMakeCache.txt like this:

CMAKE_EXE_LINKER_FLAGS:STRING=-fsycl-targets=spir64_gen-unknown-unknown-sycldevice -Xs '-device Gen9'

from amr-wind.

What I mean is I believe the linker needs to have that option too. As for what's the proper cmake way to achieve that, I don't know. Maybe -DCMAKE_EXE_LINKER_FLAGS=... as a command line argument works.

from amr-wind.

from amr-wind.

@ax3l @WeiqunZhang
I'm facing errors using the flag CMAKE_EXE_LINKER_FLAGS: STRING=-fsycl-targets=spir64_gen-unknown-unknown-sycldevice -Xs '-device Gen9' on my local machine which has public release of Intel oneapi (DPCPP version: 2021.2.0). I was able to make this work with version 2021.x.0.20210520.

Is this an experimental feature that is only available on certain versions?

from amr-wind.

The AOT feature has been available for more than a year. Maybe the particular version of oneeapi on your local machine has a bug. It might also have something to do with the graphics driver.

from amr-wind.

Can I get some help on this @WeiqunZhang @ax3l?

I have access to only one oneapi version (the one released for public use). Could you test out at your end and see if this issue persists?

It might also have something to do with the graphics driver.

Could you elaborate?

from amr-wind.

How to disable google test? The latest dpcpp fails to compile it.

from amr-wind.

I have not got any errors yet. But it has been more than 4 hours and it's still doing AOT at link time.

from amr-wind.

Closing for now due to age of the issue and lack of traffic. Please reopen if further discussion is warranted.

from amr-wind.