Segmentation fault when running more than one turbine about amr-wind HOT 11 CLOSED

Hi everyone, thank you for all your help. The issue has been resolved. Here's a report on how I fixed the issue in case anyone comes across this in the future.

TL;DR : I had to recompile OpenFAST and AMR-Wind from source with OpenMP off for both and MPI on for AMR-Wind.

REPORT:
I compiled everything from source, including all dependencies such as BLAS and LAPACK. I had to get a little bit hacky and copy some files which were installed elsewhere on the BYU supercomputer but in the end it worked. Specifically, I had to copy libgfortran.so.5 to my openfast/lib directory and libmpi.so.40 to my amrwind/lib directory. My build uses OpenFAST-v3.5.2 and AMR-Wind-v2.1.0. I should note, compiling with these two was enough to get past the two initializations of OpenFAST (Which was the original issue) but the run would still crash after the initial pressure itterations, right after writing the first plot file, with the following error:

terminate called after throwing an instance of 'std::runtime_error'
what(): FastIface: Error calling OpenFAST function:
FAST_Solution:FAST_AdvanceStates:ED_ABM4:ED_AB4:ED_Input_ExtrapInterp:ED_Input_ExtrapInterp2:t(1) must not equal t(2) to avoid a division-by-zero error.

But even with this build, a single turbine run would still finish without issue.
Anyway, at @psakievich 's suggestion, I recompiled with OpenMP off and MPI on for AMR-Wind. This fixed the issue and I am now able to run multiple turbines without the program crashing. Just in case anybody is curious as to the exact cmake command I used to build the programs, here they are:

OpenFAST compilation from openfast/build (As you can see the only additional flag was one to install to a specific directory):
cmake -DCMAKE_INSTALL_PREFIX=$HOME/exawind/openfast/my_openfast ..

AMR-Wind:
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/exawind/amr-wind/myamr7 -DAMR_WIND_ENABLE_OPENFAST=ON -DOpenFAST_DIR=$HOME/exawind/openfast/my_openfast/lib/cmake/OpenFAST -DCMAKE_EXE_LINKER_FLAGS="-static-libstdc++" -DAMR_WIND_ENABLE_MPI=ON

Once again, thank you everybody for your help. I couldn't have fixed this without you.

p.s.
Built with cmake-v3.27.8 and GCC-v13.2.0

from amr-wind.

This definitely feels like it could be due to using old versions of the stack. You mentioned using spack and I am not sure the best way to do that here. We tend to use cmake or exawind-manager (that uses spack under the hood). I would definitely prioritize updating the stack and then seeing if the segfault remains.

From the attached files you shared, it definitely looks like the error is in openfast.

@psakievich, @jrood-nrel is there a way they can use vanilla spack here? I am a bit surprised it went with an old version of openfast. Or should they try the exawind-manager route?

from amr-wind.

@justhawk98 what spack version are you using? I would prefer we fix this in spack if that is the issue rather than pushing more people to exawind-manager.

Using spack@develop I get this:

$ spack solve amr-wind@main ^[email protected]
==> Error: concretization failed for the following reasons:

   1. Cannot select a single "version" for package "openfast"
   2. Cannot satisfy '[email protected]:'
   3. Cannot satisfy '[email protected]'
   4. Cannot satisfy '[email protected]:3.4.1'
   5. Cannot satisfy '[email protected]:'
        required because amr-wind depends on [email protected]: when @2:+openfast
          required because amr-wind@main ^[email protected] requested explicitly
   6. Cannot satisfy '[email protected]'
        required because amr-wind@main ^[email protected] requested explicitly
   7. Cannot satisfy '[email protected]:' and '[email protected]
        required because amr-wind depends on [email protected]: when @2:+openfast
          required because amr-wind@main ^[email protected] requested explicitly
        required because amr-wind@main ^[email protected] requested explicitly
   8. Cannot satisfy '[email protected]' and '[email protected]:
        required because amr-wind depends on [email protected]: when @2:+openfast
          required because amr-wind@main ^[email protected] requested explicitly
        required because amr-wind@main ^[email protected] requested explicitly

So I suspect this is an older version of spack before we updated all the package requirements.

from amr-wind.

@justhawk98 you need to use [email protected], but your binaries also have to be old. amr-wind won't compile with [email protected] anymore. Several releases have been added and updated in the latest spack. Looks like those latest changes didn't make it into [email protected] so going to develop would be best. Many fixes are in for the package using [email protected] but we know many openfast bugs were fixed as part of the 3.5 release so that would be better to use.

from amr-wind.

@justhawk98 you need to use [email protected], but your binaries also have to be old. amr-wind won't compile with [email protected] anymore. Several releases have been added and updated in the latest spack. Looks like those latest changes didn't make it into [email protected] so going to develop would be best. Many fixes are in for the package using [email protected] but we know many openfast bugs were fixed as part of the 3.5 release so that would be better to use.

For what it's worth, Justin messaged me a few weeks back before AMR-Wind 2.0 and the OpenFAST 3.5 changeover, and he was having this same problem at that time

from amr-wind.

Hi @justhawk98, I noticed that you're running two openfast turbines but have only 1 rank/mpi process for the entire simulation. I believe that amr-wind will let this happen (both openfast turbine instances will end up on the same processor), but it is probably not good practice.

Lawrence

from amr-wind.

Thanks for the help and suggestions everyone. Currently, we are using spack 0.19.2. It's looking like the best option is to recompile with an updated OpenFAST. I'll reach out to our research computing office and see what they think is the best way to do that. Also I wanted to ask, are there any flags we're missing that would be good to turn on? Or any that would be better to turn off?

from amr-wind.

@justhawk98 if you are stuck on Spack 0.19.2 then you won't have the newest versions registered in spack. You can pull down the amr-wind and openfast package.py files from spack and create a custom repo in spack to use them, but you might have to do some editing.

Alternatively for openfast you could give the spec [email protected]=3.4.1 to have spack treat the 3.5.2 git tag like a known version in spack (in this example it was 3.4.1) when it builds. That might be sufficient.

from amr-wind.

@justhawk98 in terms of flags, I don't think we typically run with openmp. I would try turning that off on both. We recently had to turn that off in openfast when it was accidentally turned on. I believe the issues was segfaults. So that could be your issue...

from amr-wind.

And even if it is off, it could be still picking it up: OpenFAST/openfast#2229. We've asked that it be completely disabled so (very) recent commits of openfast should be safe(r)

from amr-wind.

I was having the same issue when running small tests in my laptop with OpenMP. I sorted it out running with MPI. For instance:
mpirun -np 2 amr_wind <inputfile.inp>
I guess when you compile with MPI you must have at least one MPI rank per turbine?
By the way, I did not have to change anything in the code, nor recompile with a different version of OpenFAST.

from amr-wind.