compilation error using intel compilers about pymfem HOT 12 CLOSED

@i3s93 Thank you. METIS recently changed to put some functionality outside as GKlib. This is caused by an adjustment I introduced due to this change. Actually, handling this change is more complicated than I thought, since depending on the version of METIS, mfem may or may not need to be linked using -GKlib. For now, we are rolling back the setup.py script to specifically use an older version in https://github.com/mfem/tpls. For now, can you please use an older PyMFEM?

from pymfem.

Are you using master branch? It is true that I merged one PR yesterday (or one day before). Can you share the error message you are seeing?

from pymfem.

The error happened when using the master branch. After the following command:
python setup.py install --with-gslib --with-parallel --ext-only --CC=icc --CXX=icc --MPICC=mpiicc --MPICXX=mpiicc

we got the following error:

icc -pthread -B /home/docker/.conda/envs/mfem/compiler_compat -shared -L/home/docker/.conda/envs/mfem/lib -Wl,-z,noexecstack,-z,relro,-z,now,-rpath,$ORIGIN/../..,-rpath,$ORIGIN/../../.. -L/home/docker/.conda/envs/mfem/lib -L/home/docker/.conda/envs/mfem/lib -Wl,-z,noexecstack,-z,relro,-z,now,-rpath,$ORIGIN/../..,-rpath,$ORIGIN/../../.. build/temp.linux-x86_64-3.9/std_vectors_wrap.o -L/home/docker/.conda/envs/mfem/lib/python3.9/site-packages/mfem/external/ser/lib -Wl,-R/home/docker/.conda/envs/mfem/lib/python3.9/site-packages/mfem/external/ser/lib -lmfem -o /home/docker/PyMFEM/mfem/_ser/_std_vectors.cpython-39-x86_64-linux-gnu.so

[91merror: command '/opt/intel/oneapi/compiler/2022.1.0/linux/bin/intel64/icc' failed with exit code 4

It works fine if I specify an older commit for mfem, i.e.,

python setup.py install --with-gslib --with-parallel --ext-only --CC=icc --CXX=icc --MPICC=mpiicc --MPICXX=mpiicc --mfem-branch=e7485db5ebc5d40411887aa97563dd6a3722e3d8

from pymfem.

I'm also encountering issues when trying to build on a cluster with Intel compilers, but have not yet identified a solution either. My errors are slightly different. Since I already have builds for serial and parallel MFEM, I decided to provide the prefix of the path to the dependencies. When I run

python setup.py install --with-parallel --CC=icc --CXX=icc --MPICC=mpiicc --MPICXX=mpicc --mfemp-prefix="~/Projects/mfem/build_parallel" --hypre-prefix="~/Projects/hypre/src" --metis-prefix="~/Projects/metis-5.1.0" --skip-ext

I get a collection of errors that look like

In file included from pnonlinearform_wrap.cxx(3384): catastrophic error: cannot open source file "_config.hpp" #include "_config.hpp".

The last error that shows up is

File "setup.py", line 1397, in run File "setup.py", line 923, in make_mfem_wrapper File "setup.py", line 286, in make_call File "subprocess.py", line 328, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['python', 'setup.py', 'build_ext', '--inplace', '--parallel', '55']' returned non-zero exit status 1.

So it's a different error message than what you are seeing, but I'm also having problems here.

from pymfem.

For more clarity, I am performing the following commands:
python setup.py install --with-gslib --with-parallel --ext-only --CC=icc --CXX=icc --MPICC=mpiicc --MPICXX=mpiicc
python setup.py clean --swig
python setup.py install --with-gslib --with-parallel --swig
python setup.py install --with-gslib --with-parallel --skip-ext --CC=icc --CXX=icc --MPICC=mpiicc --MPICXX=mpiicc

This last step produces the following errors:

icc -pthread -B /home/docker/.conda/envs/mfem/compiler_compat -shared -L/home/docker/.conda/envs/mfem/lib -Wl,-z,noexecstack,-z,relro,-z,now,-rpath,$ORIGIN/../..,-rpath,$ORIGIN/../../.. -L/home/docker/.conda/envs/mfem/lib -L/home/docker/.conda/envs/mfem/lib -Wl,-z,noexecstack,-z,relro,-z,now,-rpath,$ORIGIN/../..,-rpath,$ORIGIN/../../.. build/temp.linux-x86_64-3.9/std_vectors_wrap.o -L/home/docker/.conda/envs/mfem/lib/python3.9/site-packages/mfem/external/ser/lib -Wl,-R/home/docker/.conda/envs/mfem/lib/python3.9/site-packages/mfem/external/ser/lib -lmfem -o /home/docker/PyMFEM/mfem/_ser/_std_vectors.cpython-39-x86_64-linux-gnu.so

[91merror: command '/opt/intel/oneapi/compiler/2022.1.0/linux/bin/intel64/icc' failed with exit code 4

and

nonlininteg_wrap.cxx(3441): catastrophic error: cannot open source file "fem/ceed/interface/operator.hpp"
#include "fem/ceed/interface/operator.hpp"
^

compilation aborted for nonlininteg_wrap.cxx (code 4)

I hope this helps.

from pymfem.

After many attempts I realized that if you specify the branch=master it works. The commands are as follows:
python setup.py install --with-gslib --with-parallel --ext-only --CC=icc --CXX=icc --MPICC=mpiicc --MPICXX=mpiicc --mfem-branch=master
python setup.py clean --swig
python setup.py install --with-gslib --with-parallel --swig
python setup.py install --with-gslib --with-parallel --skip-ext --CC=icc --CXX=icc --MPICC=mpiicc --MPICXX=mpiicc

Whatever is the default mfem branch in setup.py does not work.

Cheers

from pymfem.

@i3s93, @adantra .
Thank you for reporting this. The command sequence @adantra regenerate a entire wrapper code, and it is good know that the SWGI recipe files (*.i) are still okay. @i3s93, please let us know if the command sequence works for you, too.

Incidentally, I am working on an update to run SWIG automatically in #143. I will make sure that it will be tested with Intel compiler before merging.

from pymfem.

By the way...
When mfem-branch=master is not set. setup.py build MFEM with a specific version of MFEM.
Master branch build it using MFEM 4.4 release (specified in https://github.com/mfem/PyMFEM/blob/master/setup.py#L54)
Probably I should have modified it to a newer version (see https://github.com/mfem/PyMFEM/blob/autorun_swig_dev/setup.py#L55), when I merged a recent PR.

from pymfem.

@sshiraiwa, sorry for the late response. In short, the sequence of commands above works for me in the case of the serial build. I ran your script run_examples.py in serial mode and most of the tests pass, but there seems to be a problem with examples 4 and 25. I believe you are already aware of the latter case. In example 4, it looks like the output is slightly different, but I have not investigated this any further. The parallel build is still not working for me. I have built MFEM and it's dependencies as a separate project, so I already have those library files. What I would like to do is supply the prefix for each of these dependencies, so that the individual pieces do not have to be built every time. I am using the following installation command:

python setup.py install --with-parallel --mfem-source=~/Projects/mfem/ --mfem-prefix=~/Projects/mfem/build_parallel/ --metis-prefix=/opt/apps/intel19/metis/5.1.0/ --hypre-prefix=~/Projects/hypre/src/ --CC=icc --CXX=icc --MPICC=mpiicc --MPICXX=mpiicc

When I run the script, the build does look in the right directories and seems to be successful at building some of the wrappers, but then I see this error:

calling ... /home1/apps/intel19/swig/4.0.0/bin/swig -Wall -c++ -python -fastproxy -olddefs -keyword -I/home1/09047/was2489/Projects/mfem/build_parallel/include -I/home1/09047/was2489/Projects/mfem/build_parallel/include/mfem -I/home1/09047/was2489/Projects/mfem -I/home1/09047/was2489/Projects/hypre/src/include -I/opt/apps/intel19/metis/5.1.0/include -I/opt/apps/intel19/impi19_0/python3/3.7.0/lib/python3.7/site-packages/mpi4py/include pfespace.i ../common/hypre_int.i:1: Error: Unable to find 'HYPRE_utilities.h' Failed when calling command: /home1/apps/intel19/swig/4.0.0/bin/swig -Wall -c++ -python -fastproxy -olddefs -keyword -I/home1/09047/was2489/Projects/mfem/build_parallel/include -I/home1/09047/was2489/Projects/mfem/build_parallel/include/mfem -I/home1/09047/was2489/Projects/mfem -I/home1/09047/was2489/Projects/hypre/src/include -I/opt/apps/intel19/metis/5.1.0/include -I/opt/apps/intel19/impi19_0/python3/3.7.0/lib/python3.7/site-packages/mpi4py/include pfespace.i Traceback (most recent call last): File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1614, in <module> main() File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1610, in main run_setup() File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1598, in run_setup setup( File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands self.run_command(cmd) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command cmd_obj.run() File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1399, in run _install.run(self) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/command/install.py", line 72, in run orig.install.run(self) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 692, in run self.run_command('build') File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command self.distribution.run_command(command) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command cmd_obj.run() File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/command/build.py", line 24, in run super().run() File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 132, in run self.run_command(cmd_name) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command self.distribution.run_command(command) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command cmd_obj.run() File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1451, in run generate_wrapper() File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 921, in generate_wrapper make_call(command) File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 294, in make_call subprocess.check_call(command, **kwargs) File "/home1/09047/was2489/miniconda3/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/home1/apps/intel19/swig/4.0.0/bin/swig', '-Wall', '-c++', '-python', '-fastproxy', '-olddefs', '-keyword', '-I/home1/09047/was2489/Projects/mfem/build_parallel/include', '-I/home1/09047/was2489/Projects/mfem/build_parallel/include/mfem', '-I/home1/09047/was2489/Projects/mfem', '-I/home1/09047/was2489/Projects/hypre/src/include', '-I/opt/apps/intel19/metis/5.1.0/include', '-I/opt/apps/intel19/impi19_0/python3/3.7.0/lib/python3.7/site-packages/mpi4py/include', 'pfespace.i']' returned non-zero exit status 1.

This error is due to a missing file, but my earlier build for Hypre was successful, so I'm not sure why this file is missing. As I mentioned earlier, I would really like to point the installation script to my other builds for MFEM and its dependencies so that I don't have to keep compiling the source code. Would you happen to have any suggestions for this?

As an alternative to the above, I chose to build MFEM, but point the script to the installations of Hypre and Metis on our cluster:

python setup.py install --with-parallel --metis-prefix=/opt/apps/intel19/metis/5.1.0/ --hypre-prefix=/home1/apps/intel19/impi19_0/petsc/3.14/clx/ --CC=icc --CXX=icc --MPICC=mpiicc --MPICXX=mpiicc --mfem-branch=master

This leads to the error message below.

/opt/apps/gcc/8.3.0/bin/ld: cannot find -lGKlib
make[2]: *** [libmfem.so.4.4.0] Error 1
make[2]: Leaving directory `/home1/09047/was2489/Projects/PyMFEM/external/mfem/cmbuild_par'
make[1]: *** [CMakeFiles/mfem.dir/all] Error 2
make[1]: Leaving directory `/home1/09047/was2489/Projects/PyMFEM/external/mfem/cmbuild_par'
make: *** [all] Error 2
Failed when calling command: mfem_parallel
Traceback (most recent call last):
  File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1614, in <module>
    main()
  File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1610, in main
    run_setup()
  File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1598, in run_setup
    setup(
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands
    self.run_command(cmd)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
    cmd_obj.run()
  File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1399, in run
    _install.run(self)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/command/install.py", line 72, in run
    orig.install.run(self)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 692, in run
    self.run_command('build')
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
    cmd_obj.run()
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/command/build.py", line 24, in run
    super().run()
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 132, in run
    self.run_command(cmd_name)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
    cmd_obj.run()
  File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 1446, in run
    cmake_make_mfem(serial=False)
  File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 730, in cmake_make_mfem
    make('mfem_' + txt)
  File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 327, in make
    make_call(command, target=target)
  File "/home1/09047/was2489/Projects/PyMFEM/setup.py", line 294, in make_call
    subprocess.check_call(command, **kwargs)
  File "/home1/09047/was2489/miniconda3/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j', '55']' returned non-zero exit status 2.

In this case, the script cannot find GKlib, which is apparently a dependency for Metis. I'm not sure why this file is missing because a proper build of Metis would have included this.

I apologize for the long message, but hopefully this gives you a sense of the problems I have encountered.

from pymfem.

@i3s93, @adantra
Closing this for now. As mentioned earlier, we rolled back setup.py to point an older version of METIS when building it from setup.py. Please take a look at the current master (https://github.com/mfem/PyMFEM/releases/tag/v4.4.0.1).
Note that when compiling with intel the proper option is. --CXX=icpc --MPICXX=mpiicpc. Also, (probably) mpi4py needs to compiled with the same MPI compiler.

from pymfem.

from pymfem.

Hi, @jcwright77. We are using .tar.gz file of metis under https://github.com/mfem/tpls, This includes libGK.
This is an older form of how METIS5 was packaged. It is not necessary to download libGK separately.

As a side note, in setup.py, if you change the following flag to True

 use_metis_gklib = False

then, it downloads both libGK and METIS from Github repo. It also makes some adjustment to a link flag
when building MFEM so that libmfem.so is linked both libGK and METIS. However, at moment we are
not testing this feature well, and it is not recommended to use it.

from pymfem.