hpddm / hpddm Goto Github PK

• float,
• std::complex<float>.

Calling numfact on a local matrix performs both the symbolic and numerical factorizations. When factorizing multiple local matrices with an equivalent sparsity pattern, the symbolic factorization should be done only once.

I would like to setup GCRODR with flexible preconditioning.
According to the manual you have to pass the option -ksp_hpddm_variant flexible. At the same time the manual says that this option is is superseded by KSPSetPCSide(KSP ksp, PCSide side). However, KSPSetPCSide only gives you the options PC_LEFT, PC_RIGHT and PC_SYMMETRIC.
When I pass the -ksp_hpddm_variant flexible option, PetscCall(KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD)) gives me the following output:

KSP Object: 1 MPI process
type: hpddm
HPDDM type: gcrodr
precision: DOUBLE
deflation subspace attached? TRUE
deflation target: SM
maximum iterations=500, initial guess is zero
tolerances:  relative=1e-06, absolute=1e-50, divergence=10000.
right preconditioning
using UNPRECONDITIONED norm type for convergence test

Do I really have flexible preconditioning now?
It would be nice if flexible preconditioning would be confirmed in this info statement.

This could be implemented inside the class Lapack.

It looks like the default solver for the subdomain is set to direct solver. However, I want to change it to iterative solver. I am using FreeFEM as an interface to hpddm. Is there a document or simple example/tutorial explaining how to change the subdomain solver to iterative solver in FreeFEM/hpddm environment?

Hello,
in your readme you mention that hpddm can be used via PETSc, however there is apparently no further documentation available. Also, when I check the compilation options of PETSc, there is no mention of hpddm. Is it still possible? If so, could you please tell me how to link those two packages?

This may sometimes happen when using penalized nonhomogeneous Dirichlet boundary conditions. As a result, two-level preconditioners converge badly.

I have recently been trying to install FreeFEM on Marconi100 powerpc and I have run into some troubles. Eventually we fixed the problem by forking the repository and changing hpddm/include/HPDDM.hpp. I will briefly descibe the problem and explain our fix. Hopefully you can use the information to find a neater/more permanent fix.

--
Marconi100 system:

Model: IBM Power AC922 (Whiterspoon)
Racks: 55 total (49 compute)
Nodes: 980
Processors: 2x16 cores IBM POWER9 AC922 at 2.6(3.1) GHz
Accelerators: 4 x NVIDIA Volta V100 GPUs/node, Nvlink 2.0, 16GB
Cores: 32 cores/node, Hyperthreading x4
RAM: 256 GB/node (242 usable)
Peak Performance: about 32 Pflop/s, 32 TFlops per node
Internal Network: Mellanox IB EDR DragonFly++
Disk Space: 8PB raw GPFS storage

Compilers:
spectrum_mpi - 10.3.1
gnu - 8.4.0

--

We tried different routes to do the install (hpddm via FreeFem, hpddm via PETSc, hppdm via PETSc via FreeFEM) but most routes eventually came to the same error of libpetsc.so havind undefined references to some lapack functions (error messages added below). However, libpetsc.so was correctly linked to liblapack.so. The problem is that liblapack.so has an underscore added to the function names (see other picture).

We have traced the source of this discrepancy to hpddm/include/HPDDM.hpp::70 :

#if defined(__powerpc__) || defined(INTEL_MKL_VERSION)
# define HPDDM_F77(func) func
#else
# define HPDDM_F77(func) func ## _
#endif

We solve it by forking the branch and changing this to (https://github.com/DaanVanVugt/hpddm/blob/hpddm_underscore_powerpc_2/include/HPDDM.hpp)

#define HPDDM_F77(func) func ## _

and changing the repository and commit-id of hpddm in petsc/config/BuildSystem/config/packages/hppdm.py

I hope you can take a look and create a more general fix that could work for everyone. I think solving this should also fix the issue for installing directly from FreeFEM (with PETSc, SLEPc and hpddm)
If you require any more information from me just let me know!

.

This is currently not handled when the MKL is not used, sometimes yielding wrong results especially when dealing with symmetric indefinite matrices.

If the number of Krylov directions to orthogonalize against is greater than the maximum number of iterations, then there is no need to allocate too much memory.

Would you like to wrap any pointers with the class template “std::unique_ptr”?

This may be useful for substructuring preconditioners which require pseudo-inverses.

Hi Pierre,
could you add support for python3?
python2 will be unavailable on future Debian/Ubuntu release...

Best
C

PS: for freefem++ which version of hpddm shall I use?

Because the maximum subdomain connectivity is currently hard-coded using a macro, cf.

I would like to point out that identifiers like “_A” and “_HPDDM_H_” do eventually not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

It is currently not possible to build the library with the latest version of MinGW (from 2013). This is due to two defects: one with lambda functions, the other with std::stoi/std::stof/std::stod (from C++11).

An extra null pointer check is not needed in functions like the following.

• buildTwo
• solveEVP
• Mumps

By default, IBM's XLF uses the -qnoextname option so that there is no underscore added to the names of Fortran global entities. Hence, there are currently some undefined symbols when linking with ARPACK for example.

I've updated hpddm at MacPorts to the version 2.2.2 and discovered that additional patches is required to support OpenBLAS.

Without patch bellow it fails on attempt to link because daxpby function has wrong prefix.

Here the used patch. Anyway, I feel that it isn't right one. The right one should introduce HPDDM_MKL like macros to detect and support OpenBLAS.

diff --git include/HPDDM_BLAS.hpp include/HPDDM_BLAS.hpp
index 4062a1f..2abbc98 100644
--- include/HPDDM_BLAS.hpp
+++ include/HPDDM_BLAS.hpp
@@ -100,7 +100,7 @@ HPDDM_GENERATE_EXTERN_BLAS_COMPLEX(k, std::complex<__fp16>, h, __fp16)
 #  endif
 #  if HPDDM_MKL || (defined(__APPLE__) && !defined(PETSC_HAVE_F2CBLASLAPACK))
 #   if !HPDDM_MKL
-#    define HPDDM_PREFIX_AXPBY(func) catlas_ ## func
+#    define HPDDM_PREFIX_AXPBY(func) cblas_ ## func
 #   else
 HPDDM_GENERATE_EXTERN_MKL_EXTENSIONS(c, std::complex<float>, s, float)
 HPDDM_GENERATE_EXTERN_MKL_EXTENSIONS(z, std::complex<double>, d, double)

I guess this should be Arg::numeric here:

Upcoming changes in PETSc are going to cause "nodiscard" warnings on PetscLogEventBegin() and PetscLogEventEnd() that are un-handled.

Can you please wrap lines like this in PetscCall() or discard them with "(void)"?

When running on MVAPICH2, I got a hang on MPI_Igather. (OpenMPI seemed unaffected.)
Using Wrapper<downscaled_type<K>>::mpi_type() instead of MPI_DATATYPE_NULL fixed it for me.

When imposing a BC by setting a row to 0 and its diagonal coefficient to 1, there is sometimes a problem when an unknown is also a Lagrange multiplier. A temporary workaround is to ensure that the original system remains symmetric (by also setting the appropriate column to 0).

I'm using hpddm-2.2.4 on macOS 12 via MacPorts. And attempt to run tests leads to infinity loop.

The log:

:info:test /opt/local/bin/mpiexec-mpich-mp -np 1  /opt/local/var/macports/build/_Users_catap_src_macports-ports_math_hpddm/hpddm/work/hpddm-2.2.4/bin/schwarz_cpp -hpddm_verbosity -hpddm_dump_matrices=.trash/output.txt -hpddm_version
:info:test [[email protected]] Sending Ctrl-C to processes as requested
:info:test [[email protected]] Press Ctrl-C again to force abort
:info:test ===================================================================================
:info:test =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
:info:test =   PID 66603 RUNNING AT Kirills-MBP.sa31-home.catap.net
:info:test =   EXIT CODE: 2
:info:test =   CLEANING UP REMAINING PROCESSES
:info:test =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
:info:test ===================================================================================
:info:test YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Interrupt: 2 (signal 2)
:info:test This typically refers to a problem with your application.
:info:test Please see the FAQ page for debugging suggestions
:info:test make: *** [test_bin/schwarz_cpp] Error 2

Before I've interrupted it I've waited for about an hour.

The end of output.txt:

     9998      9997 -9.99999999999999857891452847979962825775146484e+01
     9998      9998 3.99999999999999943156581139191985130310058594e+02
     9998      9999 -9.99999999999999857891452847979962825775146484e+01
     9999      9899 -9.99999999999999857891452847979962825775146484e+01
     9999      9998 -9.99999999999999857891452847979962825775146484e+01
     9999      9999 3.99999999999999943156581139191985130310058594e+02
     9999     10000 -9.99999999999999857891452847979962825775146484e+01
    10000      9900 -9.99999999999999857891452847979962825775146484e+01
    10000      9999 -9.99999999999999857891452847979962825775146484e+01
    10000     10000 3.99999999999999943156581139191985130310058594e+02

BTW it reaches that in bling of the eye, and after that it simple starts to consume CPU and it's all.