ecp-copa / exampm Goto Github PK
View Code? Open in Web Editor NEWMaterial point method proxy application based on Cabana.
License: BSD 3-Clause "New" or "Revised" License
exampm's People
Contributors
Stargazers
Watchers
exampm's Issues
Require Cabana 0.6
Issues running with MPI
Hi! I'm new to using this app, and was wondering if you have an example for running with mpirun (or similar?) I'm looking at the docs here: https://github.com/ECP-copa/ExaMPM/wiki/Run
Thank you! And apologies if this is an overly simple question (e.g., just put mpirun in front of that :P )
exampm fails to build in spack
spack install exampm leads to:
[ 66%] Building CXX object examples/CMakeFiles/DamBreak.dir/dam_break.cpp.o
cd /tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/examples && /home/junghans/spack.git/lib/spack/env/gcc/g++ -I/tmp/junghans/spack-stage/spack-stage-exa
mpm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/examples -I/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples -I/tmp/junghans/spack-stage/spack
-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src -I/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/src -isystem /home/junghans/spack.git
/opt/spack/linux-centos7-haswell/gcc-8.2.0/kokkos-3.1.01-ewfgo3wepuf7d37jgtinll7qk6zigof7/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/openmpi-3.1.6-22nn2lya7jfgf7eo
4vw42n7aq6t7dz2q/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cabana-master-ueinsnbtht7zrp5rmwof3wrt4jkd3pq7/include -isystem /home/junghans/spack.git/opt/spack/linu
x-centos7-haswell/gcc-8.2.0/silo-4.10.2-bk5whiftxlqnmvjcigaz5466nmm6vhaf/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -pthread -o CMakeFiles/FreeFall.dir/free_fall.cpp.o -c /tmp/junghans/s
pack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples/free_fall.cpp
cd /tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/examples && /home/junghans/spack.git/lib/spack/env/gcc/g++ -I/tmp/junghans/spack-stage/spack-stage-exa
mpm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/examples -I/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples -I/tmp/junghans/spack-stage/spack
-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src -I/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/src -isystem /home/junghans/spack.git
/opt/spack/linux-centos7-haswell/gcc-8.2.0/kokkos-3.1.01-ewfgo3wepuf7d37jgtinll7qk6zigof7/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/openmpi-3.1.6-22nn2lya7jfgf7eo
4vw42n7aq6t7dz2q/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cabana-master-ueinsnbtht7zrp5rmwof3wrt4jkd3pq7/include -isystem /home/junghans/spack.git/opt/spack/linu
x-centos7-haswell/gcc-8.2.0/silo-4.10.2-bk5whiftxlqnmvjcigaz5466nmm6vhaf/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -pthread -o CMakeFiles/DamBreak.dir/dam_break.cpp.o -c /tmp/junghans/s
pack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples/dam_break.cpp
In file included from /tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_Solver.hpp:16,
from /tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples/free_fall.cpp:1:
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_TimeIntegrator.hpp: In instantiation of 'void ExaMPM::TimeIntegrator::fieldSolve(const ExecutionSp
ace&, const ProblemManagerType&, double, double, const ExaMPM::BoundaryCondition&) [with ProblemManagerType = ExaMPM::ProblemManager<Kokkos::HostSpace>; ExecutionSpace = Kokkos::Serial]':
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_TimeIntegrator.hpp:373:15: required from 'void ExaMPM::TimeIntegrator::step(const ExecutionSpace
&, const ProblemManagerType&, double, double, const ExaMPM::BoundaryCondition&) [with ProblemManagerType = ExaMPM::ProblemManager<Kokkos::HostSpace>; ExecutionSpace = Kokkos::Serial]'
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_Solver.hpp:97:33: required from 'void ExaMPM::Solver<MemorySpace, ExecutionSpace>::solve(double,
int) [with MemorySpace = Kokkos::HostSpace; ExecutionSpace = Kokkos::Serial]'
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_Solver.hpp:79:10: required from here
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_TimeIntegrator.hpp:134:10: error: no matching function for call to 'Cajita::LocalGrid<Cajita::UniformMesh<double> >::indexSpace(Cajita::Ghost, Cajita::Node, Cajita::Global)'
auto global_nodes =
^~~~~~~~~~~~
Dam break example does not generate output files
Hi,
My aim is to write a SPH-DEM solver and simulate millions of particles using Cabana, and in the process I am learning to understand the internals and basics of the API.
I am successful at installing the ExaMPM software, however, the DamBreak example does not dump any output files while running.
Here is the output:
/DamBreak 0.05 2 0 0.001 1.0 50 serial
Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
For unit testing set OMP_PROC_BIND=false
Time 0.000000 / 1.000000
Time 0.050000 / 1.000000
...
Can someone please help with this issue?
Many thanks.
Build fails with cabana-0.3.0
106 cd /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-build/examples && /home/junghans/spack.git/lib/spack/env/gcc/g++ -I/tmp/junghans/spack-stage/sp
ack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-build/examples -I/tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/examples -I/tmp/
junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src -I/tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-buil
d/src -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/kokkos-3.1.01-ewfgo3wepuf7d37jgtinll7qk6zigof7/include -isystem /home/junghans/spack.git/opt/spack/linux-cent
os7-haswell/gcc-8.2.0/openmpi-3.1.6-jnys4eardy2dlj25lklmxcxuarflps3i/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cabana-0.3.0-qzqsu3nor3rltt3ecq63m6pgx
xn7u7eb/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cajita-0.1-lsqwyf355hijcigqrs6xwn24aarxxmd6/include -isystem /home/junghans/spack.git/opt/spack/lin
ux-centos7-haswell/gcc-8.2.0/silo-4.10.2-ga3tsj5twrcwtk5zgeq6xapshedtoghj/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -pthread -o CMakeFiles/DamBreak.dir/dam_break.cpp.o -c /
tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/examples/dam_break.cpp
107 In file included from /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_ProblemManager.hpp:16,
108 from /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_Solver.hpp:15,
109 from /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/examples/free_fall.cpp:1:
110 /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_ParticleCommunication.hpp: In function 'void ExaMPM::ParticleCommunication::redistrib
ute(const LocalGridType&, int, ParticleContainer&, std::integral_constant<long unsigned int, CoordIndex>, bool)':
>> 111 /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_ParticleCommunication.hpp:224:13: error: 'Distributor' is not a member of 'Cabana'
112 Cabana::Distributor<device_type> distributor( local_grid.globalGrid().comm(),
113 ^~~~~~~~~~~
No idea, but what version of cabaa is needed to build this?
Convert from Cajita to Cabana::Grid
Build with Cabana Master fails
2020-08-03T02:19:08.1445642Z [ 66%] Building CXX object examples/CMakeFiles/FreeFall.dir/free_fall.cpp.o
2020-08-03T02:19:08.1455217Z cd /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/examples && /home/runner/spack/lib/spack/env/gcc/g++ -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/examples -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/src -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/src -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/kokkos-3.1.01-4qzdxlx5fux2ydikb2ez5e42qtp66uhy/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/openmpi-3.1.6-5gsagjujszo4s347uw7vkyqn2teqzprv/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/cabana-master-xps3be3gaxfbk4izrjvn2hwxz233wfee/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/silo-4.10.2-va4cvfzv3ydoyjxjshzdlby4hihpmikt/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -mrtm -pthread -o CMakeFiles/DamBreak.dir/dam_break.cpp.o -c /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples/dam_break.cpp
2020-08-03T02:19:08.1462512Z cd /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/examples && /home/runner/spack/lib/spack/env/gcc/g++ -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/examples -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/src -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/src -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/kokkos-3.1.01-4qzdxlx5fux2ydikb2ez5e42qtp66uhy/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/openmpi-3.1.6-5gsagjujszo4s347uw7vkyqn2teqzprv/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/cabana-master-xps3be3gaxfbk4izrjvn2hwxz233wfee/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/silo-4.10.2-va4cvfzv3ydoyjxjshzdlby4hihpmikt/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -mrtm -pthread -o CMakeFiles/FreeFall.dir/free_fall.cpp.o -c /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples/free_fall.cpp
2020-08-03T02:19:09.5645222Z In file included from /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/src/ExaMPM_Solver.hpp:15,
2020-08-03T02:19:09.5804759Z from /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples/free_fall.cpp:1:
2020-08-03T02:19:09.5805597Z /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/src/ExaMPM_ProblemManager.hpp:78:49: error: wrong number of template arguments (2, should be 1)
2020-08-03T02:19:09.5806127Z 78 | using halo = Cajita::Halo<double,MemorySpace>;
2020-08-03T02:19:09.5806375Z | ^
from https://github.com/proxyapps/proxy-ci/runs/938692143?check_suite_focus=true
Serial build on spack broke
From https://github.com/proxyapps/proxy-ci/runs/1051635986
[ 95%] Building CXX object cajita/unit_test/CMakeFiles/Cajita_SplineEvaluation_test_SERIAL.dir/SERIAL/tstSplineEvaluation_SERIAL.cpp.o
cd /tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test && /home/runner/spack/lib/spack/env/gcc/g++ -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/gtest -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/unit_test -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test/SERIAL -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/unit_test/../../core/unit_test -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/src -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/src -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-9.3.0/kokkos-3.2.00-g6y3qy5fm6vw66eqwhoemkbkg7cbcqar/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-9.3.0/openmpi-3.1.6-c4biojltyxh4hvwvjlc47vmh3vkoiwfq/include -DGTEST_HAS_PTHREAD=0 -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -pthread -std=c++14 -o CMakeFiles/Cajita_SplineEvaluation_test_SERIAL.dir/SERIAL/tstSplineEvaluation_SERIAL.cpp.o -c /tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test/SERIAL/tstSplineEvaluation_SERIAL.cpp
In file included from /tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/unit_test/tstInterpolation.hpp:17,
from /tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test/SERIAL/tstInterpolation_SERIAL.cpp:2:
/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/src/Cajita_Interpolation.hpp:243:51: error: type/value mismatch at argument 4 in template parameter list for 'template<class DataType, class Layout, class DeviceType, class Op, class Duplication, class Contribution> class Kokkos::Experimental::ScatterView'
243 | DataType, Layout, ExecSpace, Op, duplication, contribution>>
| ^~~~~~~~~~~~
/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/src/Cajita_Interpolation.hpp:243:51: note: expected a type, got 'Op'
MPI boundary issue
From Mark on slack:
I've been using ExaMPM (DamBreak) for performance and profiling on one Crusher node and continue to see what look like communications errors leading to spurious new velocities at processor boundaries which then lead to numerical blow-up and crashes (almost always in g2p->scatter->packBuffer). This only occurs for problems over about 50^3 cells and 4 or more MPI ranks (I test with srun -N1 -n8 -S16 --exclusive -t30:00 --cpus-per-task=1 --threads-per-core=1 --gpus-per-task=1 --gpu-bind=closest ./DamBreak 0.01 2 3 0.00004 10.0 2500 hip) and does not go away with wider halos, evenly-divisible ny, different Y boundary conditions (periodic, slip, noslip), or different versions of Cabana (0.5.0, head). It seems to be suppressed somewhat with fewer particles per cell and by using AMD_SANITIZE_KERNEL and AMD_SANITIZE_COPY vars, but never goes away. For a while, it seemed to always happen 2 or 3 time steps after a Silo write, but still occurs without any Silo writes. It can occur anywhere between steps 5000 and 100000.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.