Git Product home page Git Product logo

mamico's Introduction

The macro-micro-coupling tool for multiscale coupled molecular-continuum flow simulation.

Documentation

  • The source code documentation can be found here
  • The unit test code coverage is here

Requirements and optional dependencies

To compile and execute MaMiCo on a linux system, you need at least:

  • a C++17 compiler installed (e.g. GCC 7 or newer)
  • CMake version >= 3.0

Optional dependencies are:

  • MPI, highly recommended for parallel runs, e.g. on clusters.
  • Eigen 3, a linear algebra library used for the POD noise filter.
  • pybind11, for the MaMiCo Python bindings.
  • OpenFOAM, for coupling to CFD simulations with OpenFOAM.
  • preCICE, for coupling with other solvers using the preCICE coupling library.
  • ls1-MarDyn, for coupling to MD simulations with ls1.
  • LAMMPS, for coupling to MD simulations with LAMMPS.
  • LCOV and gcc, for test code coverage analysis.

Build instructions

  • First clone this repository and create a new build directory:

      git clone [email protected]:HSU-HPC/MaMiCo.git
      cd MaMiCo
      mkdir build
      cd build
    
  • If you don't have a github account or don't want to use it, you can alternatively use git clone https://github.com/HSU-HPC/MaMiCo.git (read-only).

  • Then initialise CMake in your build directory, pointing it to the root directory of the repository. (Note that on some older systems, the command may be named cmake3 instead of cmake.)

      cmake ..
    
  • This will configure the build system with default settings and generate makefiles in the build directory. Optionally, you might want to change the build configuration now:

      ccmake .
    
  • Here you can enable optional dependencies, e.g. activate BUILD_WITH_MPI (default OFF), or modifiy which compiler and flags you want to use. When you are done, press the configure [c] and generate [g] keys. Now you can start the compilation:

      make
    

Additional instructions to build with ls1

  • After cloning the repository, initialize the ls1 submodule with

      git submodule init
      git submodule update
    
  • Follow the instructions on the ls1 repository to build with cmake, however remember to enable the MAMICO_COUPLING flag, and provide the MaMiCo base directory in the MAMICO_SRC_DIR variable.

  • Make MaMiCo as normal, choosing LS1_MARDYN as your MD library.

Additional instructions to build with LAMMPS

  1. Download the LAMMPS source code, build it using CMake with the option -DBUILD_SHARED_LIBS=ON, and install it using make install.
  2. Make LAMMPS available through pgk-config with export PKG_CONFIG_PATH=/home/$(whoami)/.local/lib/pkgconfig.
  3. Create a symbolic link to the LAMMPS header files in the MaMiCo root folder using ln -s <path to LAMMPS>/src <path to MaMiCo>/lammps

Input file construction and first runs

  • The build instructions have created an executable of the standard Couette flow test case, the file is called 'couette'. It expects an XML configuration file named 'couette.xml' in the current working directory.
  • An example simulation configuration file can be found here, other template input files are in the examples folder as well.
  • Copy this file to your build folder, rename it 'couette.xml'.
  • The available options and features are listed directly in the template file via XML comments, so that you can modify the configuration to suit your needs.
  • Start the simulation by executing (sequential case) ./couette or e.g. (MPI-parallel) mpirun -n 8 ./couette.
  • If you get the error message 'ERROR MoleculeService::MoleculeService: Could not open file CheckpointSimpleMD_10000_reflecting_0.checkpoint!', copy the file of the same name from the 'examples' folder into your build folder.
  • Depending on the configuration, you will obtain various output files in CSV, VTK or other formats.

Papers to cite

mamico's People

Contributors

thinkpiet avatar maurerf avatar amartyads avatar louievoit avatar jafariv avatar frida-oskar avatar rubenhorn avatar vincntfrst avatar wittmern avatar leonard2804 avatar pmoldtmann avatar

Stargazers

Lea avatar zhang zhiguo avatar  avatar Surya Kiran Peravali avatar  avatar Markus Mühlhäußer avatar  avatar  avatar Jan Lukas Augustin avatar Gerasimos Chourdakis avatar  avatar

Watchers

James Cloos avatar

mamico's Issues

Re-implement branch mpi-collective

The branch mpi-collective was never merged and is way too far behind (see master...mpi-collective).
Thus, it is necessary to re-implement the features on this branch based on the current version of the master branch.

To-Do:

  • Re-implement changes from branch mpi-collective
  • Test correctness of the code using simple static couette flow scenario
  • Test correctness of the code using dynamic scenario (adding/removing MD instances depending on error threshold)
  • Delete stale branch mpi-collective

Parallel-in-time two-way coupling

Currently, PinT does not work with TWC

Steps to do:

  • enable momentumTransfer on inner cells for MD equilibration steps in F (ParallelTimeIntegrationService.h)
  • run MD equilibration in F
  • check if MD is running in a consistent state with CFD
  • double check that filter pipeline output ends up in F.getState() after F()
  • debug TWC crashes

Class diagrams in Doxygen

Generate class diagrams for Doxygen documentation using graphviz/dot

Pros:

Dependencies/interactions between classes are easier to see at a glance.

Cons:

Increases build time from ca. 2 seconds to ca. 14 seconds (locally, not including installing graphviz).

Use valid XML for config files

Issue

Currently, the MaMiCo configuration XML files have multiple root nodes, but according to the XML standard there should be only one, so other parsers may not be able to read them.

Solution

Updating the configuration parser to deal with a single root note (similar to ls1) containing the current configuration structure (similar to ls1):

<?xml version="1.0" encoding="UTF-8"?>
<scenario-configuration>
   <!-- contents of the old configuration file format -->
</scenario-configuration>

All example configuration files should be updated accordingly.
The parser should still be able to handle the legacy format for compatibility with old configuration files.

Steps

  • Refactor the XML parsing
  • Update (or write) unit tests for the new AND old configuration formats
  • Update the example configuration files

SimpleMD fails during equ. with 120³ domain

Run CouetteScenario...
Run CouetteScenario...
Run CouetteScenario...
Finish MD timestep 0...
Finish MD timestep 50...
Capacity of buffer was exceeded when reallocation is not permitted. Terminating...
Pushing molecule to send buffer 3 failed. Terminating...
Rank 11 was unable to push molecule to send buffer 3. 
Upper bound of buffer (70) exceeded. Fatal error. Please contact developers as this is a principal issue.
srun: error: node0084: task 11: Exited with exit code 1
srun: Terminating StepId=323787.0
slurmstepd: error: *** STEP 323787.0 ON node0084 CANCELLED AT 2024-08-14T15:29:42 ***
srun: error: node0084: tasks 0-10,12-63: Terminated
srun: Force Terminated StepId=323787.0

Rename CouetteSolverInterface

coupling/solvers/CouetteSolverInterface.h is named 'CouetteSolverInterface', however it actually is not related to Couette flow in any way. It is not even directly related to any specific CFD solver coupling or implementation, besides the assumption of it running sequentially. I think this is basically just a 'default MSI' (MacroscopicSolverInterface), maybe we should rename the class and file accordingly.

Move the defintion of the OpenFoam-installation-directory from couette.xml to a CMAKE-Flag

Currently, the installation directory of OpenFOAM is defined in the couette.xml-file as macroscopic-solver->foam-setup-directory.
For consistency, it would be great to be able to define the path as a CMake-flag instead.

As the installation path of OpenFOAM is a platform-specific variable rather than a simulation-specific variable, it would make more sense to define it before compilation.

Furthermore, this simplifies use of MaMiCo with the automation tool FabMaMiCo, as the tool tries to setup MaMiCo individually for every remote machine.
The couette.xml-file is user-defined and should not contain such platform-specific variables.

The same applies to the related variable foam-setup-folder.

One might also think about a clearer distinction in the naming of the two variables open-foam-directory and open-foam-folder.

Refactoring during MaMiCo Coding Week February 2024

Branch for refactoring: refactor2402-WIP

TODO-List:

  • Init load balancing @amartyads
  • Rename MacroscopicCell to CouplingCell (#41) @rubenhorn
  • Remove all types of legacy IndexConversion @Thinkpiet
  • Refactor indexing from IndexConversion to CellIndex
    • coupling/cell-mapping @LouieVoit
    • coupling/configurations @rubenhorn
    • coupling/datastructures (#43) @rubenhorn
    • coupling/filtering @Thinkpiet
    • coupling/interface @Thinkpiet
    • coupling/paralleltopology
    • coupling/python-binding
    • coupling/scenario
    • coupling/sendrev
    • coupling/services @jafariv
    • coupling/solvers
    • coupling/transferstrategies
    • test/integration
    • test/unit
  • Use new CouplingCell datastructures

Collective MPI

Collective MPI communication has been implemented and can be activated using a cmake flag.
However, there is currently no speed-up when using collective MPI, suggesting inefficient communication.
Furthermore, collective MPI is not used by FromMD2MacroRecvOnly.cpph and FromMacro2MDSendOnly.cpph.

Enhance Wiki/Documentation

As a newbie to MaMiCo, the Wiki is helpful in a lot of ways.
However, it's missing out some pages and explanations.
Some pages have not been created at all, and some contain a lot of "TODO"-keywords.

It would be of great help if the wiki was up-to-date and filled with essential information, at least not representing a WIP-state.

Information about compilation with most dependencies would help a lot, too (e.g. OpenFOAM).
Best would be some examples (maybe even on a specific remote machine, like HSUper).

I am aware this is a very generic issue with no list of objectives, but any improvement made during MaMiCo coding week would be very much appreciated.

MultiMDCellService modifies MolecularDynamicsConfiguration

In coupling/services/MultiMDCellService.h simplemd::configurations::MolecularDynamicsConfiguration mdConfiguration is passed as a non-const reference, and lines 128-132 in the constructor modify the initFromCheckpoint setting. While modifying this setting is intended behavior, it it unexpected that the simpleMDConfig object outside MultiMDCellService changes. mdConfiguration should be const, and the checkpoint settings should be somewhere in InstanceHandling or so, not in the config object

CI test ls1 files

Configure the automated GitHub build tests to test ls1 files, by doing a submodule init, compiling with proper parameters, and also executing (only) the ls1 tests.

FromMacro2MDTest and FromMD2MacroTest do not test anything

In

test/unit/coupling/sendrecv/FromMD2MacroTest.cpp and test/unit/coupling/sendrecv/FromMacro2MDTest.cpp

there are "test()" functions. While this code is named "test", it just prints debug output, but does not execute any tests. Thus there is no reason to execute it at all. Tasks:

  • Rename test() to something more appropriate
  • Refactor “test” (demo?) code (extract to somewhere else than CCPUNIT_TEST)
  • Do the same for testUniqueRanks2D() and testUniqueRanks3D() in test/unit/IndexConversionTest.cpp

Config validation

Running couette.xml.example with 3 MPI ranks causes crash (works with 2 or 4):

$ mpirun -n 3 ../build/couette 
Run CouetteScenario...
Run CouetteScenario...
Run CouetteScenario...
Initialization: 0ms
Finish CouetteScenario::initSolvers() 
Finish CouetteScenario::initSolvers() 
Finish CouetteScenario::initSolvers() 
double free or corruption (out)
[thinkpad:1050255] *** Process received signal ***
[thinkpad:1050255] Signal: Aborted (6)
[thinkpad:1050255] Signal code:  (-6)
[thinkpad:1050255] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fda8f042520]
[thinkpad:1050255] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fda8f0969fc]
[thinkpad:1050255] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fda8f042476]
[thinkpad:1050255] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fda8f0287f3]
[thinkpad:1050255] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x89676)[0x7fda8f089676]
[thinkpad:1050255] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0xa0cfc)[0x7fda8f0a0cfc]
[thinkpad:1050255] [ 6] /lib/x86_64-linux-gnu/libc.so.6(+0xa2e70)[0x7fda8f0a2e70]
[thinkpad:1050255] [ 7] /lib/x86_64-linux-gnu/libc.so.6(free+0x73)[0x7fda8f0a5453]
[thinkpad:1050255] [ 8] ../build/couette(_ZN8coupling8services31MacroscopicCellServiceMacroOnlyILj3EED0Ev+0x3e)[0x558b56981bee]
[thinkpad:1050255] [ 9] ../build/couette(_ZN15CouetteScenario8shutdownEv+0x431)[0x558b569c3ca1]
[thinkpad:1050255] [10] ../build/couette(_Z11runScenarioP8Scenario+0x9d)[0x558b5697423d]
[thinkpad:1050255] [11] ../build/couette(main+0x3d)[0x558b569622cd]
[thinkpad:1050255] [12] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fda8f029d90]
[thinkpad:1050255] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fda8f029e40]
[thinkpad:1050255] [14] ../build/couette(_start+0x25)[0x558b56963535]
[thinkpad:1050255] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 0 on node thinkpad exited on signal 6 (Aborted).
--------------------------------------------------------------------------
$ git rev-parse --short HEAD
33f31b10

Excessive output by NLM filter

Issue:
There is WAY too much output being printed to stdout by NLM.cpph without using some debugging flags.

std::cout << "NLM info: [...]

This causes giant output files and excessive runtimes.

Proposed solution:
Restrict this output to (filtering-) debug builds using the corresponding CMake flag/preprocessor macro.
Perhaps something separate from DEBUG_FILTER_PIPELINE should be used.

(Relates to #9)

Less prints to output

currently every rank prints "Run CouetteScenario", "Finish CouetteScenario::initSolvers()" and "Finish CouetteScenario::shutdown() Shut down CouetteScenario" to the output. It floods output and makes manual scanning very cumbersome, and writes huge files with no actual data. We should remove these prints.

/examples 👉 /tools

What/Why:
The /examples folder contains too many things that are strictly speaking not examples (e.g. plotting scripts).
Additionally, we will soon have a config generator which makes files like the couette.xml.template obsolete.

Proposed solution:
We should create a /tools folder and clean up /examples. (Proposed by @Thinkpiet)

  • Move scripts to /tools
  • Document scripts in /tools/README.md or a Wiki page referenced from the main README
  • Remove redundant templates/other files from /examples (couette.xml, checkpoints, etc?)
  • Update all references in the code, README, Wiki, etc. to /examples that are now outdated

Some files may remain in /examples. Their usage should also be explicitly documented.

Only start this after merging #65!

Remove commented out code

There is a lot of commented out code in the repository making it potentially hard to read and maintain.
We should…
A) Remove commented out code
B) Add context to comment in which the code might still be useful (e.g. explain workaround for fringe use cases)
C) For code that is (or was) “work in progress”, new issues/branches should be created instead.

NaNs after ca. 220 coupling cycles

After refactoring (#45), the couette scenario fails after about 220 coupling cycles if 2-way coupling is enabled:

With SIMPLE_MD and couette.xml.template but with 1 MD instance and ONLY with 2-way coupling

Finish coupling cycle 234
Finish coupling cycle 244
ERROR simplemd::moleculemappings::VelocityStoermerVerletMapping::handleMolecule: Position 0 is out of range
Position: -nan , 21.3675 , 5.01122, molecule: 2679
Velocity: -nan , -0.811837 , 0.0131969, molecule: 2679
OldVelocity: 0.904897 , -0.800832 , 0.019862, molecule: 2679
Force: -nan , -1.75438 , -1.70058, old: -3.47021 , -2.64766 , -0.965466
Old position: 17.8997 , 21.3716 , 5.01118

With LS1_MARDYN (663745bb88c8f85d9df7abb55b32c74bb7dad03a) and couette.xml.template but with 1 MD instance and ONLY with 2-way coupling

Finish coupling cycle 223
Finish coupling cycle 228
ERROR:  20240808T125404  752.102 [0]    NaN detected, exiting.
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.