Phase field crystal (PFC) is a semi-atomistic technique, containing atomic resolution information of crystalline structures while operating on diffusive time scales. PFC has an ability to simulate solidification and elastic-plastic material response, coupled to a wide range of phenomena, including formation and co-evolution of microstructural defects such as dislocations and stacking faults, voids, defect formation in epitaxial growth, displacive phase transitions, and electromigration.

OpenPFC is an open-source framework for high performance 3D phase field crystal simulations. It is designed to scale up from a single laptop to exascale class supercomputers. OpenPFC has succesfully used to simulate domain of size 8192 x 8192 x 4096 on CSC Mahti. 200 computing nodes were used, where each node contains 128 cores, thus total 25600 cores were used. During the simulation, 25 TB of memory was utilized. The central part of the solver is Fast Fourier Transform with time complexity of O(N log N), and there are no known limiting bottlenecks, why larger models could not be calculated as well.

The project documentation can be found from https://vtt-propertune.github.io/OpenPFC/dev/.

• scales up to tens of thousands of cores, demonstrably
• modern c++17 header only framework, easy to use

• Todo

Requirements:

• Compiler supporting C++17 standard. C++17 features are available since GCC 5. Check your version number with g++ --version. The default compiler might be relatively old, and more recent version needs to be loaded with module load gcc. Do not try to compile with GCC 4.8.5. It will not work. At least GCC versions 9.4.0 (coming with Ubuntu 20.04) and 11.2 are working.
• OpenMPI. All recent versions should work. Tested with OpenMPI version 2.1.3. Again, you might need to load proper OpenMPI version with module load openmpi/2.1.3, for instance. Additionally, if cmake is not able to find proper OpenMPI installation, assistance might be needed by setting MPI_ROOT, e.g. export MPI_ROOT=/share/apps/OpenMPI/2.1.3.
• FFTW. Probably all versions will work. Tested with FFTW versions 3.3.2 and 3.3.10. Again, cmake might need some assistance to find the libraries, which can be controlled with environment variable FFTW_ROOT. Depending how FFTW is installed to system, it might be in non-standard location and module load fftw is needed. You can use commands like whereis fftw or ldconfig -p | grep fftw to locate your FFTW installation, if needed.

Typically in clusters, these are already installed and can be loaded with an on-liner

module load gcc openmpi fftw

For local Linux machines (or WSL2), packages usually can be installed from repositories, e.g. in case of Ubuntu, the following should work:

sudo apt-get install -y gcc openmpi fftw

Some OpenPFC applications uses json files to provide initial data for simulations. In principle, applications can also be built to receive initial data in other ways, but as a widely known file format, we recommend to use json. The choice for json package is JSON for Modern C++. There exists packages for certain Linux distributions (nlohmann-json3-dev for Ubuntu, json-devel for Centos) for easy install. If the system-wide installation is not found, the library is downloaded from GitHub during the configuration.

The last and most important dependency in order to use OpenPFC is HeFFTe, which is our choice for parallel FFT implementation. The instructions to install HeFFTe can be found from here. HeFFTe can be downloaded from https://bitbucket.org/icl/heffte/downloads/.

If HeFFTe is installed to some non-standard location, cmake is unable to find it when configuring OpenPFC. To overcome this problem, the install path of HeFFTe can be set into environment variable CMAKE_PREFIX_PATH. For example, if HeFFe is installed to $HOME/opt/heffte/2.3, the following is making cmake to find HeFFTe succesfully:

export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.3:$CMAKE_PREFIX_PATH

During the configuration, OpenPFC prefers local installations, thus if HeFFTe is already installed and founded, it will be used. For convenience, there is a fallback method to fetch HeFFTe sources from internet and build it concurrently with OpenPFC. In general, however, it is better to build and install programs one at a time. So, make sure you have HeFFTe installed and working on your system before continuing.

OpenPFC uses cmake to automate software building. First the source code must be downloaded to some appropriate place. Head to the releases page and pick the newest release and unzip is somewhere. Alternatively, if you are planning to develop the project itself of are just interested about the bleeding edge features, you might be interested to clone the repository to your local machine. GitHub account is needed to clone the project.

git clone https://github.com/VTT-ProperTune/OpenPFC.git
# git clone [email protected]:VTT-ProperTune/OpenPFC.git  # if you prefer ssh instead
cd OpenPFC

Next step is to configure project. One might consider at least setting option CMAKE_BUILD_TYPE to Debug or Release. For large scale simulations, make sure to use Release as it turns on compiler optimizations.

cmake -DCMAKE_BUILD_TYPE=Release -S . -B build

Keep on mind, that configuration will download HeFFTe if the local installation is not found. To use local installation instead, add HeFFTe path to environment variable CMAKE_PREFIX_PATH or add Heffte_DIR option to point where HeFFTe configuration files are installed. Typical configuration command in cluster environment is something like

module load gcc openmpi fftw
export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.3:$CMAKE_PREFIX_PATH
cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_INSTALL_PREFIX=$HOME/opt/openpfc \
      -S . -B build

Then, building can be done with command cmake --build build. After build finishes, one should find example codes from ./build/examples and apps from ./build/apps. Installation to path defined by CMAKE_INSTALL_PREFIX can be done with cmake --install build.

OpenPFC is a software framework. It doesn't give you ready-made solutions, but a platform on which you can start building your own scalable PFC code. We will familiarize ourselves with the construction of the model with the help of a simple diffusion model in a later stage of the documentation. However, let's give a tip already at this stage, how to start the development work effectively. Our "hello world" code is as follows:

#include <iostream>
#include <openpfc/openpfc.hpp>

using namespace std;
using namespace pfc;

int main() {
  World world({32, 32, 32});
  cout << world << endl;
}

To compile, CMakeLists.txt is needed. Minimal CMakeLists.txt is:

cmake_minimum_required(VERSION 3.15)
project(helloworld)
find_package(OpenPFC REQUIRED)
add_executable(main main.cpp)
target_link_libraries(main OpenPFC)

With the help of CMakeLists.txt, build and compilation of application is straightforward:

cmake -S . -B build
cmake --build build
./build/main

There is also some examples in examples directory, which can be used as a base for your own codes.

Here's some common problems and their solutions.

During the configuration step (cmake -S. -B build), you might end up to the following error message:

CMake Error at CMakeLists.txt:3 (find_package):
By not providing "FindOpenPFC.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "OpenPFC", but
CMake did not find one.

The error message is trying to say command in CMakeLists.txt (line 3) fails:

find_package(OpenPFC REQUIRED)  # <-- this is failing

The reason why this happens is that cmake is not able to find the package. By default, cmake finds packages by looking a file which is called Find<package_name>.cmake from couple of standard locations. For example, in Ubuntu, one of these locations is /usr/lib/cmake, where the files are installed when doing a global install of some package with root rights. When working with supercomputers, users in general doesn't have rights to make global installations, thus packages are almost always installed to some non-default locations. Thus, one needs to give some hints to cmake where the file could be found. This can be done (at least) two different ways.

The first way is to set up an environment variable indicating any extra locations for the files. One option is to use CMAKE_PREFIX_PATH environment variable, like before. For example, if OpenPFC is installed to /opt/OpenPFC, one can give that information before starting configuration:

export CMAKE_PREFIX_PATH=/opt/OpenPFC:$CMAKE_PREFIX_PATH
cmake -S . -B build
# rest of the things ...

Another option is to hard code the choise inside the CMakeLists.txt file directly. Just keep in mind, that this option is not very portable as users tends to install software to several different locations and there is no any general rule how it should be done. So, instead of defining CMAKE_PREFIX_PATH before doing configuration, the following change in CMakeLists.txt is equivalent:

cmake_minimum_required(VERSION 3.15)
project(helloworld)
# find_package(OpenPFC REQUIRED)                                     #  <-- Replace this command ...
find_package(OpenPFC REQUIRED PATHS /opt/OpenPFC/lib/cmake/OpenPFC)  #  <-- ... with this one
add_executable(main main.cpp)
target_link_libraries(main OpenPFC)

This way, cmake know to search necessary files from the path given above.

There might be various reasons why the simulation returns NaNs. Despite the reason, it usually makes sense to stop simulation as it doesn't do anything useful. OpenPFC does not currently have built-in json validator, which would check that simulation parameters are valid. Thus, it is possible to give invalid parameters to the simulation, which might lead to NaNs. If some model parameters which should be defined are undefined and thus zero, there might be zero division problem.

There is schema file for the input file, which can be used to validate the json file using external validator like check-jsonchema:

check-jsonschema --schemafile apps/schema.json input.json

OpenPFC implements NaN check, which is enabled by default when compiling with a debug build type:

cmake -DCMAKE_BUILD_TYPE=Debug -S . -B build

Another way to enable NaN check is to use compile option NAN_CHECK_ENABLED. In CMakeLists.txt, add the following line:

add_compile_definitions(NAN_CHECK_ENABLED)

Or, when configuring project with cmake, the following is equivalent:

cmake -DNAN_CHECK_ENABLED=ON -S . -B build

Or another, quick and dirty solution might be to simply add the following to the source file:

#define NAN_CHECK_ENABLED

Then, in code level, there's a macro CHECK_AND_ABORT_IF_NANS, which can be used to check if there's any NaNs in the simulation. The macro is defined in openpfc/utils/nancheck.hpp. This is a zero overhead when compiling with release build type. At the moment, user must explicitly call the macro, but in the future it might be called automatically in some situations. Example usage is (see also this example):

std::vector<double> psi = {1.0, 2.0, 3.0, 4.0, 5.0, 6.0};
CHECK_AND_ABORT_IF_NANS(psi);
psi[0] = std::numeric_limits<double>::quiet_NaN();
CHECK_AND_ABORT_IF_NANS(psi);

A bigger application example is Tungsten model. Todo.

• Todo

@article{
  blaablaa
}