Phase field crystal (PFC) is a semi-atomistic technique, containing atomic resolution information of crystalline structures while operating on diffusive time scales. PFC can simulate solidification and elastic-plastic material response, coupled with a wide range of phenomena, including formation and co-evolution of microstructural defects such as dislocations and stacking faults, voids, defect formation in epitaxial growth, displacive phase transitions, and electromigration.
The image above shows a simulation of a rapidly solidifying tungsten block approximately 50 x 100 x 200 nm in size, using MovingBC boundary conditions. The rightmost section depicts the pure atomic structure of the tungsten, the middle section highlights the surface of the entire object and the leftmost section provides a transparent view of the surface, revealing the lattice defects that formed during the simulation. This visualization aids in understanding the atomic arrangement, surface features, and internal defects of the tungsten block.
OpenPFC is an open-source framework for high-performance 3D phase field crystal simulations. It is designed to scale up from a single laptop to exascale class supercomputers. OpenPFC has successfully been used to simulate a domain of size 8192 x 8192 x 4096 on CSC Mahti. 200 computing nodes were used, where each node contained 128 cores, thus total of 25600 cores were used. During the simulation, 25 TB of memory was utilized. The central part of the solver is the Fast Fourier Transform with time complexity of O(N log N), and there are no known limiting bottlenecks, why larger models could not be calculated as well.
The graph above demonstrates the remarkable scalability and performance of the simulation framework. In the strong scaling analysis (panel a), the step time significantly decreases as the number of cores increases for various grid sizes, from 256³ to 8192³. This indicates efficient parallelization, though the rate of decrease diminishes at higher core counts.
In the weak scaling analysis (panel b), the step time remains relatively constant as the grid size increases while maintaining a fixed number of voxels per core. This stability illustrates excellent weak scaling performance, highlighting the framework's capability to efficiently manage larger problems by proportionally increasing computational resources. The right Y-axis projects the number of time steps calculable in a week, emphasizing the framework's suitability for extensive simulations on supercomputers. Notably, these simulations were conducted using the LUMI supercomputer, further showcasing the framework's capability to leverage top-tier computational resources for high-performance simulations.
T. Pinomaa, J. Aho, J. Suviranta, P. Jreidini, N. Provatas, and A. Laukkanen, “OpenPFC: an open-source framework for high performance 3D phase field crystal simulations”, Modelling Simul. Mater. Sci. Eng., Feb. 2024, doi: 10.1088/1361-651X/ad269e. (link)
The project documentation can be found from https://vtt-propertune.github.io/OpenPFC/dev/.
- scales up to tens of thousands of cores, demonstrably
- modern c++17 header-only framework, easy to use
- Todo
Requirements:
- Compiler supporting C++17 standard: C++17 features are
available since GCC 5. Check
your version number with
g++ --version. The default compiler might be relatively old, and a more recent version needs to be loaded withmodule load gcc. Do not try to compile with GCC 4.8.5. It will not work. At least GCC versions 9.4.0 (coming with Ubuntu 20.04) and 11.2 are working. - CMake: Version 3.15 or later should be used. Your system may already contain CMake, but if not, it can most likely be installed with the package manager.
- OpenMPI: All recent versions should work.
Tested with OpenMPI version 2.1.3. Again, you might need to load proper
OpenMPI version with
module load openmpi/2.1.3, for instance. Additionally, if CMake is not able to find proper OpenMPI installation, assistance might be needed by settingMPI_ROOT, e.g.export MPI_ROOT=/share/apps/OpenMPI/2.1.3. - FFTW: Probably all versions will work. Tested
with FFTW versions 3.3.2 and 3.3.10. Again, CMake might need some assistance
to find the libraries, which can be controlled with environment variable
FFTW_ROOT. Depending how FFTW is installed to system, it might be in non-standard location andmodule load fftwis needed. You can use commands likewhereis fftworldconfig -p | grep fftwto locate your FFTW installation, if needed. - Niels Lohmann's JSON for Modern C++ library: All recent versions should work. Tested with version 3.11.2. If you do not have the JSON library installed, CMake for OpenPFC will download the library for you.
- HeFFTe: All recent versions should work. Tested with version 2.3.0.
Typically in clusters, these are already installed and can be loaded with an on-liner
module load gcc openmpi fftwFor local Linux machines (or WSL2), packages usually can be installed from repositories, e.g. in the case of Ubuntu, the following should work:
sudo apt-get install -y gcc openmpi fftwSome OpenPFC applications use JSON files to provide initial data for
simulations. In principle, applications can also be built to receive initial
data in other ways, but as a widely known file format, we recommend using JSON.
The choice for the JSON package is JSON for Modern C++.
There exist packages for certain Linux distributions (nlohmann-json3-dev for
Ubuntu, json-devel for Centos) for easy installation. If the system-wide installation
is not found, the library is downloaded from GitHub during the configuration.
The last and most important dependency to use OpenPFC is HeFFTe, which is our choice for parallel FFT implementation. The instructions to install HeFFTe can be found from here. HeFFTe can be downloaded from https://github.com/icl-utk-edu/heffte. Your typical workflow to install HeFFTe would be something like this:
cmake -S heffte-2.4.0-src -B heffte-2.4.0-build \
-DCMAKE_INSTALL_PREFIX=/opt/heffte/2.4.0 \
-DCMAKE_BUILD_TYPE=Release -D Heffte_ENABLE_FFTW=ON
cmake --build heffte-2.4.0-build
cmake --install heffte-2.4.0-buildIf HeFFTe is installed in some non-standard location, CMake is unable to find it
when configuring OpenPFC. To overcome this problem, the install path of HeFFTe
can be set into the environment variable CMAKE_PREFIX_PATH. For example, if HeFFe
is installed to $HOME/opt/heffte/2.3, the following is making CMake to find
HeFFTe successfully:
export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.3:$CMAKE_PREFIX_PATHDuring the configuration, OpenPFC prefers local installations, thus if HeFFTe is already installed and found, it will be used. For convenience, there is a fallback method to fetch HeFFTe sources from the internet and build it concurrently with OpenPFC. In general, however, it is better to build and install programs one at a time. So, make sure you have HeFFTe installed and working on your system before continuing.
OpenPFC uses cmake to automate software building. First, the source code must be downloaded to some appropriate place. Head to the releases page and pick the newest release and unzip it somewhere. Alternatively, if you are planning to develop the project itself or are just interested in the bleeding-edge features, you might be interested in cloning the repository to your local machine. A GitHub account is needed to clone the project.
git clone https://github.com/VTT-ProperTune/OpenPFC.git
# git clone [email protected]:VTT-ProperTune/OpenPFC.git # if you prefer ssh instead
cd OpenPFCThe next step is to configure the project. One might consider at least setting an option
CMAKE_BUILD_TYPE to Debug or Release. For large-scale simulations, make
sure to use Release as it turns on compiler optimizations.
cmake -DCMAKE_BUILD_TYPE=Release -S . -B buildKeep in mind, that the configuration will download HeFFTe if the local installation
is not found. To use local installation instead, add HeFFTe path to the environment
variable CMAKE_PREFIX_PATH or add Heffte_DIR option to point where HeFFTe
configuration files are installed. A typical configuration command in a cluster
environment is something like
module load gcc openmpi fftw
export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.3:$CMAKE_PREFIX_PATH
cmake -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=$HOME/opt/openpfc \
-S . -B buildThen, building can be done with the command cmake --build build. After the build
finishes, one should find example codes from ./build/examples and apps from
./build/apps. Installation to a path defined by CMAKE_INSTALL_PREFIX can be
done with cmake --install build.
classDiagram
App~Model~ --> Simulator
Simulator --> Model
Simulator --> Time
Simulator --> FieldModifier
Simulator --> ResultsWriter
Model --> FFT
Model --> RealField
Model --> ComplexField
FFT --> Decomposition
Decomposition --> World
The OpenPFC framework aims to simplify the development of highly scalable
applications for solving partial differential equations using spectral methods.
It provides a modular application structure that users can customize by
inheriting and overriding specific classes. When examining the class diagram
from bottom to top, we first encounter classes such as World, Decomposition,
and FFT. These classes form a low-level layer responsible for domain
decomposition and performing FFT using the HeFFTe library. These details might
not be of general interest from an implementation perspective, except for the
framework developers themselves.
Next, we have classes such as Model, FieldModifier, and ResultsWriter. The
Model class is of particular interest as it describes the physics of the
model, including the partial differential equation (PDE) itself. Inside the
Model class, there is a function called step that needs to be overridden.
Currently, users are free to choose whichever time integration method they are
comfortable with. However, in the future, we may abstract the time integration
method away from the model and create a separate class to approach the problem
using "The Method of Lines" view. The Model class consists of one or several
different "fields" which can be real or complex-valued. These fields are updated
during time stepping. The FieldModifier class does exactly what the name
implies – it modifies these fields. In more detail, initial and boundary
conditions serve as field modifiers and are often also of interest, although
some already implemented ones exist. Lastly, we should mention the
ResultsWriter, which implements a way to store results during certain periods.
We have some existing implementations such as raw binary format and vti
format, but nothing is preventing us from implementing, for example, the storage
of results in hdf5 format, which is currently under planning.
In the third level, we have the Simulator, which assembles and runs the actual
simulation. It's a simple container-like class that calls lower-level objects in
a stepper to ensure that everything is called in time. Typically, there should
be no need to override this, but it is still possible to do so.
The top level is App, which handles the user interface. Since the simulations
are usually run on supercomputers, we don't have anything fancy like a graphical
user interface or interactive control of the simulation. Instead, we input user
parameters in an input file, preferably in JSON format. After reading the model
parameters, the simulator starts. This type of user interface is very basic, but
it works well in high-performance computing (HPC) environments where there are
no displays available. Typically, a batch script (e.g. Slurm) is created to run
the application in the chosen HPC environment's queue.
OpenPFC is a software framework. It doesn't give you ready-made solutions, but a platform on which you can start building your own scalable PFC code. We will familiarize ourselves with the construction of the model with the help of a simple diffusion model in a later stage of the documentation. However, let's give a tip already at this stage, how to start the development work effectively. Our "hello world" code is as follows:
#include <iostream>
#include <openpfc/openpfc.hpp>
using namespace std;
using namespace pfc;
int main() {
World world({32, 32, 32});
cout << world << endl;
}To compile, CMakeLists.txt is needed. Minimal CMakeLists.txt is:
cmake_minimum_required(VERSION 3.15)
project(helloworld)
find_package(OpenPFC REQUIRED)
add_executable(main main.cpp)
target_link_libraries(main OpenPFC)With the help of CMakeLists.txt, build and compilation of the application is
straightforward:
cmake -S . -B build
cmake --build build
./build/mainThere are also some examples in examples directory, which can be used as a base for your codes.
The Cahn-Hilliard equation is a fundamental model in materials science used to describe the phase separation process in binary mixtures. It models how the concentration field evolves over time to minimize the free energy of the system. The equation is particularly useful in understanding the dynamics of spinodal decomposition and coarsening processes.
Spectral methods, combined with the Fast Fourier Transform (FFT), are highly efficient for solving partial differential equations (PDEs) like the Cahn-Hilliard equation. The FFT allows us to transform differential operators into algebraic ones in the frequency domain, significantly simplifying the computation. This approach is particularly advantageous for problems involving periodic boundary conditions and large-scale simulations, where the efficiency and accuracy of the FFT are paramount.
Exponential time integration is well-suited for stiff PDEs like the Cahn-Hilliard equation. Traditional explicit methods require very small time steps to maintain stability, which can be computationally expensive. Exponential integrators, however, handle the stiff linear part of the equation exactly, allowing for larger time steps without sacrificing stability. This makes the integration process more efficient and stable, especially for long-term simulations.
Starting with the Cahn-Hilliard equation:
Applying the Fourier transform to the equation converts the spatial differential operators into algebraic forms:
Simplifying the right-hand side:
Using an exponential integrator, we handle the linear part exactly and integrate the non-linear part explicitly:
Here,
Assuming
Combining the terms, we obtain the discrete update rule for
Simplify the coefficient for the non-linear term:
The linear and non-linear operators can be defined as follows:
These operators are used to update the concentration field
Below is the code snippet for the Cahn-Hilliard model in OpenPFC:
class CahnHilliard : public Model {
private:
std::vector<double> opL, opN, c; // Define operators and field c
std::vector<std::complex<double>> c_F, c_NF; // Define (complex) psi
double gamma = 1.0e-2; // Surface tension
double D = 1.0; // Diffusion coefficient
public:
void initialize(double dt) override {
FFT &fft = get_fft();
const Decomposition &decomp = get_decomposition();
// Allocate space for the main variable and it's fourier transform
c.resize(fft.size_inbox());
c_F.resize(fft.size_outbox());
c_NF.resize(fft.size_outbox());
opL.resize(fft.size_outbox());
opN.resize(fft.size_outbox());
add_real_field("concentration", c);
// prepare operators
World w = get_world();
std::array<int, 3> o_low = decomp.outbox.low;
std::array<int, 3> o_high = decomp.outbox.high;
size_t idx = 0;
double pi = std::atan(1.0) * 4.0;
double fx = 2.0 * pi / (w.dx * w.Lx);
double fy = 2.0 * pi / (w.dy * w.Ly);
double fz = 2.0 * pi / (w.dz * w.Lz);
for (int k = o_low[2]; k <= o_high[2]; k++) {
for (int j = o_low[1]; j <= o_high[1]; j++) {
for (int i = o_low[0]; i <= o_high[0]; i++) {
// Laplacian operator -k^2
double ki = (i <= w.Lx / 2) ? i * fx : (i - w.Lx) * fx;
double kj = (j <= w.Ly / 2) ? j * fy : (j - w.Ly) * fy;
double kk = (k <= w.Lz / 2) ? k * fz : (k - w.Lz) * fz;
double kLap = -(ki * ki + kj * kj + kk * kk);
double L = kLap * (-D - D * gamma * kLap);
opL[idx] = std::exp(L * dt);
opN[idx] = (L != 0.0) ? (opL[idx] - 1.0) / L * kLap : 0.0;
idx++;
}
}
}
}
void step(double) override {
// Calculate cₙ₊₁ = opL * cₙ + opN * cₙ³
FFT &fft = get_fft();
fft.forward(c, c_F);
for (auto &elem : c) elem = D * elem * elem * elem;
fft.forward(c, c_NF);
for (size_t i = 0; i < c_F.size(); i++) {
c_F[i] = opL[i] * c_F[i] + opN[i] * c_NF[i];
}
fft.backward(c_F, c);
}
};
The full code can be found from examples.
Here are some common problems and their solutions.
During the configuration step (cmake -S. -B build), you might end up with the
following error message:
CMake Error at CMakeLists.txt:3 (find_package):
By not providing "FindOpenPFC.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "OpenPFC", but
CMake did not find one.
The error message is trying to say the command in CMakeLists.txt (line 3) fails:
find_package(OpenPFC REQUIRED) # <-- this is failingThe reason why this happens is that CMake is not able to find the package. By
default, CMake finds packages by looking at a file which is called
Find<package_name>.cmake from a couple of standard locations. For example, in
Ubuntu, one of these locations is /usr/lib/cmake, where the files are
installed when doing a global install of some package with root rights. When
working with supercomputers, users, in general, don't have rights to make global
installations, thus packages are almost always installed to some non-default
locations. Thus, one needs to give some hints to CMake where the file could be
found. This can be done (at least) in two different ways.
The first way is to set up an environment variable indicating any extra
locations for the files. One option is to use CMAKE_PREFIX_PATH environment
variable, like before. For example, if OpenPFC is installed to /opt/OpenPFC,
one can give that information before starting the configuration:
export CMAKE_PREFIX_PATH=/opt/OpenPFC:$CMAKE_PREFIX_PATH
cmake -S . -B build
# rest of the things ...Another option is to hardcode the choice inside the CMakeLists.txt file
directly. Just keep in mind, that this option is not very portable as users
tends to install software to several different locations and there is no any
general rule on how it should be done. So, instead of defining CMAKE_PREFIX_PATH
before doing configuration, the following change in CMakeLists.txt is
equivalent:
cmake_minimum_required(VERSION 3.15)
project(helloworld)
# find_package(OpenPFC REQUIRED) # <-- Replace this command ...
find_package(OpenPFC REQUIRED PATHS /opt/OpenPFC/lib/cmake/OpenPFC) # <-- ... with this one
add_executable(main main.cpp)
target_link_libraries(main OpenPFC)This way, CMake knows to search for necessary files from the path given above.
There might be various reasons why the simulation returns NaNs. Despite the reason, it usually makes sense to stop simulation as it doesn't do anything useful. OpenPFC does not currently have a built-in JSON validator, which would check that simulation parameters are valid. Thus, it is possible to give invalid parameters to the simulation, which might lead to NaNs. If some model parameters that should be defined are undefined and thus zero, there might be a zero division problem.
There is a schema file for the input file, which can be used to validate the JSON
file using an external validator like check-jsonchema:
check-jsonschema --schemafile apps/schema.json input.jsonOpenPFC implements NaN check, which is enabled by default when compiling with a debug build type:
cmake -DCMAKE_BUILD_TYPE=Debug -S . -B buildAnother way to enable NaN check is to use compile option NAN_CHECK_ENABLED. In CMakeLists.txt, add the following line:
add_compile_definitions(NAN_CHECK_ENABLED)Or, when configuring a project with CMake, the following is equivalent:
cmake -DNAN_CHECK_ENABLED=ON -S . -B buildOr another, quick and dirty solution might be to simply add the following to the source file:
#define NAN_CHECK_ENABLEDThen, at the code level, there's a macro CHECK_AND_ABORT_IF_NANS, which can be
used to check if there are any NaNs in the simulation. The macro is defined in
openpfc/utils/nancheck.hpp. This is a zero overhead when compiling with
release build type. At the moment, a user must explicitly call the macro, but in
the future, it might be called automatically in some situations. Example usage is
(see also this example):
std::vector<double> psi = {1.0, 2.0, 3.0, 4.0, 5.0, 6.0};
CHECK_AND_ABORT_IF_NANS(psi);
psi[0] = std::numeric_limits<double>::quiet_NaN();
CHECK_AND_ABORT_IF_NANS(psi);@article{pinomaa2024openpfc,
title={OpenPFC: an open-source framework for high performance 3D phase field crystal simulations},
author={Pinomaa, Tatu and Aho, Jukka and Suviranta, Jaarli and Jreidini, Paul and Provatas, Nikolaos and Laukkanen, Anssi},
journal={Modelling and Simulation in Materials Science and Engineering},
year={2024}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent