spdomin / nalu Goto Github PK

Interpolate previous solution to new mesh to use as an initial condition.

Allow for min/max plus to be provided to surface force and moment calculation

A user supplied wall BC specification for heat conduction that includes radiative and convective bcs does not work as expected. Specifically, the solver algorithm map searches for entry in the WALL_BC entry.

The bc types enum needs to be modified to allow for multiple types of WALL_BCs, e.g., WALL_BC_RAD

I tried to restart a simulation that used backward Euler time integration while activating a BDF2 approach in the new simulation. However, the code cored. I thought that I had the correct interface to stk_io, however, seemingly not.

Activities:

1. Improve baseline sliding mesh capability at curved surfaces.
2. Evaluate ATDM-based parallel search methods.
3. Establish matrix set-up cost timings.
4. Evaluate possible lagging of matrix update.
5. Evaluate reduction of matrix system by omitting moving block column entries in favor of multiple matrix assembly/solve iterations.

Working through the DataProbes interface has exposed (possibly) an issue with multiple specifications for turbulence averaging.

Currently, the PMR/Fluids coupling simply provides the full d/dxj(qj) source term. Alternatively, provide the linearized source term (black body - scalar flux) for more stable couplings.

nodes_per_element is used to register wall friction factor and Yp.

CHT and heat flux looks correct.

Would be nice to be able to specify multiple target surfaces within BC blocks, e.g. target_name: [surface_1, surface_2, ...]

In the head Nalu code base, P=2 post processing of heat flux is not working.

Activities:

1. Establish local row- and column-maps for dofs only at moving mesh interfaces.
2. Modify matrix assembly to exploit locally changed matrix maps.
3. Work with Tpetra team to allow for in situ matrix modifications.

Trouble running the periodicEdge and periodicElement tests with latest version of Nalu from Github. The parallel consistency check in periodic manager fails with the compared values being slightly off.

Issue tracked down to the 85d266c commit on March 1st with the initial search tolerance being modified to 1.0e-8. I have a search tolerance of 1.0e-5 specified in my input file, but Nalu chooses to use the more stringent tolerance (lines 65 & 66 in periodic manager). I just ran these models in master with the initial search tolerance set to 1.0e-4, and the parallel consistency check passed with the models running to completion.

This is a desired feature that will be useful for external aerodynamics calculations, e.g. flow over a wind turbine blade.

Many of the data members in LinearSolverConfig are not initialized in the constructor. Specifically, they are initialized when the "get_if_present" yaml call is made. Worse yet, the defaults are provided to this method.

This is dangerous and should be replaced by the options defaulted in the constructor.

Activities:

1. Benchmark, at scale, solver setup costs over the history of the project (~once per year).
2. Improve RAP operation.
3. Integrate with Kokkos and Tpetra.

outputP0() should provide only output to P0 while output() should provide output all all ranks.

However, at present, output() does not produce any output.

Be careful here as the design had to have a NULL stream provided for outputP0() to all other ranks for performance reasons.

Hi

when building on a linux platform cmake adds PATH_TO_YAML/include/yaml-cpp as an include directory. But Nalu explicitly include <yaml-cpp/SOME_FILE.hpp>.

Heres output:
[ 0%] Building CXX object CMakeFiles/nalu.dir/src/HeatCondMassBackwardEulerNodeSuppAlg.C.o /opt/cray/craype/2.1.3/bin/CC -O3 -I/home/crtrott/lib/yaml/0.3.0/gnu/4.9.0/include/yaml-cpp -I/home/crtrott/TrilinosBuild/SerialClassic-Nalu/install/lib/cmake/Trilinos/../../../include -I/home/projects/x86-64/boost/1.55.0/cray/7.0.1/gnu/4.9.0/include -I/home/crtrott/lib/superlu/4.3.0/include -I/opt/cray/netcdf-hdf5parallel/4.3.2/gnu/49/include -I/opt/cray/hdf5-parallel/1.8.13/gnu/49/include -I/home/crtrott/lib/zlib/1.2.8/gnu/4.9.0/include -I/home/crtrott/Nalu/include -o CMakeFiles/nalu.dir/src/HeatCondMassBackwardEulerNodeSuppAlg.C.o -c /home/crtrott/Nalu/src/HeatCondMassBackwardEulerNodeSuppAlg.C In file included from /home/crtrott/Nalu/src/HeatCondMassBackwardEulerNodeSuppAlg.C:12:0: /home/crtrott/Nalu/include/Realm.h:16:27: fatal error: yaml-cpp/yaml.h: No such file or directory #include <yaml-cpp/yaml.h>

This is the Nalu config:

nalu_build_dir=/home/crtrott/NaluBuild/SerialClassic/
trilinos_install_dir=/home/crtrott/TrilinosBuild/SerialClassic-Nalu/install
yaml_install_dir=/home/crtrott/lib/yaml/0.3.0/gnu/4.9.0

EXTRA_ARGS=$@

# Cleanup old cache before we configure
# Note:  This does not remove files produced by make.  Use "make clean" for this.
find . -name "CMakeFiles" -exec rm -rf {} \;
rm -f CMakeCache.txt

cmake \
  -DTrilinos_DIR:PATH=$trilinos_install_dir \
  -DYAML_DIR:PATH=$yaml_install_dir \
  -DENABLE_INSTALL:BOOL=OFF \
$EXTRA_ARGS \
/home/crtrott/Nalu

Activities:

1. Documentation of the ABL model implementation in NaluDoc.
2. Implementation of appropriate wall boundary conditions for energy and momentum transport.
3. Prototype STK_transfer of inflow boundary condition from the ABL to a subsequent simulation in which “velocity_bc” is interpolated in space and time for use in the inflow bc.
4. Testing, at available scales, ABL with solver and general algorithmic costs.

Activities:

1. Algorithm infrastructure modification to operate on a homogeneous topology set; step 1 is to elevate std::vector resize operations to the algorithm constructor.
2. Transition of matrix assemble algorithms to use Kokkos-based matrix apply_coefficient().
3. Usage of Kokkos views in solver-based algorithms.

Activities:

1. Evaluate current solver technologies for high aspect ratio meshes.
2. Develop improved coarsening and preconditioning techniques.
3. Evaluate possible algorithmic discretization approach improvements.
4. Explore approximate LHS contribution’s effect on linear and nonlinear solver convergence.

Nodal fields such as heat flux, heat transfer coefficient, etc., appear to be missing the periodic sum/sync within the AssembleWallHeatTransferAlgorithmDriver class.

Activities:

1. Implement proper infrastructure for generalized interfaces.
2. Implement master element methods for low- and higher-order search methods to determine proper face:element DG pairs.
3. Test mixed element approach for Hex8:Tet4 and Hex8:Hex27 topologies.

Within the code, integrate control volume face area vectors to check that the integral is zero for each volume.

edge32D, Quad93D (surface) quad92D and hex27 require isInElement

Quad93D is the most important one for the current project set.

Activities:

1. Establish static condensation approaches with local, fast solves.
2. Deploy preconditioner coarsening strategies which may require only the P=1 system to be stored.
3. Evaluate matrix-free methods for both advection/diffusion and possible PPE.

Currently Nalu explicitly uses Kokkos::SerialNode in some places. It would be better to use:
KokkosClassic::DefaultNode::DefaultNodeType which would allow Nalu to pick up the Node type from the Trilinos configuration. THis is for example important when trying to build with OpenMP.
It would be even better to not specifiy Node at all but simply let Tpetra use its defaults.
I.e.
Tpetra::Export< LocalOrdinal, GlobalOrdinal>
instead of
Tpetra::Export< LocalOrdinal, GlobalOrdinal, Node >

Periodic initialization is performed before current coordinates are populated. As such, the periodic translation fails since current coordinates is zero.

Simple fix is to populate current_coordinates before periodic initialization..

Edge, Quad2D and Quad3D all are setup to work with CHT and the NC alg. However, if we have Tets or wedges, we need some extra methods on the Tri3D topo, e.g., isInElement, interpolatePoint, face_grad_op, etc.

Should we pass in the YAML node to the constructor like the other PP approaches? This avoids a later Realm::load() with logic check on !NULL.

Some post processing utilities encapsulate field registration and loading by accepting a YAML node in the constructor. These classes simply need to be called in the "initialize_post_processing" Realm method via the "setup" call.

Other post processing requires a load to be called (SolutionNorm). The next post processing added should try to consolidate these approaches such that a vector of postProcessing "algorithms" are simply iterated.

In march the timers were changed to stk::cpu_time from stk::wall_time. That means that now the actual wall time is multiplied by the number of threads the process is using (or rather the number of cycles all threads together consume). While this can be a useful measure to get the total consumed hardware resources, I don't think that is what was intended. In particular you would need accumulate MPI time as well instead of providing min, ave and max. Otherwise in a perfect threaded code the time reported by NALU goes up when you use more threads and less MPI ranks, even though the wall clock time is the same.

There are a few instances of Teuchos:: missing in front of "Time" and "TimeMonitor" in LinearSolver.C. This did not show up when using KokkosClassic but does when using the new Kokkos backend for Tpetra. My guess is that somehwere Teuchos::Time[Monitor] gets imported or marked with "using" and that file is not included when building with the new Kokkos backend.

Lines: 157,270,272

Irradiation, and boundary area, need periodic assembly, e.g., something like the following...

// add periodic assembly prior to normalization
if ( realm_.hasPeriodic_) {
const unsigned scalarSize = 1;
const bool bypassFieldCheck = false; // nodal fields are only defined at periodic nodes
realm_.periodic_field_update(assembledWallArea_, scalarSize, bypassFieldCheck);
realm_.periodic_field_update(irradiation_, scalarSize, bypassFieldCheck);
}

Make sure that the unique element for the A/L point is correctly stored and that we remove past candidates. The following code looks off:

  // save off best element and its isoparametric coordinates for this point
  if ( nearestDistance < actuatorLinePointInfo->bestX_ ) {
    actuatorLinePointInfo->bestX_ = nearestDistance;
    actuatorLinePointInfo->isoParCoords_ = isoParCoords;
    if ( stk::mesh::Entity() == actuatorLinePointInfo->bestElem_) {
      actuatorLinePointInfo->bestElem_ = elem;
    }
    else {
      // swap the current best element
      actuatorLinePointInfo->elementVec_.push_back(actuatorLinePointInfo->bestElem_);
      actuatorLinePointInfo->bestElem_ = elem;
    }
  }
  else {
    // regular element
    actuatorLinePointInfo->elementVec_.push_back(elem);
  }

There is no erase, etc.

The IO realm is reporting a candidate time of -1

Currently, the element-based consistent mass matrix form requires second order time integration.

Therefore, modify code to support first-order (Backward Euler) consistent mass matrix.

Since all of the BDF2 element-based methods are implemented already, it may be easy to do the following to avoid a new class:

scalarQNm1_ = realm_.number_of_states() == 2
? &(scalarQ->field_of_state(stk::mesh::StateN))
: &(scalarQ->field_of_state(stk::mesh::StateNM1));

In the above, gamma3_ would be zero so the state of Nm1 is never used... This comes at the expense of more interpolations that are not formally required.

Work towards sub grid model for blades moving in a background Eulerian mesh

Multi-physics runs are confusing without the precise Realm name; add it at the advance_time_step() method.

Technically, the projected nodal gradient and bcs should be using the known value. This is currently only coded for pressure. However, other dogs such as velocity, tke, sdr, enthalpy, and species are using the current dof value.

For species, it can be more complex as we need a field for the current open bc species value.

This is probably not very serious of an issue, however, should be fixed.

We would like to support the Robin-style BC prototyped in the Sierra code base. This allows for an additional penalty term in addition to the full flux.

Add all_blocks option to target_name field for initial_conditions and material_properties

Harden the interface to stk_transfer by field checking, part/field registration checking, tolerance, etc.

Intel-based provides the following warnings:

/projects/netpub/nalu/Nalu/include/LinearSystem.h(85): warning #858: type qualifier on return type is meaningless
const unsigned numDof() const { return numDof_; }
^

/projects/netpub/nalu/Nalu/include/user_functions/WindEnergyTaylorVortexAuxFunction.h(18): warning #654: overloaded virtual function "sierra::nalu::AuxFunction::do_evaluate" is only partially overridden in class "sierra::nalu::WindEnergyTaylorVortexAuxFunction"
class WindEnergyTaylorVortexAuxFunction : public AuxFunction

All the code base to generalize omega specifications with proper centroid coordinates. Currently, wind energy applications require flow in the z-direction with assumption that the centroid is zero.

Allow for a general centroid specification (user defined) with arbitrary rotation.

If the user sets CMAKE_CXX_FLAGS when configuring Nalu, make fails. This seems to be because CMake inserts a ';' character into CMAKE_CXX_FLAGS between the user-given flags and those inherited from Trilinos:

> cmake --version
cmake version 3.1.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).
> cmake -DCMAKE_CXX_FLAGS="-Wno-deprecated-declarations" ...
[...]
-- CMAKE_CXX_FLAGS     = -Wno-deprecated-declarations;   -DSkip_f2c_Undefs -DHAVE_MPI_CXX -DMPICH_IGNORE_CXX_SEEK -fpic  -fpic  -O2 -mtune=generic -std=c++11 -fopenmp -O3

This patch seems to fix this issue:

--- CMakeLists.txt.orig 2016-07-05 15:58:25.000000000 -0500
+++ CMakeLists.txt      2016-07-05 15:53:12.000000000 -0500
@@ -85,8 +85,8 @@

 PROJECT(Nalu)

-SET(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} ${Trilinos_CXX_COMPILER_FLAGS})
-SET(CMAKE_Fortran_FLAGS ${CMAKE_Fortran_FLAGS} ${Trilinos_Fortran_COMPILER_FLAGS})
+SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${Trilinos_CXX_COMPILER_FLAGS}")
+SET(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} ${Trilinos_Fortran_COMPILER_FLAGS}")
 MESSAGE("-- CMAKE_CXX_FLAGS     = ${CMAKE_CXX_FLAGS}")
 MESSAGE("-- CMAKE_Fortran_FLAGS = ${CMAKE_Fortran_FLAGS}")

When running a 2D simulation with the edge-based scheme: a uniform inflow bc (left) specified pressure open (right) and symmetry (top/bottom), a velocity profile sets up at the upper and lower corner location.

At small time steps, the sensitivity is not noted.

Some preliminary scoping seems to indicate that when sensitivities at the open momentum bc are removed, the flow looks fine..

In all cases, we ghost based on the coarse search and then use this list as the set of ghosted objects. However, after the coarse search, there is a fine search which can rule out former candidates. For memory's sake, we should try and prune this custom ghosting list.

Allow the code to interpolate (via linear interpolation) from the simulation mesh to a user defined mesh.

Allow for multiple periodic bcs with surfaces touching each other, e.g., periodic flow in a box.

If we populate an IC from an InitXfer, it would be nice for the code to start at the time that loss found.