If one of the simulations segfaults while there is still a correct strategy.out file, the python script will check this file, and thus the correctness test might return that the test passed even though a simulation may have segfaulted.

CircleCI 1.0 will be deprecated on August 31, 2018.

When running the partitioner without any output node, the partitioner produces a malformatted input_parallelized.15 file. Presumably, this is an error in the input_parameters::WriteTo function.

Tiny fixes for #34

I would preferably use doxygen.

Right now without using the non_default_constructible serialization, the pointers to interfaces need to be non-const. Would be a nice feature to add for code safety, and presumably performance (?)...

To optimally balance the load, we can measure the load internally in HPX, and use those results to balance the load.

Currently, build-jemalloc.sh will try to remove the build_directory from the wrong place.

• Unit tests should occur at the same time as parallel correctness tests
• Manufactured solution timestep should be increased to make the execution time on par with the parallel weirs test

Currently problem/SWE/kernels_preprocessor/swe_pre_create_bound.hpp and problem/SWE/kernels_preprocessor/swe_pre_create_inface.hpp are erasing the iterators of raw_boundaries that they are looping over. This invalidates the current iterator and causes an invalid read in the system.

Needs to be titled section 4.1 Manufactured Solution.

Add fields to input start and end dates in <year/month/day/time>.

What should we do to ensure consistent time zones are used?

Right now the serialization buffer writes the entire RK-stages into the buffer. Technically, we only need input looking things to be serialized.

We should move the following

namespace Global {
	static constexpr double g = 9.81;
	static constexpr double Cf = 0.001;
}

enum BoundaryConditions : unsigned char {
	land = 0,
	tidal = 1,
	internal = INTERNAL
};

out of swe_definitions. Or better yet give the problem class it's own header file. It seems wrong that I have to link to SWE::Data to run the LLLF flux test

Will go a long way to help with #40

We need a functional test for weirs to ensure parallel correctness. Test should probably mimic something along the test in scripts/correctness/test_parallel_correctness.sh

find a way to emplace_back a sim unit instead of using unique_ptr

Is there a reason for this function?

template <typename RawBoundaryType>
static void create_distributed_boundaries_kernel(ProblemMeshType&,
                                                 std::tuple<>&,
                                                 std::map<uint, std::map<uint, RawBoundaryType>>&);

It seems like this function never gets called, only the version that also calls a communicator argument seems to get called.

Currently, building on stampede2 requires copying a cmake toolchain file. This should set correctly in the cmake script.

I want to introduce the load balancing frame work similar to how it was introduce in the LBL load balancing paper.

• Need to build a polymorphic base class for Tile-, Locality- and World-models
• The simplest load balancer should move tiles around randomly, and be used in #59
• We should implement a semi-static load balancer.

Before we try to do any sort of optimization, we need to have performance benchmarks set up. This way we can quantify the benefit of any given optimization.

A small test needs to be added that checks that all the necessary nodes are defined and helpful error exceptions get thrown when they aren't.

If I call resize when this->state.size() is zero, the unsigned arithmetic causes the value to flip to unsigned real max.

But resize also doesn't recreate the number of states passed as the argument. i.e. it creates nstates+1 stages. I think this would be more clear to just pass nstates+1 as the arguement, rather than letting the function secretly add an extra stage.

Would be nice to run a python script that checks if the code converges nightly.

The L2 errors for the serial and parallel implementations are not the same. Observed on the manufactured solution.

Taking the L2-Norm of the solution at the final time for the weir test case introduced in #78 , we find that L2 errors are not agreeing. The current values are

L2 Errors for each build type:
L2 error for serial: 2278.232235133
L2 error for hpx: 2280.0290280987
L2 error for ompi: 2280.0290280987

which is a relatively significant deviation in value.

At the moment, if an output/ directory doesn't exist based on where you're running the simulation, no output will get written.

It might be better to include an option in the input file to specify the output location.

This is crucial for both good static balancing as well as good dynamic rebalancing.

HPXSimulationUnit should only allow migration between timesteps--- not between stages. This allows to keep remote action invocation overhead smaller and allows us to keep smaller buffers.

See discussion channel on slack for ideas.

Based on previous performance results, a semi-static load balancer seems to be the first thing to try to help improve parallel performance.

The Circle CI build process is broken. It seems that there is some confusion regarding which C++ compiler to use. Building should be revamped explicitly specifying which compiler to use for all dependencies.

It would be nice to be able to ship a container with everything inside to allow people to play around with the code more easily.

There should be a way to integrate circleci and cmake to launch a nightly convergence test. This could also be used for assessing how much slower the code is getting. Here is what I think would be a good example http://www.libgeodecomp.org/developers.html#perf_apps

I'm unsure if this a hold-over from the dgswem code. However, not all valid ADCIRC meshes have generic boundaries. Thus we need to check the input file stream to make sure that reading the generic boundaries is valid.

We need to add a load balancing test similar to scripts/correctness/test_parallel_correctness.sh to check that dynamically load balanced implementations are giving identical results to parallel implementations.

The load balancer configuration should get set through the input.

We need to add a method to check that the adcirc mesh is well formed.
This should include:

• no missing edges
• all elements have positive jacobian(?)

Not sure if this is everything, but it seems like a start

Potential parts of SWE::Data are getting overwritten when the Geometry::Element gets loaded and Geometry::Element::SetMaster() gets called. Might not need to be fixed if we don't end up send that part through the wire anyway.

Partitioner needs to be able to take care of mesh files with weir boundary conditions.
If possible: can we keep neighbor element IDs for distributed boundaries instead of assigning them as DEFAULT_ID after partitioning.

On stampede2 after partitioning .15 file in start/end time has ** in minutes field.

communicator argument in preprocessor functions should be const

Right now the inputs in swe yaml-node the input file are ignored. I think there are probably a few scenarios. Right now there are only global variables, which we could replicate by just making the swe_inputs class a static member of the problem type. This is probably closest to the spirit of the current implementation

Alternatively, we could make it a proper member variable of the problem class, but this I think might put more strain on the API.

These are features that need to be built into HPXSimulationUnit::load to ensure that the simulation can continue after the HPXSimulationUnit has been migrated to another directory

• writer log file needs to be reopened
• Interfaces need to be reconnected
• Distributed buffers need to get reset by calling Problem::create_distributed_boundaries_kernel
• Need to add serialization for the spherical projection

when constructing meta files from input file code does not throw in case an input file is missing.

Just wrote out a 4.4 Gig output file