amrex-astro / maestroex Goto Github PK

With the 19.02 releases of AMReX, Microphysics, and MAESTROeX, compilation of extern.F90 in MAESTROeX/Exec/SCIENCE/wdconvect using the Intel v19.0.0.117 compilers fails as follows:

extern.F90(153): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [VELPERT_AMPLITUDE]
  namelist /probin/ velpert_amplitude
--------------------^
extern.F90(154): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [VELPERT_RADIUS]
  namelist /probin/ velpert_radius
--------------------^
extern.F90(155): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [VELPERT_SCALE]
  namelist /probin/ velpert_scale
--------------------^
extern.F90(156): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [VELPERT_STEEP]
  namelist /probin/ velpert_steep
--------------------^
extern.F90(157): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [TAG_DENSITY_1]
  namelist /probin/ tag_density_1
--------------------^
extern.F90(158): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [TAG_DENSITY_2]
  namelist /probin/ tag_density_2
--------------------^
extern.F90(159): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [TAG_DENSITY_3]
  namelist /probin/ tag_density_3
--------------------^
extern.F90(160): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [PARTICLE_TEMP_CUTOFF]
  namelist /probin/ particle_temp_cutoff
--------------------^
extern.F90(161): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [PARTICLE_TPERT_THRESHOLD]
  namelist /probin/ particle_tpert_threshold
--------------------^
extern.F90(162): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [USE_EOS_COULOMB]
  namelist /probin/ use_eos_coulomb
--------------------^
extern.F90(163): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [EOS_INPUT_IS_CONSTANT]
  namelist /probin/ eos_input_is_constant
--------------------^
extern.F90(164): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [EOS_TTOL]
  namelist /probin/ eos_ttol
--------------------^
extern.F90(165): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [EOS_DTOL]
  namelist /probin/ eos_dtol
--------------------^
extern.F90(166): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [SMALL_X]
  namelist /probin/ small_x
--------------------^
extern.F90(167): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [USE_TABLES]
  namelist /probin/ use_tables
--------------------^
extern.F90(168): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [USE_C12AG_DEBOER17]
  namelist /probin/ use_c12ag_deboer17
--------------------^
extern.F90(169): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [ODE_MAX_STEPS]
  namelist /probin/ ode_max_steps
--------------------^
extern.F90(170): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [SCALING_METHOD]
  namelist /probin/ scaling_method
--------------------^
extern.F90(171): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [USE_TIMESTEP_ESTIMATOR]
  namelist /probin/ use_timestep_estimator
--------------------^
extern.F90(172): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [ODE_SCALE_FLOOR]
  namelist /probin/ ode_scale_floor
--------------------^
extern.F90(173): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [ODE_METHOD]
  namelist /probin/ ode_method
--------------------^
extern.F90(174): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [SAFETY_FACTOR]
  namelist /probin/ safety_factor
--------------------^
extern.F90(175): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [DO_CONSTANT_VOLUME_BURN]
  namelist /probin/ do_constant_volume_burn
--------------------^
extern.F90(176): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [CALL_EOS_IN_RHS]
  namelist /probin/ call_eos_in_rhs
--------------------^
extern.F90(177): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [DT_CRIT]
  namelist /probin/ dT_crit
--------------------^
extern.F90(178): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [BURNING_MODE]
  namelist /probin/ burning_mode
--------------------^
extern.F90(179): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [BURNING_MODE_FACTOR]
  namelist /probin/ burning_mode_factor
--------------------^
extern.F90(180): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [INTEGRATE_TEMPERATURE]
  namelist /probin/ integrate_temperature
--------------------^
extern.F90(181): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [INTEGRATE_ENERGY]
  namelist /probin/ integrate_energy
--------------------^
extern.F90(182): error #6576: If a namelist-group-name has the PUBLIC attribute, no item in the namelist-group-object-list may have the PRIVATE attribute.   [JACOBIAN]
  namelist /probin/ jacobian
--------------------^
/tmp/ifortO5xWJ0.i90(527): catastrophic error: Too many errors, exiting

This error is new with Intel v19; it does not occur for any release of the v18 compilers.

We need to add a few tests to the regression suite for OpenMP/Tiling.

This would require using the AMReX tensor solve since div(u) != 0.

we should store the contents of the &extern namelist in the job_info file.

Currently, during initialization the plotfiles plt_InitData, plt_after_DivuIter and plt_after_InitProj are output to the directory where the simulation is being run. It would make more sense for these plots to have the prefix maestro.plot_base_name (rather than plt_) so that e.g. they are saved to the same directory as the other plotfiles (if that is not the runtime directory).

Similar to what we do in the checkpoint file, we should write out the base state to the plotfile.

Issue can be reproduced using inputs_3d_C.128.avgbase.dr for uneven base state spacing, and inputs_3d_C.128.avgbase.dx for even base state spacing. Both tests fail after approx. 20 time steps, near the base_cutoff_radius. Backtrace files show that error is due to either a failure to converge during MAC projection, or eos failure to compute enthalpy outside base_cutoff_radius in UpdateScal().

The advective reflux is currently disabled (so do_reflux=F by default and all inputs files). Instead, there is an AverageDownEdges call on the species/enthalpy fluxes. This should give the same result as refluxing (to machine precision), however it is not as efficient since averageDownFaces operates over all components and all faces (instead of flux registers which only update information ON C-F boundaries)

unless there are any objections, we should enable ppm_trace_forces by default

When compiling/running on summit with pgi, an error message appears

 ERROR: problem in the namelist
FORTRAN STOP

You can compile/run with gnu and there's no error. Or you can comment out the exit command and use pgi and it seems to run ok. We suspect this is an issue due to combining the inputs file and probin (namelist) into 1 file.

We can use the new amrex functionality to get rid of the probin file completely, and instead have it just in the inputs file.

we should do some profiling to find "hot spots" for single processor

We might use deltagamma for a model for slow rotation, so we should recover that from old Maestro

In wdconvect, inputs_3d_C.128.2levels generates gibberish after ~50 time steps. Compare this to original MAESTRO/wdconvect inputs_3d_C.128.2levels.fixed, which has the same fixed grid structure and works fine. I notice that during the first nodal projection, when looking at multigrid verbosity, the answers diverge.

There should be a routine like the writeSmallPlotFile routine in Castro that only outputs a small subset of the variables

The default value of burning_cutoff_density is 3.e6 (g/cc). This reflects Maestro's origin as a code for modeling burning in WDs, but for other applications users may be unaware of this cutoff. We should have this off by default (set it to 0.0 as in Castro). This point of this parameter is optimization, just to skip the burn in regions where it shouldn't matter.

One thing we talked about in the past was whether we should evolve a momentum equation in Maestro instead of a velocity equation. Maybe this is something to think about as we rethink the algorithm.

See https://ccse.lbl.gov/pub/RegressionTesting/MAESTROeX/

After merging "tiling" into "development", if you build with DEBUG=TRUE and run, these tests crash. If you build with DEBUG=FALSE they run to completion, so we may not have noticed this before.

We should have a routine in MaestroPlot.cpp that outputs the gravitational acceleration

We need reacting_bubble_2d/3d working with AMR. Must of the support is missing, such as regridding the base state and modifying tagging so it spans the domain.

Add a regression test on battra for double_bubble, incomp_shear_jet, rt, test_convect in 2d (and 3d if available)

It might make some sciencing easier if we had a proper 2-d axisymmetric solver.

It is useful for scaling studies to have the tile size in the job_info file along with the number of MPI tasks and OMP threads.

We should port the ppm_trace_forces functionality from Maestro and turn it on be default. This seems to help some problems a lot, like test_stability

It would be useful for the plotfiles to include the Mach number

Look at IAMR development branch commit a3f9f6d096992dd63cf4733632b8afafba5083e2. There is a bug in the 3d corner coupling terms for conservative differencing. This manifested itself in IAMR test cases as a constant density field not remaining constant under divergence-free flow. This needs to be fixed in MAESTRO and MAESTROeX.

The original MAESTRO had unit tests under the Exec/ directory. These had their own drivers. We should port them to MAESTROeX

Augment each MFIter loop to include tilling. Castro and IAMR have this implemented already as a guide.

Currently, for lots of the hydro routines (ppm, mkutrans, velpred, slope, make_edge_scal) there are multiple versions for 1, 2 and 3 dimensions. It is likely that many of these can be combined into single, dimensionally agnostic routines, reducing the amount of repeated code and making them easier to maintain.

This crashes with a backtrace in screening.

It might be useful to add a do_smallscale feature and associated test problems that is analogous to the feature in the old MAESTRO code.

(Namely, setting rho_0 = rhoh_0 = 0, not evolving the background, and using div U = S constraint.)

Since it seems like an approachable first issue, I'll attempt this and then issue a pull request once it is ready.

We have no examples with inflow / user-specified boundary conditions. We should port the mach_jet problem over to demonstrate / implement this functionality.

Support for 1-d problems was added to original Maestro to allow us to model smallscale flames. In the original Bell et al. 2004 low Mach algorithm we used for flames, we never had 1-d -- we just ran flames in long, narrow 2-d domains if we were just interested in the laminar properties.

1-d doesn't really make sense for atmospheric flows.

So I would propose that we remove 1-d support from MAESTROeX. We can still do smallscale flames in 2-d if needed (and this is not any user's science priority at the moment).

I've been trying to run a two-level 3D calc with planar geometry but currently tells me that init_multilevel has not been implemented for this yet. Is there a work around for this (e.g. single chunk at finest level)?

The docs for managing jobs (from the original Maestro) point to the old Titan job scripts path and don't clarify that the restart flag is now maestro.restart_file=[checkpoint file].

We should update this as well as add MAESTROeX specific job scripts.

The HPSS archiving script should also use the new C++ ftime in AMReX.

I'm testing these changes and will update the docs when I confirm they work.

I just realised that MAESTROeX has no license currently. Can we just copy from the license file in Castro?

Currently wdconvect with inputs_3d_regression fails during the first MAC projection (too many multigrid V-cycles). (pure MPI)

If I back out amrex to 953c10b3c07a914973aea6e8c6bad69b9cbcd88d the wdconvect example runs correctly. No further time to debug it at the moment.

The master branch still seems to have a dependency on some things from F_BaseLib, so when I try compiling one of the TEST_PROBLEMS, I get this error. It seems fine on the development branch

make: *** No rule to make target '/home/jt/data/amrex/Src/F_BaseLib/FParallelMG.mak'. Stop.

See AMReX-Astro/Castro#651.

This would make the installation process easier for new users.

The plot variables in MAESTROeX should match those in MAESTRO as much as possible. Quite a few variables are missing. There are also "plot_X" logicals/integers in the _parameters that control whether things get plotted (and are not implemented).

We should follow the lead of Castro and offload the reactions to GPUs

In Maestro_F.H there are several instances of NULL arguments (for the mask) to fortran. This does not work with all fortran compilers according to Weiqun; I'm not sure of a good solution right now.

Note that this also breaks compilation when you do "make typecheck" to check for inconsistent argument lists between Maestro_F.H and the corresponding F90 subroutines.

Hello,

Im looking to develop some capability to model the dynamics of hot bubble motion in a stratified atmosphere, where the scale height is an important consideration, and the densities/temperatures (though not the pressures) are very different to the background. I suspect that MAESTRO has the capability to do so and so i intend to modify the code accordingly. My question is should i be looking at using the Fortran MAESTRO or the c++/Fortran MAESTROeX to do this? I note that not all tests work properly with MAESTROeX which used to work with MAESTRO. Are both codes in active development or is MAESTRO likely to be deprecated? Many thanks for your time.

Are there any thoughts on moving the current version of MaestroeX to take advantage of the Amr/AmrLevel classes (like IAMR) rather than the current implementation using AmrCore? Is there any obvious advantage to doing this?

The job_info file is missing some information. In particular:

• CPU time used
• inputs file used
• plotfile output time and dir
• EOS / network, etc. used
• boundary conditions
• species information
• for input parameters, whether we overrode the default
• probin / extern parameters

this can pretty much all be directly copied from Castro

The docs have not been updated much to reflect the changes in MAESTROeX vs MAESTRO, so there are a few sections where they should be updated

We should do some scaling with MPI / MPI+OMP

Instead of manually adding problem-specific parametrers in a probdata.f90, and then setting the initial value in probdata_init, we should consider adding the parameters in an _parameters file, that is automatically captured by the build system. This makes it easier to maintain and document.

It would be useful to have a chk_deltat parameter (similar to plot_deltat) so checkpoints could be output at fixed simulation time intervals

This is the script that writes the generates code in the header files and routine that describes the runtime parameters in _cpp_parameters. Currently, it is having issues with small_plot_vars, where the default value is set to a list of space-separated values. E.g. rather than

std::string Maestro::small_plot_vars = "rho p0 magvel";

it is generating

#ifdef magvel"
std::string Maestro::small_plot_vars = "rho;
#endif