Limitation: Different number of tasks change answers about mizuroute HOT 3 OPEN

Hi @ekluzek , Yes, this is expected. A different number of MPI tasks changes orders of basinID (hru dimension) and reachID (seg dimension) in history file.

If basinID and reachID in one history file are sorted to the ones in another history file with different MPI task run, both should be identical. I have a python script to sort the netcdf.

Roughly speaking, the array order associated with hru and seg are determined based on independent river basins defined based on the MPI tasks numbers

from mizuroute.

Ahhh, OK. So right now we could run the python script to show that answers are identical.

In the short term I think it would be good to have that script in the repository so that it would be easy to run the PEM test and then verify that answers are identical with the script. Could you add the script somewhere in the repo?

In the medium to longer term it would be good to have a mode where the sorting was done inside the code, so that you don't need the external sorting script. In the coupler we have a trigger like this that slows things down a little bit, but ensures that answers will be identical on different number of processors. That trigger is already turned on for testing, we just need something similar for mizuRoute. A slow simple solution would be fine to start with. A future advancement could be to speed it up. Since, you wouldn't normally run in this mode it's OK for it to be slower.

How hard would it be to add such a flag that would do the sorting needed to make sure answers are identical?

from mizuroute.

We talked about this and we want to look into using the python script to first just be able to do this checking. And then add some automatic tests that run the script so that we know answers don't change. If we can do that we wouldn't need to get this into the code. @nmizukami did work on this, but it was really slow.

from mizuroute.