Jobs assignment in server mode about tangos HOT 6 CLOSED

Could you post the full log? Might have more hints in it than this snippet.

Tests are in test_pynbody_server.py but probably don't capture this issue whatever it is

from tangos.

Here is the log for a mpirun with two processes

[ 1] 2018-03-02 14:06:51,497 : Processing <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00001' z=99.00 t=0.02 Gyr>
[ 1] 2018-03-02 14:06:51,614 : Gathering existing properties for all halos in timestep <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00001' z=99.00 t=0.02 Gyr>
[ 1] 2018-03-02 14:06:51,632 : Gathering existing properties...
[ 1] 2018-03-02 14:06:51,633 : Successfully gathered existing properties; calculating halo properties now...
[ 1] 2018-03-02 14:06:51,633 : 0 halos to consider; 1 property calculations for each of them
/Users/mrey/Documents/tangos/tangos/parallel_tasks/backends/mpi4py.py:54: UserWarning: Number of processors requested (2) will be ignored as this is an MPI run that has already selected 2 processors
warnings.warn("Number of processors requested (%d) will be ignored as this is an MPI run that has already selected %d processors"%(num_procs,size()))
[ 0] 2018-03-02 14:06:51,633 : Finished jobs; notify node 1
[ 1] 2018-03-02 14:06:51,634 : Done with <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00001' z=99.00 t=0.02 Gyr>
[ 1] 2018-03-02 14:06:51,665 : Succeeded: 0 property calculations
[ 1] 2018-03-02 14:06:51,666 : Errored: 0 property calculations
[ 1] 2018-03-02 14:06:51,666 : Errored during load: 0 property calculations
[ 1] 2018-03-02 14:06:51,666 : Already exists: 0 property calculations
[ 1] 2018-03-02 14:06:51,666 : Missing pre-requisite: 0 property calculations
[ 1] 2018-03-02 14:06:51,666 : Processing <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00002' z=47.77 t=0.05 Gyr>
[ 1] 2018-03-02 14:06:51,667 : Gathering existing properties for all halos in timestep <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00002' z=47.77 t=0.05 Gyr>
[ 1] 2018-03-02 14:06:51,677 : Gathering existing properties...
[ 1] 2018-03-02 14:06:51,677 : Successfully gathered existing properties; calculating halo properties now...
[ 1] 2018-03-02 14:06:51,678 : 0 halos to consider; 1 property calculations for each of them
Number of jobs in object.process:0.000000
[ 0] 2018-03-02 14:06:51,678 : Finished jobs; notify node 1
[ 1] 2018-03-02 14:06:51,678 : Done with <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00002' z=47.77 t=0.05 Gyr>
[ 1] 2018-03-02 14:06:51,706 : Succeeded: 0 property calculations
[ 1] 2018-03-02 14:06:51,706 : Errored: 0 property calculations
[ 1] 2018-03-02 14:06:51,706 : Errored during load: 0 property calculations
[ 1] 2018-03-02 14:06:51,706 : Already exists: 0 property calculations
[ 1] 2018-03-02 14:06:51,706 : Missing pre-requisite: 0 property calculations
[ 1] 2018-03-02 14:06:51,706 : Processing <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00003' z=23.79 t=0.14 Gyr>
[ 1] 2018-03-02 14:06:51,708 : Gathering existing properties for all halos in timestep <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00003' z=23.79 t=0.14 Gyr>
[ 1] 2018-03-02 14:06:51,718 : Gathering existing properties...
[ 1] 2018-03-02 14:06:51,719 : Successfully gathered existing properties; calculating halo properties now...
[ 1] 2018-03-02 14:06:51,719 : 0 halos to consider; 1 property calculations for each of them
Number of jobs in object.process:0.000000
[ 0] 2018-03-02 14:06:51,719 : Finished jobs; notify node 1
[ 1] 2018-03-02 14:06:51,719 : Done with <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00003' z=23.79 t=0.14 Gyr>
[ 1] 2018-03-02 14:06:51,747 : Succeeded: 0 property calculations
[ 1] 2018-03-02 14:06:51,747 : Errored: 0 property calculations
[ 1] 2018-03-02 14:06:51,747 : Errored during load: 0 property calculations
[ 1] 2018-03-02 14:06:51,747 : Already exists: 0 property calculations
[ 1] 2018-03-02 14:06:51,747 : Missing pre-requisite: 0 property calculations
[ 1] 2018-03-02 14:06:51,747 : Processing <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00004' z=15.37 t=0.26 Gyr>
[ 1] 2018-03-02 14:06:51,748 : Gathering existing properties for all halos in timestep <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00004' z=15.37 t=0.26 Gyr>
[ 1] 2018-03-02 14:06:51,758 : Gathering existing properties...
[ 1] 2018-03-02 14:06:51,759 : Successfully gathered existing properties; calculating halo properties now...
[ 1] 2018-03-02 14:06:51,759 : 0 halos to consider; 1 property calculations for each of them
Number of jobs in object.process:0.000000
[ 0] 2018-03-02 14:06:51,759 : Finished jobs; notify node 1
[ 1] 2018-03-02 14:06:51,759 : Done with <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00004' z=15.37 t=0.26 Gyr>
[ 1] 2018-03-02 14:06:51,789 : Succeeded: 0 property calculations
[ 1] 2018-03-02 14:06:51,789 : Errored: 0 property calculations
[ 1] 2018-03-02 14:06:51,789 : Errored during load: 0 property calculations
[ 1] 2018-03-02 14:06:51,789 : Already exists: 0 property calculations
[ 1] 2018-03-02 14:06:51,789 : Missing pre-requisite: 0 property calculations
[ 1] 2018-03-02 14:06:51,790 : Processing <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00005' z=11.47 t=0.39 Gyr>
[ 1] 2018-03-02 14:06:51,791 : Gathering existing properties for all halos in timestep <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00005' z=11.47 t=0.39 Gyr>
[ 1] 2018-03-02 14:06:51,800 : Gathering existing properties...
[ 1] 2018-03-02 14:06:51,801 : Successfully gathered existing properties; calculating halo properties now...
[ 1] 2018-03-02 14:06:51,801 : 0 halos to consider; 1 property calculations for each of them
Number of jobs in object.process:0.000000
[ 0] 2018-03-02 14:06:51,801 : Finished jobs; notify node 1
[ 1] 2018-03-02 14:06:51,801 : Done with <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00005' z=11.47 t=0.39 Gyr>
[ 1] 2018-03-02 14:06:51,833 : Succeeded: 0 property calculations
[ 1] 2018-03-02 14:06:51,834 : Errored: 0 property calculations
[ 1] 2018-03-02 14:06:51,834 : Errored during load: 0 property calculations
[ 1] 2018-03-02 14:06:51,834 : Already exists: 0 property calculations
[ 1] 2018-03-02 14:06:51,834 : Missing pre-requisite: 0 property calculations
[ 1] 2018-03-02 14:06:51,834 : Processing <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00006' z=8.97 t=0.55 Gyr>
[ 1] 2018-03-02 14:06:51,835 : Gathering existing properties for all halos in timestep <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00006' z=8.97 t=0.55 Gyr>
[ 1] 2018-03-02 14:06:51,845 : Gathering existing properties...
[ 1] 2018-03-02 14:06:51,845 : Successfully gathered existing properties; calculating halo properties now...
[ 1] 2018-03-02 14:06:51,845 : 0 halos to consider; 1 property calculations for each of them
Number of jobs in object.process:0.000000
[ 0] 2018-03-02 14:06:51,846 : Finished jobs; notify node 1
[ 1] 2018-03-02 14:06:51,846 : Done with <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00006' z=8.97 t=0.55 Gyr>
[ 1] 2018-03-02 14:06:51,874 : Succeeded: 0 property calculations
[ 1] 2018-03-02 14:06:51,874 : Errored: 0 property calculations
[ 1] 2018-03-02 14:06:51,874 : Errored during load: 0 property calculations
[ 1] 2018-03-02 14:06:51,874 : Already exists: 0 property calculations
[ 1] 2018-03-02 14:06:51,875 : Missing pre-requisite: 0 property calculations
[ 1] 2018-03-02 14:06:51,875 : Processing <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00007' z=7.32 t=0.72 Gyr>
[ 1] 2018-03-02 14:06:51,876 : Gathering existing properties for all halos in timestep <TimeStep 'ramses_dmo_L50_N256_ZOOM4_Halo740_variancex0.7/output_00007' z=7.32 t=0.72 Gyr>
[ 1] 2018-03-02 14:06:51,886 : Gathering existing properties...
[ 1] 2018-03-02 14:06:51,896 : Successfully gathered existing properties; calculating halo properties now...
[ 1] 2018-03-02 14:06:51,896 : 7 halos to consider; 1 property calculations for each of them
Number of jobs in object.process:0.000000
Traceback (most recent call last):
File "/Users/mrey/anaconda3/bin/tangos", line 11, in
load_entry_point('tangos', 'console_scripts', 'tangos')()
File "/Users/mrey/Documents/tangos/tangos/scripts/init.py", line 57, in main
args.func(args)
File "/Users/mrey/Documents/tangos/tangos/scripts/init.py", line 14, in run
parallel_tasks.launch(obj.run_calculation_loop, 2, [])
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/init.py", line 54, in launch
backend.launch(_exec_function_or_server, num_procs, [function, args])
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/backends/mpi4py.py", line 55, in launch
function(*args)
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/init.py", line 84, in _exec_function_or_server
_server_thread()
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/init.py", line 111, in _server_thread
obj.process()
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/jobs.py", line 19, in process
self.contents, self.source, num_jobs))
RuntimeError: Number of jobs (7) expected by rank 1 is inconsistent with 0
^Cmpirun: killing job...

The failure occurs at the first timestep for which there is work to be done (output_00007 in this case)

from tangos.

Could you try running with the flag --backwards?

from tangos.

It runs with no problem under --backwards flag.

from tangos.

Should be fixed in branch issue-35. Please confirm.

from tangos.

Yep, it is running fine now on my setup. Thanks for taking the time !

from tangos.