Comments (4)
That's because every process you spawn tries to download the asset at the same time, and they override each others progress on disk. Run the setup once without MPI (to pre-download the assets) to work around this.
This could be circumvented by using some locking mechanism in the future.
from veros.
Thanks, that solved my problem! But another one just popped up.
It seems interpolation routine doesn't work for bathymetry and I don't know why it cannot interpolate
global data in MPI's subdomains. It must be straitforward given that global data are available on each MPI process.
Veros log:
Setting up everything
Traceback (most recent call last):
File "/groups/ocean/software/python/gcc/3.6.7/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/groups/ocean/software/python/gcc/3.6.7/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/groups/ocean/software/mpi4py_mvapich231/gcc/3.0.1/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
main()
File "/groups/ocean/software/mpi4py_mvapich231/gcc/3.0.1/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
run_command_line(args)
File "/groups/ocean/software/mpi4py_mvapich231/gcc/3.0.1/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
run_path(sys.argv[0], run_name='__main__')
File "/groups/ocean/software/python/gcc/3.6.7/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/groups/ocean/software/python/gcc/3.6.7/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/groups/ocean/software/python/gcc/3.6.7/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "global_flexible.py", line 389, in <module>
run()
File "/groups/ocean/software/python/gcc/3.6.7/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/groups/ocean/software/python/gcc/3.6.7/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/groups/ocean/software/python/gcc/3.6.7/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/groups/ocean/software/python/gcc/3.6.7/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/lustre/hpc/ocean/software/veros/repo23052019/veros/tools/cli.py", line 89, in wrapped
run(*args, **kwargs)
File "global_flexible.py", line 384, in run
simulation.setup()
File "/lustre/hpc/ocean/software/veros/repo23052019/veros/veros.py", line 196, in setup
self.set_topography(vs)
File "/lustre/hpc/ocean/software/veros/repo23052019/veros/decorators.py", line 119, in veros_method_wrapper
res = function(*newargs, **kwargs)
File "global_flexible.py", line 180, in set_topography
(topo_x_shifted, topo_y), topo_z_shifted, coords, kind='nearest', fill=False
ValueError: could not broadcast input array from shape (360,160) into shape (90,40)
srun: Complete job step 13539345.0 received
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 13539345.0 ON node172 CANCELLED AT 2019-05-31T14:34:04 ***
srun: Complete job step 13539345.0 received
srun: Received task exit notification for 16 tasks (status=0x0009).
srun: error: node172: tasks 0-15: Killed
srun: Terminating job step 13539345.0
from veros.
Confirmed and fixed on master!
from veros.
Thanks, it works like a charm!
from veros.
Related Issues (20)
- Average diagnostics sometimes triggers twice in succession
- Domain and bathymetry HOT 2
- Tests fail hard when disabling TKE
- Compute initial streamfunction from initial velocity HOT 5
- pressure solver HOT 2
- Missing net short-wave radiation forcing in global_1deg setup? HOT 3
- Non-hydrostatic solver HOT 5
- Linear solver issue when parallelising VEROS HOT 1
- Document overturning variables HOT 2
- Bathymetry in Veros HOT 1
- Changing grid resolution and a closed ITF in global_4deg setup HOT 1
- How to run Veros via MPI on a cluster with slurm? HOT 1
- How to run veros with multi-GPU HOT 5
- More useful tools to simplify setup definitions
- Enforce code formatting standard
- High quality simu GIF link broken HOT 1
- Better calendar handling HOT 1
- Ensure reproducibility
- Work on flexible setup
- Ensure CF compliance
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from veros.