grid.from_netcdf does not efficiently allocate memory for models with many time snapshots about parcels HOT 5 CLOSED

A somewhat related issue which I am coming up against increasingly is that of loading multiple netCDF files that contain the same variable across different time periods.

Many times long time-series of high resolution variable data are split up into separate files, e.g. by month, day etc. When these are on a computing cluster, it's not really appropriate to stitch them together using NCO tools into new (and enormous) netcdfs.

It would be neat if multiple files could be specified to PARCELS during grid creating/additions, and multiple instances of the same variable were stitched together in the time dimension. Not too complicated an addition, but possibly fiddly to do. What's the best approach for this?

from parcels.

Hi @Jacketless, maybe I misunderstand, but isn't the wildcard operator * what you are after? See e.g. the examples/example_globcurrent.py script. The code that handles this is in the from_netcdf classmethod in grid.py (line 106).

Let me know if this fixes your issue with multiple netCDF files, or if you want something else

from parcels.

Apologies for my slow reply, this week has been very busy with PI meetings and prep for the Nouméa visit.

@erikvansebille You're absolutely right, PARCELS does already have this functionality. Sorry about that. I have not yet had a chance to try it out, but it should work fine.
If we can slice only the layers we are interested in from netcdfs that we then load into memory, in most cases we should be able to keep the memory footprint in reasonable territory even for long-ish runs.

from parcels.

Slicing layers is indeed a feature I'd like to add in the near future. See also #75. If you have any thoughts on how to go about, it would be great to hear them there!

from parcels.

Fixed in #155

from parcels.