Comments (1)
Maybe to clarify, what this issue is about:
Currently, when creating a QGDataset
, all the QGField
objects managed by the Dataset are immediately instanciated and kept for later use. This generally means there are (at least) two copies of any input/output data in memory when using the xarray interface: one in the QGField instances and one in the xarray datasets.
I would like to keep the data in xarray datasets/dataarrays as much and exclusively as possible and only temporarily make copies for the QGFields when computing. More data could then be handled with the same amout of memory I think. An issue here is that the three processing steps (interpolation, reference state, LWA/fluxes) depend on each other and use the QGField to store the intermediate results. So the objects must be kept around at least for some time in most applications.
Integrating further with dask would allow for distributing the computations onto multiple processors, enabling parallelism and allowing for even larger datasets to be processed. For this to work, the computation steps must be converted into a dask graph. I think this could be done either by using dask.delayed
or maybe even better with xarray.apply_ufunc
. I'm not sure though how well the OO interface can be wrapped with these, I think they are more geared towards a functional interface.
from hn2016_falwa.
Related Issues (20)
- Automate deployment onto PyPI via GitHub workflow
- Issue in reference state calculations for model data HOT 1
- A potential bug in the 'u_baro' calculation of FALWA v.1.2.0 HOT 2
- Add diabatic heating in the LWA budget HOT 3
- Issues with deployment onto conda channel HOT 2
- Issues deploying package to PyPI channel HOT 1
- Diagnostics (+ interpretations) to incorporate into MDTF
- Research into Scikit-build solution for F2PY
- Incorporate listed diagnostic to MDTF and submit pull request
- Bug in GitHub action
- IMPORTANT bugfix release 0.7.2 + corrigendum HOT 3
- QGDataset does not support data input with even number of latitude gridpoints HOT 7
- Use Cartopy instead of Basemap (deprecated) in all the visualization examples
- Provide the option to compute QGPV from relative vorticity in QGFIeld
- Compute vertically averaged local non-conservative force contribution
- Rename the python package to falwa for release v1.0.0 HOT 1
- IMPORTANT: Code error in calculating advective flux in bug fix release 0.7.2
- Vertical coordinate conversions HOT 3
- numpy.disutils will deprecate for python 3.12
- Bug in processing latitude grid that does not include equator
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hn2016_falwa.