Comments (6)
NaNs in your data are always a bit tricky. I'm not sure for pyEOFs, but what xMCA does in the case of individual NaNs (e.g. grid points where only a few time steps are missing), is just to remove the entire grid cell before the analysis. This admittedly rude approach works reasonably well when the overall fraction of features with individual NaNs is low. In your case, what's the fraction of grid cells that has individual NaNs?
Alternatively, you have to think about how you can fill these values, either subjectively (e.g. by using some fixed values) or more objectively (e.g. by using Probabilistic PCA). Ultimately, the choice will depend on the data you have. In xeofs we don't automatically treat individual NaNs, because the decision of how to treat NaNs should ultimately be with the analyst.
from xeofs.
Thanks for the response @nicrie
xMCA does help.
But there is not cos_lat weighing option in xmca right?
from xeofs.
Yes, there is a cosine-latitude weighting option in xMCA. You can find more information here. Note, however, that your solution obtained from xMCA will be exactly the same as using xeofs if you remove all grid cells with individual NaNs. This can be done by calling:
da = da.where(da.notnull().all("time"))
assuming that time
is the dimension along which you want to maximize the variance.
from xeofs.
Thanks for the time @nicrie
I tried 'apply_coslat()' from the link as suggested.
pca = xMCA(x.z) #x is my dataarray
pca.apply_coslat()
pca.solve(complexify=False) # True for complex PCA
svals = pca.singular_values() # singular vales = eigenvalues for PCA
expvar = pca.explained_variance() # explained variance
pcs = pca.pcs() # Principal component scores (PCs)
eofs = pca.eofs()
Yet, the results look like those without 'cos_lat'
Is there anything I'm missing?
With xeofs:
\anaconda3\envs\xEOFs\Lib\site-packages\numpy\lib\nanfunctions.py:1879: RuntimeWarning: Degrees of freedom <= 0 for slice.
var = nanvar(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
error
which means, the nan values are not deleted.
Here is a screenshot after applying
da = da.where(da.notnull().all("time"))
My questions:
- In xmca: mca.apply_coslat() works for pca too?
- In xeofs, is there anything I'm missing for not being able to get data without 'NaNs'?
from xeofs.
Why do you think that the warning message implies that NaN values are not deleted? The code
da = da.where(da.notnull().all("time"))
masks out grid cells with individual NaNs. In the preprocessing step of xeofs, these grid cells are removed prior to the SVD (otherwise you would get an error there), and are reinserted afterwards into your results. My guess is that the warning arises due to the computation of standard deviations on only NaN slices. Would you mind sharing a minimal reproducable example?
For your question about xmca: Yes, apply_coslat()
works for both PCA and MCA. However, I need to mention that I've stopped maintaining xmca in favor of xeofs for about a year now. This means I can't provide detailed debugging support for xmca anymore. I'd encourage you to focus on xeofs for your current and future analyses, as it's actively maintained and supported.
from xeofs.
Related Issues (20)
- ModuleNotFound Error datatree HOT 5
- `scores()` don't match `transform()` with dask data HOT 4
- Improving release process
- Keep the documentation in sync with the code HOT 3
- Add dependencies section in the documentation HOT 1
- numpy and pandas dependency question
- numpy and pandas dependencies HOT 3
- Migrating repository to xarray-contrib HOT 2
- why the explained_variance_ratio of CCA so small HOT 4
- Serialization fails with `xarray>=2024.1.0` HOT 2
- Single mode data reconstruction fails with normalized scores HOT 9
- Support for complex input data
- MCA incorrect coords alignment in transform method HOT 3
- Is the reconstruction of unseen data possible using EOF? HOT 5
- Model cannot fit DataArray without coordiantes
- Add option `n_modes="all"` to perform the full decomposition HOT 2
- Trouble creating the eof object HOT 8
- TypeError: __init__() missing 1 required positional argument: 'X' HOT 1
- Questions to the REOF power parameter in the examples HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from xeofs.