sbailey / empca Goto Github PK

Hi, Can you please tell me how can I use for missing data imputation for this program ? It is written as

Missing data is simply the limit of weight=0.

But I am not getting where to set weight = 0. When I did
m0 = empca(noisy_data, weights = 0, niter=20)
it gives error as

File "empca.py", line 290, in empca
assert data.shape == weights.shape

Can you please help me ? I want to use your code for imputation problem. Thank you.

Hi,

I just started using empca and it works great. However in a few instances, I have the following Warning which seems to make the PCA stop:
empca.py:129: RuntimeWarning: invalid value encountered in double_scalars
self.eigvec[k, j] = x.dot(cw) / c.dot(cw)

Any advice on how to get rid of this message?

Thanks in advance.

I am trying to run with a large data set, ~200,000 eBOSS spectra, and stumbled upon an issue with memory.
What would be the best strategy to deal with that?
Is there an option float32, or should I split the spectra I am looking into computing
in half according to lambdaRF and tape as best as I can after?

INFO: Starting EMPCA
       iter        R2             rchi2
Traceback (most recent call last):
  File "<HOME>/redvsblue/bin//redvsblue_compute_PCA.py", line 205, in <module>
    model = empca.empca(pcaflux, weights=pcaivar, niter=args.niter, nvec=args.nvec)
  File "<HOME>/Programs/sbailey/empca/empca.py", line 307, in empca
    model.solve_eigenvectors(smooth=smooth)
  File "<HOME>/Programs/sbailey/empca/empca.py", line 142, in solve_eigenvectors
    data -= np.outer(self.coeff[:,k], self.eigvec[k])    
  File "<HOME>/.local/lib/python3.6/site-packages/numpy/core/numeric.py", line 1203, in outer
    return multiply(a.ravel()[:, newaxis], b.ravel()[newaxis, :], out)
MemoryError

I get the following warning when using weights, I bet it can simply be corrected:

<me>/Programs/sbailey/empca/empca.py:256: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  x = np.linalg.lstsq(A, b)[0]

Here is a minimal example of code:

import scipy as sp
import empca

flux = sp.loadtxt('exemple_flux.txt')
weights = sp.loadtxt('exemple_weight.txt')
model = empca.empca(flux, niter=1, nvec=1, weights=weights)

exemple_flux.txt
exemple_weight.txt

As reported by Lingfeng Cheng at Cornell, input data cannot have NaN values even if the corresponding values are masked with weights=0. Normally masked data (weights=0) are effectively ignored via line 247:

b = A.T.dot( w*b )

But if any of the data in b are NaN, this results in a NaN output, not a 0 output for that element even if w (=weights) is 0. This results in an error like:

ValueError: On entry to DGELSD parameter number 6 had an illegal value

At minimum, catch NaNs in the input and report as a meaningful error. Explore whether masked NaNs can be ignored without requiring rewriting or recopying the input array.