Ideas for storing/accessing large distance matrices about bigstatsr HOT 6 CLOSED

Package {bigstatsr} supports storing and accessing dense matrices on disk.
There is no way to store symmetric matrices by storing only the upper or lower triangle. You would have to store everything, which would take twice as much space.

The only package I know to provide such feature is package {Matrix}, but the objects are stored in RAM.

from bigstatsr.

Please try big_write by installing the latest version.

from bigstatsr.

Thanks for big_write and congrats on the CRAN release!

from bigstatsr.

Among the options available to store large distance matrices to disk, bigstatsr seems to be the best way to maintain it as compared to bigmemory, lvec and ff even if I need to store the everything twice. I believe this is not a concern as space is not a constraint. bigstatsr offers transparent implementations of SVD, lasso which will be helpful for me in implementing MDS on FBMs.

I would request two things:

1. A version of CRAN (#19) as this would be a dependency.
2. A big_write function to write distances matrices to disk.

Please let me know your thoughts.

from bigstatsr.

• For CRAN, for now a warning due to package {BH} is blocking me. I've opened an issue on the boost repo several days ago.
• For big_write, you want to write to a CSV format? You need this format for another soft?
  This should be easy enough with fread and big_apply. I can do this on Monday.

from bigstatsr.

Thanks for the quick turn around!

1. Hope the BH issue resolves and CRAN release happens!
2. big_write to CSV is sufficient for me.

from bigstatsr.