Comments (9)
@jjallaire I see this is closed, but FYI indeed Matrix does have a C API -- it uses the SuiteSparse package by Tim Davis. You need to include at least "Matrix.h" and "Matrix_stubs.c". See https://github.com/bwlewis/irlba/blob/master/src/irlb.c for an example of a package linking to the Matrix package (and there are others that do this too). I think https://github.com/scikit-sparse/scikit-sparse uses the same cholmod subset of suite sparse that R (and Matlab) does. I'm not sure what the rest of scipy uses for sparse matrices.
See this ancient pre-reticulate fork of the tensorflow package for an example that supports the most common sparse matrix object between R/Python (dgCMatrix/scipy.sparse.csc.csc_matrix):
https://github.com/bwlewis/python/tree/1e92aec80fd29fdf8e3f38e894d3a600e25b5aae
A crude/old but working example of sparse martrices.
from reticulate.
@eddelbuettel yeah, in the case of Matrix we have available a curated subset of SuiteSparse that we can simply directly link to. No need to embed again, just link to Matrix.
from reticulate.
@bwlewis Do you know whether the Matrix package exposes a C API for sparse matrices?
from reticulate.
I don't think it does. The best short-term bet may be to get to the functions from Matrix via Rcpp.
A Google Summer of Code student is currently putting more sparse matrix support into RcppArmadillo.
from reticulate.
What we would need to do is somehow get our hands on the underlying memory and then create a SciPy sparse matrix from it. That said, I don't think Keras currently accepts SciPy sparse matrices (discussion I've seen about the issue seems to center on converting each batch to dense array before feeding it into Keras).
@gsimchoni Perhaps you could write your own training loop (via train_on_batch where you go from sparse to dense as required in each iteration?
from reticulate.
Thank you @jjallaire , this is exactly the advice I got from @dfalbel in the original issue.
from reticulate.
@bwlewis thanks for the SuiteSparse reminder. That requires external linking, or as Matrix does, embedding of SuiteSparse. May make for a good test case for RcppArmadillo too.
from reticulate.
@bwlewis Yep, which is what lme4 has done all those years -- that was always "exhibit one" of how to access code from another package which provides a cleanly exported API.
from reticulate.
Re-capping, it looks like Keras doesn't currently support Sparse matrices (they just suggest that you use train_on_batch
. It also looks like Sparse matrix support is in SciPy rather than NumPy, meaning that even if we can get ahold of R's memory there isn't anywhere in NumPy to put it.
from reticulate.
Related Issues (20)
- Need to reinstall reticulate whenever I want to access arcpy HOT 4
- Timeout when calling Async python function HOT 1
- Corrupt dataframe when converting from pandas to R HOT 5
- Release reticulate 1.38.0 HOT 1
- Reticulate 1.38.0 for R 4.2.1 can install but not import numpy, ... HOT 5
- Error in reticulate::use_python(): failed to initialize requested version of Python HOT 2
- Namespace conflict with python config package HOT 1
- Python variables sourced via `reticulate::source_python()` are not contained within `py` list HOT 6
- Reticulate does not play nice with Rich progressbars HOT 1
- R Studio Global Options Conflict with `use_virtualenv()` embedded in `.onLoad()` HOT 1
- Converting a python datatype using mutate in dplyr HOT 6
- Feature request: export `conda_run2` HOT 3
- Extracting data from Google Earth Engine HOT 2
- can't use rpy2 inside reticulate-embedded python HOT 6
- Bug: Existence of system-wide version of a shared library causes `undefined symbol` error HOT 10
- Default Altair chart dimensions are not respected HOT 1
- This is not logical HOT 1
- Prompting user input HOT 5
- Reticulate can't handle large integer values: Seems to experience integer overflow when it exceeds the limit for 32-bit signed integers
- as.raw method for bytes
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from reticulate.