Comments (9)
Thanks, looking forward to it!
The reason why I am interested in doing it in Numpy is that Keras/Tensorflow isn't very great for sparse training. My use case is training 9 figures of embeddings. It eats up a lot of memory to have all those embeddings loaded into memory at the same time, and it's not necessary since only a small fraction is trained during each update step.
So I was thinking of using your library and altering it so that it saves the unused weights to disk, and only loads them when they are trained or about to be trained.
from numpy-ml.
So I was thinking of using your library and altering it so that it saves the unused weights to disk, and only loads them when they are trained or about to be trained.
That sounds like a good idea! If you end up implementing this, definitely consider submitting a PR :) I think this could be quite useful for a number of different model components, including the sparse evolutionary training layer (which currently uses dense matrices 😬).
In the meantime, you might look into the magnitude package (I haven't used it myself, but it seems potentially relevant).
from numpy-ml.
Don't be sorry - this question is very justified -- there is almost no usage documentation right now!
First, the bad news: at the moment, implementing word2vec will require a little bit of extra leg-work on your end. In particular, you'll need to implement either a negative sampler/noise contrastive estimation loss or a hierarchical softmax loss. For the latter, you could use the numpy_ml.preprocessing.nlp.HuffmanEncoder
module.
Now, the good news: I'm actively working on writing an NCE loss object, and hope to push it ASAP. I'll also probably include a convenience Embedding
layer to make embedding lookups a bit faster. I will update this thread when it has been pushed.
Ultimately, once these two components are in place, you should be able to write a relatively straightforward model. To see what a model object might look like, you can look at some examples in either the numpy_ml.neural_nets.modules
or numpy_ml.neural_nets.models
directories.
from numpy-ml.
Finally, one last caveat - if you're interested in training a non-toy word embedding model, I'd highly recommend using a library like keras, since it will make use of performance-optimized implementations for each model component. The code in this repo is meant to be clear and straightforward, but this often comes at the expense of efficiency!
from numpy-ml.
It looks like this is exactly what I was looking for. I'm not familiar with a lot of computer science terms, but it uses sqlite as the datastore, so I'm guessing it does what I am looking for.
Edit: reading the paper
"Magnitude queries return almost instantly and
are memory efficient. It uses lazy loading directly from disk, instead of having to load the entire model into memory"
Wow! Thanks for this recommendation!!!
from numpy-ml.
Looks like you can't use it for training, oh well
from numpy-ml.
But looking at the code gives me a strong idea of how I could implement it in this library, by using SQLite database as the vector store, and just copying the values back and forth between that and the numpy embedding array.
from numpy-ml.
Heya @Santosh-Gupta - I've just pushed a preliminary version of an NCELoss
and word2vec model here and here.
Unfortunately, I suspect that if you're going to use the models for any sizeable dataset you'll have to do some performance modifications first. Let me know if you decide to try it out / have any questions in the meantime!
from numpy-ml.
Sounds good!
from numpy-ml.
Related Issues (20)
- Using numpy.tensordot for Conv2D HOT 1
- error in DecisionTree
- Bug in transfer learning
- Bug in initializers init_from_dict()
- [Question] Gradient of Gradient Penalty in WGAN-GP.
- Naive Bayes HOT 3
- Feature Request: Clustering Kmeans (hard and soft version) HOT 2
- Feature Request: Online Linear Regression HOT 7
- Feature request: Accept multiple samples for online least squares HOT 1
- Best choice for my use case? HOT 1
- `numpy_ml.linear_model.LinearRegression.predict()` generates `ValueError` when used with copy-pasted code, but pip installed version works as expected!! HOT 1
- neural nets optimizer shape mismatch during backward pass
- Import of collections.Hashable fails in Python 3.10 HOT 3
- Columns and DataType Not Explicitly Set on line 228 of rl_utils.py
- Feature request: save/load model to/from json
- Question to improve the code.
- Declare your version of modules
- Automatic diferentiation for neural networks
- no 'load_dataset' in numpy_mL
- Example of MLP architecture
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from numpy-ml.