Comments (4)
FWIW I like this breakout - it combines the best of PyTorch's flexibility and provides logical callbacks during the training process. Do you see the eval_objective
providing a mechanism for early stopping, or should that be another callback?
In line with what many other ML libs do, I agree that Axon should define a Protocol for datasets that any external data management library could implement support for. I also think including a basic in-memory implementation for testing, quick ramp-up, and providing a reference example would add great value.
IMO PyTorch has one of the most reasonable interfaces for working with custom datasets; I'm actually in the process of migrating one project from TF to PyTorch due to the complexity of implementing custom dataset generators.
from axon.
@arpieb Yes! I actually think it would be best to adjust after_epoch
to take the evaluation state as well so early stopping can be implemented as a callback. That would also be useful for implementing model checkpointing based on validation metrics.
I've really only ever worked in TensorFlow/TFDS, but I have heard there is more of a preference for PyTorch for some of these things. I haven't really considered what kind of dataset integration Axon should have, mainly because I figured we would want to create a more general abstraction that works for the whole ecosystem, but it's definitely worth some thought to have something convenient.
from axon.
I think it's going to be hard to come up with a general abstraction for all use cases of Nx tensor collections. For example, Axon SL models will rely on sample-label pairs, which is not a requirement for most other data ops. A dataframe equivalent to pandas (exdas? nexdas? ;)) would need to be columnar to handle the mixture of numeric (Nx tensor) and non-numeric columns.
Ultimately all Python-based ML solutions seem to fall back to requiring a collection (len, get_item) or iterable (iter) to satisfy dataset needs, which in turn usually return one or more numpy arrays. Even DL4j forces any kind of transform pipeline or generator to ultimately spit up something implementing the INDArray interface (their attempt at replicating numpy.ndarray) before it can be handed off to a model.
Many UL models will not require labels, so 2+ rank tensors would fill the need just fine for them.
from axon.
This looks a lot like PyTorch Lightning! And that's a compliment. As much as I have distaste for much of the Python deep learning ecosystem, I think PTL has emerged from some early teething problems to be one of the most sane training APIs out there. Axon could do much worse than to emulate their approach to callbacks.
I agree wholeheartedly with @arpieb's take here (and in elixir-nx/nx#301) that defining some protocol for the inevitable dataframe libraries is essential here. Dataloading and its complexities have always been a bugbear for Pytorch and Tensorflow.
from axon.
Related Issues (20)
- Compiled model will always result in `%Axon.None{}`
- Block with multiple inputs fails to compile HOT 2
- mixed precision policies cause bumblebee models to fail HOT 2
- Cannot return map from Axon.nx
- batch_norm states "mean" and "var" never updated HOT 1
- Optimize frozen parameter memory usage HOT 1
- Allow overriding layer metadata such as `:op` and `:op_name` for built-ins HOT 1
- XLA Slow operation alarms: "The operation took ..." HOT 1
- a
- Loss increasing with binary_cross_entropy above version 0.4.1 HOT 6
- Compile Error due to a type mismatch HOT 2
- Req usage in OTP 26 HOT 6
- Most `Axon.ModelState` functions should be `defn` or `deftransform`
- Merge model state updates into a single `Axon.ModelState` call
- Optionally return gradients / gradient state in train step
- Crash with mixed precision with new Nx
- Remove namespaces
- Can't pass multiple inputs in custom trainer functions HOT 3
- Ambiguous inputs should raise
- Tagged versions missing in the repo compared to the ones in hexdocs.pm HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from axon.