Comments (9)
`# Import what is needed to build the Keras model
from keras import backend as K
from keras.layers import Dense, Activation, Flatten
from keras.models import Sequential
Import a toy dataset and the importance training
from importance_sampling.datasets import MNIST
from importance_sampling.training import ImportanceTraining
import keras.utils
def create_nn():
"""Build a simple fully connected NN"""
model = Sequential([
Flatten(input_shape=(28, 28, 1)),
Dense(40, activation="tanh"),
Dense(40, activation="tanh"),
Dense(10),
Activation("softmax") # Needs to be separate to automatically
# get the preactivation outputs
])
model.compile(
optimizer="adam",
loss="categorical_crossentropy",
metrics=["accuracy"]
)
return model
if name == "main":
# Load the data
dataset = MNIST()
x_train, y_train = dataset.train_data[:]
x_test, y_test = dataset.test_data[:]
# Create the NN and keep the initial weights
model = create_nn()
weights = model.get_weights()
# Train with uniform sampling
K.set_value(model.optimizer.lr, 0.001)
model.fit(
x_train, y_train,
batch_size=64, epochs=8,
validation_data=(x_test, y_test)
)
# Train with importance sampling
model.set_weights(weights)
K.set_value(model.optimizer.lr, 0.001)
ImportanceTraining(model).fit(
x_train, y_train,
batch_size=64, epochs=2,
validation_data=(x_test, y_test)
)`
from importance-sampling.
Hi,
The complete code is in this repository. The number 937 is 937 batches per epoch. Which is basically floor(len(x_train)/batch_size) = floor(60000/64) = 937.
Let me know if you have any more questions.
Cheers,
Angelos
from importance-sampling.
I'm so glad to receive your reply that I can't reply to you immediately. Thank you again for your reply. I really have a lot of questions to ask you about importance sampling. Firstly,what does the increment parameter τ mean mentioned in the article? Secondly, in the source code of importance sampling, the th_u = (B+3b)/3b, but in this article, maximum speedup = (B+3B)/3B,do these two parameters have the same meaning? Or maybe it's my misunderstanding.Looking forward to your reply.
from importance-sampling.
Hi again,
So τ
is the factor one needs to increment the batch size to achieve equivalent variance reduction. Basically this means that using uniform sampling one needs to sample τ * b
samples to achieve gradients of the same quality as the ones of importance sampling.
τ_th
is set to (B + 3b)/(3b)
because that is the ratio of extra time spent to do importance sampling. If the variance reduction is higher than this ratio then this means that IS is worth it. As an example say we compute τ = 1.5
this means that uniform sampling needs approximately τ * 3 * b = 4.5 b
units of time to compute gradients of this quality while IS needs B + 3b
. So, if B <= 1.5 b
then we can start IS.
To compute the max speedup, we know that the smallest variance is equivalent to using B
samples per batch. However, we only spend B+3b
time. Thus the ratio is (B+3b)/(3B)
because uniform sampling would need to use all B
samples.
Cheers,
Angelos
from importance-sampling.
I am closing the issue, feel free to reopen it if you have further questions.
from importance-sampling.
Hi, I am sorry that I did not respond in time.That's because I did not know enough about the importance sampling, I dare not to ask questions casually. I also want to discuss with you how to add importance sampling to LSTM. Finally, i have a small wish,can you tell me your email address so I can ask questions in time?My English is not very good. Please forgive me for my poor English expression.Thank you very much!
Have a good day!
from importance-sampling.
Hello, I'm sorry to disturb you again. Could you tell me why do we need to do pre-sampling? And What determines the small batch b? Thanks again!
from importance-sampling.
Hi,
My email is angelos.katharopoulos (at) idiap.ch .
We do pre-sampling to avoid computing the importance distribution on the whole dataset. See section 3.3 of the paper. b is the batch size so it can be arbitrarily chosen.
Cheers,
Angelos
from importance-sampling.
Hello,after I edit the mail and send it to you, the system displays this"Your email is classified as spam by the recipient's server, and the recipient will not receive it."
Here are the details of the email.
Dear Angelos:
I am a college student and hope to know your importance sampling method in detail. Please forgive me for interrupting you many times.After reading your paper again, I still have some queastions as follow:
(1).In section 3.2 of the paper,what does the variance reduction mean?And whose variance is it?The loss function or the gradient norm?
(2).In Equation 20,all the upper bounds of the gradient norm are proposed.What parameters do all the parameters here refer to?Is parameter theta of the model?
(3).Why is the left end of equation 25 equal to the right end equal to 1/(1-τ^2)?
Thanks for your help and patience.
from importance-sampling.
Related Issues (20)
- Import error with resnet HOT 4
- Addition of RAIS in the implemented methods
- The result seems okay, but there are a few confusing things HOT 2
- Fail when metric given as function HOT 9
- Add initial_epoch to ImportanceTraining classes
- Remove unnecessary loading of validation data to memory HOT 2
- Failed when using ModelCheckpoint callback HOT 3
- Port to tensorflow 2 HOT 20
- Support of networks with multiple outputs HOT 10
- How do I get the serial Numbers of important samples HOT 1
- High GPU memory usage HOT 4
- Memory leak? HOT 2
- Question regarding Eq. 29 HOT 1
- Question regarding maximum variance reduction & speedup HOT 4
- RuntimeError: The layer has never been called and thus has no defined output shape. HOT 6
- Adjust Learning rate
- Compatibility with tensorflow = 2.5.0 and Keras = 2.4 HOT 4
- tensorflow and keras versions HOT 2
- Some confusion about the fast grads calculation when converting to Pytorch. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from importance-sampling.