Git Product home page Git Product logo

tensorflow-examples's Introduction

TensorFlow Examples

This tutorial was designed for easily diving into TensorFlow, through examples. For readability, it includes both notebooks and source codes with explanation, for both TF v1 & v2.

It is suitable for beginners who want to find clear and concise examples about TensorFlow. Besides the traditional 'raw' TensorFlow implementations, you can also find the latest TensorFlow API practices (such as layers, estimator, dataset, ...).

Update (05/16/2020): Moving all default examples to TF2. For TF v1 examples: check here.

Tutorial index

0 - Prerequisite

1 - Introduction

  • Hello World (notebook). Very simple example to learn how to print "hello world" using TensorFlow 2.0+.
  • Basic Operations (notebook). A simple example that cover TensorFlow 2.0+ basic operations.

2 - Basic Models

  • Linear Regression (notebook). Implement a Linear Regression with TensorFlow 2.0+.
  • Logistic Regression (notebook). Implement a Logistic Regression with TensorFlow 2.0+.
  • Word2Vec (Word Embedding) (notebook). Build a Word Embedding Model (Word2Vec) from Wikipedia data, with TensorFlow 2.0+.
  • GBDT (Gradient Boosted Decision Trees) (notebooks). Implement a Gradient Boosted Decision Trees with TensorFlow 2.0+ to predict house value using Boston Housing dataset.

3 - Neural Networks

Supervised
  • Simple Neural Network (notebook). Use TensorFlow 2.0 'layers' and 'model' API to build a simple neural network to classify MNIST digits dataset.
  • Simple Neural Network (low-level) (notebook). Raw implementation of a simple neural network to classify MNIST digits dataset.
  • Convolutional Neural Network (notebook). Use TensorFlow 2.0+ 'layers' and 'model' API to build a convolutional neural network to classify MNIST digits dataset.
  • Convolutional Neural Network (low-level) (notebook). Raw implementation of a convolutional neural network to classify MNIST digits dataset.
  • Recurrent Neural Network (LSTM) (notebook). Build a recurrent neural network (LSTM) to classify MNIST digits dataset, using TensorFlow 2.0 'layers' and 'model' API.
  • Bi-directional Recurrent Neural Network (LSTM) (notebook). Build a bi-directional recurrent neural network (LSTM) to classify MNIST digits dataset, using TensorFlow 2.0+ 'layers' and 'model' API.
  • Dynamic Recurrent Neural Network (LSTM) (notebook). Build a recurrent neural network (LSTM) that performs dynamic calculation to classify sequences of variable length, using TensorFlow 2.0+ 'layers' and 'model' API.
Unsupervised
  • Auto-Encoder (notebook). Build an auto-encoder to encode an image to a lower dimension and re-construct it.
  • DCGAN (Deep Convolutional Generative Adversarial Networks) (notebook). Build a Deep Convolutional Generative Adversarial Network (DCGAN) to generate images from noise.

4 - Utilities

  • Save and Restore a model (notebook). Save and Restore a model with TensorFlow 2.0+.
  • Build Custom Layers & Modules (notebook). Learn how to build your own layers / modules and integrate them into TensorFlow 2.0+ Models.
  • Tensorboard (notebook). Track and visualize neural network computation graph, metrics, weights and more using TensorFlow 2.0+ tensorboard.

5 - Data Management

  • Load and Parse data (notebook). Build efficient data pipeline with TensorFlow 2.0 (Numpy arrays, Images, CSV files, custom data, ...).
  • Build and Load TFRecords (notebook). Convert data into TFRecords format, and load them with TensorFlow 2.0+.
  • Image Transformation (i.e. Image Augmentation) (notebook). Apply various image augmentation techniques with TensorFlow 2.0+, to generate distorted images for training.

6 - Hardware

  • Multi-GPU Training (notebook). Train a convolutional neural network with multiple GPUs on CIFAR-10 dataset.

TensorFlow v1

The tutorial index for TF v1 is available here: TensorFlow v1.15 Examples. Or see below for a list of the examples.

Dataset

Some examples require MNIST dataset for training and testing. Don't worry, this dataset will automatically be downloaded when running examples. MNIST is a database of handwritten digits, for a quick description of that dataset, you can check this notebook.

Official Website: http://yann.lecun.com/exdb/mnist/.

Installation

To download all the examples, simply clone this repository:

git clone https://github.com/aymericdamien/TensorFlow-Examples

To run them, you also need the latest version of TensorFlow. To install it:

pip install tensorflow

or (with GPU support):

pip install tensorflow_gpu

For more details about TensorFlow installation, you can check TensorFlow Installation Guide

TensorFlow v1 Examples - Index

The tutorial index for TF v1 is available here: TensorFlow v1.15 Examples.

0 - Prerequisite

1 - Introduction

  • Hello World (notebook) (code). Very simple example to learn how to print "hello world" using TensorFlow.
  • Basic Operations (notebook) (code). A simple example that cover TensorFlow basic operations.
  • TensorFlow Eager API basics (notebook) (code). Get started with TensorFlow's Eager API.

2 - Basic Models

  • Linear Regression (notebook) (code). Implement a Linear Regression with TensorFlow.
  • Linear Regression (eager api) (notebook) (code). Implement a Linear Regression using TensorFlow's Eager API.
  • Logistic Regression (notebook) (code). Implement a Logistic Regression with TensorFlow.
  • Logistic Regression (eager api) (notebook) (code). Implement a Logistic Regression using TensorFlow's Eager API.
  • Nearest Neighbor (notebook) (code). Implement Nearest Neighbor algorithm with TensorFlow.
  • K-Means (notebook) (code). Build a K-Means classifier with TensorFlow.
  • Random Forest (notebook) (code). Build a Random Forest classifier with TensorFlow.
  • Gradient Boosted Decision Tree (GBDT) (notebook) (code). Build a Gradient Boosted Decision Tree (GBDT) with TensorFlow.
  • Word2Vec (Word Embedding) (notebook) (code). Build a Word Embedding Model (Word2Vec) from Wikipedia data, with TensorFlow.

3 - Neural Networks

Supervised
  • Simple Neural Network (notebook) (code). Build a simple neural network (a.k.a Multi-layer Perceptron) to classify MNIST digits dataset. Raw TensorFlow implementation.
  • Simple Neural Network (tf.layers/estimator api) (notebook) (code). Use TensorFlow 'layers' and 'estimator' API to build a simple neural network (a.k.a Multi-layer Perceptron) to classify MNIST digits dataset.
  • Simple Neural Network (eager api) (notebook) (code). Use TensorFlow Eager API to build a simple neural network (a.k.a Multi-layer Perceptron) to classify MNIST digits dataset.
  • Convolutional Neural Network (notebook) (code). Build a convolutional neural network to classify MNIST digits dataset. Raw TensorFlow implementation.
  • Convolutional Neural Network (tf.layers/estimator api) (notebook) (code). Use TensorFlow 'layers' and 'estimator' API to build a convolutional neural network to classify MNIST digits dataset.
  • Recurrent Neural Network (LSTM) (notebook) (code). Build a recurrent neural network (LSTM) to classify MNIST digits dataset.
  • Bi-directional Recurrent Neural Network (LSTM) (notebook) (code). Build a bi-directional recurrent neural network (LSTM) to classify MNIST digits dataset.
  • Dynamic Recurrent Neural Network (LSTM) (notebook) (code). Build a recurrent neural network (LSTM) that performs dynamic calculation to classify sequences of different length.
Unsupervised
  • Auto-Encoder (notebook) (code). Build an auto-encoder to encode an image to a lower dimension and re-construct it.
  • Variational Auto-Encoder (notebook) (code). Build a variational auto-encoder (VAE), to encode and generate images from noise.
  • GAN (Generative Adversarial Networks) (notebook) (code). Build a Generative Adversarial Network (GAN) to generate images from noise.
  • DCGAN (Deep Convolutional Generative Adversarial Networks) (notebook) (code). Build a Deep Convolutional Generative Adversarial Network (DCGAN) to generate images from noise.

4 - Utilities

  • Save and Restore a model (notebook) (code). Save and Restore a model with TensorFlow.
  • Tensorboard - Graph and loss visualization (notebook) (code). Use Tensorboard to visualize the computation Graph and plot the loss.
  • Tensorboard - Advanced visualization (notebook) (code). Going deeper into Tensorboard; visualize the variables, gradients, and more...

5 - Data Management

  • Build an image dataset (notebook) (code). Build your own images dataset with TensorFlow data queues, from image folders or a dataset file.
  • TensorFlow Dataset API (notebook) (code). Introducing TensorFlow Dataset API for optimizing the input data pipeline.
  • Load and Parse data (notebook). Build efficient data pipeline (Numpy arrays, Images, CSV files, custom data, ...).
  • Build and Load TFRecords (notebook). Convert data into TFRecords format, and load them.
  • Image Transformation (i.e. Image Augmentation) (notebook). Apply various image augmentation techniques, to generate distorted images for training.

6 - Multi GPU

  • Basic Operations on multi-GPU (notebook) (code). A simple example to introduce multi-GPU in TensorFlow.
  • Train a Neural Network on multi-GPU (notebook) (code). A clear and simple TensorFlow implementation to train a convolutional neural network on multiple GPUs.

More Examples

The following examples are coming from TFLearn, a library that provides a simplified interface for TensorFlow. You can have a look, there are many examples and pre-built operations and layers.

Tutorials

  • TFLearn Quickstart. Learn the basics of TFLearn through a concrete machine learning task. Build and train a deep neural network classifier.

Examples

tensorflow-examples's People

Contributors

ae1020 avatar arthurtalkgoal avatar aymericdamien avatar bhageena avatar bhavtoshrath avatar bobotig avatar bquast avatar ckirksey3 avatar dakerfp avatar ddurham2 avatar dragon-yu avatar drheart avatar federicoponzi avatar hosseinsheikhi avatar jasonvann avatar jfsantos avatar jilljenn avatar joefreeman avatar kilarinikhil avatar kokimame avatar kulbear avatar lf-devjourney avatar luanrly avatar maxthyen avatar mhue avatar normanheckscher avatar solaris33 avatar sudomakeinstall avatar vikramtiwari avatar weijianzz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tensorflow-examples's Issues

change n_steps value in training and test

Hello, Aymeric Damien,

Thanks in advance

What I want to do is using different n_steps values in training and testing, How to set it?
for example, in current training n_steps = 16, however in test I want to use n_steps = 55 (or changed as various example). I reset n_steps =55 in test, but error happened:

tensorflow.python.framework.errors.InvalidArgumentError: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 55) and num_split 16

Thank you very much

Leo

Linear regression: use tf.reduce_mean instead of tf.reduce_sum

In this fileTensorFlow-Examples/examples/2_BasicModels/linear_regression.py line 39, I try to use tf.reduce_mean instead.
cost = tf.reduce_mean(tf.pow(pred - Y, 2)) / 2
This is the original code
cost = tf.reduce_sum(tf.pow(pred - Y, 2)) / (2 * n_samples)

But I found the result is different.
Origin(with tf.reduce_sum)
Epoch: 0050 cost= 0.078097254 W= 0.268404 b= 0.666144
Epoch: 0100 cost= 0.077967979 W= 0.267285 b= 0.674192
Epoch: 0150 cost= 0.077853732 W= 0.266233 b= 0.681762
Epoch: 0200 cost= 0.077752732 W= 0.265244 b= 0.688881
Epoch: 0250 cost= 0.077663496 W= 0.264313 b= 0.695577
Epoch: 0300 cost= 0.077584624 W= 0.263437 b= 0.701875
Epoch: 0350 cost= 0.077514939 W= 0.262614 b= 0.707798
Epoch: 0400 cost= 0.077453338 W= 0.26184 b= 0.713369
Epoch: 0450 cost= 0.077398948 W= 0.261111 b= 0.718609
Epoch: 0500 cost= 0.077350870 W= 0.260426 b= 0.723537
Epoch: 0550 cost= 0.077308379 W= 0.259782 b= 0.728173
Epoch: 0600 cost= 0.077270873 W= 0.259176 b= 0.732533
Epoch: 0650 cost= 0.077237763 W= 0.258606 b= 0.736633
Epoch: 0700 cost= 0.077208482 W= 0.25807 b= 0.740489
Epoch: 0750 cost= 0.077182651 W= 0.257565 b= 0.744117
Epoch: 0800 cost= 0.077159837 W= 0.257091 b= 0.747528
Epoch: 0850 cost= 0.077139676 W= 0.256645 b= 0.750738
Epoch: 0900 cost= 0.077121906 W= 0.256226 b= 0.753756
Epoch: 0950 cost= 0.077106208 W= 0.255831 b= 0.756594
Epoch: 1000 cost= 0.077092350 W= 0.25546 b= 0.759265
Optimization Finished!
Training cost= 0.0770923 W= 0.25546 b= 0.759265
train
Testing ...
Testing cost= 0.100858
Absolute mean square loss difference: 0.0112338
test


Now(with tf.reduce_mean)
Epoch: 0050 cost= 0.089926675 W= 0.229255 b= 0.785728
Epoch: 0100 cost= 0.089696057 W= 0.225371 b= 0.814913
Epoch: 0150 cost= 0.089644745 W= 0.224121 b= 0.8243
Epoch: 0200 cost= 0.089630596 W= 0.223719 b= 0.827321
Epoch: 0250 cost= 0.089626297 W= 0.22359 b= 0.828292
Epoch: 0300 cost= 0.089624956 W= 0.223548 b= 0.828604
Epoch: 0350 cost= 0.089624502 W= 0.223535 b= 0.828705
Epoch: 0400 cost= 0.089624360 W= 0.223531 b= 0.828736
Epoch: 0450 cost= 0.089624345 W= 0.22353 b= 0.828744
Epoch: 0500 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 0550 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 0600 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 0650 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 0700 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 0750 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 0800 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 0850 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 0900 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 0950 cost= 0.089624323 W= 0.22353 b= 0.828746
Epoch: 1000 cost= 0.089624323 W= 0.22353 b= 0.828746
Optimization Finished!
Training cost= 0.0896243 W= 0.22353 b= 0.828746
train1
Testing ...
Testing cost= 0.100858
Absolute mean square loss difference: 0.0112338
test1

You can see that the cost and W b is different.

Why?
Could anyone tell me?

confused about sess.run() function

I'm trying to understand this code https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/5%20-%20User%20Interface/loss_visualization.py. There are 3 sess.run() functions in the "for" loop (see attached picture). Does this mean the whole graph runs 3 times in one "for" loop?
capture
I tried to use result = sess.run([optimizer, cost, merged_summary_op]), it seems it can get the same results. Is this more efficient than calling 3 times sess.run()?
And, I tried to only use sess.run([cost,merged_summary_op]) (removed optimizer) and the results are wrong. I guess run() function doesn't run the whole graph once it's called. Am I wrong?

logistic_regression.py fails with python 2.7

$ python logistic_regression.py 
  File "logistic_regression.py", line 61
    print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(avg_cost)
                 ^
SyntaxError: invalid syntax

python-2.7.6, ubuntu-14.04

Variable-Length Bi-Directional RNN

Hi,
and thank you so much for this examples. These are all very helpful and demonstrate the core ideas of the models very well!

I am still new to Tensorflow and RNN training and am currently wondering, how to deal with input series that has variable length. I found this example on reddit and it seems it uses padding to bring sequences to the same length. Unfortunately, I cannot make that particular example run.

Would it be possible to provide an example?

Thanks and best,
Sigi

delete of matmul for hidden weights

HI,
I note that in your updated code, you deleted _X = tf.matmul(_X, _weights['hidden']) + _biases['hidden'], and reshpae as n_steps * (batch_size, input_dim) instead of n_steps * (batch_size, n_hidden)

Why did you do this?
Thanks

initial state issue

The code of implementing recurrent network in notebook shows that:

Tensorflow LSTM cell requires 2x n_hidden length (state & cell)

istate = tf.placeholder("float", [None, 2_n_hidden])
Here, I am confused about the length of initial state. As I have n hidden units, should not the length of initial state is n.
Why is it 2_hidden instead? Thanks for your reply.

why was each point treated separately?

inside the for loop, you take each (x,y) in train_X and train_y and calculate its cost function.
it makes the process stochastic (as it is done in Neural Networks) - and I don't understand why you do it like this since in linear regression the process shouldn't be stochastic.

in linear regression, the cost function should include the entire data set in each iteration.
I mean, the second 'for' loop shouldn't be there.

like this:

linear reg

Launch the graph

with tf.Session() as sess:
sess.run(init)

# Fit all training data
for epoch in range(training_epochs):
    sess.run(optimizer, feed_dict={X: train_X, Y: train_Y})

Display logs per epoch step

    if ((epoch+1) % display_step == 0) or (epoch == 0):
        c = sess.run(cost, feed_dict={X: train_X, Y:train_Y})
        print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c), \
            "W=", sess.run(W), "b=", sess.run(b)

print "Optimization Finished!"
training_cost = sess.run(cost, feed_dict={X: train_X, Y: train_Y})
print "Training cost=", training_cost, "W=", sess.run(W), "b=", sess.run(b), '\n'

Graphic display

plt.plot(train_X, train_Y, 'ro', label='Original data')
plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label='Fitted line')
plt.legend()
plt.show()

(*of course, other modifications must be done before using the code segment like creating a dot product since the weight is a 2x1 vector, etc.)

this way, it takes more iteration to get the result to converge, but each iteration is much faster. (and the difference in the results is less then 6*10^-5 percent).

also, expanding this method to a 2-features data set (instead of 1-feature), it might not converge at all and if it does, the result will depend on the initial conditions and it might converge to a local minimum.

so, I was wondering if there's a reason why you wrote the code to use a stochastic process.

thanks

MNIST convolutional DNN example: the weights of the convolutional layers don't change much

Hello Aymeric.

thank you for your great examples. I visualized the weights of the first convolutional layer in your MNIST example and noticed that they change only slightly during training. On the other hand, in original TF MNIST convolutional example they change drastically, "feature patterns" are clearly visible at the end. I noticed that they use MomentumOptimizer, while you are using AdamOptimizer. Could this be the reason for different behaviors? Still, it seems rather strange to me that we don't have to change the weights much during training.

I wonder, what do you think about it, may be you can provide some links to documents that can help me to find an explanation for the issue?

recurrent_network.py

The comments of x=tf.split(0,n_steps,x) is 'split to get a list of 'n_steps' tensors of shape(batch_size,n_hidden)', the shape is (batch_sizs,n_hidden)? why not the (batch_size,n_input). Do i missing something?

Porting Model to Android

Hi,

Thanks for such wonderful tutorials. I would like to port the Bi-Directional LSTM model trained on a GPU to Android. Can you pl add a sample for it?

What's the difference between acc and val_acc

When I run the tfLearn example, i get logs printing acc and val_acc
As i understand, val_acc is the accuracy of validation set
However, I am not sure what acc identifies exactly.
Does "acc" mean the accuracy of all of training data of some parts of training data?

And what is validation_batch_size?
Im confused why batch size is needed...( I know what batch size of training data means)

Thanks in advance

I have a question about rnn code

istate = tf.placeholder("float", [None, 2*n_hidden]) #state & cell => 2x n_hidden

why i have to 2*n_hidden?
does it mean that we are using lstm cell and tensorflow.model.rnn require us to provide initial hidden state and initial cell state together when using lstm cell?

convolutional network memory usage

Hi !
I am new in tensorflow and I am following this nice tutorial.
I am trying to run the convolutional network tutorial on my GTX 1080 4 Gb but seems I have memory problem. I already set up the configuration to allow memory growth and per process memory fraction but I still find out of memory problem. I tried to reduce the batch size and run it on python instead of the notebook.
So my question is.. it my GPU too small to be able to run these few layers of CNN?

What are the meanings of the variables which get from RNN

Hi, Aymeric Damien, thank you so much for this examples, it's great and helpful.
I use your RNN (recurrent_network.py) to train my dataset and get a good result. I use this way to get all variables:

def RNN(x, weights, biases):
    x = tf.transpose(x, [1, 0, 2])
    x = tf.reshape(x, [-1, n_input])
    x = tf.split(0, n_steps, x)
    lstm_cell = rnn_cell.LSTMCell(n_hidden, use_peepholes=False, forget_bias=1.0)
    outputs, states = rnn.rnn(lstm_cell, x, dtype=tf.float32)
    return tf.matmul(outputs[-1], weights['out']) + biases['out']
all_variables = tf.get_collection(tf.GraphKeys.VARIABLES)

all_variable = sess.run(all_variables)
print all_variable

Then I can get 14 arrays which are all variables of the RNN. But I don't know their meanings. Could you help me? Thanks a lot!

My network parameters:

n_input = 140 
n_hidden = 32 
n_classes = 2 

The dimensions of 14 variables are as follows:

[0] 32*2(hidden->output?)
[1] 2
[2] 172*128  (n_input+n_hidden)*(n_hidden*4)  
[3] 128
[4] 1
[5] 1
[6] 32*2
[7] 32*2
[8] 2
[9] 2
[10] 172*128
[11] 172*128
[12] 128
[13] 128

Thanks a lot!

Bi-Directional LSTM

Thanks for sharing this valuable resource. The recurrent network example was very useful to me for sequence classification.
Can you pl add a new example which is same as recurrent_network.py but uses Bi-Directional LSTM instead of uni-directional. That will be very useful for me.
Thanks once again.

bidirectional RNN

I'm using 0.6.0 with python 3.4
run the script and it showed:

Extracting /tmp/data/train-images-idx3-ubyte.gz
Extracting /tmp/data/train-labels-idx1-ubyte.gz
Extracting /tmp/data/t10k-images-idx3-ubyte.gz
Extracting /tmp/data/t10k-labels-idx1-ubyte.gz

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-4c9116ccec55> in BiRNN(x, weights, biases)
     62         outputs, _, _ = rnn.bidirectional_rnn(lstm_fw_cell, lstm_bw_cell, x, 
---> 63                                               dtype=tf.float32)
     64     except Exception: # Old TensorFlow version only returns outputs not states

/Users/aaronlai/anaconda/lib/python3.4/site-packages/tensorflow/python/ops/rnn.py in bidirectional_rnn(cell_fw, cell_bw, inputs, initial_state_fw, initial_state_bw, dtype, sequence_length, scope)
    225   if not sequence_length:
--> 226     raise ValueError("sequence_length has to be defined")
    227   if not inputs:

ValueError: sequence_length has to be defined

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-4-4c9116ccec55> in <module>()
     69     return tf.matmul(outputs[-1], weights['out']) + biases['out']
     70 
---> 71 pred = BiRNN(x, weights, biases)
     72 
     73 # Define loss and optimizer

<ipython-input-4-4c9116ccec55> in BiRNN(x, weights, biases)
     64     except Exception: # Old TensorFlow version only returns outputs not states
     65         outputs = rnn.bidirectional_rnn(lstm_fw_cell, lstm_bw_cell, x,
---> 66                                         dtype=tf.float32)
     67 
     68     # Linear activation, using rnn inner loop last output

/Users/aaronlai/anaconda/lib/python3.4/site-packages/tensorflow/python/ops/rnn.py in bidirectional_rnn(cell_fw, cell_bw, inputs, initial_state_fw, initial_state_bw, dtype, sequence_length, scope)
    224     raise TypeError("inputs must be a list")
    225   if not sequence_length:
--> 226     raise ValueError("sequence_length has to be defined")
    227   if not inputs:
    228     raise ValueError("inputs must not be empty")

ValueError: sequence_length has to be defined

Convert Tensor to numpy array

I am trying to calculate ruc score after every epoch. For than the tensor object need to be converted to numpy array.
Following is the code I am trying.


# Launch the graph
with tf.Session() as sess:
    sess.run(init)

    # Training cycle
    for epoch in range(training_epochs):
        avg_cost = 0.
        total_batch = int(len(trX)/batch_size)
        # Loop over all batches
        for i in range(total_batch):
            batch_x, batch_y = trX[batch_size*i:batch_size*(i+1)],trY[batch_size*i:batch_size*(i+1)]
            # Run optimization op (backprop) and cost op (to get loss value)
            _, c = sess.run([optimizer, cost], feed_dict={x: batch_x,
                                                          y: batch_y})
            # Compute average loss
            avg_cost += c / total_batch
        # Display logs per epoch step
        if epoch % display_step == 0:
            print "Epoch:", '%04d' % (epoch+1), "cost=", \
                "{:.9f}".format(avg_cost)
            print roc_auc_score(teY,pred)

    print "Optimization Finished!"
    # Test model
    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
    # Calculate accuracy
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    print "Accuracy:", accuracy.eval({x: teX, y: teY})

It is giving following error:

TypeError: Expected sequence or array-like, got <class 'tensorflow.python.framework.ops.Tensor'>

Can you please tell how to convert tensor to numpy array.
I tried pred.eval() but it is showing error also.

Multi GPU

I have tried multi GPU example on 2 x GTX 750 Ti.
But it is failing and utilized only CPU.
Could you help me out in this regard?

PFB output of nvidia-smi ,

image

Major update

Current examples start to be quite old, I started to re-write all examples and make them fully suitable for TensorFlow v0.8. There are some new examples too.
You can check it out in branch 'v2'.

question about accuracy

int "recurrent network" the loss was calculated by the function tf.nn.softmax_cross_entropy_with_logits ,and the correct_pred is calculated by tf.equal(tf.argmax(pred,1), tf.argmax(y,1)), which take "pred " directly, how dose this make sense? I think it should be softmax(pred) instead of pred, but I code just works fine. This really confuse me. Can somebody explain whats going on ? thanks!

Dst tensor is not initialized

Hi @aymericdamien I have run this script logistic regression.py
But i met a problem "Dst tensor is not initialized".
The detail log is here:

Epoch: 0001 cost= 29.917553501
Epoch: 0002 cost= 21.929896693
Epoch: 0003 cost= 21.063875407
Epoch: 0004 cost= 20.457020144
Epoch: 0005 cost= 20.084428289
Epoch: 0006 cost= 19.814794980
Epoch: 0007 cost= 19.674670629
Epoch: 0008 cost= 19.510438999
Epoch: 0009 cost= 19.309689613
Epoch: 0010 cost= 19.223995275
Epoch: 0011 cost= 19.161345129
Epoch: 0012 cost= 18.985856709
Epoch: 0013 cost= 18.917688493
Epoch: 0014 cost= 18.832972273
Epoch: 0015 cost= 18.742634454
Epoch: 0016 cost= 18.695894625
Epoch: 0017 cost= 18.643278683
Epoch: 0018 cost= 18.609112186
Epoch: 0019 cost= 18.444614899
Epoch: 0020 cost= 18.532375607
Epoch: 0021 cost= 18.437554449
Epoch: 0022 cost= 18.310914770
Epoch: 0023 cost= 18.289282742
Epoch: 0024 cost= 18.214274961
Epoch: 0025 cost= 18.293197173
Optimization Finished!
Accuracy:
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
<ipython-input-17-f661f1e1e9de> in <module>()
     24     # Calculate accuracy
     25     accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
---> 26     print "Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})

/home/burness/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in eval(self, feed_dict, session)
    500 
    501     """
--> 502     return _eval_using_default_session(self, feed_dict, self.graph, session)
    503 
    504 

/home/burness/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in _eval_using_default_session(tensors, feed_dict, graph, session)
   3332                        "the tensor's graph is different from the session's "
   3333                        "graph.")
-> 3334   return session.run(tensors, feed_dict)
   3335 
   3336 

/home/burness/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    338     try:
    339       result = self._run(None, fetches, feed_dict, options_ptr,
--> 340                          run_metadata_ptr)
    341       if run_metadata:
    342         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/burness/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
    562     try:
    563       results = self._do_run(handle, target_list, unique_fetches,
--> 564                              feed_dict_string, options, run_metadata)
    565     finally:
    566       # The movers are no longer used. Delete them.

/home/burness/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
    635     if handle is None:
    636       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
--> 637                            target_list, options, run_metadata)
    638     else:
    639       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/home/burness/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
    657       # pylint: disable=protected-access
    658       raise errors._make_specific_exception(node_def, op, error_message,
--> 659                                             e.code)
    660       # pylint: enable=protected-access
    661 

InternalError: Dst tensor is not initialized.
     [[Node: _recv_Placeholder_1_0/_27513 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_267__recv_Placeholder_1_0", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
     [[Node: Mean_6/_27517 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_277_Mean_6", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

bidirectional RNN states never updated?

Hopefully this is not stupid. In the Tensorflow tutorial, the states of the RNN are updated after each time-step and used as the states for the next time-step

output, state = lstm(current_batch_of_words, state)

In yours the state of the bidirectional RNN seems to just be discarded. Is this the case, or am I missing something? If so, why?

Also, a small detail; you use outputs[-1] for

return tf.matmul(outputs[-1], _weights['out']) + _biases['out']

which corresponds according to the Tensorflow API

(outputs, output_state_fw, output_state_bw) = outputs

So shouldn't this be

return tf.matmul(outputs[0], _weights['out']) + _biases['out']

Instead?

Different batch_size in training and testing for Bi-directional RNN

Hi,

I am working closed to bi-directional RNN. It is so great to find your demo code here. However, I noticed that in your bi-directional RNN examples, the batch size for training and testing should be remain the same. It is due to the new tensorflow 0.6.0 asks for setting the seq_len with a constant value which is equal to batch_size.

I totally understand that when we unroll the network the batch_size should be the sequence length. But in testing, can we adjust the batch size to a different value? how to implement it?

Is that possible to re-initialize the rnn with different configurations, saying different batch_size?

Thanks very much.

Typo in autoencoder.ipynb?

# Building the encoder
def encoder(x):
    # Encoder Hidden layer with sigmoid activation #1
    layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['encoder_h1']),
                                   biases['encoder_b1']))
    # Decoder Hidden layer with sigmoid activation #2
    layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['encoder_h2']),
                                   biases['encoder_b2']))
    return layer_2


# Building the decoder
def decoder(x):
    # Encoder Hidden layer with sigmoid activation #1
    layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['decoder_h1']),
                                   biases['decoder_b1']))
    # Decoder Hidden layer with sigmoid activation #2
    layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['decoder_h2']),
                                   biases['decoder_b2']))
    return layer_2

seems comment of layer_2 in encoder() and layer_1 in decoder() mismatch their functionality. You basically build an autoencoder with 2 encoding layers and 2 decoding layers

How to fit my own dataset

Dear Aymeric

I am following ur work. You use MNIST data set for your work.
My question is how I can use my own dataset and fit it to your coding.

Please help me regarding this.

Thanks and Regards
Nahid

convolution parameter

Hi in your convolutional_network.ipynb file. Do you mind how you come with the number of 7_7_64 in
'wd1': tf.Variable(tf.random_normal([7_7_64, 1024])) . I am still confuse as how you are calculating this number. I think it is not arbitary number right?

MultiGPU Tensorflow

Hi,

When I run your code, instead of going to two GPUs, it takes over all that are available ( I have more than 2). How can I specify if I want to grab just two (say 4 and 5)?

Need explanation on weights

Can anyone please elaborate on the weights dimensions/shape and explain me those. I have data different than the MNIST, and I am trying to apply this CNN model on it, but I am having hard time understanding these shaping movements. For example:

weights = {
# 5x5 conv, 1 input, 32 outputs
'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32])),
# 5x5 conv, 32 inputs, 64 outputs
'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])),
# fully connected, 7_7_64 inputs, 1024 outputs
'wd1': tf.Variable(tf.random_normal([7_7_64, 1024])),
#1024 inputs, 10 outputs (class prediction)
'out': tf.Variable(tf.random_normal([1024, n_classes]))

--> The size of the output is defined according to what, then this 7_7_64 in the fully connected layer is defined by what again?
Thank you very much.

questions about softmax + cross_entropy v.s. softmax_cross_entropy_with_logits

Hi, in the Logistic regression
the original code is

pred=tf.nn.softmax(tf.add(tf.matmul(x,W),b))
cost =tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred),reduction_indices=1))

we use cost to be minimized.
But when I use the code below:

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(tf.nn.softmax(tf.add(tf.matmul(x,W),b)),y)
cost=tf.reduce_mean(cross_entropy)

It could runs successfully, but the accuracy is not stable as the original one, it changed from 0.5 - 0.9.
I think the 2 code works as similar logic, but I could not get the essential cause for that.

recurrent_network.py Linear Activation

Hi, I am a little confused.

in the def RNN function, you perform a linear activation of
_X = tf.matmul(_X, _weights['hidden']) + _biases['hidden']

than, pass it in as input.

Is that normal ? Isn't there only weight and bias for the 4 gates ?

Use cases for each algorithm

I'm very new to machine learning and I've found this repository really useful.

I'm wondering if you could please add some notes to either the README or the individual examples detailing some example use cases that are appropriate for each algorithm?

Multi-Layer BDLSTM

Hi,
Thanks for such awesome tutorials. You have included a BDLSTM example. I would like to know how can we create a multi-layer BDLSTM model with non-linearities in between.

Issue in logistic regression

Hi,

avg_cost += sess.run(cost, feed_dict={x: batch_xs, y: batch_ys})/total_batch

Why do you divide the cose with total_batch. I think you should first divide the cost with batch_size and then after tool divide it again with total_batch.

Thanks,
Nitin

How to train/test sequences with different n_steps?

Hi,
Your examples are really helpful! Now I have some testing sequences with different durations than the training sequences. How could I have a dynamic n_steps in BiRNN? I've tried several different ways, but didn't success.

Thanks!
Chi

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.