New Examples (#160)

* Added basic models examples (kmeans, random forest, ...)
* Added API examples (layers, estimator, ...)
* Added other examples (Multi-GPU, build a dataset, ...)
* Notebook refactoring with new header and more details

README.md

@@ -1,110 +1,90 @@
 # TensorFlow Examples
-TensorFlow Tutorial with popular machine learning algorithms implementation. This tutorial was designed for easily diving into TensorFlow, through examples.
 
-It is suitable for beginners who want to find clear and concise examples about TensorFlow. For readability, the tutorial includes both notebook and code with explanations.
+This tutorial was designed for easily diving into TensorFlow, through examples. For readability, it includes both notebooks and source codes with explanation.
 
-Note: If you are using older TensorFlow version (before 0.12), please have a [look here](https://github.com/aymericdamien/TensorFlow-Examples/tree/0.11)
+It is suitable for beginners who want to find clear and concise examples about TensorFlow. Besides the traditional 'raw' TensorFlow implementations, you can also find the latest TensorFlow API practices (such as `layers`, `estimator`, `dataset`, ...).
+
+**Update (27.08.17):** TensorFlow v1.3 is recommended. Added many new examples (kmeans, random forest, multi-gpu training, layers api, estimator api, dataset api ...).
+
+*If you are using older TensorFlow version (0.11 and under), please have a [look here](https://github.com/aymericdamien/TensorFlow-Examples/tree/0.11).*
 
 ## Tutorial index
 
 #### 0 - Prerequisite
-- Introduction to Machine Learning ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/0_Prerequisite/ml_introduction.ipynb))
-- Introduction to MNIST Dataset ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/0_Prerequisite/mnist_dataset_intro.ipynb))
+- [Introduction to Machine Learning](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/0_Prerequisite/ml_introduction.ipynb).
+- [Introduction to MNIST Dataset](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/0_Prerequisite/mnist_dataset_intro.ipynb).
 
 #### 1 - Introduction
-- Hello World ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/1_Introduction/helloworld.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/1_Introduction/helloworld.py))
-- Basic Operations ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/1_Introduction/basic_operations.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/1_Introduction/basic_operations.py))
+- **Hello World** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/1_Introduction/helloworld.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/1_Introduction/helloworld.py)). Very simple example to learn how to print "hello world" using TensorFlow.
+- **Basic Operations** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/1_Introduction/basic_operations.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/1_Introduction/basic_operations.py)). A simple example that cover TensorFlow basic operations.
 
 #### 2 - Basic Models
-- Nearest Neighbor ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/nearest_neighbor.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/nearest_neighbor.py))
-- Linear Regression ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/linear_regression.py))
-- Logistic Regression ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/logistic_regression.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/logistic_regression.py))
+- **Linear Regression** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/linear_regression.py)). Implement a Linear Regression with TensorFlow.
+- **Logistic Regression** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/logistic_regression.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/logistic_regression.py)). Implement a Logistic Regression with TensorFlow.
+- **Nearest Neighbor** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/nearest_neighbor.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/nearest_neighbor.py)). Implement Nearest Neighbor algorithm with TensorFlow.
+- **K-Means** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/kmeans.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/kmeans.py)). Build a K-Means classifier with TensorFlow.
+- **Random Forest** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/random_forest.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/random_forest.py)). Build a Random Forest classifier with TensorFlow.
 
 #### 3 - Neural Networks
-- Multilayer Perceptron ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/multilayer_perceptron.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/multilayer_perceptron.py))
-- Convolutional Neural Network ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/convolutional_network.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/convolutional_network.py))
-- Recurrent Neural Network (LSTM) ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/recurrent_network.py))
-- Bidirectional Recurrent Neural Network (LSTM) ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/bidirectional_rnn.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/bidirectional_rnn.py))
-- Dynamic Recurrent Neural Network (LSTM) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py))
-- AutoEncoder ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/autoencoder.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/autoencoder.py))
+##### Supervised
+
+- **Simple Neural Network** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/neural_network_raw.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/neural_network_raw.py)). Build a simple neural network (a.k.a Multi-layer Perceptron) to classify MNIST digits dataset. Raw TensorFlow implementation.
+- **Simple Neural Network (tf.layers/estimator api)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/neural_network.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/neural_network.py)). Use TensorFlow 'layers' and 'estimator' API to build a simple neural network (a.k.a Multi-layer Perceptron) to classify MNIST digits dataset.
+- **Convolutional Neural Network** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/convolutional_network_raw.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/convolutional_network_raw.py)). Build a convolutional neural network to classify MNIST digits dataset. Raw TensorFlow implementation.
+- **Convolutional Neural Network (tf.layers/estimator api)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/convolutional_network.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/convolutional_network.py)). Use TensorFlow 'layers' and 'estimator' API to build a convolutional neural network to classify MNIST digits dataset.
+- **Recurrent Neural Network (LSTM)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/recurrent_network.py)). Build a recurrent neural network (LSTM) to classify MNIST digits dataset.
+- **Bi-directional Recurrent Neural Network (LSTM)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/bidirectional_rnn.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/bidirectional_rnn.py)). Build a bi-directional recurrent neural network (LSTM) to classify MNIST digits dataset.
+- **Dynamic Recurrent Neural Network (LSTM)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/dynamic_rnn.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py)). Build a recurrent neural network (LSTM) that performs dynamic calculation to classify sequences of different length.
+
+##### Unsupervised
+- **Auto-Encoder** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/autoencoder.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/autoencoder.py)). Build an auto-encoder to encode an image to a lower dimension and re-contruct it.
+- **Variational Auto-Encoder** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/Variational_autoencoder.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/Variational_autoencoder.py)). Build a variational auto-encoder (VAE), to encode and generate images from noise.
+- **GAN (Generative Adversarial Networks)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/gan.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/gan.py)). Build a Generative Adversarial Network (GAN) to generate images from noise.
+- **DCGAN (Deep Convolutional Generative Adversarial Networks)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/dcgan.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dcgan.py)). Build a Deep Convolutional Generative Adversarial Network (DCGAN) to generate images from noise.
 
 #### 4 - Utilities
-- Save and Restore a model ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/4_Utils/save_restore_model.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/4_Utils/save_restore_model.py))
-- Tensorboard - Graph and loss visualization ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/4_Utils/tensorboard_basic.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/4_Utils/tensorboard_basic.py))
-- Tensorboard - Advanced visualization
-([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/4_Utils/tensorboard_advanced.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/4_Utils/tensorboard_advanced.py))
+- **Save and Restore a model** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/4_Utils/save_restore_model.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/4_Utils/save_restore_model.py)). Save and Restore a model with TensorFlow.
+- **Tensorboard - Graph and loss visualization** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/4_Utils/tensorboard_basic.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/4_Utils/tensorboard_basic.py)). Use Tensorboard to visualize the computation Graph and plot the loss.
+- **Tensorboard - Advanced visualization** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/4_Utils/tensorboard_advanced.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/4_Utils/tensorboard_advanced.py)). Going deeper into Tensorboard; visualize the variables, gradients, and more...
+
+#### 5 - Data Management
+- **Build an image dataset** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/5_DataManagement/build_an_image_dataset.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/5_DataManagement/build_an_image_dataset.py)). Build your own images dataset with TensorFlow data queues, from image folders or a dataset file.
+- **TensorFlow Dataset API** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/5_DataManagement/tensorflow_dataset_api.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/5_DataManagement/tensorflow_dataset_api.py)). Introducing TensorFlow Dataset API for optimizing the input data pipeline.
 
-#### 5 - Multi GPU
-- Basic Operations on multi-GPU ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/5_MultiGPU/multigpu_basics.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/5_MultiGPU/multigpu_basics.py))
+#### 6 - Multi GPU
+- **Basic Operations on multi-GPU** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/6_MultiGPU/multigpu_basics.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/5_MultiGPU/multigpu_basics.py)). A simple example to introduce multi-GPU in TensorFlow.
+- **Train a Neural Network on multi-GPU** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/6_MultiGPU/multigpu_basics.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/5_MultiGPU/multigpu_cnn.py)). A clear and simple TensorFlow implementation to train a convolutional neural network on multiple GPUs.
 
 ## Dataset
-Some examples require MNIST dataset for training and testing. Don't worry, this dataset will automatically be downloaded when running examples (with input_data.py).
+Some examples require MNIST dataset for training and testing. Don't worry, this dataset will automatically be downloaded when running examples.
 MNIST is a database of handwritten digits, for a quick description of that dataset, you can check [this notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/0_Prerequisite/mnist_dataset_intro.ipynb).
 
 Official Website: [http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/)
 
+## Installation
+
+To download all the examples, simply clone this repository:
+```
+git clone https://github.com/aymericdamien/TensorFlow-Examples
+```
+
+To run them, you also need the latest version of TensorFlow. To install it:
+```
+pip install tensorflow
+```
+
+or (if you want GPU support):
+```
+pip install tensorflow_gpu
+```
+
+For more details about TensorFlow installation, you can check [TensorFlow Installation Guide](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/get_started/os_setup.md)
+
 ## More Examples
 The following examples are coming from [TFLearn](https://github.com/tflearn/tflearn), a library that provides a simplified interface for TensorFlow. You can have a look, there are many [examples](https://github.com/tflearn/tflearn/tree/master/examples) and [pre-built operations and layers](http://tflearn.org/doc_index/#api).
 
 ### Tutorials
 - [TFLearn Quickstart](https://github.com/tflearn/tflearn/blob/master/tutorials/intro/quickstart.md). Learn the basics of TFLearn through a concrete machine learning task. Build and train a deep neural network classifier.
 
-### Basics
-- [Linear Regression](https://github.com/tflearn/tflearn/blob/master/examples/basics/linear_regression.py). Implement a linear regression using TFLearn.
-- [Logical Operators](https://github.com/tflearn/tflearn/blob/master/examples/basics/logical.py). Implement logical operators with TFLearn (also includes a usage of 'merge').
-- [Weights Persistence](https://github.com/tflearn/tflearn/blob/master/examples/basics/weights_persistence.py). Save and Restore a model.
-- [Fine-Tuning](https://github.com/tflearn/tflearn/blob/master/examples/basics/finetuning.py). Fine-Tune a pre-trained model on a new task.
-- [Using HDF5](https://github.com/tflearn/tflearn/blob/master/examples/basics/use_hdf5.py). Use HDF5 to handle large datasets.
-- [Using DASK](https://github.com/tflearn/tflearn/blob/master/examples/basics/use_dask.py). Use DASK to handle large datasets.
-
-### Computer Vision
-- [Multi-layer perceptron](https://github.com/tflearn/tflearn/blob/master/examples/images/dnn.py). A multi-layer perceptron implementation for MNIST classification task.
-- [Convolutional Network (MNIST)](https://github.com/tflearn/tflearn/blob/master/examples/images/convnet_mnist.py). A Convolutional neural network implementation for classifying MNIST dataset.
-- [Convolutional Network (CIFAR-10)](https://github.com/tflearn/tflearn/blob/master/examples/images/convnet_cifar10.py). A Convolutional neural network implementation for classifying CIFAR-10 dataset.
-- [Network in Network](https://github.com/tflearn/tflearn/blob/master/examples/images/network_in_network.py). 'Network in Network' implementation for classifying CIFAR-10 dataset.
-- [Alexnet](https://github.com/tflearn/tflearn/blob/master/examples/images/alexnet.py). Apply Alexnet to Oxford Flowers 17 classification task.
-- [VGGNet](https://github.com/tflearn/tflearn/blob/master/examples/images/vgg_network.py). Apply VGG Network to Oxford Flowers 17 classification task.
-- [VGGNet Finetuning (Fast Training)](https://github.com/tflearn/tflearn/blob/master/examples/images/vgg_network_finetuning.py). Use a pre-trained VGG Network and retrain it on your own data, for fast training.
-- [RNN Pixels](https://github.com/tflearn/tflearn/blob/master/examples/images/rnn_pixels.py). Use RNN (over sequence of pixels) to classify images.
-- [Highway Network](https://github.com/tflearn/tflearn/blob/master/examples/images/highway_dnn.py). Highway Network implementation for classifying MNIST dataset.
-- [Highway Convolutional Network](https://github.com/tflearn/tflearn/blob/master/examples/images/convnet_highway_mnist.py). Highway Convolutional Network implementation for classifying MNIST dataset.
-- [Residual Network (MNIST)](https://github.com/tflearn/tflearn/blob/master/examples/images/residual_network_mnist.py). A bottleneck residual network applied to MNIST classification task.
-- [Residual Network (CIFAR-10)](https://github.com/tflearn/tflearn/blob/master/examples/images/residual_network_cifar10.py). A residual network applied to CIFAR-10 classification task.
-- [Google Inception (v3)](https://github.com/tflearn/tflearn/blob/master/examples/images/googlenet.py). Google's Inception v3 network applied to Oxford Flowers 17 classification task.
-- [Auto Encoder](https://github.com/tflearn/tflearn/blob/master/examples/images/autoencoder.py). An auto encoder applied to MNIST handwritten digits.
-
-### Natural Language Processing
-- [Recurrent Neural Network (LSTM)](https://github.com/tflearn/tflearn/blob/master/examples/nlp/lstm.py). Apply an LSTM to IMDB sentiment dataset classification task.
-- [Bi-Directional RNN (LSTM)](https://github.com/tflearn/tflearn/blob/master/examples/nlp/bidirectional_lstm.py). Apply a bi-directional LSTM to IMDB sentiment dataset classification task.
-- [Dynamic RNN (LSTM)](https://github.com/tflearn/tflearn/blob/master/examples/nlp/dynamic_lstm.py). Apply a dynamic LSTM to classify variable length text from IMDB dataset.
-- [City Name Generation](https://github.com/tflearn/tflearn/blob/master/examples/nlp/lstm_generator_cityname.py). Generates new US-cities name, using LSTM network.
-- [Shakespeare Scripts Generation](https://github.com/tflearn/tflearn/blob/master/examples/nlp/lstm_generator_shakespeare.py). Generates new Shakespeare scripts, using LSTM network.
-- [Seq2seq](https://github.com/tflearn/tflearn/blob/master/examples/nlp/seq2seq_example.py). Pedagogical example of seq2seq reccurent network. See [this repo](https://github.com/ichuang/tflearn_seq2seq) for full instructions.
-- [CNN Seq](https://github.com/tflearn/tflearn/blob/master/examples/nlp/cnn_sentence_classification.py). Apply a 1-D convolutional network to classify sequence of words from IMDB sentiment dataset.
-
-### Reinforcement Learning
-- [Atari Pacman 1-step Q-Learning](https://github.com/tflearn/tflearn/blob/master/examples/reinforcement_learning/atari_1step_qlearning.py). Teach a machine to play Atari games (Pacman by default) using 1-step Q-learning.
-
-### Others
-- [Recommender - Wide & Deep Network](https://github.com/tflearn/tflearn/blob/master/examples/others/recommender_wide_and_deep.py). Pedagogical example of wide & deep networks for recommender systems.
-
-### Notebooks
-- [Spiral Classification Problem](https://github.com/tflearn/tflearn/blob/master/examples/notebooks/spiral.ipynb). TFLearn implementation of spiral classification problem from Stanford CS231n.
-
-### Extending TensorFlow
-- [Layers](https://github.com/tflearn/tflearn/blob/master/examples/extending_tensorflow/layers.py). Use TFLearn layers along with TensorFlow.
-- [Trainer](https://github.com/tflearn/tflearn/blob/master/examples/extending_tensorflow/trainer.py). Use TFLearn trainer class to train any TensorFlow graph.
-- [Built-in Ops](https://github.com/tflearn/tflearn/blob/master/examples/extending_tensorflow/builtin_ops.py). Use TFLearn built-in operations along with TensorFlow.
-- [Summaries](https://github.com/tflearn/tflearn/blob/master/examples/extending_tensorflow/summaries.py). Use TFLearn summarizers along with TensorFlow.
-- [Variables](https://github.com/tflearn/tflearn/blob/master/examples/extending_tensorflow/variables.py). Use TFLearn variables along with TensorFlow.
-
-
-## Dependencies
-```
-tensorflow 1.0alpha
-numpy
-matplotlib
-cuda
-tflearn (if using tflearn examples)
-```
-For more details about TensorFlow installation, you can check [TensorFlow Installation Guide](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/get_started/os_setup.md)
+### Examples
+- [TFLearn Examples](https://github.com/tflearn/tflearn/blob/master/examples). A large collection of examples using TFLearn.

examples/2_BasicModels/kmeans.py

@@ -0,0 +1,86 @@
+""" K-Means.
+
+Implement K-Means algorithm with TensorFlow, and apply it to classify
+handwritten digit images. This example is using the MNIST database of
+handwritten digits as training samples (http://yann.lecun.com/exdb/mnist/).
+
+Note: This example requires TensorFlow v1.1.0 or over.
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+
+from __future__ import print_function
+
+import numpy as np
+import tensorflow as tf
+from tensorflow.contrib.factorization import KMeans
+
+# Ignore all GPUs, tf random forest does not benefit from it.
+import os
+os.environ["CUDA_VISIBLE_DEVICES"] = ""
+
+# Import MNIST data
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
+full_data_x = mnist.train.images
+
+# Parameters
+num_steps = 50 # Total steps to train
+batch_size = 1024 # The number of samples per batch
+k = 25 # The number of clusters
+num_classes = 10 # The 10 digits
+num_features = 784 # Each image is 28x28 pixels
+
+# Input images
+X = tf.placeholder(tf.float32, shape=[None, num_features])
+# Labels (for assigning a label to a centroid and testing)
+Y = tf.placeholder(tf.float32, shape=[None, num_classes])
+
+# K-Means Parameters
+kmeans = KMeans(inputs=X, num_clusters=k, distance_metric='cosine',
+                use_mini_batch=True)
+
+# Build KMeans graph
+(all_scores, cluster_idx, scores, cluster_centers_initialized, init_op,
+train_op) = kmeans.training_graph()
+cluster_idx = cluster_idx[0] # fix for cluster_idx being a tuple
+avg_distance = tf.reduce_mean(scores)
+
+# Initialize the variables (i.e. assign their default value)
+init_vars = tf.global_variables_initializer()
+
+# Start TensorFlow session
+sess = tf.Session()
+
+# Run the initializer
+sess.run(init_vars, feed_dict={X: full_data_x})
+sess.run(init_op, feed_dict={X: full_data_x})
+
+# Training
+for i in range(1, num_steps + 1):
+    _, d, idx = sess.run([train_op, avg_distance, cluster_idx],
+                         feed_dict={X: full_data_x})
+    if i % 10 == 0 or i == 1:
+        print("Step %i, Avg Distance: %f" % (i, d))
+
+# Assign a label to each centroid
+# Count total number of labels per centroid, using the label of each training
+# sample to their closest centroid (given by 'idx')
+counts = np.zeros(shape=(k, num_classes))
+for i in range(len(idx)):
+    counts[idx[i]] += mnist.train.labels[i]
+# Assign the most frequent label to the centroid
+labels_map = [np.argmax(c) for c in counts]
+labels_map = tf.convert_to_tensor(labels_map)
+
+# Evaluation ops
+# Lookup: centroid_id -> label
+cluster_label = tf.nn.embedding_lookup(labels_map, cluster_idx)
+# Compute accuracy
+correct_prediction = tf.equal(cluster_label, tf.cast(tf.argmax(Y, 1), tf.int32))
+accuracy_op = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
+
+# Test Model
+test_x, test_y = mnist.test.images, mnist.test.labels
+print("Test Accuracy:", sess.run(accuracy_op, feed_dict={X: test_x, Y: test_y}))

examples/2_BasicModels/linear_regression.py

@@ -41,11 +41,13 @@ cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
 #  Note, minimize() knows to modify W and b because Variable objects are trainable=True by default
 optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
-# Launch the graph
+# Start training
 with tf.Session() as sess:
+
+    # Run the initializer
     sess.run(init)
 
     # Fit all training data

examples/2_BasicModels/logistic_regression.py

@@ -37,11 +37,13 @@ cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
 # Gradient Descent
 optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
-# Launch the graph
+# Start training
 with tf.Session() as sess:
+
+    # Run the initializer
     sess.run(init)
 
     # Training cycle

examples/2_BasicModels/nearest_neighbor.py

@@ -32,11 +32,13 @@ pred = tf.arg_min(distance, 0)
 
 accuracy = 0.
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
-# Launch the graph
+# Start training
 with tf.Session() as sess:
+
+    # Run the initializer
     sess.run(init)
 
     # loop over test data

examples/2_BasicModels/random_forest.py

@@ -0,0 +1,75 @@
+""" Random Forest.
+
+Implement Random Forest algorithm with TensorFlow, and apply it to classify 
+handwritten digit images. This example is using the MNIST database of 
+handwritten digits as training samples (http://yann.lecun.com/exdb/mnist/).
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+
+from __future__ import print_function
+
+import tensorflow as tf
+from tensorflow.contrib.tensor_forest.python import tensor_forest
+
+# Ignore all GPUs, tf random forest does not benefit from it.
+import os
+os.environ["CUDA_VISIBLE_DEVICES"] = ""
+
+# Import MNIST data
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=False)
+
+# Parameters
+num_steps = 500 # Total steps to train
+batch_size = 1024 # The number of samples per batch
+num_classes = 10 # The 10 digits
+num_features = 784 # Each image is 28x28 pixels
+num_trees = 10
+max_nodes = 1000
+
+# Input and Target data
+X = tf.placeholder(tf.float32, shape=[None, num_features])
+# For random forest, labels must be integers (the class id)
+Y = tf.placeholder(tf.int32, shape=[None])
+
+# Random Forest Parameters
+hparams = tensor_forest.ForestHParams(num_classes=num_classes,
+                                      num_features=num_features,
+                                      num_trees=num_trees,
+                                      max_nodes=max_nodes).fill()
+
+# Build the Random Forest
+forest_graph = tensor_forest.RandomForestGraphs(hparams)
+# Get training graph and loss
+train_op = forest_graph.training_graph(X, Y)
+loss_op = forest_graph.training_loss(X, Y)
+
+# Measure the accuracy
+infer_op = forest_graph.inference_graph(X)
+correct_prediction = tf.equal(tf.argmax(infer_op, 1), tf.cast(Y, tf.int64))
+accuracy_op = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
+
+# Initialize the variables (i.e. assign their default value)
+init_vars = tf.global_variables_initializer()
+
+# Start TensorFlow session
+sess = tf.Session()
+
+# Run the initializer
+sess.run(init_vars)
+
+# Training
+for i in range(1, num_steps + 1):
+    # Prepare Data
+    # Get the next batch of MNIST data (only images are needed, not labels)
+    batch_x, batch_y = mnist.train.next_batch(batch_size)
+    _, l = sess.run([train_op, loss_op], feed_dict={X: batch_x, Y: batch_y})
+    if i % 50 == 0 or i == 1:
+        acc = sess.run(accuracy_op, feed_dict={X: batch_x, Y: batch_y})
+        print('Step %i, Loss: %f, Acc: %f' % (i, l, acc))
+
+# Test Model
+test_x, test_y = mnist.test.images, mnist.test.labels
+print("Test Accuracy:", sess.run(accuracy_op, feed_dict={X: test_x, Y: test_y}))

examples/3_NeuralNetworks/autoencoder.py

@@ -1,13 +1,18 @@
-# -*- coding: utf-8 -*-
-
 """ Auto Encoder Example.
-Using an auto encoder on MNIST handwritten digits.
+
+Build a 2 layers auto-encoder with TensorFlow to compress images to a
+lower latent space and then reconstruct them.
+
 References:
     Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based
     learning applied to document recognition." Proceedings of the IEEE,
     86(11):2278-2324, November 1998.
+
 Links:
     [MNIST Dataset] http://yann.lecun.com/exdb/mnist/
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
 """
 from __future__ import division, print_function, absolute_import
 
@@ -17,37 +22,37 @@ import matplotlib.pyplot as plt
 
 # Import MNIST data
 from tensorflow.examples.tutorials.mnist import input_data
-mnist = input_data.read_data_sets("MNIST_data", one_hot=True)
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
 
-# Parameters
+# Training Parameters
 learning_rate = 0.01
-training_epochs = 20
+num_steps = 30000
 batch_size = 256
-display_step = 1
+
+display_step = 1000
 examples_to_show = 10
 
 # Network Parameters
-n_hidden_1 = 256 # 1st layer num features
-n_hidden_2 = 128 # 2nd layer num features
-n_input = 784 # MNIST data input (img shape: 28*28)
+num_hidden_1 = 256 # 1st layer num features
+num_hidden_2 = 128 # 2nd layer num features (the latent dim)
+num_input = 784 # MNIST data input (img shape: 28*28)
 
 # tf Graph input (only pictures)
-X = tf.placeholder("float", [None, n_input])
+X = tf.placeholder("float", [None, num_input])
 
 weights = {
-    'encoder_h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
-    'encoder_h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
-    'decoder_h1': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_1])),
-    'decoder_h2': tf.Variable(tf.random_normal([n_hidden_1, n_input])),
+    'encoder_h1': tf.Variable(tf.random_normal([num_input, num_hidden_1])),
+    'encoder_h2': tf.Variable(tf.random_normal([num_hidden_1, num_hidden_2])),
+    'decoder_h1': tf.Variable(tf.random_normal([num_hidden_2, num_hidden_1])),
+    'decoder_h2': tf.Variable(tf.random_normal([num_hidden_1, num_input])),
 }
 biases = {
-    'encoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
-    'encoder_b2': tf.Variable(tf.random_normal([n_hidden_2])),
-    'decoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
-    'decoder_b2': tf.Variable(tf.random_normal([n_input])),
+    'encoder_b1': tf.Variable(tf.random_normal([num_hidden_1])),
+    'encoder_b2': tf.Variable(tf.random_normal([num_hidden_2])),
+    'decoder_b1': tf.Variable(tf.random_normal([num_hidden_1])),
+    'decoder_b2': tf.Variable(tf.random_normal([num_input])),
 }
 
-
 # Building the encoder
 def encoder(x):
     # Encoder Hidden layer with sigmoid activation #1
@@ -79,38 +84,59 @@ y_pred = decoder_op
 y_true = X
 
 # Define loss and optimizer, minimize the squared error
-cost = tf.reduce_mean(tf.pow(y_true - y_pred, 2))
-optimizer = tf.train.RMSPropOptimizer(learning_rate).minimize(cost)
+loss = tf.reduce_mean(tf.pow(y_true - y_pred, 2))
+optimizer = tf.train.RMSPropOptimizer(learning_rate).minimize(loss)
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
-# Launch the graph
+# Start Training
+# Start a new TF session
 with tf.Session() as sess:
+
+    # Run the initializer
     sess.run(init)
-    total_batch = int(mnist.train.num_examples/batch_size)
-    # Training cycle
-    for epoch in range(training_epochs):
-        # Loop over all batches
-        for i in range(total_batch):
-            batch_xs, batch_ys = mnist.train.next_batch(batch_size)
-            # Run optimization op (backprop) and cost op (to get loss value)
-            _, c = sess.run([optimizer, cost], feed_dict={X: batch_xs})
-        # Display logs per epoch step
-        if epoch % display_step == 0:
-            print("Epoch:", '%04d' % (epoch+1),
-                  "cost=", "{:.9f}".format(c))
-
-    print("Optimization Finished!")
-
-    # Applying encode and decode over test set
-    encode_decode = sess.run(
-        y_pred, feed_dict={X: mnist.test.images[:examples_to_show]})
-    # Compare original images with their reconstructions
-    f, a = plt.subplots(2, 10, figsize=(10, 2))
-    for i in range(examples_to_show):
-        a[0][i].imshow(np.reshape(mnist.test.images[i], (28, 28)))
-        a[1][i].imshow(np.reshape(encode_decode[i], (28, 28)))
-    f.show()
-    plt.draw()
-    plt.waitforbuttonpress()
+
+    # Training
+    for i in range(1, num_steps+1):
+        # Prepare Data
+        # Get the next batch of MNIST data (only images are needed, not labels)
+        batch_x, _ = mnist.train.next_batch(batch_size)
+
+        # Run optimization op (backprop) and cost op (to get loss value)
+        _, l = sess.run([optimizer, loss], feed_dict={X: batch_x})
+        # Display logs per step
+        if i % display_step == 0 or i == 1:
+            print('Step %i: Minibatch Loss: %f' % (i, l))
+
+    # Testing
+    # Encode and decode images from test set and visualize their reconstruction.
+    n = 4
+    canvas_orig = np.empty((28 * n, 28 * n))
+    canvas_recon = np.empty((28 * n, 28 * n))
+    for i in range(n):
+        # MNIST test set
+        batch_x, _ = mnist.test.next_batch(n)
+        # Encode and decode the digit image
+        g = sess.run(decoder_op, feed_dict={X: batch_x})
+
+        # Display original images
+        for j in range(n):
+            # Draw the original digits
+            canvas_orig[i * 28:(i + 1) * 28, j * 28:(j + 1) * 28] = \
+                batch_x[j].reshape([28, 28])
+        # Display reconstructed images
+        for j in range(n):
+            # Draw the reconstructed digits
+            canvas_recon[i * 28:(i + 1) * 28, j * 28:(j + 1) * 28] = \
+                g[j].reshape([28, 28])
+
+    print("Original Images")
+    plt.figure(figsize=(n, n))
+    plt.imshow(canvas_orig, origin="upper", cmap="gray")
+    plt.show()
+
+    print("Reconstructed Images")
+    plt.figure(figsize=(n, n))
+    plt.imshow(canvas_recon, origin="upper", cmap="gray")
+    plt.show()

examples/3_NeuralNetworks/bidirectional_rnn.py

@@ -1,11 +1,16 @@
-'''
-A Bidirectional Recurrent Neural Network (LSTM) implementation example using TensorFlow library.
-This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)
-Long Short Term Memory paper: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf
+""" Bi-directional Recurrent Neural Network.
+
+A Bi-directional Recurrent Neural Network (LSTM) implementation example using 
+TensorFlow library. This example is using the MNIST database of handwritten 
+digits (http://yann.lecun.com/exdb/mnist/)
+
+Links:
+    [Long Short Term Memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf)
+    [MNIST Dataset](http://yann.lecun.com/exdb/mnist/).
 
 Author: Aymeric Damien
 Project: https://github.com/aymericdamien/TensorFlow-Examples/
-'''
+"""
 
 from __future__ import print_function
 
@@ -23,46 +28,46 @@ every image row as a sequence of pixels. Because MNIST image shape is 28*28px,
 we will then handle 28 sequences of 28 steps for every sample.
 '''
 
-# Parameters
+# Training Parameters
 learning_rate = 0.001
-training_iters = 100000
+training_steps = 10000
 batch_size = 128
-display_step = 10
+display_step = 200
 
 # Network Parameters
-n_input = 28 # MNIST data input (img shape: 28*28)
-n_steps = 28 # timesteps
-n_hidden = 128 # hidden layer num of features
-n_classes = 10 # MNIST total classes (0-9 digits)
+num_input = 28 # MNIST data input (img shape: 28*28)
+timesteps = 28 # timesteps
+num_hidden = 128 # hidden layer num of features
+num_classes = 10 # MNIST total classes (0-9 digits)
 
 # tf Graph input
-x = tf.placeholder("float", [None, n_steps, n_input])
-y = tf.placeholder("float", [None, n_classes])
+X = tf.placeholder("float", [None, timesteps, num_input])
+Y = tf.placeholder("float", [None, num_classes])
 
 # Define weights
 weights = {
     # Hidden layer weights => 2*n_hidden because of forward + backward cells
-    'out': tf.Variable(tf.random_normal([2*n_hidden, n_classes]))
+    'out': tf.Variable(tf.random_normal([2*num_hidden, num_classes]))
 }
 biases = {
-    'out': tf.Variable(tf.random_normal([n_classes]))
+    'out': tf.Variable(tf.random_normal([num_classes]))
 }
 
 
 def BiRNN(x, weights, biases):
 
-    # Prepare data shape to match `bidirectional_rnn` function requirements
-    # Current data input shape: (batch_size, n_steps, n_input)
-    # Required shape: 'n_steps' tensors list of shape (batch_size, n_input)
+    # Prepare data shape to match `rnn` function requirements
+    # Current data input shape: (batch_size, timesteps, n_input)
+    # Required shape: 'timesteps' tensors list of shape (batch_size, num_input)
 
-    # Unstack to get a list of 'n_steps' tensors of shape (batch_size, n_input)
-    x = tf.unstack(x, n_steps, 1)
+    # Unstack to get a list of 'timesteps' tensors of shape (batch_size, num_input)
+    x = tf.unstack(x, timesteps, 1)
 
     # Define lstm cells with tensorflow
     # Forward direction cell
-    lstm_fw_cell = rnn.BasicLSTMCell(n_hidden, forget_bias=1.0)
+    lstm_fw_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)
     # Backward direction cell
-    lstm_bw_cell = rnn.BasicLSTMCell(n_hidden, forget_bias=1.0)
+    lstm_bw_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)
 
     # Get lstm cell output
     try:
@@ -75,44 +80,47 @@ def BiRNN(x, weights, biases):
     # Linear activation, using rnn inner loop last output
     return tf.matmul(outputs[-1], weights['out']) + biases['out']
 
-pred = BiRNN(x, weights, biases)
+logits = BiRNN(X, weights, biases)
+prediction = tf.nn.softmax(logits)
 
 # Define loss and optimizer
-cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
-optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
+loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
+    logits=logits, labels=Y))
+optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
+train_op = optimizer.minimize(loss_op)
 
-# Evaluate model
-correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
+# Evaluate model (with test logits, for dropout to be disabled)
+correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))
 accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
-# Launch the graph
+# Start training
 with tf.Session() as sess:
+
+    # Run the initializer
     sess.run(init)
-    step = 1
-    # Keep training until reach max iterations
-    while step * batch_size < training_iters:
+
+    for step in range(1, training_steps+1):
         batch_x, batch_y = mnist.train.next_batch(batch_size)
         # Reshape data to get 28 seq of 28 elements
-        batch_x = batch_x.reshape((batch_size, n_steps, n_input))
+        batch_x = batch_x.reshape((batch_size, timesteps, num_input))
         # Run optimization op (backprop)
-        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
-        if step % display_step == 0:
-            # Calculate batch accuracy
-            acc = sess.run(accuracy, feed_dict={x: batch_x, y: batch_y})
-            # Calculate batch loss
-            loss = sess.run(cost, feed_dict={x: batch_x, y: batch_y})
-            print("Iter " + str(step*batch_size) + ", Minibatch Loss= " + \
-                  "{:.6f}".format(loss) + ", Training Accuracy= " + \
-                  "{:.5f}".format(acc))
-        step += 1
+        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
+        if step % display_step == 0 or step == 1:
+            # Calculate batch loss and accuracy
+            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,
+                                                                 Y: batch_y})
+            print("Step " + str(step) + ", Minibatch Loss= " + \
+                  "{:.4f}".format(loss) + ", Training Accuracy= " + \
+                  "{:.3f}".format(acc))
+
     print("Optimization Finished!")
 
     # Calculate accuracy for 128 mnist test images
     test_len = 128
-    test_data = mnist.test.images[:test_len].reshape((-1, n_steps, n_input))
+    test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))
     test_label = mnist.test.labels[:test_len]
     print("Testing Accuracy:", \
-        sess.run(accuracy, feed_dict={x: test_data, y: test_label}))
+        sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))

examples/3_NeuralNetworks/convolutional_network.py

@@ -1,134 +1,125 @@
-'''
-A Convolutional Network implementation example using TensorFlow library.
+""" Convolutional Neural Network.
+
+Build and train a convolutional neural network with TensorFlow.
 This example is using the MNIST database of handwritten digits
 (http://yann.lecun.com/exdb/mnist/)
 
+This example is using TensorFlow layers API, see 'convolutional_network_raw' 
+example for a raw implementation with variables.
+
 Author: Aymeric Damien
 Project: https://github.com/aymericdamien/TensorFlow-Examples/
-'''
-
-from __future__ import print_function
-
-import tensorflow as tf
+"""
+from __future__ import division, print_function, absolute_import
 
 # Import MNIST data
 from tensorflow.examples.tutorials.mnist import input_data
-mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=False)
 
-# Parameters
+import tensorflow as tf
+
+# Training Parameters
 learning_rate = 0.001
-training_iters = 200000
+num_steps = 2000
 batch_size = 128
-display_step = 10
 
 # Network Parameters
-n_input = 784 # MNIST data input (img shape: 28*28)
-n_classes = 10 # MNIST total classes (0-9 digits)
+num_input = 784 # MNIST data input (img shape: 28*28)
+num_classes = 10 # MNIST total classes (0-9 digits)
 dropout = 0.75 # Dropout, probability to keep units
 
-# tf Graph input
-x = tf.placeholder(tf.float32, [None, n_input])
-y = tf.placeholder(tf.float32, [None, n_classes])
-keep_prob = tf.placeholder(tf.float32) #dropout (keep probability)
-
 
-# Create some wrappers for simplicity
-def conv2d(x, W, b, strides=1):
-    # Conv2D wrapper, with bias and relu activation
-    x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')
-    x = tf.nn.bias_add(x, b)
-    return tf.nn.relu(x)
+# Create the neural network
+def conv_net(x_dict, n_classes, dropout, reuse, is_training):
+    # Define a scope for reusing the variables
+    with tf.variable_scope('ConvNet', reuse=reuse):
+        # TF Estimator input is a dict, in case of multiple inputs
+        x = x_dict['images']
 
+        # MNIST data input is a 1-D vector of 784 features (28*28 pixels)
+        # Reshape to match picture format [Height x Width x Channel]
+        # Tensor input become 4-D: [Batch Size, Height, Width, Channel]
+        x = tf.reshape(x, shape=[-1, 28, 28, 1])
 
-def maxpool2d(x, k=2):
-    # MaxPool2D wrapper
-    return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],
-                          padding='SAME')
+        # Convolution Layer with 32 filters and a kernel size of 5
+        conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)
+        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2
+        conv1 = tf.layers.max_pooling2d(conv1, 2, 2)
 
+        # Convolution Layer with 32 filters and a kernel size of 5
+        conv2 = tf.layers.conv2d(conv1, 64, 3, activation=tf.nn.relu)
+        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2
+        conv2 = tf.layers.max_pooling2d(conv2, 2, 2)
 
-# Create model
-def conv_net(x, weights, biases, dropout):
-    # Reshape input picture
-    x = tf.reshape(x, shape=[-1, 28, 28, 1])
+        # Flatten the data to a 1-D vector for the fully connected layer
+        fc1 = tf.contrib.layers.flatten(conv2)
 
-    # Convolution Layer
-    conv1 = conv2d(x, weights['wc1'], biases['bc1'])
-    # Max Pooling (down-sampling)
-    conv1 = maxpool2d(conv1, k=2)
+        # Fully connected layer (in tf contrib folder for now)
+        fc1 = tf.layers.dense(fc1, 1024)
+        # Apply Dropout (if is_training is False, dropout is not applied)
+        fc1 = tf.layers.dropout(fc1, rate=dropout, training=is_training)
 
-    # Convolution Layer
-    conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
-    # Max Pooling (down-sampling)
-    conv2 = maxpool2d(conv2, k=2)
+        # Output layer, class prediction
+        out = tf.layers.dense(fc1, n_classes)
 
-    # Fully connected layer
-    # Reshape conv2 output to fit fully connected layer input
-    fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])
-    fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
-    fc1 = tf.nn.relu(fc1)
-    # Apply Dropout
-    fc1 = tf.nn.dropout(fc1, dropout)
-
-    # Output, class prediction
-    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
     return out
 
-# Store layers weight & bias
-weights = {
-    # 5x5 conv, 1 input, 32 outputs
-    'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32])),
-    # 5x5 conv, 32 inputs, 64 outputs
-    'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])),
-    # fully connected, 7*7*64 inputs, 1024 outputs
-    'wd1': tf.Variable(tf.random_normal([7*7*64, 1024])),
-    # 1024 inputs, 10 outputs (class prediction)
-    'out': tf.Variable(tf.random_normal([1024, n_classes]))
-}
-
-biases = {
-    'bc1': tf.Variable(tf.random_normal([32])),
-    'bc2': tf.Variable(tf.random_normal([64])),
-    'bd1': tf.Variable(tf.random_normal([1024])),
-    'out': tf.Variable(tf.random_normal([n_classes]))
-}
-
-# Construct model
-pred = conv_net(x, weights, biases, keep_prob)
-
-# Define loss and optimizer
-cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
-optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
-
-# Evaluate model
-correct_pred = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
-accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
-
-# Initializing the variables
-init = tf.global_variables_initializer()
-
-# Launch the graph
-with tf.Session() as sess:
-    sess.run(init)
-    step = 1
-    # Keep training until reach max iterations
-    while step * batch_size < training_iters:
-        batch_x, batch_y = mnist.train.next_batch(batch_size)
-        # Run optimization op (backprop)
-        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y,
-                                       keep_prob: dropout})
-        if step % display_step == 0:
-            # Calculate batch loss and accuracy
-            loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x,
-                                                              y: batch_y,
-                                                              keep_prob: 1.})
-            print("Iter " + str(step*batch_size) + ", Minibatch Loss= " + \
-                  "{:.6f}".format(loss) + ", Training Accuracy= " + \
-                  "{:.5f}".format(acc))
-        step += 1
-    print("Optimization Finished!")
-
-    # Calculate accuracy for 256 mnist test images
-    print("Testing Accuracy:", \
-        sess.run(accuracy, feed_dict={x: mnist.test.images[:256],
-                                      y: mnist.test.labels[:256],
-                                      keep_prob: 1.}))
+
+# Define the model function (following TF Estimator Template)
+def model_fn(features, labels, mode):
+    # Build the neural network
+    # Because Dropout have different behavior at training and prediction time, we
+    # need to create 2 distinct computation graphs that still share the same weights.
+    logits_train = conv_net(features, num_classes, dropout, reuse=False,
+                            is_training=True)
+    logits_test = conv_net(features, num_classes, dropout, reuse=True,
+                           is_training=False)
+
+    # Predictions
+    pred_classes = tf.argmax(logits_test, axis=1)
+    pred_probas = tf.nn.softmax(logits_test)
+
+    # If prediction mode, early return
+    if mode == tf.estimator.ModeKeys.PREDICT:
+        return tf.estimator.EstimatorSpec(mode, predictions=pred_classes)
+
+        # Define loss and optimizer
+    loss_op = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
+        logits=logits_train, labels=tf.cast(labels, dtype=tf.int32)))
+    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
+    train_op = optimizer.minimize(loss_op,
+                                  global_step=tf.train.get_global_step())
+
+    # Evaluate the accuracy of the model
+    acc_op = tf.metrics.accuracy(labels=labels, predictions=pred_classes)
+
+    # TF Estimators requires to return a EstimatorSpec, that specify
+    # the different ops for training, evaluating, ...
+    estim_specs = tf.estimator.EstimatorSpec(
+        mode=mode,
+        predictions=pred_classes,
+        loss=loss_op,
+        train_op=train_op,
+        eval_metric_ops={'accuracy': acc_op})
+
+    return estim_specs
+
+# Build the Estimator
+model = tf.estimator.Estimator(model_fn)
+
+# Define the input function for training
+input_fn = tf.estimator.inputs.numpy_input_fn(
+    x={'images': mnist.train.images}, y=mnist.train.labels,
+    batch_size=batch_size, num_epochs=None, shuffle=True)
+# Train the Model
+model.train(input_fn, steps=num_steps)
+
+# Evaluate the Model
+# Define the input function for evaluating
+input_fn = tf.estimator.inputs.numpy_input_fn(
+    x={'images': mnist.test.images}, y=mnist.test.labels,
+    batch_size=batch_size, shuffle=False)
+# Use the Estimator 'evaluate' method
+e = model.evaluate(input_fn)
+
+print("Testing Accuracy:", e['accuracy'])

examples/3_NeuralNetworks/convolutional_network_raw.py

@@ -0,0 +1,141 @@
+""" Convolutional Neural Network.
+
+Build and train a convolutional neural network with TensorFlow.
+This example is using the MNIST database of handwritten digits
+(http://yann.lecun.com/exdb/mnist/)
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+
+from __future__ import division, print_function, absolute_import
+
+import tensorflow as tf
+
+# Import MNIST data
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
+
+# Training Parameters
+learning_rate = 0.001
+num_steps = 200
+batch_size = 128
+display_step = 10
+
+# Network Parameters
+num_input = 784 # MNIST data input (img shape: 28*28)
+num_classes = 10 # MNIST total classes (0-9 digits)
+dropout = 0.75 # Dropout, probability to keep units
+
+# tf Graph input
+X = tf.placeholder(tf.float32, [None, num_input])
+Y = tf.placeholder(tf.float32, [None, num_classes])
+keep_prob = tf.placeholder(tf.float32) # dropout (keep probability)
+
+
+# Create some wrappers for simplicity
+def conv2d(x, W, b, strides=1):
+    # Conv2D wrapper, with bias and relu activation
+    x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')
+    x = tf.nn.bias_add(x, b)
+    return tf.nn.relu(x)
+
+
+def maxpool2d(x, k=2):
+    # MaxPool2D wrapper
+    return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],
+                          padding='SAME')
+
+
+# Create model
+def conv_net(x, weights, biases, dropout):
+    # MNIST data input is a 1-D vector of 784 features (28*28 pixels)
+    # Reshape to match picture format [Height x Width x Channel]
+    # Tensor input become 4-D: [Batch Size, Height, Width, Channel]
+    x = tf.reshape(x, shape=[-1, 28, 28, 1])
+
+    # Convolution Layer
+    conv1 = conv2d(x, weights['wc1'], biases['bc1'])
+    # Max Pooling (down-sampling)
+    conv1 = maxpool2d(conv1, k=2)
+
+    # Convolution Layer
+    conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
+    # Max Pooling (down-sampling)
+    conv2 = maxpool2d(conv2, k=2)
+
+    # Fully connected layer
+    # Reshape conv2 output to fit fully connected layer input
+    fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])
+    fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
+    fc1 = tf.nn.relu(fc1)
+    # Apply Dropout
+    fc1 = tf.nn.dropout(fc1, dropout)
+
+    # Output, class prediction
+    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
+    return out
+
+# Store layers weight & bias
+weights = {
+    # 5x5 conv, 1 input, 32 outputs
+    'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32])),
+    # 5x5 conv, 32 inputs, 64 outputs
+    'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])),
+    # fully connected, 7*7*64 inputs, 1024 outputs
+    'wd1': tf.Variable(tf.random_normal([7*7*64, 1024])),
+    # 1024 inputs, 10 outputs (class prediction)
+    'out': tf.Variable(tf.random_normal([1024, num_classes]))
+}
+
+biases = {
+    'bc1': tf.Variable(tf.random_normal([32])),
+    'bc2': tf.Variable(tf.random_normal([64])),
+    'bd1': tf.Variable(tf.random_normal([1024])),
+    'out': tf.Variable(tf.random_normal([num_classes]))
+}
+
+# Construct model
+logits = conv_net(X, weights, biases, keep_prob)
+prediction = tf.nn.softmax(logits)
+
+# Define loss and optimizer
+loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
+    logits=logits, labels=Y))
+optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
+train_op = optimizer.minimize(loss_op)
+
+
+# Evaluate model
+correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))
+accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
+
+# Initialize the variables (i.e. assign their default value)
+init = tf.global_variables_initializer()
+
+# Start training
+with tf.Session() as sess:
+
+    # Run the initializer
+    sess.run(init)
+
+    for step in range(1, num_steps+1):
+        batch_x, batch_y = mnist.train.next_batch(batch_size)
+        # Run optimization op (backprop)
+        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y, keep_prob: 0.8})
+        if step % display_step == 0 or step == 1:
+            # Calculate batch loss and accuracy
+            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,
+                                                                 Y: batch_y,
+                                                                 keep_prob: 1.0})
+            print("Step " + str(step) + ", Minibatch Loss= " + \
+                  "{:.4f}".format(loss) + ", Training Accuracy= " + \
+                  "{:.3f}".format(acc))
+
+    print("Optimization Finished!")
+
+    # Calculate accuracy for 256 MNIST test images
+    print("Testing Accuracy:", \
+        sess.run(accuracy, feed_dict={X: mnist.test.images[:256],
+                                      Y: mnist.test.labels[:256],
+                                      keep_prob: 1.0}))

examples/3_NeuralNetworks/dcgan.py

@@ -0,0 +1,167 @@
+""" Deep Convolutional Generative Adversarial Network (DCGAN).
+
+Using deep convolutional generative adversarial networks (DCGAN) to generate
+digit images from a noise distribution.
+
+References:
+    - Unsupervised representation learning with deep convolutional generative
+    adversarial networks. A Radford, L Metz, S Chintala. arXiv:1511.06434.
+
+Links:
+    - [DCGAN Paper](https://arxiv.org/abs/1511.06434).
+    - [MNIST Dataset](http://yann.lecun.com/exdb/mnist/).
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+
+from __future__ import division, print_function, absolute_import
+
+import matplotlib.pyplot as plt
+import numpy as np
+import tensorflow as tf
+
+# Import MNIST data
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
+
+# Training Params
+num_steps = 20000
+batch_size = 32
+
+# Network Params
+image_dim = 784 # 28*28 pixels * 1 channel
+gen_hidden_dim = 256
+disc_hidden_dim = 256
+noise_dim = 200 # Noise data points
+
+
+# Generator Network
+# Input: Noise, Output: Image
+def generator(x, reuse=False):
+    with tf.variable_scope('Generator', reuse=reuse):
+        # TensorFlow Layers automatically create variables and calculate their
+        # shape, based on the input.
+        x = tf.layers.dense(x, units=6 * 6 * 128)
+        x = tf.nn.tanh(x)
+        # Reshape to a 4-D array of images: (batch, height, width, channels)
+        # New shape: (batch, 6, 6, 128)
+        x = tf.reshape(x, shape=[-1, 6, 6, 128])
+        # Deconvolution, image shape: (batch, 14, 14, 64)
+        x = tf.layers.conv2d_transpose(x, 64, 4, strides=2)
+        # Deconvolution, image shape: (batch, 28, 28, 1)
+        x = tf.layers.conv2d_transpose(x, 1, 2, strides=2)
+        # Apply sigmoid to clip values between 0 and 1
+        x = tf.nn.sigmoid(x)
+        return x
+
+
+# Discriminator Network
+# Input: Image, Output: Prediction Real/Fake Image
+def discriminator(x, reuse=False):
+    with tf.variable_scope('Discriminator', reuse=reuse):
+        # Typical convolutional neural network to classify images.
+        x = tf.layers.conv2d(x, 64, 5)
+        x = tf.nn.tanh(x)
+        x = tf.layers.average_pooling2d(x, 2, 2)
+        x = tf.layers.conv2d(x, 128, 5)
+        x = tf.nn.tanh(x)
+        x = tf.layers.average_pooling2d(x, 2, 2)
+        x = tf.contrib.layers.flatten(x)
+        x = tf.layers.dense(x, 1024)
+        x = tf.nn.tanh(x)
+        # Output 2 classes: Real and Fake images
+        x = tf.layers.dense(x, 2)
+    return x
+
+# Build Networks
+# Network Inputs
+noise_input = tf.placeholder(tf.float32, shape=[None, noise_dim])
+real_image_input = tf.placeholder(tf.float32, shape=[None, 28, 28, 1])
+
+# Build Generator Network
+gen_sample = generator(noise_input)
+
+# Build 2 Discriminator Networks (one from noise input, one from generated samples)
+disc_real = discriminator(real_image_input)
+disc_fake = discriminator(gen_sample, reuse=True)
+disc_concat = tf.concat([disc_real, disc_fake], axis=0)
+
+# Build the stacked generator/discriminator
+stacked_gan = discriminator(gen_sample, reuse=True)
+
+# Build Targets (real or fake images)
+disc_target = tf.placeholder(tf.int32, shape=[None])
+gen_target = tf.placeholder(tf.int32, shape=[None])
+
+# Build Loss
+disc_loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
+    logits=disc_concat, labels=disc_target))
+gen_loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
+    logits=stacked_gan, labels=gen_target))
+
+# Build Optimizers
+optimizer_gen = tf.train.AdamOptimizer(learning_rate=0.001)
+optimizer_disc = tf.train.AdamOptimizer(learning_rate=0.001)
+
+# Training Variables for each optimizer
+# By default in TensorFlow, all variables are updated by each optimizer, so we
+# need to precise for each one of them the specific variables to update.
+# Generator Network Variables
+gen_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope='Generator')
+# Discriminator Network Variables
+disc_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope='Discriminator')
+
+# Create training operations
+train_gen = optimizer_gen.minimize(gen_loss, var_list=gen_vars)
+train_disc = optimizer_disc.minimize(disc_loss, var_list=disc_vars)
+
+# Initialize the variables (i.e. assign their default value)
+init = tf.global_variables_initializer()
+
+# Start training
+with tf.Session() as sess:
+
+    # Run the initializer
+    sess.run(init)
+
+    for i in range(1, num_steps+1):
+
+        # Prepare Input Data
+        # Get the next batch of MNIST data (only images are needed, not labels)
+        batch_x, _ = mnist.train.next_batch(batch_size)
+        batch_x = np.reshape(batch_x, newshape=[-1, 28, 28, 1])
+        # Generate noise to feed to the generator
+        z = np.random.uniform(-1., 1., size=[batch_size, noise_dim])
+
+        # Prepare Targets (Real image: 1, Fake image: 0)
+        # The first half of data fed to the generator are real images,
+        # the other half are fake images (coming from the generator).
+        batch_disc_y = np.concatenate(
+            [np.ones([batch_size]), np.zeros([batch_size])], axis=0)
+        # Generator tries to fool the discriminator, thus targets are 1.
+        batch_gen_y = np.ones([batch_size])
+
+        # Training
+        feed_dict = {real_image_input: batch_x, noise_input: z,
+                     disc_target: batch_disc_y, gen_target: batch_gen_y}
+        _, _, gl, dl = sess.run([train_gen, train_disc, gen_loss, disc_loss],
+                                feed_dict=feed_dict)
+        if i % 100 == 0 or i == 1:
+            print('Step %i: Generator Loss: %f, Discriminator Loss: %f' % (i, gl, dl))
+
+    # Generate images from noise, using the generator network.
+    f, a = plt.subplots(4, 10, figsize=(10, 4))
+    for i in range(10):
+        # Noise input.
+        z = np.random.uniform(-1., 1., size=[4, noise_dim])
+        g = sess.run(gen_sample, feed_dict={noise_input: z})
+        for j in range(4):
+            # Generate image from noise. Extend to 3 channels for matplot figure.
+            img = np.reshape(np.repeat(g[j][:, :, np.newaxis], 3, axis=2),
+                             newshape=(28, 28, 3))
+            a[j][i].imshow(img)
+
+    f.show()
+    plt.draw()
+    plt.waitforbuttonpress()

examples/3_NeuralNetworks/dynamic_rnn.py

@@ -1,13 +1,16 @@
-'''
-A Dynamic Recurrent Neural Network (LSTM) implementation example using
-TensorFlow library. This example is using a toy dataset to classify linear
-sequences. The generated sequences have variable length.
+""" Dynamic Recurrent Neural Network.
 
-Long Short Term Memory paper: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf
+TensorFlow implementation of a Recurrent Neural Network (LSTM) that performs
+dynamic computation over sequences with variable length. This example is using
+a toy dataset to classify linear sequences. The generated sequences have
+variable length.
+
+Links:
+    [Long Short Term Memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf)
 
 Author: Aymeric Damien
 Project: https://github.com/aymericdamien/TensorFlow-Examples/
-'''
+"""
 
 from __future__ import print_function
 
@@ -81,9 +84,9 @@ class ToySequenceData(object):
 
 # Parameters
 learning_rate = 0.01
-training_iters = 1000000
+training_steps = 10000
 batch_size = 128
-display_step = 10
+display_step = 200
 
 # Network Parameters
 seq_max_len = 20 # Sequence max length
@@ -157,30 +160,31 @@ optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minim
 correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
 accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
-# Launch the graph
+# Start training
 with tf.Session() as sess:
+
+    # Run the initializer
     sess.run(init)
-    step = 1
-    # Keep training until reach max iterations
-    while step * batch_size < training_iters:
+
+    for step in range(1, training_steps + 1):
         batch_x, batch_y, batch_seqlen = trainset.next(batch_size)
         # Run optimization op (backprop)
         sess.run(optimizer, feed_dict={x: batch_x, y: batch_y,
                                        seqlen: batch_seqlen})
-        if step % display_step == 0:
+        if step % display_step == 0 or step == 1:
             # Calculate batch accuracy
             acc = sess.run(accuracy, feed_dict={x: batch_x, y: batch_y,
                                                 seqlen: batch_seqlen})
             # Calculate batch loss
             loss = sess.run(cost, feed_dict={x: batch_x, y: batch_y,
                                              seqlen: batch_seqlen})
-            print("Iter " + str(step*batch_size) + ", Minibatch Loss= " + \
+            print("Step " + str(step*batch_size) + ", Minibatch Loss= " + \
                   "{:.6f}".format(loss) + ", Training Accuracy= " + \
                   "{:.5f}".format(acc))
-        step += 1
+
     print("Optimization Finished!")
 
     # Calculate accuracy

examples/3_NeuralNetworks/gan.py

@@ -0,0 +1,157 @@
+""" Generative Adversarial Networks (GAN).
+
+Using generative adversarial networks (GAN) to generate digit images from a
+noise distribution.
+
+References:
+    - Generative adversarial nets. I Goodfellow, J Pouget-Abadie, M Mirza,
+    B Xu, D Warde-Farley, S Ozair, Y. Bengio. Advances in neural information
+    processing systems, 2672-2680.
+    - Understanding the difficulty of training deep feedforward neural networks.
+    X Glorot, Y Bengio. Aistats 9, 249-256
+
+Links:
+    - [GAN Paper](https://arxiv.org/pdf/1406.2661.pdf).
+    - [MNIST Dataset](http://yann.lecun.com/exdb/mnist/).
+    - [Xavier Glorot Init](www.cs.cmu.edu/~bhiksha/courses/deeplearning/Fall.../AISTATS2010_Glorot.pdf).
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+
+from __future__ import division, print_function, absolute_import
+
+import matplotlib.pyplot as plt
+import numpy as np
+import tensorflow as tf
+
+# Import MNIST data
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
+
+# Training Params
+num_steps = 100000
+batch_size = 128
+learning_rate = 0.0002
+
+# Network Params
+image_dim = 784 # 28*28 pixels
+gen_hidden_dim = 256
+disc_hidden_dim = 256
+noise_dim = 100 # Noise data points
+
+# A custom initialization (see Xavier Glorot init)
+def glorot_init(shape):
+    return tf.random_normal(shape=shape, stddev=1. / tf.sqrt(shape[0] / 2.))
+
+# Store layers weight & bias
+weights = {
+    'gen_hidden1': tf.Variable(glorot_init([noise_dim, gen_hidden_dim])),
+    'gen_out': tf.Variable(glorot_init([gen_hidden_dim, image_dim])),
+    'disc_hidden1': tf.Variable(glorot_init([image_dim, disc_hidden_dim])),
+    'disc_out': tf.Variable(glorot_init([disc_hidden_dim, 1])),
+}
+biases = {
+    'gen_hidden1': tf.Variable(tf.zeros([gen_hidden_dim])),
+    'gen_out': tf.Variable(tf.zeros([image_dim])),
+    'disc_hidden1': tf.Variable(tf.zeros([disc_hidden_dim])),
+    'disc_out': tf.Variable(tf.zeros([1])),
+}
+
+
+# Generator
+def generator(x):
+    hidden_layer = tf.matmul(x, weights['gen_hidden1'])
+    hidden_layer = tf.add(hidden_layer, biases['gen_hidden1'])
+    hidden_layer = tf.nn.relu(hidden_layer)
+    out_layer = tf.matmul(hidden_layer, weights['gen_out'])
+    out_layer = tf.add(out_layer, biases['gen_out'])
+    out_layer = tf.nn.sigmoid(out_layer)
+    return out_layer
+
+
+# Discriminator
+def discriminator(x):
+    hidden_layer = tf.matmul(x, weights['disc_hidden1'])
+    hidden_layer = tf.add(hidden_layer, biases['disc_hidden1'])
+    hidden_layer = tf.nn.relu(hidden_layer)
+    out_layer = tf.matmul(hidden_layer, weights['disc_out'])
+    out_layer = tf.add(out_layer, biases['disc_out'])
+    out_layer = tf.nn.sigmoid(out_layer)
+    return out_layer
+
+# Build Networks
+# Network Inputs
+gen_input = tf.placeholder(tf.float32, shape=[None, noise_dim], name='input_noise')
+disc_input = tf.placeholder(tf.float32, shape=[None, image_dim], name='disc_input')
+
+# Build Generator Network
+gen_sample = generator(gen_input)
+
+# Build 2 Discriminator Networks (one from noise input, one from generated samples)
+disc_real = discriminator(disc_input)
+disc_fake = discriminator(gen_sample)
+
+# Build Loss
+gen_loss = -tf.reduce_mean(tf.log(disc_fake))
+disc_loss = -tf.reduce_mean(tf.log(disc_real) + tf.log(1. - disc_fake))
+
+# Build Optimizers
+optimizer_gen = tf.train.AdamOptimizer(learning_rate=learning_rate)
+optimizer_disc = tf.train.AdamOptimizer(learning_rate=learning_rate)
+
+# Training Variables for each optimizer
+# By default in TensorFlow, all variables are updated by each optimizer, so we
+# need to precise for each one of them the specific variables to update.
+# Generator Network Variables
+gen_vars = [weights['gen_hidden1'], weights['gen_out'],
+            biases['gen_hidden1'], biases['gen_out']]
+# Discriminator Network Variables
+disc_vars = [weights['disc_hidden1'], weights['disc_out'],
+            biases['disc_hidden1'], biases['disc_out']]
+
+# Create training operations
+train_gen = optimizer_gen.minimize(gen_loss, var_list=gen_vars)
+train_disc = optimizer_disc.minimize(disc_loss, var_list=disc_vars)
+
+# Initialize the variables (i.e. assign their default value)
+init = tf.global_variables_initializer()
+
+# Start training
+with tf.Session() as sess:
+
+    # Run the initializer
+    sess.run(init)
+
+    for i in range(1, num_steps+1):
+        # Prepare Data
+        # Get the next batch of MNIST data (only images are needed, not labels)
+        batch_x, _ = mnist.train.next_batch(batch_size)
+        # Generate noise to feed to the generator
+        z = np.random.uniform(-1., 1., size=[batch_size, noise_dim])
+
+        # Train
+        feed_dict = {disc_input: batch_x, gen_input: z}
+        _, _, gl, dl = sess.run([train_gen, train_disc, gen_loss, disc_loss],
+                                feed_dict=feed_dict)
+        if i % 1000 == 0 or i == 1:
+            print('Step %i: Generator Loss: %f, Discriminator Loss: %f' % (i, gl, dl))
+
+    # Generate images from noise, using the generator network.
+    f, a = plt.subplots(4, 10, figsize=(10, 4))
+    for i in range(10):
+        # Noise input.
+        z = np.random.uniform(-1., 1., size=[4, noise_dim])
+        g = sess.run([gen_sample], feed_dict={gen_input: z})
+        g = np.reshape(g, newshape=(4, 28, 28, 1))
+        # Reverse colours for better display
+        g = -1 * (g - 1)
+        for j in range(4):
+            # Generate image from noise. Extend to 3 channels for matplot figure.
+            img = np.reshape(np.repeat(g[j][:, :, np.newaxis], 3, axis=2),
+                             newshape=(28, 28, 3))
+            a[j][i].imshow(img)
+
+    f.show()
+    plt.draw()
+    plt.waitforbuttonpress()

examples/3_NeuralNetworks/multilayer_perceptron.py

@@ -1,11 +1,22 @@
-'''
-A Multilayer Perceptron implementation example using TensorFlow library.
-This example is using the MNIST database of handwritten digits
-(http://yann.lecun.com/exdb/mnist/)
+""" Multilayer Perceptron.
+
+A Multilayer Perceptron (Neural Network) implementation example using
+TensorFlow library. This example is using the MNIST database of handwritten
+digits (http://yann.lecun.com/exdb/mnist/).
+
+Links:
+    [MNIST Dataset](http://yann.lecun.com/exdb/mnist/).
 
 Author: Aymeric Damien
 Project: https://github.com/aymericdamien/TensorFlow-Examples/
-'''
+"""
+
+# ------------------------------------------------------------------
+#
+# THIS EXAMPLE HAS BEEN RENAMED 'neural_network.py', FOR SIMPLICITY.
+#
+# ------------------------------------------------------------------
+
 
 from __future__ import print_function
 
@@ -22,27 +33,14 @@ batch_size = 100
 display_step = 1
 
 # Network Parameters
-n_hidden_1 = 256 # 1st layer number of features
-n_hidden_2 = 256 # 2nd layer number of features
+n_hidden_1 = 256 # 1st layer number of neurons
+n_hidden_2 = 256 # 2nd layer number of neurons
 n_input = 784 # MNIST data input (img shape: 28*28)
 n_classes = 10 # MNIST total classes (0-9 digits)
 
 # tf Graph input
-x = tf.placeholder("float", [None, n_input])
-y = tf.placeholder("float", [None, n_classes])
-
-
-# Create model
-def multilayer_perceptron(x, weights, biases):
-    # Hidden layer with RELU activation
-    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
-    layer_1 = tf.nn.relu(layer_1)
-    # Hidden layer with RELU activation
-    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
-    layer_2 = tf.nn.relu(layer_2)
-    # Output layer with linear activation
-    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
-    return out_layer
+X = tf.placeholder("float", [None, n_input])
+Y = tf.placeholder("float", [None, n_classes])
 
 # Store layers weight & bias
 weights = {
@@ -56,17 +54,28 @@ biases = {
     'out': tf.Variable(tf.random_normal([n_classes]))
 }
 
+
+# Create model
+def multilayer_perceptron(x):
+    # Hidden fully connected layer with 256 neurons
+    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
+    # Hidden fully connected layer with 256 neurons
+    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
+    # Output fully connected layer with a neuron for each class
+    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
+    return out_layer
+
 # Construct model
-pred = multilayer_perceptron(x, weights, biases)
+logits = multilayer_perceptron(X)
 
 # Define loss and optimizer
-cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
-optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
-
+loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
+    logits=logits, labels=Y))
+optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
+train_op = optimizer.minimize(loss_op)
 # Initializing the variables
 init = tf.global_variables_initializer()
 
-# Launch the graph
 with tf.Session() as sess:
     sess.run(init)
 
@@ -78,18 +87,18 @@ with tf.Session() as sess:
         for i in range(total_batch):
             batch_x, batch_y = mnist.train.next_batch(batch_size)
             # Run optimization op (backprop) and cost op (to get loss value)
-            _, c = sess.run([optimizer, cost], feed_dict={x: batch_x,
-                                                          y: batch_y})
+            _, c = sess.run([train_op, loss_op], feed_dict={X: batch_x,
+                                                            Y: batch_y})
             # Compute average loss
             avg_cost += c / total_batch
         # Display logs per epoch step
         if epoch % display_step == 0:
-            print("Epoch:", '%04d' % (epoch+1), "cost=", \
-                "{:.9f}".format(avg_cost))
+            print("Epoch:", '%04d' % (epoch+1), "cost={:.9f}".format(avg_cost))
     print("Optimization Finished!")
 
     # Test model
-    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
+    pred = tf.nn.softmax(logits)  # Apply softmax to logits
+    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(Y, 1))
     # Calculate accuracy
     accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
-    print("Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels}))
+    print("Accuracy:", accuracy.eval({X: mnist.test.images, Y: mnist.test.labels}))

examples/3_NeuralNetworks/neural_network.py

@@ -0,0 +1,103 @@
+""" Neural Network.
+
+A 2-Hidden Layers Fully Connected Neural Network (a.k.a Multilayer Perceptron)
+implementation with TensorFlow. This example is using the MNIST database
+of handwritten digits (http://yann.lecun.com/exdb/mnist/).
+
+This example is using TensorFlow layers, see 'neural_network_raw' example for
+a raw implementation with variables.
+
+Links:
+    [MNIST Dataset](http://yann.lecun.com/exdb/mnist/).
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+
+from __future__ import print_function
+
+# Import MNIST data
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=False)
+
+import tensorflow as tf
+
+# Parameters
+learning_rate = 0.1
+num_steps = 1000
+batch_size = 128
+display_step = 100
+
+# Network Parameters
+n_hidden_1 = 256 # 1st layer number of neurons
+n_hidden_2 = 256 # 2nd layer number of neurons
+num_input = 784 # MNIST data input (img shape: 28*28)
+num_classes = 10 # MNIST total classes (0-9 digits)
+
+
+# Define the neural network
+def neural_net(x_dict):
+    # TF Estimator input is a dict, in case of multiple inputs
+    x = x_dict['images']
+    # Hidden fully connected layer with 256 neurons
+    layer_1 = tf.layers.dense(x, n_hidden_1)
+    # Hidden fully connected layer with 256 neurons
+    layer_2 = tf.layers.dense(layer_1, n_hidden_2)
+    # Output fully connected layer with a neuron for each class
+    out_layer = tf.layers.dense(layer_2, num_classes)
+    return out_layer
+
+
+# Define the model function (following TF Estimator Template)
+def model_fn(features, labels, mode):
+    # Build the neural network
+    logits = neural_net(features)
+
+    # Predictions
+    pred_classes = tf.argmax(logits, axis=1)
+    pred_probas = tf.nn.softmax(logits)
+
+    # If prediction mode, early return
+    if mode == tf.estimator.ModeKeys.PREDICT:
+        return tf.estimator.EstimatorSpec(mode, predictions=pred_classes)
+
+        # Define loss and optimizer
+    loss_op = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
+        logits=logits, labels=tf.cast(labels, dtype=tf.int32)))
+    optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
+    train_op = optimizer.minimize(loss_op,
+                                  global_step=tf.train.get_global_step())
+
+    # Evaluate the accuracy of the model
+    acc_op = tf.metrics.accuracy(labels=labels, predictions=pred_classes)
+
+    # TF Estimators requires to return a EstimatorSpec, that specify
+    # the different ops for training, evaluating, ...
+    estim_specs = tf.estimator.EstimatorSpec(
+        mode=mode,
+        predictions=pred_classes,
+        loss=loss_op,
+        train_op=train_op,
+        eval_metric_ops={'accuracy': acc_op})
+
+    return estim_specs
+
+# Build the Estimator
+model = tf.estimator.Estimator(model_fn)
+
+# Define the input function for training
+input_fn = tf.estimator.inputs.numpy_input_fn(
+    x={'images': mnist.train.images}, y=mnist.train.labels,
+    batch_size=batch_size, num_epochs=None, shuffle=True)
+# Train the Model
+model.train(input_fn, steps=num_steps)
+
+# Evaluate the Model
+# Define the input function for evaluating
+input_fn = tf.estimator.inputs.numpy_input_fn(
+    x={'images': mnist.test.images}, y=mnist.test.labels,
+    batch_size=batch_size, shuffle=False)
+# Use the Estimator 'evaluate' method
+e = model.evaluate(input_fn)
+
+print("Testing Accuracy:", e['accuracy'])

examples/3_NeuralNetworks/neural_network_raw.py

@@ -0,0 +1,101 @@
+""" Neural Network.
+
+A 2-Hidden Layers Fully Connected Neural Network (a.k.a Multilayer Perceptron)
+implementation with TensorFlow. This example is using the MNIST database
+of handwritten digits (http://yann.lecun.com/exdb/mnist/).
+
+Links:
+    [MNIST Dataset](http://yann.lecun.com/exdb/mnist/).
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+
+from __future__ import print_function
+
+# Import MNIST data
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
+
+import tensorflow as tf
+
+# Parameters
+learning_rate = 0.1
+num_steps = 500
+batch_size = 128
+display_step = 100
+
+# Network Parameters
+n_hidden_1 = 256 # 1st layer number of neurons
+n_hidden_2 = 256 # 2nd layer number of neurons
+num_input = 784 # MNIST data input (img shape: 28*28)
+num_classes = 10 # MNIST total classes (0-9 digits)
+
+# tf Graph input
+X = tf.placeholder("float", [None, num_input])
+Y = tf.placeholder("float", [None, num_classes])
+
+# Store layers weight & bias
+weights = {
+    'h1': tf.Variable(tf.random_normal([num_input, n_hidden_1])),
+    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
+    'out': tf.Variable(tf.random_normal([n_hidden_2, num_classes]))
+}
+biases = {
+    'b1': tf.Variable(tf.random_normal([n_hidden_1])),
+    'b2': tf.Variable(tf.random_normal([n_hidden_2])),
+    'out': tf.Variable(tf.random_normal([num_classes]))
+}
+
+
+# Create model
+def neural_net(x):
+    # Hidden fully connected layer with 256 neurons
+    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
+    # Hidden fully connected layer with 256 neurons
+    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
+    # Output fully connected layer with a neuron for each class
+    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
+    return out_layer
+
+# Construct model
+logits = neural_net(X)
+prediction = tf.nn.softmax(logits)
+
+# Define loss and optimizer
+loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
+    logits=logits, labels=Y))
+optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
+train_op = optimizer.minimize(loss_op)
+
+# Evaluate model
+correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))
+accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
+
+# Initialize the variables (i.e. assign their default value)
+init = tf.global_variables_initializer()
+
+# Start training
+with tf.Session() as sess:
+
+    # Run the initializer
+    sess.run(init)
+
+    for step in range(1, num_steps+1):
+        batch_x, batch_y = mnist.train.next_batch(batch_size)
+        # Run optimization op (backprop)
+        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
+        if step % display_step == 0 or step == 1:
+            # Calculate batch loss and accuracy
+            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,
+                                                                 Y: batch_y})
+            print("Step " + str(step) + ", Minibatch Loss= " + \
+                  "{:.4f}".format(loss) + ", Training Accuracy= " + \
+                  "{:.3f}".format(acc))
+
+    print("Optimization Finished!")
+
+    # Calculate accuracy for MNIST test images
+    print("Testing Accuracy:", \
+        sess.run(accuracy, feed_dict={X: mnist.test.images,
+                                      Y: mnist.test.labels}))

examples/3_NeuralNetworks/recurrent_network.py

@@ -1,11 +1,15 @@
-'''
+""" Recurrent Neural Network.
+
 A Recurrent Neural Network (LSTM) implementation example using TensorFlow library.
 This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)
-Long Short Term Memory paper: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf
+
+Links:
+    [Long Short Term Memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf)
+    [MNIST Dataset](http://yann.lecun.com/exdb/mnist/).
 
 Author: Aymeric Damien
 Project: https://github.com/aymericdamien/TensorFlow-Examples/
-'''
+"""
 
 from __future__ import print_function
 
@@ -22,42 +26,42 @@ row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then
 handle 28 sequences of 28 steps for every sample.
 '''
 
-# Parameters
+# Training Parameters
 learning_rate = 0.001
-training_iters = 100000
+training_steps = 10000
 batch_size = 128
-display_step = 10
+display_step = 200
 
 # Network Parameters
-n_input = 28 # MNIST data input (img shape: 28*28)
-n_steps = 28 # timesteps
-n_hidden = 128 # hidden layer num of features
-n_classes = 10 # MNIST total classes (0-9 digits)
+num_input = 28 # MNIST data input (img shape: 28*28)
+timesteps = 28 # timesteps
+num_hidden = 128 # hidden layer num of features
+num_classes = 10 # MNIST total classes (0-9 digits)
 
 # tf Graph input
-x = tf.placeholder("float", [None, n_steps, n_input])
-y = tf.placeholder("float", [None, n_classes])
+X = tf.placeholder("float", [None, timesteps, num_input])
+Y = tf.placeholder("float", [None, num_classes])
 
 # Define weights
 weights = {
-    'out': tf.Variable(tf.random_normal([n_hidden, n_classes]))
+    'out': tf.Variable(tf.random_normal([num_hidden, num_classes]))
 }
 biases = {
-    'out': tf.Variable(tf.random_normal([n_classes]))
+    'out': tf.Variable(tf.random_normal([num_classes]))
 }
 
 
 def RNN(x, weights, biases):
 
     # Prepare data shape to match `rnn` function requirements
-    # Current data input shape: (batch_size, n_steps, n_input)
-    # Required shape: 'n_steps' tensors list of shape (batch_size, n_input)
+    # Current data input shape: (batch_size, timesteps, n_input)
+    # Required shape: 'timesteps' tensors list of shape (batch_size, n_input)
 
-    # Unstack to get a list of 'n_steps' tensors of shape (batch_size, n_input)
-    x = tf.unstack(x, n_steps, 1)
+    # Unstack to get a list of 'timesteps' tensors of shape (batch_size, n_input)
+    x = tf.unstack(x, timesteps, 1)
 
     # Define a lstm cell with tensorflow
-    lstm_cell = rnn.BasicLSTMCell(n_hidden, forget_bias=1.0)
+    lstm_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)
 
     # Get lstm cell output
     outputs, states = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)
@@ -65,44 +69,47 @@ def RNN(x, weights, biases):
     # Linear activation, using rnn inner loop last output
     return tf.matmul(outputs[-1], weights['out']) + biases['out']
 
-pred = RNN(x, weights, biases)
+logits = RNN(X, weights, biases)
+prediction = tf.nn.softmax(logits)
 
 # Define loss and optimizer
-cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
-optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
+loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
+    logits=logits, labels=Y))
+optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
+train_op = optimizer.minimize(loss_op)
 
-# Evaluate model
-correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
+# Evaluate model (with test logits, for dropout to be disabled)
+correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))
 accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
-# Launch the graph
+# Start training
 with tf.Session() as sess:
+
+    # Run the initializer
     sess.run(init)
-    step = 1
-    # Keep training until reach max iterations
-    while step * batch_size < training_iters:
+
+    for step in range(1, training_steps+1):
         batch_x, batch_y = mnist.train.next_batch(batch_size)
         # Reshape data to get 28 seq of 28 elements
-        batch_x = batch_x.reshape((batch_size, n_steps, n_input))
+        batch_x = batch_x.reshape((batch_size, timesteps, num_input))
         # Run optimization op (backprop)
-        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
-        if step % display_step == 0:
-            # Calculate batch accuracy
-            acc = sess.run(accuracy, feed_dict={x: batch_x, y: batch_y})
-            # Calculate batch loss
-            loss = sess.run(cost, feed_dict={x: batch_x, y: batch_y})
-            print("Iter " + str(step*batch_size) + ", Minibatch Loss= " + \
-                  "{:.6f}".format(loss) + ", Training Accuracy= " + \
-                  "{:.5f}".format(acc))
-        step += 1
+        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
+        if step % display_step == 0 or step == 1:
+            # Calculate batch loss and accuracy
+            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,
+                                                                 Y: batch_y})
+            print("Step " + str(step) + ", Minibatch Loss= " + \
+                  "{:.4f}".format(loss) + ", Training Accuracy= " + \
+                  "{:.3f}".format(acc))
+
     print("Optimization Finished!")
 
     # Calculate accuracy for 128 mnist test images
     test_len = 128
-    test_data = mnist.test.images[:test_len].reshape((-1, n_steps, n_input))
+    test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))
     test_label = mnist.test.labels[:test_len]
     print("Testing Accuracy:", \
-        sess.run(accuracy, feed_dict={x: test_data, y: test_label}))
+        sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))

examples/3_NeuralNetworks/variational_autoencoder.py

@@ -0,0 +1,143 @@
+""" Variational Auto-Encoder Example.
+
+Using a variational auto-encoder to generate digits images from noise.
+MNIST handwritten digits are used as training examples.
+
+References:
+    - Auto-Encoding Variational Bayes The International Conference on Learning
+    Representations (ICLR), Banff, 2014. D.P. Kingma, M. Welling
+    - Understanding the difficulty of training deep feedforward neural networks.
+    X Glorot, Y Bengio. Aistats 9, 249-256
+    - Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based
+    learning applied to document recognition." Proceedings of the IEEE,
+    86(11):2278-2324, November 1998.
+
+Links:
+    - [VAE Paper] https://arxiv.org/abs/1312.6114
+    - [Xavier Glorot Init](www.cs.cmu.edu/~bhiksha/courses/deeplearning/Fall.../AISTATS2010_Glorot.pdf).
+    - [MNIST Dataset] http://yann.lecun.com/exdb/mnist/
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+from __future__ import division, print_function, absolute_import
+
+import numpy as np
+import matplotlib.pyplot as plt
+from scipy.stats import norm
+import tensorflow as tf
+
+# Import MNIST data
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
+
+# Parameters
+learning_rate = 0.001
+num_steps = 30000
+batch_size = 64
+
+# Network Parameters
+image_dim = 784 # MNIST images are 28x28 pixels
+hidden_dim = 512
+latent_dim = 2
+
+# A custom initialization (see Xavier Glorot init)
+def glorot_init(shape):
+    return tf.random_normal(shape=shape, stddev=1. / tf.sqrt(shape[0] / 2.))
+
+# Variables
+weights = {
+    'encoder_h1': tf.Variable(glorot_init([image_dim, hidden_dim])),
+    'z_mean': tf.Variable(glorot_init([hidden_dim, latent_dim])),
+    'z_std': tf.Variable(glorot_init([hidden_dim, latent_dim])),
+    'decoder_h1': tf.Variable(glorot_init([latent_dim, hidden_dim])),
+    'decoder_out': tf.Variable(glorot_init([hidden_dim, image_dim]))
+}
+biases = {
+    'encoder_b1': tf.Variable(glorot_init([hidden_dim])),
+    'z_mean': tf.Variable(glorot_init([latent_dim])),
+    'z_std': tf.Variable(glorot_init([latent_dim])),
+    'decoder_b1': tf.Variable(glorot_init([hidden_dim])),
+    'decoder_out': tf.Variable(glorot_init([image_dim]))
+}
+
+# Building the encoder
+input_image = tf.placeholder(tf.float32, shape=[None, image_dim])
+encoder = tf.matmul(input_image, weights['encoder_h1']) + biases['encoder_b1']
+encoder = tf.nn.tanh(encoder)
+z_mean = tf.matmul(encoder, weights['z_mean']) + biases['z_mean']
+z_std = tf.matmul(encoder, weights['z_std']) + biases['z_std']
+
+# Sampler: Normal (gaussian) random distribution
+eps = tf.random_normal(tf.shape(z_std), dtype=tf.float32, mean=0., stddev=1.0,
+                       name='epsilon')
+z = z_mean + tf.exp(z_std / 2) * eps
+
+# Building the decoder (with scope to re-use these layers later)
+decoder = tf.matmul(z, weights['decoder_h1']) + biases['decoder_b1']
+decoder = tf.nn.tanh(decoder)
+decoder = tf.matmul(decoder, weights['decoder_out']) + biases['decoder_out']
+decoder = tf.nn.sigmoid(decoder)
+
+
+# Define VAE Loss
+def vae_loss(x_reconstructed, x_true):
+    # Reconstruction loss
+    encode_decode_loss = x_true * tf.log(1e-10 + x_reconstructed) \
+                         + (1 - x_true) * tf.log(1e-10 + 1 - x_reconstructed)
+    encode_decode_loss = -tf.reduce_sum(encode_decode_loss, 1)
+    # KL Divergence loss
+    kl_div_loss = 1 + z_std - tf.square(z_mean) - tf.exp(z_std)
+    kl_div_loss = -0.5 * tf.reduce_sum(kl_div_loss, 1)
+    return tf.reduce_mean(encode_decode_loss + kl_div_loss)
+
+loss_op = vae_loss(decoder, input_image)
+optimizer = tf.train.RMSPropOptimizer(learning_rate=learning_rate)
+train_op = optimizer.minimize(loss_op)
+
+# Initialize the variables (i.e. assign their default value)
+init = tf.global_variables_initializer()
+
+# Start training
+with tf.Session() as sess:
+
+    # Run the initializer
+    sess.run(init)
+
+    for i in range(1, num_steps+1):
+        # Prepare Data
+        # Get the next batch of MNIST data (only images are needed, not labels)
+        batch_x, _ = mnist.train.next_batch(batch_size)
+
+        # Train
+        feed_dict = {input_image: batch_x}
+        _, l = sess.run([train_op, loss_op], feed_dict=feed_dict)
+        if i % 1000 == 0 or i == 1:
+            print('Step %i, Loss: %f' % (i, l))
+
+    # Testing
+    # Generator takes noise as input
+    noise_input = tf.placeholder(tf.float32, shape=[None, latent_dim])
+    # Rebuild the decoder to create image from noise
+    decoder = tf.matmul(noise_input, weights['decoder_h1']) + biases['decoder_b1']
+    decoder = tf.nn.tanh(decoder)
+    decoder = tf.matmul(decoder, weights['decoder_out']) + biases['decoder_out']
+    decoder = tf.nn.sigmoid(decoder)
+
+    # Building a manifold of generated digits
+    n = 20
+    x_axis = np.linspace(-3, 3, n)
+    y_axis = np.linspace(-3, 3, n)
+
+    canvas = np.empty((28 * n, 28 * n))
+    for i, yi in enumerate(x_axis):
+        for j, xi in enumerate(y_axis):
+            z_mu = np.array([[xi, yi]] * batch_size)
+            x_mean = sess.run(decoder, feed_dict={noise_input: z_mu})
+            canvas[(n - i - 1) * 28:(n - i) * 28, j * 28:(j + 1) * 28] = \
+            x_mean[0].reshape(28, 28)
+
+    plt.figure(figsize=(8, 10))
+    Xi, Yi = np.meshgrid(x_axis, y_axis)
+    plt.imshow(canvas, origin="upper", cmap="gray")
+    plt.show()

examples/4_Utils/save_restore_model.py

@@ -63,7 +63,7 @@ pred = multilayer_perceptron(x, weights, biases)
 cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
 optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
 # 'Saver' op to save and restore all the variables
@@ -72,7 +72,8 @@ saver = tf.train.Saver()
 # Running first session
 print("Starting 1st session...")
 with tf.Session() as sess:
-    # Initialize variables
+
+    # Run the initializer
     sess.run(init)
 
     # Training cycle

examples/4_Utils/tensorboard_advanced.py

@@ -20,7 +20,7 @@ learning_rate = 0.01
 training_epochs = 25
 batch_size = 100
 display_step = 1
-logs_path = '/tmp/tensorflow_logs/example'
+logs_path = '/tmp/tensorflow_logs/example/'
 
 # Network Parameters
 n_hidden_1 = 256 # 1st layer number of features
@@ -87,7 +87,7 @@ with tf.name_scope('Accuracy'):
     acc = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
     acc = tf.reduce_mean(tf.cast(acc, tf.float32))
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
 # Create a summary to monitor cost tensor
@@ -103,8 +103,10 @@ for grad, var in grads:
 # Merge all summaries into a single op
 merged_summary_op = tf.summary.merge_all()
 
-# Launch the graph
+# Start training
 with tf.Session() as sess:
+
+    # Run the initializer
     sess.run(init)
 
     # op to write logs to Tensorboard

examples/4_Utils/tensorboard_basic.py

@@ -19,8 +19,8 @@ mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
 learning_rate = 0.01
 training_epochs = 25
 batch_size = 100
-display_step = 1
-logs_path = '/tmp/tensorflow_logs/example'
+display_epoch = 1
+logs_path = '/tmp/tensorflow_logs/example/'
 
 # tf Graph Input
 # mnist data image of shape 28*28=784
@@ -48,7 +48,7 @@ with tf.name_scope('Accuracy'):
     acc = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
     acc = tf.reduce_mean(tf.cast(acc, tf.float32))
 
-# Initializing the variables
+# Initialize the variables (i.e. assign their default value)
 init = tf.global_variables_initializer()
 
 # Create a summary to monitor cost tensor
@@ -58,8 +58,10 @@ tf.summary.scalar("accuracy", acc)
 # Merge all summaries into a single op
 merged_summary_op = tf.summary.merge_all()
 
-# Launch the graph
+# Start training
 with tf.Session() as sess:
+
+    # Run the initializer
     sess.run(init)
 
     # op to write logs to Tensorboard
@@ -81,7 +83,7 @@ with tf.Session() as sess:
             # Compute average loss
             avg_cost += c / total_batch
         # Display logs per epoch step
-        if (epoch+1) % display_step == 0:
+        if (epoch+1) % display_epoch == 0:
             print("Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(avg_cost))
 
     print("Optimization Finished!")

examples/5_DataManagement/build_an_image_dataset.py

@@ -0,0 +1,211 @@
+""" Build an Image Dataset in TensorFlow.
+
+For this example, you need to make your own set of images (JPEG).
+We will show 2 different ways to build that dataset:
+
+- From a root folder, that will have a sub-folder containing images for each class
+    ```
+    ROOT_FOLDER
+       |-------- SUBFOLDER (CLASS 0)
+       |             |
+       |             | ----- image1.jpg
+       |             | ----- image2.jpg
+       |             | ----- etc...
+       |             
+       |-------- SUBFOLDER (CLASS 1)
+       |             |
+       |             | ----- image1.jpg
+       |             | ----- image2.jpg
+       |             | ----- etc...
+    ```
+
+- From a plain text file, that will list all images with their class ID:
+    ```
+    /path/to/image/1.jpg CLASS_ID
+    /path/to/image/2.jpg CLASS_ID
+    /path/to/image/3.jpg CLASS_ID
+    /path/to/image/4.jpg CLASS_ID
+    etc...
+    ```
+
+Below, there are some parameters that you need to change (Marked 'CHANGE HERE'), 
+such as the dataset path.
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+from __future__ import print_function
+
+import tensorflow as tf
+import os
+
+# Dataset Parameters - CHANGE HERE
+MODE = 'folder' # or 'file', if you choose a plain text file (see above).
+DATASET_PATH = '/path/to/dataset/' # the dataset file or root folder path.
+
+# Image Parameters
+N_CLASSES = 2 # CHANGE HERE, total number of classes
+IMG_HEIGHT = 64 # CHANGE HERE, the image height to be resized to
+IMG_WIDTH = 64 # CHANGE HERE, the image width to be resized to
+CHANNELS = 3 # The 3 color channels, change to 1 if grayscale
+
+
+# Reading the dataset
+# 2 modes: 'file' or 'folder'
+def read_images(dataset_path, mode, batch_size):
+    imagepaths, labels = list(), list()
+    if mode == 'file':
+        # Read dataset file
+        data = open(dataset_path, 'r').read().splitlines()
+        for d in data:
+            imagepaths.append(d.split(' ')[0])
+            labels.append(int(d.split(' ')[1]))
+    elif mode == 'folder':
+        # An ID will be affected to each sub-folders by alphabetical order
+        label = 0
+        # List the directory
+        try:  # Python 2
+            classes = sorted(os.walk(dataset_path).next()[1])
+        except Exception:  # Python 3
+            classes = sorted(os.walk(dataset_path).__next__()[1])
+        # List each sub-directory (the classes)
+        for c in classes:
+            c_dir = os.path.join(dataset_path, c)
+            try:  # Python 2
+                walk = os.walk(c_dir).next()
+            except Exception:  # Python 3
+                walk = os.walk(c_dir).__next__()
+            # Add each image to the training set
+            for sample in walk[2]:
+                # Only keeps jpeg images
+                if sample.endswith('.jpg') or sample.endswith('.jpeg'):
+                    imagepaths.append(os.path.join(c_dir, sample))
+                    labels.append(label)
+            label += 1
+    else:
+        raise Exception("Unknown mode.")
+
+    # Convert to Tensor
+    imagepaths = tf.convert_to_tensor(imagepaths, dtype=tf.string)
+    labels = tf.convert_to_tensor(labels, dtype=tf.int32)
+    # Build a TF Queue, shuffle data
+    image, label = tf.train.slice_input_producer([imagepaths, labels],
+                                                 shuffle=True)
+
+    # Read images from disk
+    image = tf.read_file(image)
+    image = tf.image.decode_jpeg(image, channels=CHANNELS)
+
+    # Resize images to a common size
+    image = tf.image.resize_images(image, [IMG_HEIGHT, IMG_WIDTH])
+
+    # Normalize
+    image = image * 1.0/127.5 - 1.0
+
+    # Create batches
+    X, Y = tf.train.batch([image, label], batch_size=batch_size,
+                          capacity=batch_size * 8,
+                          num_threads=4)
+
+    return X, Y
+
+# -----------------------------------------------
+# THIS IS A CLASSIC CNN (see examples, section 3)
+# -----------------------------------------------
+# Note that a few elements have changed (usage of queues).
+
+# Parameters
+learning_rate = 0.001
+num_steps = 10000
+batch_size = 128
+display_step = 100
+
+# Network Parameters
+dropout = 0.75 # Dropout, probability to keep units
+
+# Build the data input
+X, Y = read_images(DATASET_PATH, MODE, batch_size)
+
+
+# Create model
+def conv_net(x, n_classes, dropout, reuse, is_training):
+    # Define a scope for reusing the variables
+    with tf.variable_scope('ConvNet', reuse=reuse):
+
+        # Convolution Layer with 32 filters and a kernel size of 5
+        conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)
+        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2
+        conv1 = tf.layers.max_pooling2d(conv1, 2, 2)
+
+        # Convolution Layer with 32 filters and a kernel size of 5
+        conv2 = tf.layers.conv2d(conv1, 64, 3, activation=tf.nn.relu)
+        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2
+        conv2 = tf.layers.max_pooling2d(conv2, 2, 2)
+
+        # Flatten the data to a 1-D vector for the fully connected layer
+        fc1 = tf.contrib.layers.flatten(conv2)
+
+        # Fully connected layer (in contrib folder for now)
+        fc1 = tf.layers.dense(fc1, 1024)
+        # Apply Dropout (if is_training is False, dropout is not applied)
+        fc1 = tf.layers.dropout(fc1, rate=dropout, training=is_training)
+
+        # Output layer, class prediction
+        out = tf.layers.dense(fc1, n_classes)
+        # Because 'softmax_cross_entropy_with_logits' already apply softmax,
+        # we only apply softmax to testing network
+        out = tf.nn.softmax(out) if not is_training else out
+
+    return out
+
+
+# Because Dropout have different behavior at training and prediction time, we
+# need to create 2 distinct computation graphs that share the same weights.
+
+# Create a graph for training
+logits_train = conv_net(X, N_CLASSES, dropout, reuse=False, is_training=True)
+# Create another graph for testing that reuse the same weights
+logits_test = conv_net(X, N_CLASSES, dropout, reuse=True, is_training=False)
+
+# Define loss and optimizer (with train logits, for dropout to take effect)
+loss_op = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
+    logits=logits_train, labels=Y))
+optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
+train_op = optimizer.minimize(loss_op)
+
+# Evaluate model (with test logits, for dropout to be disabled)
+correct_pred = tf.equal(tf.argmax(logits_test, 1), tf.cast(Y, tf.int64))
+accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
+
+# Initialize the variables (i.e. assign their default value)
+init = tf.global_variables_initializer()
+
+# Saver object
+saver = tf.train.Saver()
+
+# Start training
+with tf.Session() as sess:
+
+    # Run the initializer
+    sess.run(init)
+
+    # Start the data queue
+    tf.train.start_queue_runners()
+
+    # Training cycle
+    for step in range(1, num_steps+1):
+
+        if step % display_step == 0:
+            # Run optimization and calculate batch loss and accuracy
+            _, loss, acc = sess.run([train_op, loss_op, accuracy])
+            print("Step " + str(step) + ", Minibatch Loss= " + \
+                  "{:.4f}".format(loss) + ", Training Accuracy= " + \
+                  "{:.3f}".format(acc))
+        else:
+            # Only run the optimization op (backprop)
+            sess.run(train_op)
+
+    print("Optimization Finished!")
+
+    # Save your model
+    saver.save(sess, 'my_tf_model')

examples/5_DataManagement/tensorflow_dataset_api.py

@@ -0,0 +1,137 @@
+""" TensorFlow Dataset API.
+
+In this example, we will show how to load numpy array data into the new 
+TensorFlow 'Dataset' API. The Dataset API implements an optimized data pipeline
+with queues, that make data processing and training faster (especially on GPU).
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+"""
+from __future__ import print_function
+
+import tensorflow as tf
+
+# Import MNIST data (Numpy format)
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
+
+# Parameters
+learning_rate = 0.001
+num_steps = 2000
+batch_size = 128
+display_step = 100
+
+# Network Parameters
+n_input = 784 # MNIST data input (img shape: 28*28)
+n_classes = 10 # MNIST total classes (0-9 digits)
+dropout = 0.75 # Dropout, probability to keep units
+
+sess = tf.Session()
+
+# Create a dataset tensor from the images and the labels
+dataset = tf.contrib.data.Dataset.from_tensor_slices(
+    (mnist.train.images, mnist.train.labels))
+# Create batches of data
+dataset = dataset.batch(batch_size)
+# Create an iterator, to go over the dataset
+iterator = dataset.make_initializable_iterator()
+# It is better to use 2 placeholders, to avoid to load all data into memory,
+# and avoid the 2Gb restriction length of a tensor.
+_data = tf.placeholder(tf.float32, [None, n_input])
+_labels = tf.placeholder(tf.float32, [None, n_classes])
+# Initialize the iterator
+sess.run(iterator.initializer, feed_dict={_data: mnist.train.images,
+                                          _labels: mnist.train.labels})
+
+# Neural Net Input
+X, Y = iterator.get_next()
+
+
+# -----------------------------------------------
+# THIS IS A CLASSIC CNN (see examples, section 3)
+# -----------------------------------------------
+# Note that a few elements have changed (usage of sess run).
+
+# Create model
+def conv_net(x, n_classes, dropout, reuse, is_training):
+    # Define a scope for reusing the variables
+    with tf.variable_scope('ConvNet', reuse=reuse):
+        # MNIST data input is a 1-D vector of 784 features (28*28 pixels)
+        # Reshape to match picture format [Height x Width x Channel]
+        # Tensor input become 4-D: [Batch Size, Height, Width, Channel]
+        x = tf.reshape(x, shape=[-1, 28, 28, 1])
+
+        # Convolution Layer with 32 filters and a kernel size of 5
+        conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)
+        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2
+        conv1 = tf.layers.max_pooling2d(conv1, 2, 2)
+
+        # Convolution Layer with 32 filters and a kernel size of 5
+        conv2 = tf.layers.conv2d(conv1, 64, 3, activation=tf.nn.relu)
+        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2
+        conv2 = tf.layers.max_pooling2d(conv2, 2, 2)
+
+        # Flatten the data to a 1-D vector for the fully connected layer
+        fc1 = tf.contrib.layers.flatten(conv2)
+
+        # Fully connected layer (in contrib folder for now)
+        fc1 = tf.layers.dense(fc1, 1024)
+        # Apply Dropout (if is_training is False, dropout is not applied)
+        fc1 = tf.layers.dropout(fc1, rate=dropout, training=is_training)
+
+        # Output layer, class prediction
+        out = tf.layers.dense(fc1, n_classes)
+        # Because 'softmax_cross_entropy_with_logits' already apply softmax,
+        # we only apply softmax to testing network
+        out = tf.nn.softmax(out) if not is_training else out
+
+    return out
+
+
+# Because Dropout have different behavior at training and prediction time, we
+# need to create 2 distinct computation graphs that share the same weights.
+
+# Create a graph for training
+logits_train = conv_net(X, n_classes, dropout, reuse=False, is_training=True)
+# Create another graph for testing that reuse the same weights, but has
+# different behavior for 'dropout' (not applied).
+logits_test = conv_net(X, n_classes, dropout, reuse=True, is_training=False)
+
+# Define loss and optimizer (with train logits, for dropout to take effect)
+loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
+    logits=logits_train, labels=Y))
+optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
+train_op = optimizer.minimize(loss_op)
+
+# Evaluate model (with test logits, for dropout to be disabled)
+correct_pred = tf.equal(tf.argmax(logits_test, 1), tf.argmax(Y, 1))
+accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
+
+# Initialize the variables (i.e. assign their default value)
+init = tf.global_variables_initializer()
+
+# Run the initializer
+sess.run(init)
+
+# Training cycle
+for step in range(1, num_steps + 1):
+
+    try:
+        # Run optimization
+        sess.run(train_op)
+    except tf.errors.OutOfRangeError:
+        # Reload the iterator when it reaches the end of the dataset
+        sess.run(iterator.initializer,
+                 feed_dict={_data: mnist.train.images,
+                            _labels: mnist.train.labels})
+        sess.run(train_op)
+
+    if step % display_step == 0 or step == 1:
+        # Calculate batch loss and accuracy
+        # (note that this consume a new batch of data)
+        loss, acc = sess.run([loss_op, accuracy])
+        print("Step " + str(step) + ", Minibatch Loss= " + \
+              "{:.4f}".format(loss) + ", Training Accuracy= " + \
+              "{:.3f}".format(acc))
+
+print("Optimization Finished!")

examples/6_MultiGPU/multigpu_cnn.py

@@ -0,0 +1,182 @@
+''' Multi-GPU Training Example.
+
+Train a convolutional neural network on multiple GPU with TensorFlow.
+
+This example is using TensorFlow layers, see 'convolutional_network_raw' example
+for a raw TensorFlow implementation with variables.
+
+This example is using the MNIST database of handwritten digits
+(http://yann.lecun.com/exdb/mnist/)
+
+Author: Aymeric Damien
+Project: https://github.com/aymericdamien/TensorFlow-Examples/
+'''
+
+from __future__ import division, print_function, absolute_import
+
+import numpy as np
+import tensorflow as tf
+import time
+
+# Import MNIST data
+from tensorflow.examples.tutorials.mnist import input_data
+mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
+
+# Training Parameters
+num_gpus = 2
+num_steps = 200
+learning_rate = 0.001
+batch_size = 1024
+display_step = 10
+
+# Network Parameters
+num_input = 784 # MNIST data input (img shape: 28*28)
+num_classes = 10 # MNIST total classes (0-9 digits)
+dropout = 0.75 # Dropout, probability to keep units
+
+
+# Build a convolutional neural network
+def conv_net(x, n_classes, dropout, reuse, is_training):
+    # Define a scope for reusing the variables
+    with tf.variable_scope('ConvNet', reuse=reuse):
+        # MNIST data input is a 1-D vector of 784 features (28*28 pixels)
+        # Reshape to match picture format [Height x Width x Channel]
+        # Tensor input become 4-D: [Batch Size, Height, Width, Channel]
+        x = tf.reshape(x, shape=[-1, 28, 28, 1])
+
+        # Convolution Layer with 64 filters and a kernel size of 5
+        x = tf.layers.conv2d(x, 64, 5, activation=tf.nn.relu)
+        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2
+        x = tf.layers.max_pooling2d(x, 2, 2)
+
+        # Convolution Layer with 256 filters and a kernel size of 5
+        x = tf.layers.conv2d(x, 256, 3, activation=tf.nn.relu)
+        # Convolution Layer with 512 filters and a kernel size of 5
+        x = tf.layers.conv2d(x, 512, 3, activation=tf.nn.relu)
+        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2
+        x = tf.layers.max_pooling2d(x, 2, 2)
+
+        # Flatten the data to a 1-D vector for the fully connected layer
+        x = tf.contrib.layers.flatten(x)
+
+        # Fully connected layer (in contrib folder for now)
+        x = tf.layers.dense(x, 2048)
+        # Apply Dropout (if is_training is False, dropout is not applied)
+        x = tf.layers.dropout(x, rate=dropout, training=is_training)
+
+        # Fully connected layer (in contrib folder for now)
+        x = tf.layers.dense(x, 1024)
+        # Apply Dropout (if is_training is False, dropout is not applied)
+        x = tf.layers.dropout(x, rate=dropout, training=is_training)
+
+        # Output layer, class prediction
+        out = tf.layers.dense(x, n_classes)
+        # Because 'softmax_cross_entropy_with_logits' loss already apply
+        # softmax, we only apply softmax to testing network
+        out = tf.nn.softmax(out) if not is_training else out
+
+    return out
+
+
+def average_gradients(tower_grads):
+    average_grads = []
+    for grad_and_vars in zip(*tower_grads):
+        # Note that each grad_and_vars looks like the following:
+        #   ((grad0_gpu0, var0_gpu0), ... , (grad0_gpuN, var0_gpuN))
+        grads = []
+        for g, _ in grad_and_vars:
+            # Add 0 dimension to the gradients to represent the tower.
+            expanded_g = tf.expand_dims(g, 0)
+
+            # Append on a 'tower' dimension which we will average over below.
+            grads.append(expanded_g)
+
+        # Average over the 'tower' dimension.
+        grad = tf.concat(grads, 0)
+        grad = tf.reduce_mean(grad, 0)
+
+        # Keep in mind that the Variables are redundant because they are shared
+        # across towers. So .. we will just return the first tower's pointer to
+        # the Variable.
+        v = grad_and_vars[0][1]
+        grad_and_var = (grad, v)
+        average_grads.append(grad_and_var)
+    return average_grads
+
+
+# Place all ops on CPU by default
+with tf.device('/cpu:0'):
+    tower_grads = []
+    reuse_vars = False
+
+    # tf Graph input
+    X = tf.placeholder(tf.float32, [None, num_input])
+    Y = tf.placeholder(tf.float32, [None, num_classes])
+
+    # Loop over all GPUs and construct their own computation graph
+    for i in range(num_gpus):
+        with tf.device('/gpu:%d' % i):
+
+            # Split data between GPUs
+            _x = X[i * batch_size: (i+1) * batch_size]
+            _y = Y[i * batch_size: (i+1) * batch_size]
+
+            # Because Dropout have different behavior at training and prediction time, we
+            # need to create 2 distinct computation graphs that share the same weights.
+
+            # Create a graph for training
+            logits_train = conv_net(_x, num_classes, dropout,
+                                    reuse=reuse_vars, is_training=True)
+            # Create another graph for testing that reuse the same weights
+            logits_test = conv_net(_x, num_classes, dropout,
+                                   reuse=True, is_training=False)
+
+            # Define loss and optimizer (with train logits, for dropout to take effect)
+            loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
+                logits=logits_train, labels=_y))
+            optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
+            grads = optimizer.compute_gradients(loss_op)
+
+            # Only first GPU compute accuracy
+            if i == 0:
+                # Evaluate model (with test logits, for dropout to be disabled)
+                correct_pred = tf.equal(tf.argmax(logits_test, 1), tf.argmax(_y, 1))
+                accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
+
+            reuse_vars = True
+            tower_grads.append(grads)
+
+    tower_grads = average_gradients(tower_grads)
+    train_op = optimizer.apply_gradients(tower_grads)
+
+    # Initialize the variables (i.e. assign their default value)
+    init = tf.global_variables_initializer()
+
+    # Start Training
+    with tf.Session() as sess:
+
+        # Run the initializer
+        sess.run(init)
+
+        # Keep training until reach max iterations
+        for step in range(1, num_steps + 1):
+            # Get a batch for each GPU
+            batch_x, batch_y = mnist.train.next_batch(batch_size * num_gpus)
+            # Run optimization op (backprop)
+            ts = time.time()
+            sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
+            te = time.time() - ts
+            if step % display_step == 0 or step == 1:
+                # Calculate batch loss and accuracy
+                loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,
+                                                                     Y: batch_y})
+                print("Step " + str(step) + ": Minibatch Loss= " + \
+                      "{:.4f}".format(loss) + ", Training Accuracy= " + \
+                      "{:.3f}".format(acc) + ", %i Examples/sec" % int(len(batch_x)/te))
+            step += 1
+        print("Optimization Finished!")
+
+        # Calculate accuracy for MNIST test images
+        print("Testing Accuracy:", \
+            np.mean([sess.run(accuracy, feed_dict={X: mnist.test.images[i:i+batch_size],
+            Y: mnist.test.labels[i:i+batch_size]}) for i in range(0, len(mnist.test.images), batch_size)]))

notebooks/0_Prerequisite/mnist_dataset_intro.ipynb

@@ -7,7 +7,7 @@
     "\n",
     "# MNIST Dataset Introduction\n",
     "\n",
-    "Most examples are using MNIST dataset of handwritten digits. It has 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image, so each sample is represented as a matrix of size 28x28 with values from 0 to 1.\n",
+    "Most examples are using MNIST dataset of handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flatten and converted to a 1-D numpy array of 784 features (28*28).\n",
     "\n",
     "## Overview\n",
     "\n",
@@ -72,7 +72,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "IPython (Python 2.7)",
+   "display_name": "Python 2",
    "language": "python",
    "name": "python2"
   },
@@ -86,7 +86,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython2",
-   "version": "2.7.11"
+   "version": "2.7.13"
   }
  },
  "nbformat": 4,

notebooks/2_BasicModels/kmeans.ipynb

@@ -0,0 +1,203 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# K-Means Example\n",
+    "\n",
+    "Implement K-Means algorithm with TensorFlow, and apply it to classify\n",
+    "handwritten digit images. This example is using the MNIST database of\n",
+    "handwritten digits as training samples (http://yann.lecun.com/exdb/mnist/).\n",
+    "\n",
+    "Note: This example requires TensorFlow v1.1.0 or over.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "from __future__ import print_function\n",
+    "\n",
+    "import numpy as np\n",
+    "import tensorflow as tf\n",
+    "from tensorflow.contrib.factorization import KMeans\n",
+    "\n",
+    "# Ignore all GPUs, tf random forest does not benefit from it.\n",
+    "import os\n",
+    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Import MNIST data\n",
+    "from tensorflow.examples.tutorials.mnist import input_data\n",
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
+    "full_data_x = mnist.train.images"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Parameters\n",
+    "num_steps = 50 # Total steps to train\n",
+    "batch_size = 1024 # The number of samples per batch\n",
+    "k = 25 # The number of clusters\n",
+    "num_classes = 10 # The 10 digits\n",
+    "num_features = 784 # Each image is 28x28 pixels\n",
+    "\n",
+    "# Input images\n",
+    "X = tf.placeholder(tf.float32, shape=[None, num_features])\n",
+    "# Labels (for assigning a label to a centroid and testing)\n",
+    "Y = tf.placeholder(tf.float32, shape=[None, num_classes])\n",
+    "\n",
+    "# K-Means Parameters\n",
+    "kmeans = KMeans(inputs=X, num_clusters=k, distance_metric='cosine',\n",
+    "                use_mini_batch=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Build KMeans graph\n",
+    "(all_scores, cluster_idx, scores, cluster_centers_initialized, init_op,\n",
+    "train_op) = kmeans.training_graph()\n",
+    "cluster_idx = cluster_idx[0] # fix for cluster_idx being a tuple\n",
+    "avg_distance = tf.reduce_mean(scores)\n",
+    "\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
+    "init_vars = tf.global_variables_initializer()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Step 1, Avg Distance: 0.341471\n",
+      "Step 10, Avg Distance: 0.221609\n",
+      "Step 20, Avg Distance: 0.220328\n",
+      "Step 30, Avg Distance: 0.219776\n",
+      "Step 40, Avg Distance: 0.219419\n",
+      "Step 50, Avg Distance: 0.219154\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Start TensorFlow session\n",
+    "sess = tf.Session()\n",
+    "\n",
+    "# Run the initializer\n",
+    "sess.run(init_vars, feed_dict={X: full_data_x})\n",
+    "sess.run(init_op, feed_dict={X: full_data_x})\n",
+    "\n",
+    "# Training\n",
+    "for i in range(1, num_steps + 1):\n",
+    "    _, d, idx = sess.run([train_op, avg_distance, cluster_idx],\n",
+    "                         feed_dict={X: full_data_x})\n",
+    "    if i % 10 == 0 or i == 1:\n",
+    "        print(\"Step %i, Avg Distance: %f\" % (i, d))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Test Accuracy: 0.7127\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Assign a label to each centroid\n",
+    "# Count total number of labels per centroid, using the label of each training\n",
+    "# sample to their closest centroid (given by 'idx')\n",
+    "counts = np.zeros(shape=(k, num_classes))\n",
+    "for i in range(len(idx)):\n",
+    "    counts[idx[i]] += mnist.train.labels[i]\n",
+    "# Assign the most frequent label to the centroid\n",
+    "labels_map = [np.argmax(c) for c in counts]\n",
+    "labels_map = tf.convert_to_tensor(labels_map)\n",
+    "\n",
+    "# Evaluation ops\n",
+    "# Lookup: centroid_id -> label\n",
+    "cluster_label = tf.nn.embedding_lookup(labels_map, cluster_idx)\n",
+    "# Compute accuracy\n",
+    "correct_prediction = tf.equal(cluster_label, tf.cast(tf.argmax(Y, 1), tf.int32))\n",
+    "accuracy_op = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+    "\n",
+    "# Test Model\n",
+    "test_x, test_y = mnist.test.images, mnist.test.labels\n",
+    "print(\"Test Accuracy:\", sess.run(accuracy_op, feed_dict={X: test_x, Y: test_y}))"
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python [default]",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

notebooks/2_BasicModels/linear_regression.ipynb

@@ -1,17 +1,17 @@
 {
  "cells": [
   {
-   "cell_type": "code",
-   "execution_count": 1,
+   "cell_type": "markdown",
    "metadata": {
     "collapsed": false
    },
-   "outputs": [],
    "source": [
-    "# A linear regression learning algorithm example using TensorFlow library.\n",
+    "# Linear Regression Example\n",
+    "\n",
+    "A linear regression learning algorithm example using TensorFlow library.\n",
     "\n",
-    "# Author: Aymeric Damien\n",
-    "# Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
    ]
   },
   {
@@ -109,7 +109,7 @@
    },
    "outputs": [],
    "source": [
-    "# Initializing the variables\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
     "init = tf.global_variables_initializer()"
    ]
   },
@@ -161,7 +161,7 @@
     }
    ],
    "source": [
-    "# Launch the graph\n",
+    "# Start training\n",
     "with tf.Session() as sess:\n",
     "    sess.run(init)\n",
     "\n",
@@ -212,8 +212,9 @@
   }
  ],
  "metadata": {
+  "anaconda-cloud": {},
   "kernelspec": {
-   "display_name": "Python 2",
+   "display_name": "Python [default]",
    "language": "python",
    "name": "python2"
   },
@@ -227,7 +228,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython2",
-   "version": "2.7.13"
+   "version": "2.7.12"
   }
  },
  "nbformat": 4,

notebooks/2_BasicModels/logistic_regression.ipynb

@@ -1,19 +1,18 @@
 {
  "cells": [
   {
-   "cell_type": "code",
-   "execution_count": 1,
+   "cell_type": "markdown",
    "metadata": {
     "collapsed": true
    },
-   "outputs": [],
    "source": [
-    "# A logistic regression learning algorithm example using TensorFlow library.\n",
-    "# This example is using the MNIST database of handwritten digits \n",
-    "# (http://yann.lecun.com/exdb/mnist/)\n",
+    "# Logistic Regression Example\n",
     "\n",
-    "# Author: Aymeric Damien\n",
-    "# Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+    "A logistic regression learning algorithm example using TensorFlow library.\n",
+    "This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
    ]
   },
   {
@@ -39,7 +38,7 @@
     "\n",
     "# Import MINST data\n",
     "from tensorflow.examples.tutorials.mnist import input_data\n",
-    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
    ]
   },
   {
@@ -72,7 +71,7 @@
     "# Gradient Descent\n",
     "optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)\n",
     "\n",
-    "# Initializing the variables\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
     "init = tf.global_variables_initializer()"
    ]
   },
@@ -108,7 +107,7 @@
     }
    ],
    "source": [
-    "# Launch the graph\n",
+    "# Start training\n",
     "with tf.Session() as sess:\n",
     "    sess.run(init)\n",
     "\n",
@@ -136,20 +135,12 @@
     "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
     "    print \"Accuracy:\", accuracy.eval({x: mnist.test.images[:3000], y: mnist.test.labels[:3000]})"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
+  "anaconda-cloud": {},
   "kernelspec": {
-   "display_name": "Python 2",
+   "display_name": "Python [default]",
    "language": "python",
    "name": "python2"
   },
@@ -163,7 +154,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython2",
-   "version": "2.7.13"
+   "version": "2.7.12"
   }
  },
  "nbformat": 4,

notebooks/2_BasicModels/nearest_neighbor.ipynb

@@ -1,19 +1,18 @@
 {
  "cells": [
   {
-   "cell_type": "code",
-   "execution_count": 1,
+   "cell_type": "markdown",
    "metadata": {
     "collapsed": true
    },
-   "outputs": [],
    "source": [
-    "# A nearest neighbor learning algorithm example using TensorFlow library.\n",
-    "# This example is using the MNIST database of handwritten digits\n",
-    "# (http://yann.lecun.com/exdb/mnist/)\n",
+    "# Nearest Neighbor Example\n",
     "\n",
-    "# Author: Aymeric Damien\n",
-    "# Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+    "A nearest neighbor learning algorithm example using TensorFlow library.\n",
+    "This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
    ]
   },
   {
@@ -40,7 +39,7 @@
     "\n",
     "# Import MINST data\n",
     "from tensorflow.examples.tutorials.mnist import input_data\n",
-    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
    ]
   },
   {
@@ -67,7 +66,7 @@
     "\n",
     "accuracy = 0.\n",
     "\n",
-    "# Initializing the variables\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
     "init = tf.global_variables_initializer()"
    ]
   },
@@ -75,7 +74,8 @@
    "cell_type": "code",
    "execution_count": 3,
    "metadata": {
-    "collapsed": false
+    "collapsed": false,
+    "scrolled": false
    },
    "outputs": [
     {
@@ -288,7 +288,7 @@
     }
    ],
    "source": [
-    "# Launch the graph\n",
+    "# Start training\n",
     "with tf.Session() as sess:\n",
     "    sess.run(init)\n",
     "\n",
@@ -305,20 +305,12 @@
     "    print \"Done!\"\n",
     "    print \"Accuracy:\", accuracy"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
+  "anaconda-cloud": {},
   "kernelspec": {
-   "display_name": "Python 2",
+   "display_name": "Python [default]",
    "language": "python",
    "name": "python2"
   },
@@ -332,7 +324,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython2",
-   "version": "2.7.13"
+   "version": "2.7.12"
   }
  },
  "nbformat": 4,

notebooks/3_NeuralNetworks/bidirectional_rnn.ipynb

@@ -6,44 +6,64 @@
     "collapsed": true
    },
    "source": [
-    "'''\n",
-    "A Bidirectional Reccurent Neural Network (LSTM) implementation example using TensorFlow library.\n",
-    "This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)\n",
-    "Long Short Term Memory paper: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf\n",
+    "# Bi-directional Recurrent Neural Network Example\n",
     "\n",
-    "Author: Aymeric Damien\n",
-    "Project: https://github.com/aymericdamien/TensorFlow-Examples/\n",
-    "'''"
+    "Build a bi-directional recurrent neural network (LSTM) with TensorFlow.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## BiRNN Overview\n",
+    "\n",
+    "<img src=\"https://ai2-s2-public.s3.amazonaws.com/figures/2016-11-08/191dd7df9cb91ac22f56ed0dfa4a5651e8767a51/1-Figure2-1.png\" alt=\"nn\" style=\"width: 600px;\"/>\n",
+    "\n",
+    "References:\n",
+    "- [Long Short Term Memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf), Sepp Hochreiter & Jurgen Schmidhuber, Neural Computation 9(8): 1735-1780, 1997.\n",
+    "\n",
+    "## MNIST Dataset Overview\n",
+    "\n",
+    "This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flatten and converted to a 1-D numpy array of 784 features (28*28).\n",
+    "\n",
+    "![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n",
+    "\n",
+    "To classify images using a recurrent neural network, we consider every image row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then handle 28 sequences of 28 timesteps for every sample.\n",
+    "\n",
+    "More info: http://yann.lecun.com/exdb/mnist/"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {
     "collapsed": false
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
+     ]
+    }
+   ],
    "source": [
+    "from __future__ import print_function\n",
+    "\n",
     "import tensorflow as tf\n",
     "from tensorflow.contrib import rnn\n",
     "import numpy as np\n",
     "\n",
-    "# Import MINST data\n",
+    "# Import MNIST data\n",
     "from tensorflow.examples.tutorials.mnist import input_data\n",
-    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "collapsed": true
-   },
-   "source": [
-    "'''\n",
-    "To classify images using a bidirectional reccurent neural network, we consider\n",
-    "every image row as a sequence of pixels. Because MNIST image shape is 28*28px,\n",
-    "we will then handle 28 sequences of 28 steps for every sample.\n",
-    "'''"
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
    ]
   },
   {
@@ -54,35 +74,44 @@
    },
    "outputs": [],
    "source": [
-    "# Parameters\n",
+    "# Training Parameters\n",
     "learning_rate = 0.001\n",
-    "training_iters = 100000\n",
+    "training_steps = 10000\n",
     "batch_size = 128\n",
-    "display_step = 10\n",
+    "display_step = 200\n",
     "\n",
     "# Network Parameters\n",
-    "n_input = 28 # MNIST data input (img shape: 28*28)\n",
-    "n_steps = 28 # timesteps\n",
-    "n_hidden = 128 # hidden layer num of features\n",
-    "n_classes = 10 # MNIST total classes (0-9 digits)\n",
+    "num_input = 28 # MNIST data input (img shape: 28*28)\n",
+    "timesteps = 28 # timesteps\n",
+    "num_hidden = 128 # hidden layer num of features\n",
+    "num_classes = 10 # MNIST total classes (0-9 digits)\n",
     "\n",
     "# tf Graph input\n",
-    "x = tf.placeholder(\"float\", [None, n_steps, n_input])\n",
-    "y = tf.placeholder(\"float\", [None, n_classes])\n",
-    "\n",
+    "X = tf.placeholder(\"float\", [None, timesteps, num_input])\n",
+    "Y = tf.placeholder(\"float\", [None, num_classes])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
     "# Define weights\n",
     "weights = {\n",
-    "    # Hidden layer weights => 2*n_hidden because of foward + backward cells\n",
-    "    'out': tf.Variable(tf.random_normal([2*n_hidden, n_classes]))\n",
+    "    # Hidden layer weights => 2*n_hidden because of forward + backward cells\n",
+    "    'out': tf.Variable(tf.random_normal([2*num_hidden, num_classes]))\n",
     "}\n",
     "biases = {\n",
-    "    'out': tf.Variable(tf.random_normal([n_classes]))\n",
+    "    'out': tf.Variable(tf.random_normal([num_classes]))\n",
     "}"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "metadata": {
     "collapsed": false
    },
@@ -90,18 +119,18 @@
    "source": [
     "def BiRNN(x, weights, biases):\n",
     "\n",
-    "    # Prepare data shape to match `bidirectional_rnn` function requirements\n",
-    "    # Current data input shape: (batch_size, n_steps, n_input)\n",
-    "    # Required shape: 'n_steps' tensors list of shape (batch_size, n_input)\n",
-    "    \n",
-    "    # Unstack to get a list of 'n_steps' tensors of shape (batch_size, n_input)\n",
-    "    x = tf.unstack(x, n_steps, 1)\n",
+    "    # Prepare data shape to match `rnn` function requirements\n",
+    "    # Current data input shape: (batch_size, timesteps, n_input)\n",
+    "    # Required shape: 'timesteps' tensors list of shape (batch_size, num_input)\n",
+    "\n",
+    "    # Unstack to get a list of 'timesteps' tensors of shape (batch_size, num_input)\n",
+    "    x = tf.unstack(x, timesteps, 1)\n",
     "\n",
     "    # Define lstm cells with tensorflow\n",
     "    # Forward direction cell\n",
-    "    lstm_fw_cell = rnn.BasicLSTMCell(n_hidden, forget_bias=1.0)\n",
+    "    lstm_fw_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)\n",
     "    # Backward direction cell\n",
-    "    lstm_bw_cell = rnn.BasicLSTMCell(n_hidden, forget_bias=1.0)\n",
+    "    lstm_bw_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)\n",
     "\n",
     "    # Get lstm cell output\n",
     "    try:\n",
@@ -112,25 +141,37 @@
     "                                        dtype=tf.float32)\n",
     "\n",
     "    # Linear activation, using rnn inner loop last output\n",
-    "    return tf.matmul(outputs[-1], weights['out']) + biases['out']\n",
-    "\n",
-    "pred = BiRNN(x, weights, biases)\n",
+    "    return tf.matmul(outputs[-1], weights['out']) + biases['out']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "logits = BiRNN(X, weights, biases)\n",
+    "prediction = tf.nn.softmax(logits)\n",
     "\n",
     "# Define loss and optimizer\n",
-    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))\n",
-    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n",
+    "loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
+    "    logits=logits, labels=Y))\n",
+    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
+    "train_op = optimizer.minimize(loss_op)\n",
     "\n",
-    "# Evaluate model\n",
-    "correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))\n",
+    "# Evaluate model (with test logits, for dropout to be disabled)\n",
+    "correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
     "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
     "\n",
-    "# Initializing the variables\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
     "init = tf.global_variables_initializer()"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 6,
    "metadata": {
     "collapsed": false
    },
@@ -139,118 +180,91 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Iter 1280, Minibatch Loss= 1.557283, Training Accuracy= 0.49219\n",
-      "Iter 2560, Minibatch Loss= 1.358445, Training Accuracy= 0.56250\n",
-      "Iter 3840, Minibatch Loss= 1.043732, Training Accuracy= 0.64062\n",
-      "Iter 5120, Minibatch Loss= 0.796770, Training Accuracy= 0.72656\n",
-      "Iter 6400, Minibatch Loss= 0.626206, Training Accuracy= 0.72656\n",
-      "Iter 7680, Minibatch Loss= 1.025919, Training Accuracy= 0.65625\n",
-      "Iter 8960, Minibatch Loss= 0.744850, Training Accuracy= 0.76562\n",
-      "Iter 10240, Minibatch Loss= 0.530111, Training Accuracy= 0.84375\n",
-      "Iter 11520, Minibatch Loss= 0.383806, Training Accuracy= 0.86719\n",
-      "Iter 12800, Minibatch Loss= 0.607816, Training Accuracy= 0.82812\n",
-      "Iter 14080, Minibatch Loss= 0.410879, Training Accuracy= 0.89062\n",
-      "Iter 15360, Minibatch Loss= 0.335351, Training Accuracy= 0.89844\n",
-      "Iter 16640, Minibatch Loss= 0.428004, Training Accuracy= 0.91406\n",
-      "Iter 17920, Minibatch Loss= 0.307468, Training Accuracy= 0.91406\n",
-      "Iter 19200, Minibatch Loss= 0.249527, Training Accuracy= 0.92188\n",
-      "Iter 20480, Minibatch Loss= 0.148163, Training Accuracy= 0.96094\n",
-      "Iter 21760, Minibatch Loss= 0.445275, Training Accuracy= 0.83594\n",
-      "Iter 23040, Minibatch Loss= 0.173083, Training Accuracy= 0.93750\n",
-      "Iter 24320, Minibatch Loss= 0.373696, Training Accuracy= 0.87500\n",
-      "Iter 25600, Minibatch Loss= 0.509869, Training Accuracy= 0.85938\n",
-      "Iter 26880, Minibatch Loss= 0.198096, Training Accuracy= 0.92969\n",
-      "Iter 28160, Minibatch Loss= 0.228221, Training Accuracy= 0.92188\n",
-      "Iter 29440, Minibatch Loss= 0.280088, Training Accuracy= 0.89844\n",
-      "Iter 30720, Minibatch Loss= 0.300495, Training Accuracy= 0.91406\n",
-      "Iter 32000, Minibatch Loss= 0.171746, Training Accuracy= 0.95312\n",
-      "Iter 33280, Minibatch Loss= 0.263745, Training Accuracy= 0.89844\n",
-      "Iter 34560, Minibatch Loss= 0.177300, Training Accuracy= 0.93750\n",
-      "Iter 35840, Minibatch Loss= 0.160621, Training Accuracy= 0.95312\n",
-      "Iter 37120, Minibatch Loss= 0.321745, Training Accuracy= 0.91406\n",
-      "Iter 38400, Minibatch Loss= 0.188322, Training Accuracy= 0.93750\n",
-      "Iter 39680, Minibatch Loss= 0.104025, Training Accuracy= 0.96875\n",
-      "Iter 40960, Minibatch Loss= 0.291053, Training Accuracy= 0.89062\n",
-      "Iter 42240, Minibatch Loss= 0.131189, Training Accuracy= 0.95312\n",
-      "Iter 43520, Minibatch Loss= 0.154949, Training Accuracy= 0.92969\n",
-      "Iter 44800, Minibatch Loss= 0.150411, Training Accuracy= 0.93750\n",
-      "Iter 46080, Minibatch Loss= 0.117008, Training Accuracy= 0.96094\n",
-      "Iter 47360, Minibatch Loss= 0.181344, Training Accuracy= 0.96094\n",
-      "Iter 48640, Minibatch Loss= 0.209197, Training Accuracy= 0.94531\n",
-      "Iter 49920, Minibatch Loss= 0.159350, Training Accuracy= 0.96094\n",
-      "Iter 51200, Minibatch Loss= 0.124001, Training Accuracy= 0.95312\n",
-      "Iter 52480, Minibatch Loss= 0.165183, Training Accuracy= 0.94531\n",
-      "Iter 53760, Minibatch Loss= 0.046438, Training Accuracy= 0.97656\n",
-      "Iter 55040, Minibatch Loss= 0.199995, Training Accuracy= 0.91406\n",
-      "Iter 56320, Minibatch Loss= 0.057071, Training Accuracy= 0.97656\n",
-      "Iter 57600, Minibatch Loss= 0.177065, Training Accuracy= 0.92188\n",
-      "Iter 58880, Minibatch Loss= 0.091666, Training Accuracy= 0.96094\n",
-      "Iter 60160, Minibatch Loss= 0.069232, Training Accuracy= 0.96875\n",
-      "Iter 61440, Minibatch Loss= 0.127353, Training Accuracy= 0.94531\n",
-      "Iter 62720, Minibatch Loss= 0.095795, Training Accuracy= 0.96094\n",
-      "Iter 64000, Minibatch Loss= 0.202651, Training Accuracy= 0.96875\n",
-      "Iter 65280, Minibatch Loss= 0.118779, Training Accuracy= 0.95312\n",
-      "Iter 66560, Minibatch Loss= 0.043173, Training Accuracy= 0.98438\n",
-      "Iter 67840, Minibatch Loss= 0.152280, Training Accuracy= 0.95312\n",
-      "Iter 69120, Minibatch Loss= 0.085301, Training Accuracy= 0.96875\n",
-      "Iter 70400, Minibatch Loss= 0.093421, Training Accuracy= 0.96094\n",
-      "Iter 71680, Minibatch Loss= 0.096358, Training Accuracy= 0.96875\n",
-      "Iter 72960, Minibatch Loss= 0.053386, Training Accuracy= 0.98438\n",
-      "Iter 74240, Minibatch Loss= 0.065237, Training Accuracy= 0.97656\n",
-      "Iter 75520, Minibatch Loss= 0.228090, Training Accuracy= 0.92188\n",
-      "Iter 76800, Minibatch Loss= 0.106751, Training Accuracy= 0.95312\n",
-      "Iter 78080, Minibatch Loss= 0.187795, Training Accuracy= 0.94531\n",
-      "Iter 79360, Minibatch Loss= 0.092611, Training Accuracy= 0.96094\n",
-      "Iter 80640, Minibatch Loss= 0.137386, Training Accuracy= 0.96875\n",
-      "Iter 81920, Minibatch Loss= 0.106634, Training Accuracy= 0.98438\n",
-      "Iter 83200, Minibatch Loss= 0.111749, Training Accuracy= 0.94531\n",
-      "Iter 84480, Minibatch Loss= 0.191184, Training Accuracy= 0.94531\n",
-      "Iter 85760, Minibatch Loss= 0.063982, Training Accuracy= 0.96094\n",
-      "Iter 87040, Minibatch Loss= 0.092380, Training Accuracy= 0.96875\n",
-      "Iter 88320, Minibatch Loss= 0.089899, Training Accuracy= 0.97656\n",
-      "Iter 89600, Minibatch Loss= 0.141107, Training Accuracy= 0.94531\n",
-      "Iter 90880, Minibatch Loss= 0.075549, Training Accuracy= 0.96094\n",
-      "Iter 92160, Minibatch Loss= 0.186539, Training Accuracy= 0.94531\n",
-      "Iter 93440, Minibatch Loss= 0.079639, Training Accuracy= 0.97656\n",
-      "Iter 94720, Minibatch Loss= 0.156895, Training Accuracy= 0.95312\n",
-      "Iter 96000, Minibatch Loss= 0.088042, Training Accuracy= 0.97656\n",
-      "Iter 97280, Minibatch Loss= 0.076670, Training Accuracy= 0.96875\n",
-      "Iter 98560, Minibatch Loss= 0.051336, Training Accuracy= 0.97656\n",
-      "Iter 99840, Minibatch Loss= 0.086923, Training Accuracy= 0.98438\n",
+      "Step 1, Minibatch Loss= 2.6218, Training Accuracy= 0.086\n",
+      "Step 200, Minibatch Loss= 2.1900, Training Accuracy= 0.211\n",
+      "Step 400, Minibatch Loss= 2.0144, Training Accuracy= 0.375\n",
+      "Step 600, Minibatch Loss= 1.8729, Training Accuracy= 0.445\n",
+      "Step 800, Minibatch Loss= 1.8000, Training Accuracy= 0.469\n",
+      "Step 1000, Minibatch Loss= 1.7244, Training Accuracy= 0.453\n",
+      "Step 1200, Minibatch Loss= 1.5657, Training Accuracy= 0.523\n",
+      "Step 1400, Minibatch Loss= 1.5473, Training Accuracy= 0.547\n",
+      "Step 1600, Minibatch Loss= 1.5288, Training Accuracy= 0.500\n",
+      "Step 1800, Minibatch Loss= 1.4203, Training Accuracy= 0.555\n",
+      "Step 2000, Minibatch Loss= 1.2525, Training Accuracy= 0.641\n",
+      "Step 2200, Minibatch Loss= 1.2696, Training Accuracy= 0.594\n",
+      "Step 2400, Minibatch Loss= 1.2000, Training Accuracy= 0.664\n",
+      "Step 2600, Minibatch Loss= 1.1017, Training Accuracy= 0.625\n",
+      "Step 2800, Minibatch Loss= 1.2656, Training Accuracy= 0.578\n",
+      "Step 3000, Minibatch Loss= 1.0830, Training Accuracy= 0.656\n",
+      "Step 3200, Minibatch Loss= 1.1522, Training Accuracy= 0.633\n",
+      "Step 3400, Minibatch Loss= 0.9484, Training Accuracy= 0.680\n",
+      "Step 3600, Minibatch Loss= 1.0470, Training Accuracy= 0.641\n",
+      "Step 3800, Minibatch Loss= 1.0609, Training Accuracy= 0.586\n",
+      "Step 4000, Minibatch Loss= 1.1853, Training Accuracy= 0.648\n",
+      "Step 4200, Minibatch Loss= 0.9438, Training Accuracy= 0.750\n",
+      "Step 4400, Minibatch Loss= 0.7986, Training Accuracy= 0.766\n",
+      "Step 4600, Minibatch Loss= 0.8070, Training Accuracy= 0.750\n",
+      "Step 4800, Minibatch Loss= 0.8382, Training Accuracy= 0.734\n",
+      "Step 5000, Minibatch Loss= 0.7397, Training Accuracy= 0.766\n",
+      "Step 5200, Minibatch Loss= 0.7870, Training Accuracy= 0.727\n",
+      "Step 5400, Minibatch Loss= 0.6380, Training Accuracy= 0.828\n",
+      "Step 5600, Minibatch Loss= 0.7975, Training Accuracy= 0.719\n",
+      "Step 5800, Minibatch Loss= 0.7934, Training Accuracy= 0.766\n",
+      "Step 6000, Minibatch Loss= 0.6628, Training Accuracy= 0.805\n",
+      "Step 6200, Minibatch Loss= 0.7958, Training Accuracy= 0.672\n",
+      "Step 6400, Minibatch Loss= 0.6582, Training Accuracy= 0.773\n",
+      "Step 6600, Minibatch Loss= 0.5908, Training Accuracy= 0.812\n",
+      "Step 6800, Minibatch Loss= 0.6182, Training Accuracy= 0.820\n",
+      "Step 7000, Minibatch Loss= 0.5513, Training Accuracy= 0.812\n",
+      "Step 7200, Minibatch Loss= 0.6683, Training Accuracy= 0.789\n",
+      "Step 7400, Minibatch Loss= 0.5337, Training Accuracy= 0.828\n",
+      "Step 7600, Minibatch Loss= 0.6428, Training Accuracy= 0.805\n",
+      "Step 7800, Minibatch Loss= 0.6708, Training Accuracy= 0.797\n",
+      "Step 8000, Minibatch Loss= 0.4664, Training Accuracy= 0.852\n",
+      "Step 8200, Minibatch Loss= 0.4249, Training Accuracy= 0.859\n",
+      "Step 8400, Minibatch Loss= 0.7723, Training Accuracy= 0.773\n",
+      "Step 8600, Minibatch Loss= 0.4706, Training Accuracy= 0.859\n",
+      "Step 8800, Minibatch Loss= 0.4800, Training Accuracy= 0.867\n",
+      "Step 9000, Minibatch Loss= 0.4636, Training Accuracy= 0.891\n",
+      "Step 9200, Minibatch Loss= 0.5734, Training Accuracy= 0.828\n",
+      "Step 9400, Minibatch Loss= 0.5548, Training Accuracy= 0.875\n",
+      "Step 9600, Minibatch Loss= 0.3575, Training Accuracy= 0.922\n",
+      "Step 9800, Minibatch Loss= 0.4566, Training Accuracy= 0.844\n",
+      "Step 10000, Minibatch Loss= 0.5125, Training Accuracy= 0.844\n",
       "Optimization Finished!\n",
-      "Testing Accuracy: 0.960938\n"
+      "Testing Accuracy: 0.890625\n"
      ]
     }
    ],
    "source": [
-    "# Launch the graph\n",
+    "# Start training\n",
     "with tf.Session() as sess:\n",
+    "\n",
+    "    # Run the initializer\n",
     "    sess.run(init)\n",
-    "    step = 1\n",
-    "    # Keep training until reach max iterations\n",
-    "    while step * batch_size < training_iters:\n",
+    "\n",
+    "    for step in range(1, training_steps+1):\n",
     "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
     "        # Reshape data to get 28 seq of 28 elements\n",
-    "        batch_x = batch_x.reshape((batch_size, n_steps, n_input))\n",
+    "        batch_x = batch_x.reshape((batch_size, timesteps, num_input))\n",
     "        # Run optimization op (backprop)\n",
-    "        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})\n",
-    "        if step % display_step == 0:\n",
-    "            # Calculate batch accuracy\n",
-    "            acc = sess.run(accuracy, feed_dict={x: batch_x, y: batch_y})\n",
-    "            # Calculate batch loss\n",
-    "            loss = sess.run(cost, feed_dict={x: batch_x, y: batch_y})\n",
-    "            print \"Iter \" + str(step*batch_size) + \", Minibatch Loss= \" + \\\n",
-    "                  \"{:.6f}\".format(loss) + \", Training Accuracy= \" + \\\n",
-    "                  \"{:.5f}\".format(acc)\n",
-    "        step += 1\n",
-    "    print \"Optimization Finished!\"\n",
+    "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n",
+    "        if step % display_step == 0 or step == 1:\n",
+    "            # Calculate batch loss and accuracy\n",
+    "            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n",
+    "                                                                 Y: batch_y})\n",
+    "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
+    "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
+    "                  \"{:.3f}\".format(acc))\n",
+    "\n",
+    "    print(\"Optimization Finished!\")\n",
     "\n",
     "    # Calculate accuracy for 128 mnist test images\n",
     "    test_len = 128\n",
-    "    test_data = mnist.test.images[:test_len].reshape((-1, n_steps, n_input))\n",
+    "    test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))\n",
     "    test_label = mnist.test.labels[:test_len]\n",
-    "    print \"Testing Accuracy:\", \\\n",
-    "        sess.run(accuracy, feed_dict={x: test_data, y: test_label})"
+    "    print(\"Testing Accuracy:\", \\\n",
+    "        sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))\n"
    ]
   },
   {

notebooks/3_NeuralNetworks/convolutional_network_raw.ipynb

@@ -0,0 +1,303 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "# Convolutional Neural Network Example\n",
+    "\n",
+    "Build a convolutional neural network with TensorFlow.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## CNN Overview\n",
+    "\n",
+    "![CNN](http://personal.ie.cuhk.edu.hk/~ccloy/project_target_code/images/fig3.png)\n",
+    "\n",
+    "## MNIST Dataset Overview\n",
+    "\n",
+    "This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flatten and converted to a 1-D numpy array of 784 features (28*28).\n",
+    "\n",
+    "![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n",
+    "\n",
+    "More info: http://yann.lecun.com/exdb/mnist/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
+     ]
+    }
+   ],
+   "source": [
+    "from __future__ import division, print_function, absolute_import\n",
+    "\n",
+    "import tensorflow as tf\n",
+    "\n",
+    "# Import MNIST data\n",
+    "from tensorflow.examples.tutorials.mnist import input_data\n",
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Training Parameters\n",
+    "learning_rate = 0.001\n",
+    "num_steps = 500\n",
+    "batch_size = 128\n",
+    "display_step = 10\n",
+    "\n",
+    "# Network Parameters\n",
+    "num_input = 784 # MNIST data input (img shape: 28*28)\n",
+    "num_classes = 10 # MNIST total classes (0-9 digits)\n",
+    "dropout = 0.75 # Dropout, probability to keep units\n",
+    "\n",
+    "# tf Graph input\n",
+    "X = tf.placeholder(tf.float32, [None, num_input])\n",
+    "Y = tf.placeholder(tf.float32, [None, num_classes])\n",
+    "keep_prob = tf.placeholder(tf.float32) # dropout (keep probability)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Create some wrappers for simplicity\n",
+    "def conv2d(x, W, b, strides=1):\n",
+    "    # Conv2D wrapper, with bias and relu activation\n",
+    "    x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')\n",
+    "    x = tf.nn.bias_add(x, b)\n",
+    "    return tf.nn.relu(x)\n",
+    "\n",
+    "\n",
+    "def maxpool2d(x, k=2):\n",
+    "    # MaxPool2D wrapper\n",
+    "    return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],\n",
+    "                          padding='SAME')\n",
+    "\n",
+    "\n",
+    "# Create model\n",
+    "def conv_net(x, weights, biases, dropout):\n",
+    "    # MNIST data input is a 1-D vector of 784 features (28*28 pixels)\n",
+    "    # Reshape to match picture format [Height x Width x Channel]\n",
+    "    # Tensor input become 4-D: [Batch Size, Height, Width, Channel]\n",
+    "    x = tf.reshape(x, shape=[-1, 28, 28, 1])\n",
+    "\n",
+    "    # Convolution Layer\n",
+    "    conv1 = conv2d(x, weights['wc1'], biases['bc1'])\n",
+    "    # Max Pooling (down-sampling)\n",
+    "    conv1 = maxpool2d(conv1, k=2)\n",
+    "\n",
+    "    # Convolution Layer\n",
+    "    conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])\n",
+    "    # Max Pooling (down-sampling)\n",
+    "    conv2 = maxpool2d(conv2, k=2)\n",
+    "\n",
+    "    # Fully connected layer\n",
+    "    # Reshape conv2 output to fit fully connected layer input\n",
+    "    fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])\n",
+    "    fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])\n",
+    "    fc1 = tf.nn.relu(fc1)\n",
+    "    # Apply Dropout\n",
+    "    fc1 = tf.nn.dropout(fc1, dropout)\n",
+    "\n",
+    "    # Output, class prediction\n",
+    "    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])\n",
+    "    return out"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Store layers weight & bias\n",
+    "weights = {\n",
+    "    # 5x5 conv, 1 input, 32 outputs\n",
+    "    'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32])),\n",
+    "    # 5x5 conv, 32 inputs, 64 outputs\n",
+    "    'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])),\n",
+    "    # fully connected, 7*7*64 inputs, 1024 outputs\n",
+    "    'wd1': tf.Variable(tf.random_normal([7*7*64, 1024])),\n",
+    "    # 1024 inputs, 10 outputs (class prediction)\n",
+    "    'out': tf.Variable(tf.random_normal([1024, num_classes]))\n",
+    "}\n",
+    "\n",
+    "biases = {\n",
+    "    'bc1': tf.Variable(tf.random_normal([32])),\n",
+    "    'bc2': tf.Variable(tf.random_normal([64])),\n",
+    "    'bd1': tf.Variable(tf.random_normal([1024])),\n",
+    "    'out': tf.Variable(tf.random_normal([num_classes]))\n",
+    "}\n",
+    "\n",
+    "# Construct model\n",
+    "logits = conv_net(X, weights, biases, keep_prob)\n",
+    "prediction = tf.nn.softmax(logits)\n",
+    "\n",
+    "# Define loss and optimizer\n",
+    "loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
+    "    logits=logits, labels=Y))\n",
+    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
+    "train_op = optimizer.minimize(loss_op)\n",
+    "\n",
+    "\n",
+    "# Evaluate model\n",
+    "correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
+    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
+    "\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
+    "init = tf.global_variables_initializer()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Step 1, Minibatch Loss= 63763.3047, Training Accuracy= 0.141\n",
+      "Step 10, Minibatch Loss= 26429.6680, Training Accuracy= 0.242\n",
+      "Step 20, Minibatch Loss= 12171.8584, Training Accuracy= 0.586\n",
+      "Step 30, Minibatch Loss= 6306.6318, Training Accuracy= 0.734\n",
+      "Step 40, Minibatch Loss= 5113.7583, Training Accuracy= 0.711\n",
+      "Step 50, Minibatch Loss= 4022.2131, Training Accuracy= 0.805\n",
+      "Step 60, Minibatch Loss= 3125.4949, Training Accuracy= 0.867\n",
+      "Step 70, Minibatch Loss= 2225.4875, Training Accuracy= 0.875\n",
+      "Step 80, Minibatch Loss= 1843.3540, Training Accuracy= 0.867\n",
+      "Step 90, Minibatch Loss= 1715.7744, Training Accuracy= 0.875\n",
+      "Step 100, Minibatch Loss= 2611.2708, Training Accuracy= 0.906\n",
+      "Step 110, Minibatch Loss= 4804.0913, Training Accuracy= 0.875\n",
+      "Step 120, Minibatch Loss= 1067.5258, Training Accuracy= 0.938\n",
+      "Step 130, Minibatch Loss= 2519.1514, Training Accuracy= 0.898\n",
+      "Step 140, Minibatch Loss= 2687.9292, Training Accuracy= 0.906\n",
+      "Step 150, Minibatch Loss= 1983.4077, Training Accuracy= 0.938\n",
+      "Step 160, Minibatch Loss= 2844.6553, Training Accuracy= 0.930\n",
+      "Step 170, Minibatch Loss= 3602.2524, Training Accuracy= 0.914\n",
+      "Step 180, Minibatch Loss= 175.3922, Training Accuracy= 0.961\n",
+      "Step 190, Minibatch Loss= 645.1918, Training Accuracy= 0.945\n",
+      "Step 200, Minibatch Loss= 1147.6567, Training Accuracy= 0.938\n",
+      "Step 210, Minibatch Loss= 1140.4148, Training Accuracy= 0.914\n",
+      "Step 220, Minibatch Loss= 1572.8756, Training Accuracy= 0.906\n",
+      "Step 230, Minibatch Loss= 1292.9274, Training Accuracy= 0.898\n",
+      "Step 240, Minibatch Loss= 1501.4623, Training Accuracy= 0.953\n",
+      "Step 250, Minibatch Loss= 1908.2997, Training Accuracy= 0.898\n",
+      "Step 260, Minibatch Loss= 2182.2380, Training Accuracy= 0.898\n",
+      "Step 270, Minibatch Loss= 487.5807, Training Accuracy= 0.961\n",
+      "Step 280, Minibatch Loss= 1284.1130, Training Accuracy= 0.945\n",
+      "Step 290, Minibatch Loss= 1232.4919, Training Accuracy= 0.891\n",
+      "Step 300, Minibatch Loss= 1198.8336, Training Accuracy= 0.945\n",
+      "Step 310, Minibatch Loss= 2010.5345, Training Accuracy= 0.906\n",
+      "Step 320, Minibatch Loss= 786.3917, Training Accuracy= 0.945\n",
+      "Step 330, Minibatch Loss= 1408.3556, Training Accuracy= 0.898\n",
+      "Step 340, Minibatch Loss= 1453.7538, Training Accuracy= 0.953\n",
+      "Step 350, Minibatch Loss= 999.8901, Training Accuracy= 0.906\n",
+      "Step 360, Minibatch Loss= 914.3958, Training Accuracy= 0.961\n",
+      "Step 370, Minibatch Loss= 488.0052, Training Accuracy= 0.938\n",
+      "Step 380, Minibatch Loss= 1070.8710, Training Accuracy= 0.922\n",
+      "Step 390, Minibatch Loss= 151.4658, Training Accuracy= 0.961\n",
+      "Step 400, Minibatch Loss= 555.3539, Training Accuracy= 0.953\n",
+      "Step 410, Minibatch Loss= 765.5746, Training Accuracy= 0.945\n",
+      "Step 420, Minibatch Loss= 326.9393, Training Accuracy= 0.969\n",
+      "Step 430, Minibatch Loss= 530.8968, Training Accuracy= 0.977\n",
+      "Step 440, Minibatch Loss= 463.3909, Training Accuracy= 0.977\n",
+      "Step 450, Minibatch Loss= 362.2226, Training Accuracy= 0.977\n",
+      "Step 460, Minibatch Loss= 414.0034, Training Accuracy= 0.953\n",
+      "Step 470, Minibatch Loss= 583.4587, Training Accuracy= 0.945\n",
+      "Step 480, Minibatch Loss= 566.1262, Training Accuracy= 0.969\n",
+      "Step 490, Minibatch Loss= 691.1143, Training Accuracy= 0.961\n",
+      "Step 500, Minibatch Loss= 282.8893, Training Accuracy= 0.984\n",
+      "Optimization Finished!\n",
+      "Testing Accuracy: 0.976562\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Start training\n",
+    "with tf.Session() as sess:\n",
+    "\n",
+    "    # Run the initializer\n",
+    "    sess.run(init)\n",
+    "\n",
+    "    for step in range(1, num_steps+1):\n",
+    "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
+    "        # Run optimization op (backprop)\n",
+    "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y, keep_prob: dropout})\n",
+    "        if step % display_step == 0 or step == 1:\n",
+    "            # Calculate batch loss and accuracy\n",
+    "            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n",
+    "                                                                 Y: batch_y,\n",
+    "                                                                 keep_prob: 1.0})\n",
+    "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
+    "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
+    "                  \"{:.3f}\".format(acc))\n",
+    "\n",
+    "    print(\"Optimization Finished!\")\n",
+    "\n",
+    "    # Calculate accuracy for 256 MNIST test images\n",
+    "    print(\"Testing Accuracy:\", \\\n",
+    "        sess.run(accuracy, feed_dict={X: mnist.test.images[:256],\n",
+    "                                      Y: mnist.test.labels[:256],\n",
+    "                                      keep_prob: 1.0}))\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

notebooks/3_NeuralNetworks/dynamic_rnn.ipynb

@@ -0,0 +1,355 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Dynamic Recurrent Neural Network.\n",
+    "\n",
+    "TensorFlow implementation of a Recurrent Neural Network (LSTM) that performs dynamic computation over sequences with variable length. This example is using a toy dataset to classify linear sequences. The generated sequences have variable length.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## RNN Overview\n",
+    "\n",
+    "<img src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png\" alt=\"nn\" style=\"width: 600px;\"/>\n",
+    "\n",
+    "References:\n",
+    "- [Long Short Term Memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf), Sepp Hochreiter & Jurgen Schmidhuber, Neural Computation 9(8): 1735-1780, 1997."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "from __future__ import print_function\n",
+    "\n",
+    "import tensorflow as tf\n",
+    "import random"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# ====================\n",
+    "#  TOY DATA GENERATOR\n",
+    "# ====================\n",
+    "\n",
+    "class ToySequenceData(object):\n",
+    "    \"\"\" Generate sequence of data with dynamic length.\n",
+    "    This class generate samples for training:\n",
+    "    - Class 0: linear sequences (i.e. [0, 1, 2, 3,...])\n",
+    "    - Class 1: random sequences (i.e. [1, 3, 10, 7,...])\n",
+    "\n",
+    "    NOTICE:\n",
+    "    We have to pad each sequence to reach 'max_seq_len' for TensorFlow\n",
+    "    consistency (we cannot feed a numpy array with inconsistent\n",
+    "    dimensions). The dynamic calculation will then be perform thanks to\n",
+    "    'seqlen' attribute that records every actual sequence length.\n",
+    "    \"\"\"\n",
+    "    def __init__(self, n_samples=1000, max_seq_len=20, min_seq_len=3,\n",
+    "                 max_value=1000):\n",
+    "        self.data = []\n",
+    "        self.labels = []\n",
+    "        self.seqlen = []\n",
+    "        for i in range(n_samples):\n",
+    "            # Random sequence length\n",
+    "            len = random.randint(min_seq_len, max_seq_len)\n",
+    "            # Monitor sequence length for TensorFlow dynamic calculation\n",
+    "            self.seqlen.append(len)\n",
+    "            # Add a random or linear int sequence (50% prob)\n",
+    "            if random.random() < .5:\n",
+    "                # Generate a linear sequence\n",
+    "                rand_start = random.randint(0, max_value - len)\n",
+    "                s = [[float(i)/max_value] for i in\n",
+    "                     range(rand_start, rand_start + len)]\n",
+    "                # Pad sequence for dimension consistency\n",
+    "                s += [[0.] for i in range(max_seq_len - len)]\n",
+    "                self.data.append(s)\n",
+    "                self.labels.append([1., 0.])\n",
+    "            else:\n",
+    "                # Generate a random sequence\n",
+    "                s = [[float(random.randint(0, max_value))/max_value]\n",
+    "                     for i in range(len)]\n",
+    "                # Pad sequence for dimension consistency\n",
+    "                s += [[0.] for i in range(max_seq_len - len)]\n",
+    "                self.data.append(s)\n",
+    "                self.labels.append([0., 1.])\n",
+    "        self.batch_id = 0\n",
+    "\n",
+    "    def next(self, batch_size):\n",
+    "        \"\"\" Return a batch of data. When dataset end is reached, start over.\n",
+    "        \"\"\"\n",
+    "        if self.batch_id == len(self.data):\n",
+    "            self.batch_id = 0\n",
+    "        batch_data = (self.data[self.batch_id:min(self.batch_id +\n",
+    "                                                  batch_size, len(self.data))])\n",
+    "        batch_labels = (self.labels[self.batch_id:min(self.batch_id +\n",
+    "                                                  batch_size, len(self.data))])\n",
+    "        batch_seqlen = (self.seqlen[self.batch_id:min(self.batch_id +\n",
+    "                                                  batch_size, len(self.data))])\n",
+    "        self.batch_id = min(self.batch_id + batch_size, len(self.data))\n",
+    "        return batch_data, batch_labels, batch_seqlen"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# ==========\n",
+    "#   MODEL\n",
+    "# ==========\n",
+    "\n",
+    "# Parameters\n",
+    "learning_rate = 0.01\n",
+    "training_steps = 10000\n",
+    "batch_size = 128\n",
+    "display_step = 200\n",
+    "\n",
+    "# Network Parameters\n",
+    "seq_max_len = 20 # Sequence max length\n",
+    "n_hidden = 64 # hidden layer num of features\n",
+    "n_classes = 2 # linear sequence or not\n",
+    "\n",
+    "trainset = ToySequenceData(n_samples=1000, max_seq_len=seq_max_len)\n",
+    "testset = ToySequenceData(n_samples=500, max_seq_len=seq_max_len)\n",
+    "\n",
+    "# tf Graph input\n",
+    "x = tf.placeholder(\"float\", [None, seq_max_len, 1])\n",
+    "y = tf.placeholder(\"float\", [None, n_classes])\n",
+    "# A placeholder for indicating each sequence length\n",
+    "seqlen = tf.placeholder(tf.int32, [None])\n",
+    "\n",
+    "# Define weights\n",
+    "weights = {\n",
+    "    'out': tf.Variable(tf.random_normal([n_hidden, n_classes]))\n",
+    "}\n",
+    "biases = {\n",
+    "    'out': tf.Variable(tf.random_normal([n_classes]))\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "def dynamicRNN(x, seqlen, weights, biases):\n",
+    "\n",
+    "    # Prepare data shape to match `rnn` function requirements\n",
+    "    # Current data input shape: (batch_size, n_steps, n_input)\n",
+    "    # Required shape: 'n_steps' tensors list of shape (batch_size, n_input)\n",
+    "    \n",
+    "    # Unstack to get a list of 'n_steps' tensors of shape (batch_size, n_input)\n",
+    "    x = tf.unstack(x, seq_max_len, 1)\n",
+    "\n",
+    "    # Define a lstm cell with tensorflow\n",
+    "    lstm_cell = tf.contrib.rnn.BasicLSTMCell(n_hidden)\n",
+    "\n",
+    "    # Get lstm cell output, providing 'sequence_length' will perform dynamic\n",
+    "    # calculation.\n",
+    "    outputs, states = tf.contrib.rnn.static_rnn(lstm_cell, x, dtype=tf.float32,\n",
+    "                                sequence_length=seqlen)\n",
+    "\n",
+    "    # When performing dynamic calculation, we must retrieve the last\n",
+    "    # dynamically computed output, i.e., if a sequence length is 10, we need\n",
+    "    # to retrieve the 10th output.\n",
+    "    # However TensorFlow doesn't support advanced indexing yet, so we build\n",
+    "    # a custom op that for each sample in batch size, get its length and\n",
+    "    # get the corresponding relevant output.\n",
+    "\n",
+    "    # 'outputs' is a list of output at every timestep, we pack them in a Tensor\n",
+    "    # and change back dimension to [batch_size, n_step, n_input]\n",
+    "    outputs = tf.stack(outputs)\n",
+    "    outputs = tf.transpose(outputs, [1, 0, 2])\n",
+    "\n",
+    "    # Hack to build the indexing and retrieve the right output.\n",
+    "    batch_size = tf.shape(outputs)[0]\n",
+    "    # Start indices for each sample\n",
+    "    index = tf.range(0, batch_size) * seq_max_len + (seqlen - 1)\n",
+    "    # Indexing\n",
+    "    outputs = tf.gather(tf.reshape(outputs, [-1, n_hidden]), index)\n",
+    "\n",
+    "    # Linear activation, using outputs computed above\n",
+    "    return tf.matmul(outputs, weights['out']) + biases['out']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/aymeric.damien/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
+      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
+     ]
+    }
+   ],
+   "source": [
+    "pred = dynamicRNN(x, seqlen, weights, biases)\n",
+    "\n",
+    "# Define loss and optimizer\n",
+    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))\n",
+    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)\n",
+    "\n",
+    "# Evaluate model\n",
+    "correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))\n",
+    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
+    "\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
+    "init = tf.global_variables_initializer()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "collapsed": false,
+    "scrolled": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Step 1, Minibatch Loss= 0.864517, Training Accuracy= 0.42188\n",
+      "Step 200, Minibatch Loss= 0.686012, Training Accuracy= 0.43269\n",
+      "Step 400, Minibatch Loss= 0.682970, Training Accuracy= 0.48077\n",
+      "Step 600, Minibatch Loss= 0.679640, Training Accuracy= 0.50962\n",
+      "Step 800, Minibatch Loss= 0.675208, Training Accuracy= 0.53846\n",
+      "Step 1000, Minibatch Loss= 0.668636, Training Accuracy= 0.56731\n",
+      "Step 1200, Minibatch Loss= 0.657525, Training Accuracy= 0.62500\n",
+      "Step 1400, Minibatch Loss= 0.635423, Training Accuracy= 0.67308\n",
+      "Step 1600, Minibatch Loss= 0.580433, Training Accuracy= 0.75962\n",
+      "Step 1800, Minibatch Loss= 0.475599, Training Accuracy= 0.81731\n",
+      "Step 2000, Minibatch Loss= 0.434865, Training Accuracy= 0.83654\n",
+      "Step 2200, Minibatch Loss= 0.423690, Training Accuracy= 0.85577\n",
+      "Step 2400, Minibatch Loss= 0.417472, Training Accuracy= 0.85577\n",
+      "Step 2600, Minibatch Loss= 0.412906, Training Accuracy= 0.85577\n",
+      "Step 2800, Minibatch Loss= 0.409193, Training Accuracy= 0.85577\n",
+      "Step 3000, Minibatch Loss= 0.406035, Training Accuracy= 0.86538\n",
+      "Step 3200, Minibatch Loss= 0.403287, Training Accuracy= 0.87500\n",
+      "Step 3400, Minibatch Loss= 0.400862, Training Accuracy= 0.87500\n",
+      "Step 3600, Minibatch Loss= 0.398704, Training Accuracy= 0.86538\n",
+      "Step 3800, Minibatch Loss= 0.396768, Training Accuracy= 0.86538\n",
+      "Step 4000, Minibatch Loss= 0.395017, Training Accuracy= 0.86538\n",
+      "Step 4200, Minibatch Loss= 0.393422, Training Accuracy= 0.86538\n",
+      "Step 4400, Minibatch Loss= 0.391957, Training Accuracy= 0.85577\n",
+      "Step 4600, Minibatch Loss= 0.390600, Training Accuracy= 0.85577\n",
+      "Step 4800, Minibatch Loss= 0.389334, Training Accuracy= 0.86538\n",
+      "Step 5000, Minibatch Loss= 0.388143, Training Accuracy= 0.86538\n",
+      "Step 5200, Minibatch Loss= 0.387015, Training Accuracy= 0.86538\n",
+      "Step 5400, Minibatch Loss= 0.385940, Training Accuracy= 0.86538\n",
+      "Step 5600, Minibatch Loss= 0.384907, Training Accuracy= 0.86538\n",
+      "Step 5800, Minibatch Loss= 0.383904, Training Accuracy= 0.85577\n",
+      "Step 6000, Minibatch Loss= 0.382921, Training Accuracy= 0.86538\n",
+      "Step 6200, Minibatch Loss= 0.381941, Training Accuracy= 0.86538\n",
+      "Step 6400, Minibatch Loss= 0.380947, Training Accuracy= 0.86538\n",
+      "Step 6600, Minibatch Loss= 0.379912, Training Accuracy= 0.86538\n",
+      "Step 6800, Minibatch Loss= 0.378796, Training Accuracy= 0.86538\n",
+      "Step 7000, Minibatch Loss= 0.377540, Training Accuracy= 0.86538\n",
+      "Step 7200, Minibatch Loss= 0.376041, Training Accuracy= 0.86538\n",
+      "Step 7400, Minibatch Loss= 0.374130, Training Accuracy= 0.85577\n",
+      "Step 7600, Minibatch Loss= 0.371514, Training Accuracy= 0.85577\n",
+      "Step 7800, Minibatch Loss= 0.367723, Training Accuracy= 0.85577\n",
+      "Step 8000, Minibatch Loss= 0.362049, Training Accuracy= 0.85577\n",
+      "Step 8200, Minibatch Loss= 0.353558, Training Accuracy= 0.85577\n",
+      "Step 8400, Minibatch Loss= 0.341072, Training Accuracy= 0.86538\n",
+      "Step 8600, Minibatch Loss= 0.323062, Training Accuracy= 0.87500\n",
+      "Step 8800, Minibatch Loss= 0.299278, Training Accuracy= 0.89423\n",
+      "Step 9000, Minibatch Loss= 0.273857, Training Accuracy= 0.90385\n",
+      "Step 9200, Minibatch Loss= 0.248392, Training Accuracy= 0.91346\n",
+      "Step 9400, Minibatch Loss= 0.221348, Training Accuracy= 0.92308\n",
+      "Step 9600, Minibatch Loss= 0.191947, Training Accuracy= 0.92308\n",
+      "Step 9800, Minibatch Loss= 0.159308, Training Accuracy= 0.93269\n",
+      "Step 10000, Minibatch Loss= 0.136938, Training Accuracy= 0.96154\n",
+      "Optimization Finished!\n",
+      "Testing Accuracy: 0.952\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Start training\n",
+    "with tf.Session() as sess:\n",
+    "\n",
+    "    # Run the initializer\n",
+    "    sess.run(init)\n",
+    "\n",
+    "    for step in range(1, training_steps+1):\n",
+    "        batch_x, batch_y, batch_seqlen = trainset.next(batch_size)\n",
+    "        # Run optimization op (backprop)\n",
+    "        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y,\n",
+    "                                       seqlen: batch_seqlen})\n",
+    "        if step % display_step == 0 or step == 1:\n",
+    "            # Calculate batch accuracy\n",
+    "            acc = sess.run(accuracy, feed_dict={x: batch_x, y: batch_y,\n",
+    "                                                seqlen: batch_seqlen})\n",
+    "            # Calculate batch loss\n",
+    "            loss = sess.run(cost, feed_dict={x: batch_x, y: batch_y,\n",
+    "                                             seqlen: batch_seqlen})\n",
+    "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
+    "                  \"{:.6f}\".format(loss) + \", Training Accuracy= \" + \\\n",
+    "                  \"{:.5f}\".format(acc))\n",
+    "\n",
+    "    print(\"Optimization Finished!\")\n",
+    "\n",
+    "    # Calculate accuracy\n",
+    "    test_data = testset.data\n",
+    "    test_label = testset.labels\n",
+    "    test_seqlen = testset.seqlen\n",
+    "    print(\"Testing Accuracy:\", \\\n",
+    "        sess.run(accuracy, feed_dict={x: test_data, y: test_label,\n",
+    "                                      seqlen: test_seqlen}))"
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python [default]",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

notebooks/3_NeuralNetworks/multilayer_perceptron.ipynb

@@ -1,216 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": [
-    "'''\n",
-    "A Multilayer Perceptron implementation example using TensorFlow library.\n",
-    "This example is using the MNIST database of handwritten digits\n",
-    "(http://yann.lecun.com/exdb/mnist/)\n",
-    "\n",
-    "Author: Aymeric Damien\n",
-    "Project: https://github.com/aymericdamien/TensorFlow-Examples/\n",
-    "'''"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
-      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
-      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
-      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Import MINST data\n",
-    "from tensorflow.examples.tutorials.mnist import input_data\n",
-    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
-    "\n",
-    "import tensorflow as tf"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": [
-    "# Parameters\n",
-    "learning_rate = 0.001\n",
-    "training_epochs = 15\n",
-    "batch_size = 100\n",
-    "display_step = 1\n",
-    "\n",
-    "# Network Parameters\n",
-    "n_hidden_1 = 256 # 1st layer number of features\n",
-    "n_hidden_2 = 256 # 2nd layer number of features\n",
-    "n_input = 784 # MNIST data input (img shape: 28*28)\n",
-    "n_classes = 10 # MNIST total classes (0-9 digits)\n",
-    "\n",
-    "# tf Graph input\n",
-    "x = tf.placeholder(\"float\", [None, n_input])\n",
-    "y = tf.placeholder(\"float\", [None, n_classes])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": [
-    "# Create model\n",
-    "def multilayer_perceptron(x, weights, biases):\n",
-    "    # Hidden layer with RELU activation\n",
-    "    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])\n",
-    "    layer_1 = tf.nn.relu(layer_1)\n",
-    "    # Hidden layer with RELU activation\n",
-    "    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])\n",
-    "    layer_2 = tf.nn.relu(layer_2)\n",
-    "    # Output layer with linear activation\n",
-    "    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']\n",
-    "    return out_layer"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "# Store layers weight & bias\n",
-    "weights = {\n",
-    "    'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),\n",
-    "    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),\n",
-    "    'out': tf.Variable(tf.random_normal([n_hidden_2, n_classes]))\n",
-    "}\n",
-    "biases = {\n",
-    "    'b1': tf.Variable(tf.random_normal([n_hidden_1])),\n",
-    "    'b2': tf.Variable(tf.random_normal([n_hidden_2])),\n",
-    "    'out': tf.Variable(tf.random_normal([n_classes]))\n",
-    "}\n",
-    "\n",
-    "# Construct model\n",
-    "pred = multilayer_perceptron(x, weights, biases)\n",
-    "\n",
-    "# Define loss and optimizer\n",
-    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))\n",
-    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n",
-    "\n",
-    "# Initializing the variables\n",
-    "init = tf.global_variables_initializer()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Epoch: 0001 cost= 173.056566575\n",
-      "Epoch: 0002 cost= 44.054413928\n",
-      "Epoch: 0003 cost= 27.455470655\n",
-      "Epoch: 0004 cost= 19.008652363\n",
-      "Epoch: 0005 cost= 13.654873594\n",
-      "Epoch: 0006 cost= 10.059267435\n",
-      "Epoch: 0007 cost= 7.436018432\n",
-      "Epoch: 0008 cost= 5.587794416\n",
-      "Epoch: 0009 cost= 4.209882509\n",
-      "Epoch: 0010 cost= 3.203879515\n",
-      "Epoch: 0011 cost= 2.319920681\n",
-      "Epoch: 0012 cost= 1.676204545\n",
-      "Epoch: 0013 cost= 1.248805338\n",
-      "Epoch: 0014 cost= 1.052676844\n",
-      "Epoch: 0015 cost= 0.890117338\n",
-      "Optimization Finished!\n",
-      "Accuracy: 0.9459\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Launch the graph\n",
-    "with tf.Session() as sess:\n",
-    "    sess.run(init)\n",
-    "\n",
-    "    # Training cycle\n",
-    "    for epoch in range(training_epochs):\n",
-    "        avg_cost = 0.\n",
-    "        total_batch = int(mnist.train.num_examples/batch_size)\n",
-    "        # Loop over all batches\n",
-    "        for i in range(total_batch):\n",
-    "            batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
-    "            # Run optimization op (backprop) and cost op (to get loss value)\n",
-    "            _, c = sess.run([optimizer, cost], feed_dict={x: batch_x,\n",
-    "                                                          y: batch_y})\n",
-    "            # Compute average loss\n",
-    "            avg_cost += c / total_batch\n",
-    "        # Display logs per epoch step\n",
-    "        if epoch % display_step == 0:\n",
-    "            print \"Epoch:\", '%04d' % (epoch+1), \"cost=\", \\\n",
-    "                \"{:.9f}\".format(avg_cost)\n",
-    "    print \"Optimization Finished!\"\n",
-    "\n",
-    "    # Test model\n",
-    "    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))\n",
-    "    # Calculate accuracy\n",
-    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
-    "    print \"Accuracy:\", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 2",
-   "language": "python",
-   "name": "python2"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 2
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython2",
-   "version": "2.7.13"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}

notebooks/3_NeuralNetworks/neural_network_raw.ipynb

@@ -0,0 +1,224 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "# Neural Network Example\n",
+    "\n",
+    "Build a 2-hidden layers fully connected neural network (a.k.a multilayer perceptron) with TensorFlow.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Neural Network Overview\n",
+    "\n",
+    "<img src=\"http://cs231n.github.io/assets/nn1/neural_net2.jpeg\" alt=\"nn\" style=\"width: 400px;\"/>\n",
+    "\n",
+    "## MNIST Dataset Overview\n",
+    "\n",
+    "This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flatten and converted to a 1-D numpy array of 784 features (28*28).\n",
+    "\n",
+    "![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n",
+    "\n",
+    "More info: http://yann.lecun.com/exdb/mnist/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
+     ]
+    }
+   ],
+   "source": [
+    "from __future__ import print_function\n",
+    "\n",
+    "# Import MNIST data\n",
+    "from tensorflow.examples.tutorials.mnist import input_data\n",
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
+    "\n",
+    "import tensorflow as tf"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Parameters\n",
+    "learning_rate = 0.1\n",
+    "num_steps = 500\n",
+    "batch_size = 128\n",
+    "display_step = 100\n",
+    "\n",
+    "# Network Parameters\n",
+    "n_hidden_1 = 256 # 1st layer number of neurons\n",
+    "n_hidden_2 = 256 # 2nd layer number of neurons\n",
+    "num_input = 784 # MNIST data input (img shape: 28*28)\n",
+    "num_classes = 10 # MNIST total classes (0-9 digits)\n",
+    "\n",
+    "# tf Graph input\n",
+    "X = tf.placeholder(\"float\", [None, num_input])\n",
+    "Y = tf.placeholder(\"float\", [None, num_classes])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Store layers weight & bias\n",
+    "weights = {\n",
+    "    'h1': tf.Variable(tf.random_normal([num_input, n_hidden_1])),\n",
+    "    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),\n",
+    "    'out': tf.Variable(tf.random_normal([n_hidden_2, num_classes]))\n",
+    "}\n",
+    "biases = {\n",
+    "    'b1': tf.Variable(tf.random_normal([n_hidden_1])),\n",
+    "    'b2': tf.Variable(tf.random_normal([n_hidden_2])),\n",
+    "    'out': tf.Variable(tf.random_normal([num_classes]))\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# Create model\n",
+    "def neural_net(x):\n",
+    "    # Hidden fully connected layer with 256 neurons\n",
+    "    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])\n",
+    "    # Hidden fully connected layer with 256 neurons\n",
+    "    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])\n",
+    "    # Output fully connected layer with a neuron for each class\n",
+    "    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']\n",
+    "    return out_layer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# Construct model\n",
+    "logits = neural_net(X)\n",
+    "\n",
+    "# Define loss and optimizer\n",
+    "loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
+    "    logits=logits, labels=Y))\n",
+    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
+    "train_op = optimizer.minimize(loss_op)\n",
+    "\n",
+    "# Evaluate model (with test logits, for dropout to be disabled)\n",
+    "correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(Y, 1))\n",
+    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
+    "\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
+    "init = tf.global_variables_initializer()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Step 1, Minibatch Loss= 13208.1406, Training Accuracy= 0.266\n",
+      "Step 100, Minibatch Loss= 462.8610, Training Accuracy= 0.867\n",
+      "Step 200, Minibatch Loss= 232.8298, Training Accuracy= 0.844\n",
+      "Step 300, Minibatch Loss= 85.2141, Training Accuracy= 0.891\n",
+      "Step 400, Minibatch Loss= 38.0552, Training Accuracy= 0.883\n",
+      "Step 500, Minibatch Loss= 55.3689, Training Accuracy= 0.867\n",
+      "Optimization Finished!\n",
+      "Testing Accuracy: 0.8729\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Start training\n",
+    "with tf.Session() as sess:\n",
+    "\n",
+    "    # Run the initializer\n",
+    "    sess.run(init)\n",
+    "\n",
+    "    for step in range(1, num_steps+1):\n",
+    "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
+    "        # Run optimization op (backprop)\n",
+    "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n",
+    "        if step % display_step == 0 or step == 1:\n",
+    "            # Calculate batch loss and accuracy\n",
+    "            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n",
+    "                                                                 Y: batch_y})\n",
+    "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
+    "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
+    "                  \"{:.3f}\".format(acc))\n",
+    "\n",
+    "    print(\"Optimization Finished!\")\n",
+    "\n",
+    "    # Calculate accuracy for MNIST test images\n",
+    "    print(\"Testing Accuracy:\", \\\n",
+    "        sess.run(accuracy, feed_dict={X: mnist.test.images,\n",
+    "                                      Y: mnist.test.labels}))"
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

notebooks/3_NeuralNetworks/recurrent_network.ipynb

@@ -6,44 +6,63 @@
     "collapsed": true
    },
    "source": [
-    "'''\n",
-    "A Reccurent Neural Network (LSTM) implementation example using TensorFlow library.\n",
-    "This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)\n",
-    "Long Short Term Memory paper: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf\n",
+    "# Recurrent Neural Network Example\n",
     "\n",
-    "Author: Aymeric Damien\n",
-    "Project: https://github.com/aymericdamien/TensorFlow-Examples/\n",
-    "'''"
+    "Build a recurrent neural network (LSTM) with TensorFlow.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## RNN Overview\n",
+    "\n",
+    "<img src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png\" alt=\"nn\" style=\"width: 600px;\"/>\n",
+    "\n",
+    "References:\n",
+    "- [Long Short Term Memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf), Sepp Hochreiter & Jurgen Schmidhuber, Neural Computation 9(8): 1735-1780, 1997.\n",
+    "\n",
+    "## MNIST Dataset Overview\n",
+    "\n",
+    "This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flatten and converted to a 1-D numpy array of 784 features (28*28).\n",
+    "\n",
+    "![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n",
+    "\n",
+    "To classify images using a recurrent neural network, we consider every image row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then handle 28 sequences of 28 timesteps for every sample.\n",
+    "\n",
+    "More info: http://yann.lecun.com/exdb/mnist/"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {
     "collapsed": false
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
+     ]
+    }
+   ],
    "source": [
+    "from __future__ import print_function\n",
+    "\n",
     "import tensorflow as tf\n",
     "from tensorflow.contrib import rnn\n",
-    "import numpy as np\n",
     "\n",
-    "# Import MINST data\n",
+    "# Import MNIST data\n",
     "from tensorflow.examples.tutorials.mnist import input_data\n",
-    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "collapsed": true
-   },
-   "source": [
-    "'''\n",
-    "To classify images using a reccurent neural network, we consider every image\n",
-    "row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then\n",
-    "handle 28 sequences of 28 steps for every sample.\n",
-    "'''"
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
    ]
   },
   {
@@ -54,34 +73,43 @@
    },
    "outputs": [],
    "source": [
-    "# Parameters\n",
+    "# Training Parameters\n",
     "learning_rate = 0.001\n",
-    "training_iters = 100000\n",
+    "training_steps = 10000\n",
     "batch_size = 128\n",
-    "display_step = 10\n",
+    "display_step = 200\n",
     "\n",
     "# Network Parameters\n",
-    "n_input = 28 # MNIST data input (img shape: 28*28)\n",
-    "n_steps = 28 # timesteps\n",
-    "n_hidden = 128 # hidden layer num of features\n",
-    "n_classes = 10 # MNIST total classes (0-9 digits)\n",
+    "num_input = 28 # MNIST data input (img shape: 28*28)\n",
+    "timesteps = 28 # timesteps\n",
+    "num_hidden = 128 # hidden layer num of features\n",
+    "num_classes = 10 # MNIST total classes (0-9 digits)\n",
     "\n",
     "# tf Graph input\n",
-    "x = tf.placeholder(\"float\", [None, n_steps, n_input])\n",
-    "y = tf.placeholder(\"float\", [None, n_classes])\n",
-    "\n",
+    "X = tf.placeholder(\"float\", [None, timesteps, num_input])\n",
+    "Y = tf.placeholder(\"float\", [None, num_classes])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
     "# Define weights\n",
     "weights = {\n",
-    "    'out': tf.Variable(tf.random_normal([n_hidden, n_classes]))\n",
+    "    'out': tf.Variable(tf.random_normal([num_hidden, num_classes]))\n",
     "}\n",
     "biases = {\n",
-    "    'out': tf.Variable(tf.random_normal([n_classes]))\n",
+    "    'out': tf.Variable(tf.random_normal([num_classes]))\n",
     "}"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "metadata": {
     "collapsed": false
    },
@@ -90,38 +118,50 @@
     "def RNN(x, weights, biases):\n",
     "\n",
     "    # Prepare data shape to match `rnn` function requirements\n",
-    "    # Current data input shape: (batch_size, n_steps, n_input)\n",
-    "    # Required shape: 'n_steps' tensors list of shape (batch_size, n_input)\n",
-    "    \n",
-    "    # Unstack to get a list of 'n_steps' tensors of shape (batch_size, n_input)\n",
-    "    x = tf.unstack(x, n_steps, 1)\n",
+    "    # Current data input shape: (batch_size, timesteps, n_input)\n",
+    "    # Required shape: 'timesteps' tensors list of shape (batch_size, n_input)\n",
+    "\n",
+    "    # Unstack to get a list of 'timesteps' tensors of shape (batch_size, n_input)\n",
+    "    x = tf.unstack(x, timesteps, 1)\n",
     "\n",
     "    # Define a lstm cell with tensorflow\n",
-    "    lstm_cell = rnn.BasicLSTMCell(n_hidden, forget_bias=1.0)\n",
+    "    lstm_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)\n",
     "\n",
     "    # Get lstm cell output\n",
     "    outputs, states = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)\n",
     "\n",
     "    # Linear activation, using rnn inner loop last output\n",
-    "    return tf.matmul(outputs[-1], weights['out']) + biases['out']\n",
-    "\n",
-    "pred = RNN(x, weights, biases)\n",
+    "    return tf.matmul(outputs[-1], weights['out']) + biases['out']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "logits = RNN(X, weights, biases)\n",
+    "prediction = tf.nn.softmax(logits)\n",
     "\n",
     "# Define loss and optimizer\n",
-    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))\n",
-    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n",
+    "loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
+    "    logits=logits, labels=Y))\n",
+    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
+    "train_op = optimizer.minimize(loss_op)\n",
     "\n",
-    "# Evaluate model\n",
-    "correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))\n",
+    "# Evaluate model (with test logits, for dropout to be disabled)\n",
+    "correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
     "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
     "\n",
-    "# Initializing the variables\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
     "init = tf.global_variables_initializer()"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 6,
    "metadata": {
     "collapsed": false
    },
@@ -130,118 +170,91 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Iter 1280, Minibatch Loss= 1.576423, Training Accuracy= 0.51562\n",
-      "Iter 2560, Minibatch Loss= 1.450179, Training Accuracy= 0.53906\n",
-      "Iter 3840, Minibatch Loss= 1.160066, Training Accuracy= 0.64844\n",
-      "Iter 5120, Minibatch Loss= 0.898589, Training Accuracy= 0.73438\n",
-      "Iter 6400, Minibatch Loss= 0.685712, Training Accuracy= 0.75781\n",
-      "Iter 7680, Minibatch Loss= 1.085666, Training Accuracy= 0.64844\n",
-      "Iter 8960, Minibatch Loss= 0.681488, Training Accuracy= 0.73438\n",
-      "Iter 10240, Minibatch Loss= 0.557049, Training Accuracy= 0.82812\n",
-      "Iter 11520, Minibatch Loss= 0.340857, Training Accuracy= 0.92188\n",
-      "Iter 12800, Minibatch Loss= 0.596482, Training Accuracy= 0.78906\n",
-      "Iter 14080, Minibatch Loss= 0.486564, Training Accuracy= 0.84375\n",
-      "Iter 15360, Minibatch Loss= 0.302493, Training Accuracy= 0.90625\n",
-      "Iter 16640, Minibatch Loss= 0.334277, Training Accuracy= 0.92188\n",
-      "Iter 17920, Minibatch Loss= 0.222026, Training Accuracy= 0.90625\n",
-      "Iter 19200, Minibatch Loss= 0.228581, Training Accuracy= 0.92188\n",
-      "Iter 20480, Minibatch Loss= 0.150356, Training Accuracy= 0.96094\n",
-      "Iter 21760, Minibatch Loss= 0.415417, Training Accuracy= 0.86719\n",
-      "Iter 23040, Minibatch Loss= 0.159742, Training Accuracy= 0.94531\n",
-      "Iter 24320, Minibatch Loss= 0.333764, Training Accuracy= 0.89844\n",
-      "Iter 25600, Minibatch Loss= 0.379070, Training Accuracy= 0.88281\n",
-      "Iter 26880, Minibatch Loss= 0.241612, Training Accuracy= 0.91406\n",
-      "Iter 28160, Minibatch Loss= 0.200397, Training Accuracy= 0.93750\n",
-      "Iter 29440, Minibatch Loss= 0.197994, Training Accuracy= 0.93750\n",
-      "Iter 30720, Minibatch Loss= 0.330214, Training Accuracy= 0.89062\n",
-      "Iter 32000, Minibatch Loss= 0.174626, Training Accuracy= 0.92969\n",
-      "Iter 33280, Minibatch Loss= 0.202369, Training Accuracy= 0.93750\n",
-      "Iter 34560, Minibatch Loss= 0.240835, Training Accuracy= 0.94531\n",
-      "Iter 35840, Minibatch Loss= 0.207867, Training Accuracy= 0.93750\n",
-      "Iter 37120, Minibatch Loss= 0.313306, Training Accuracy= 0.90625\n",
-      "Iter 38400, Minibatch Loss= 0.089850, Training Accuracy= 0.96875\n",
-      "Iter 39680, Minibatch Loss= 0.184803, Training Accuracy= 0.92188\n",
-      "Iter 40960, Minibatch Loss= 0.236523, Training Accuracy= 0.92969\n",
-      "Iter 42240, Minibatch Loss= 0.174834, Training Accuracy= 0.94531\n",
-      "Iter 43520, Minibatch Loss= 0.127905, Training Accuracy= 0.93750\n",
-      "Iter 44800, Minibatch Loss= 0.120045, Training Accuracy= 0.96875\n",
-      "Iter 46080, Minibatch Loss= 0.068337, Training Accuracy= 0.98438\n",
-      "Iter 47360, Minibatch Loss= 0.141118, Training Accuracy= 0.95312\n",
-      "Iter 48640, Minibatch Loss= 0.182404, Training Accuracy= 0.92188\n",
-      "Iter 49920, Minibatch Loss= 0.176778, Training Accuracy= 0.93750\n",
-      "Iter 51200, Minibatch Loss= 0.098927, Training Accuracy= 0.97656\n",
-      "Iter 52480, Minibatch Loss= 0.158776, Training Accuracy= 0.96094\n",
-      "Iter 53760, Minibatch Loss= 0.031863, Training Accuracy= 0.99219\n",
-      "Iter 55040, Minibatch Loss= 0.101799, Training Accuracy= 0.96094\n",
-      "Iter 56320, Minibatch Loss= 0.176387, Training Accuracy= 0.96094\n",
-      "Iter 57600, Minibatch Loss= 0.096277, Training Accuracy= 0.96875\n",
-      "Iter 58880, Minibatch Loss= 0.137416, Training Accuracy= 0.94531\n",
-      "Iter 60160, Minibatch Loss= 0.062801, Training Accuracy= 0.97656\n",
-      "Iter 61440, Minibatch Loss= 0.036346, Training Accuracy= 0.98438\n",
-      "Iter 62720, Minibatch Loss= 0.153030, Training Accuracy= 0.92969\n",
-      "Iter 64000, Minibatch Loss= 0.117716, Training Accuracy= 0.95312\n",
-      "Iter 65280, Minibatch Loss= 0.048387, Training Accuracy= 0.99219\n",
-      "Iter 66560, Minibatch Loss= 0.070802, Training Accuracy= 0.97656\n",
-      "Iter 67840, Minibatch Loss= 0.221085, Training Accuracy= 0.96875\n",
-      "Iter 69120, Minibatch Loss= 0.184049, Training Accuracy= 0.93750\n",
-      "Iter 70400, Minibatch Loss= 0.094883, Training Accuracy= 0.95312\n",
-      "Iter 71680, Minibatch Loss= 0.087278, Training Accuracy= 0.96875\n",
-      "Iter 72960, Minibatch Loss= 0.153267, Training Accuracy= 0.95312\n",
-      "Iter 74240, Minibatch Loss= 0.161794, Training Accuracy= 0.94531\n",
-      "Iter 75520, Minibatch Loss= 0.103779, Training Accuracy= 0.96875\n",
-      "Iter 76800, Minibatch Loss= 0.165586, Training Accuracy= 0.96094\n",
-      "Iter 78080, Minibatch Loss= 0.137721, Training Accuracy= 0.95312\n",
-      "Iter 79360, Minibatch Loss= 0.124014, Training Accuracy= 0.96094\n",
-      "Iter 80640, Minibatch Loss= 0.051460, Training Accuracy= 0.99219\n",
-      "Iter 81920, Minibatch Loss= 0.185836, Training Accuracy= 0.96094\n",
-      "Iter 83200, Minibatch Loss= 0.147694, Training Accuracy= 0.94531\n",
-      "Iter 84480, Minibatch Loss= 0.061550, Training Accuracy= 0.98438\n",
-      "Iter 85760, Minibatch Loss= 0.093457, Training Accuracy= 0.96875\n",
-      "Iter 87040, Minibatch Loss= 0.094497, Training Accuracy= 0.98438\n",
-      "Iter 88320, Minibatch Loss= 0.093934, Training Accuracy= 0.96094\n",
-      "Iter 89600, Minibatch Loss= 0.061550, Training Accuracy= 0.96875\n",
-      "Iter 90880, Minibatch Loss= 0.082452, Training Accuracy= 0.97656\n",
-      "Iter 92160, Minibatch Loss= 0.087423, Training Accuracy= 0.97656\n",
-      "Iter 93440, Minibatch Loss= 0.032694, Training Accuracy= 0.99219\n",
-      "Iter 94720, Minibatch Loss= 0.069597, Training Accuracy= 0.97656\n",
-      "Iter 96000, Minibatch Loss= 0.193636, Training Accuracy= 0.96094\n",
-      "Iter 97280, Minibatch Loss= 0.134405, Training Accuracy= 0.96094\n",
-      "Iter 98560, Minibatch Loss= 0.072992, Training Accuracy= 0.96875\n",
-      "Iter 99840, Minibatch Loss= 0.041049, Training Accuracy= 0.99219\n",
+      "Step 1, Minibatch Loss= 2.6268, Training Accuracy= 0.102\n",
+      "Step 200, Minibatch Loss= 2.0722, Training Accuracy= 0.328\n",
+      "Step 400, Minibatch Loss= 1.9181, Training Accuracy= 0.336\n",
+      "Step 600, Minibatch Loss= 1.8858, Training Accuracy= 0.336\n",
+      "Step 800, Minibatch Loss= 1.7022, Training Accuracy= 0.422\n",
+      "Step 1000, Minibatch Loss= 1.6365, Training Accuracy= 0.477\n",
+      "Step 1200, Minibatch Loss= 1.6691, Training Accuracy= 0.516\n",
+      "Step 1400, Minibatch Loss= 1.4626, Training Accuracy= 0.547\n",
+      "Step 1600, Minibatch Loss= 1.4707, Training Accuracy= 0.539\n",
+      "Step 1800, Minibatch Loss= 1.4087, Training Accuracy= 0.570\n",
+      "Step 2000, Minibatch Loss= 1.3033, Training Accuracy= 0.570\n",
+      "Step 2200, Minibatch Loss= 1.3773, Training Accuracy= 0.508\n",
+      "Step 2400, Minibatch Loss= 1.3092, Training Accuracy= 0.570\n",
+      "Step 2600, Minibatch Loss= 1.2272, Training Accuracy= 0.609\n",
+      "Step 2800, Minibatch Loss= 1.1827, Training Accuracy= 0.633\n",
+      "Step 3000, Minibatch Loss= 1.0453, Training Accuracy= 0.641\n",
+      "Step 3200, Minibatch Loss= 1.0400, Training Accuracy= 0.648\n",
+      "Step 3400, Minibatch Loss= 1.1145, Training Accuracy= 0.656\n",
+      "Step 3600, Minibatch Loss= 0.9884, Training Accuracy= 0.688\n",
+      "Step 3800, Minibatch Loss= 1.0395, Training Accuracy= 0.703\n",
+      "Step 4000, Minibatch Loss= 1.0096, Training Accuracy= 0.664\n",
+      "Step 4200, Minibatch Loss= 0.8806, Training Accuracy= 0.758\n",
+      "Step 4400, Minibatch Loss= 0.9090, Training Accuracy= 0.766\n",
+      "Step 4600, Minibatch Loss= 1.0060, Training Accuracy= 0.703\n",
+      "Step 4800, Minibatch Loss= 0.8954, Training Accuracy= 0.703\n",
+      "Step 5000, Minibatch Loss= 0.8163, Training Accuracy= 0.750\n",
+      "Step 5200, Minibatch Loss= 0.7620, Training Accuracy= 0.773\n",
+      "Step 5400, Minibatch Loss= 0.7388, Training Accuracy= 0.758\n",
+      "Step 5600, Minibatch Loss= 0.7604, Training Accuracy= 0.695\n",
+      "Step 5800, Minibatch Loss= 0.7459, Training Accuracy= 0.734\n",
+      "Step 6000, Minibatch Loss= 0.7448, Training Accuracy= 0.734\n",
+      "Step 6200, Minibatch Loss= 0.7208, Training Accuracy= 0.773\n",
+      "Step 6400, Minibatch Loss= 0.6557, Training Accuracy= 0.773\n",
+      "Step 6600, Minibatch Loss= 0.8616, Training Accuracy= 0.758\n",
+      "Step 6800, Minibatch Loss= 0.6089, Training Accuracy= 0.773\n",
+      "Step 7000, Minibatch Loss= 0.5020, Training Accuracy= 0.844\n",
+      "Step 7200, Minibatch Loss= 0.5980, Training Accuracy= 0.812\n",
+      "Step 7400, Minibatch Loss= 0.6786, Training Accuracy= 0.766\n",
+      "Step 7600, Minibatch Loss= 0.4891, Training Accuracy= 0.859\n",
+      "Step 7800, Minibatch Loss= 0.7042, Training Accuracy= 0.797\n",
+      "Step 8000, Minibatch Loss= 0.4200, Training Accuracy= 0.859\n",
+      "Step 8200, Minibatch Loss= 0.6442, Training Accuracy= 0.742\n",
+      "Step 8400, Minibatch Loss= 0.5569, Training Accuracy= 0.828\n",
+      "Step 8600, Minibatch Loss= 0.5838, Training Accuracy= 0.836\n",
+      "Step 8800, Minibatch Loss= 0.5579, Training Accuracy= 0.812\n",
+      "Step 9000, Minibatch Loss= 0.4337, Training Accuracy= 0.867\n",
+      "Step 9200, Minibatch Loss= 0.4366, Training Accuracy= 0.844\n",
+      "Step 9400, Minibatch Loss= 0.5051, Training Accuracy= 0.844\n",
+      "Step 9600, Minibatch Loss= 0.5244, Training Accuracy= 0.805\n",
+      "Step 9800, Minibatch Loss= 0.4932, Training Accuracy= 0.805\n",
+      "Step 10000, Minibatch Loss= 0.4833, Training Accuracy= 0.852\n",
       "Optimization Finished!\n",
-      "Testing Accuracy: 0.960938\n"
+      "Testing Accuracy: 0.882812\n"
      ]
     }
    ],
    "source": [
-    "# Launch the graph\n",
+    "# Start training\n",
     "with tf.Session() as sess:\n",
+    "\n",
+    "    # Run the initializer\n",
     "    sess.run(init)\n",
-    "    step = 1\n",
-    "    # Keep training until reach max iterations\n",
-    "    while step * batch_size < training_iters:\n",
+    "\n",
+    "    for step in range(1, training_steps+1):\n",
     "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
     "        # Reshape data to get 28 seq of 28 elements\n",
-    "        batch_x = batch_x.reshape((batch_size, n_steps, n_input))\n",
+    "        batch_x = batch_x.reshape((batch_size, timesteps, num_input))\n",
     "        # Run optimization op (backprop)\n",
-    "        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})\n",
-    "        if step % display_step == 0:\n",
-    "            # Calculate batch accuracy\n",
-    "            acc = sess.run(accuracy, feed_dict={x: batch_x, y: batch_y})\n",
-    "            # Calculate batch loss\n",
-    "            loss = sess.run(cost, feed_dict={x: batch_x, y: batch_y})\n",
-    "            print \"Iter \" + str(step*batch_size) + \", Minibatch Loss= \" + \\\n",
-    "                  \"{:.6f}\".format(loss) + \", Training Accuracy= \" + \\\n",
-    "                  \"{:.5f}\".format(acc)\n",
-    "        step += 1\n",
-    "    print \"Optimization Finished!\"\n",
+    "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n",
+    "        if step % display_step == 0 or step == 1:\n",
+    "            # Calculate batch loss and accuracy\n",
+    "            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n",
+    "                                                                 Y: batch_y})\n",
+    "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
+    "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
+    "                  \"{:.3f}\".format(acc))\n",
+    "\n",
+    "    print(\"Optimization Finished!\")\n",
     "\n",
     "    # Calculate accuracy for 128 mnist test images\n",
     "    test_len = 128\n",
-    "    test_data = mnist.test.images[:test_len].reshape((-1, n_steps, n_input))\n",
+    "    test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))\n",
     "    test_label = mnist.test.labels[:test_len]\n",
-    "    print \"Testing Accuracy:\", \\\n",
-    "        sess.run(accuracy, feed_dict={x: test_data, y: test_label})"
+    "    print(\"Testing Accuracy:\", \\\n",
+    "        sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))"
    ]
   },
   {
@@ -255,8 +268,9 @@
   }
  ],
  "metadata": {
+  "anaconda-cloud": {},
   "kernelspec": {
-   "display_name": "Python 2",
+   "display_name": "Python [default]",
    "language": "python",
    "name": "python2"
   },
@@ -270,7 +284,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython2",
-   "version": "2.7.13"
+   "version": "2.7.12"
   }
  },
  "nbformat": 4,

notebooks/4_Utils/save_restore_model.ipynb

@@ -1,21 +1,19 @@
 {
  "cells": [
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {
     "collapsed": true
    },
-   "outputs": [],
    "source": [
-    "'''\n",
+    "# Save & Restore a Model\n",
+    "\n",
     "Save and Restore a model using TensorFlow.\n",
     "This example is using the MNIST database of handwritten digits\n",
-    "(http://yann.lecun.com/exdb/mnist/)\n",
+    "(http://yann.lecun.com/exdb/mnist/).\n",
     "\n",
-    "Author: Aymeric Damien\n",
-    "Project: https://github.com/aymericdamien/TensorFlow-Examples/\n",
-    "'''"
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
    ]
   },
   {
@@ -228,20 +226,11 @@
     "    print(\"Accuracy:\", accuracy.eval(\n",
     "        {x: mnist.test.images, y: mnist.test.labels}))"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python [default]",
    "language": "python",
    "name": "python3"
   },
@@ -254,8 +243,8 @@
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.5.3"
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
   }
  },
  "nbformat": 4,

notebooks/4_Utils/tensorboard_advanced.ipynb

@@ -1,25 +1,24 @@
 {
  "cells": [
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "'''\n",
-    "Graph and Loss visualization using Tensorboard.\n",
-    "This example is using the MNIST database of handwritten digits\n",
-    "(http://yann.lecun.com/exdb/mnist/)\n",
+    "# Tensorboard Advanced\n",
     "\n",
-    "Author: Aymeric Damien\n",
-    "Project: https://github.com/aymericdamien/TensorFlow-Examples/\n",
-    "'''"
+    "Advanced visualization using Tensorboard (weights, gradient, ...). This example is using the MNIST database of handwritten digits\n",
+    "(http://yann.lecun.com/exdb/mnist/).\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -37,8 +36,6 @@
     "\n",
     "import tensorflow as tf\n",
     "\n",
-    "tf.logging.set_verbosity(tf.logging.WARN)\n",
-    "\n",
     "# Import MNIST data\n",
     "from tensorflow.examples.tutorials.mnist import input_data\n",
     "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
@@ -46,7 +43,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "metadata": {
     "collapsed": true
    },
@@ -57,17 +54,8 @@
     "training_epochs = 25\n",
     "batch_size = 100\n",
     "display_step = 1\n",
-    "logs_path = '/tmp/tensorflow_logs/example'"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": [
+    "logs_path = '/tmp/tensorflow_logs/example/'\n",
+    "\n",
     "# Network Parameters\n",
     "n_hidden_1 = 256 # 1st layer number of features\n",
     "n_hidden_2 = 256 # 2nd layer number of features\n",
@@ -83,7 +71,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "metadata": {
     "collapsed": true
    },
@@ -120,8 +108,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
    "outputs": [],
    "source": [
     "# Encapsulating all ops into scopes, making Tensorboard's Graph\n",
@@ -146,9 +136,18 @@
     "with tf.name_scope('Accuracy'):\n",
     "    # Accuracy\n",
     "    acc = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))\n",
-    "    acc = tf.reduce_mean(tf.cast(acc, tf.float32))\n",
-    "\n",
-    "# Initializing the variables\n",
+    "    acc = tf.reduce_mean(tf.cast(acc, tf.float32))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# Initialize the variables (i.e. assign their default value)\n",
     "init = tf.global_variables_initializer()\n",
     "\n",
     "# Create a summary to monitor cost tensor\n",
@@ -167,40 +166,42 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Epoch: 0001 cost= 67.380016100\n",
-      "Epoch: 0002 cost= 15.307113347\n",
-      "Epoch: 0003 cost= 9.986865815\n",
-      "Epoch: 0004 cost= 7.381951704\n",
-      "Epoch: 0005 cost= 5.849047792\n",
-      "Epoch: 0006 cost= 4.881959525\n",
-      "Epoch: 0007 cost= 4.045799575\n",
-      "Epoch: 0008 cost= 3.430059265\n",
-      "Epoch: 0009 cost= 3.076626336\n",
-      "Epoch: 0010 cost= 2.863002729\n",
-      "Epoch: 0011 cost= 2.510218838\n",
-      "Epoch: 0012 cost= 2.276251159\n",
-      "Epoch: 0013 cost= 1.978880318\n",
-      "Epoch: 0014 cost= 1.733890927\n",
-      "Epoch: 0015 cost= 1.540066199\n",
-      "Epoch: 0016 cost= 1.439536399\n",
-      "Epoch: 0017 cost= 1.279739846\n",
-      "Epoch: 0018 cost= 1.224386179\n",
-      "Epoch: 0019 cost= 1.095804572\n",
-      "Epoch: 0020 cost= 1.100819187\n",
-      "Epoch: 0021 cost= 0.885994007\n",
-      "Epoch: 0022 cost= 1.079832625\n",
-      "Epoch: 0023 cost= 0.948164673\n",
-      "Epoch: 0024 cost= 0.613826872\n",
-      "Epoch: 0025 cost= 0.644082715\n",
+      "Epoch: 0001 cost= 59.570364205\n",
+      "Epoch: 0002 cost= 13.585465186\n",
+      "Epoch: 0003 cost= 8.379069252\n",
+      "Epoch: 0004 cost= 6.005265894\n",
+      "Epoch: 0005 cost= 4.498054792\n",
+      "Epoch: 0006 cost= 3.503682522\n",
+      "Epoch: 0007 cost= 2.822272765\n",
+      "Epoch: 0008 cost= 2.306899852\n",
+      "Epoch: 0009 cost= 1.912765543\n",
+      "Epoch: 0010 cost= 1.597006118\n",
+      "Epoch: 0011 cost= 1.330172869\n",
+      "Epoch: 0012 cost= 1.142490618\n",
+      "Epoch: 0013 cost= 0.939443911\n",
+      "Epoch: 0014 cost= 0.820920588\n",
+      "Epoch: 0015 cost= 0.702543302\n",
+      "Epoch: 0016 cost= 0.604815631\n",
+      "Epoch: 0017 cost= 0.505682561\n",
+      "Epoch: 0018 cost= 0.439700446\n",
+      "Epoch: 0019 cost= 0.378268929\n",
+      "Epoch: 0020 cost= 0.299557848\n",
+      "Epoch: 0021 cost= 0.269859066\n",
+      "Epoch: 0022 cost= 0.230899029\n",
+      "Epoch: 0023 cost= 0.183722090\n",
+      "Epoch: 0024 cost= 0.164173368\n",
+      "Epoch: 0025 cost= 0.142141250\n",
       "Optimization Finished!\n",
-      "Accuracy: 0.9513\n",
+      "Accuracy: 0.9336\n",
       "Run the command line:\n",
       "--> tensorboard --logdir=/tmp/tensorflow_logs \n",
       "Then open http://0.0.0.0:6006/ into your web browser\n"
@@ -208,8 +209,10 @@
     }
    ],
    "source": [
-    "# Launch the graph\n",
+    "# Start training\n",
     "with tf.Session() as sess:\n",
+    "\n",
+    "    # Run the initializer\n",
     "    sess.run(init)\n",
     "\n",
     "    # op to write logs to Tensorboard\n",
@@ -247,34 +250,58 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Loss and Accuracy Visualization\n",
+    "<img src=\"../../resources/img/tensorboard_advanced_1.png\"/>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Computation Graph Visualization\n",
+    "<img src=\"../../resources/img/tensorboard_advanced_2.png\"/>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Weights and Gradients Visualization\n",
+    "<img src=\"../../resources/img/tensorboard_advanced_3.png\"/>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Activations Visualization\n",
+    "<img src=\"../../resources/img/tensorboard_advanced_4.png\"/>"
+   ]
   }
  ],
  "metadata": {
+  "anaconda-cloud": {},
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python [default]",
    "language": "python",
-   "name": "python3"
+   "name": "python2"
   },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
-    "version": 3
+    "version": 2
    },
    "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.5.3"
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 1
 }

notebooks/5_DataManagement/build_an_image_dataset.ipynb

@@ -0,0 +1,290 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": false
+   },
+   "source": [
+    "# Build an Image Dataset in TensorFlow.\n",
+    "\n",
+    "For this example, you need to make your own set of images (JPEG).\n",
+    "We will show 2 different ways to build that dataset:\n",
+    "\n",
+    "- From a root folder, that will have a sub-folder containing images for each class\n",
+    "\n",
+    "```\n",
+    "    ROOT_FOLDER\n",
+    "       |-------- SUBFOLDER (CLASS 0)\n",
+    "       |             |\n",
+    "       |             | ----- image1.jpg\n",
+    "       |             | ----- image2.jpg\n",
+    "       |             | ----- etc...\n",
+    "       |             \n",
+    "       |-------- SUBFOLDER (CLASS 1)\n",
+    "       |             |\n",
+    "       |             | ----- image1.jpg\n",
+    "       |             | ----- image2.jpg\n",
+    "       |             | ----- etc...\n",
+    "\n",
+    "```\n",
+    "\n",
+    "- From a plain text file, that will list all images with their class ID:\n",
+    "\n",
+    "```\n",
+    "    /path/to/image/1.jpg CLASS_ID\n",
+    "    /path/to/image/2.jpg CLASS_ID\n",
+    "    /path/to/image/3.jpg CLASS_ID\n",
+    "    /path/to/image/4.jpg CLASS_ID\n",
+    "    etc...\n",
+    "```\n",
+    "\n",
+    "Below, there are some parameters that you need to change (Marked 'CHANGE HERE'), \n",
+    "such as the dataset path.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "from __future__ import print_function\n",
+    "\n",
+    "import tensorflow as tf\n",
+    "import os\n",
+    "\n",
+    "# Dataset Parameters - CHANGE HERE\n",
+    "MODE = 'folder' # or 'file', if you choose a plain text file (see above).\n",
+    "DATASET_PATH = '/path/to/dataset/' # the dataset file or root folder path.\n",
+    "\n",
+    "# Image Parameters\n",
+    "N_CLASSES = 2 # CHANGE HERE, total number of classes\n",
+    "IMG_HEIGHT = 64 # CHANGE HERE, the image height to be resized to\n",
+    "IMG_WIDTH = 64 # CHANGE HERE, the image width to be resized to\n",
+    "CHANNELS = 3 # The 3 color channels, change to 1 if grayscale"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Reading the dataset\n",
+    "# 2 modes: 'file' or 'folder'\n",
+    "def read_images(dataset_path, mode, batch_size):\n",
+    "    imagepaths, labels = list(), list()\n",
+    "    if mode == 'file':\n",
+    "        # Read dataset file\n",
+    "        data = open(dataset_path, 'r').read().splitlines()\n",
+    "        for d in data:\n",
+    "            imagepaths.append(d.split(' ')[0])\n",
+    "            labels.append(int(d.split(' ')[1]))\n",
+    "    elif mode == 'folder':\n",
+    "        # An ID will be affected to each sub-folders by alphabetical order\n",
+    "        label = 0\n",
+    "        # List the directory\n",
+    "        try:  # Python 2\n",
+    "            classes = sorted(os.walk(dataset_path).next()[1])\n",
+    "        except Exception:  # Python 3\n",
+    "            classes = sorted(os.walk(dataset_path).__next__()[1])\n",
+    "        # List each sub-directory (the classes)\n",
+    "        for c in classes:\n",
+    "            c_dir = os.path.join(dataset_path, c)\n",
+    "            try:  # Python 2\n",
+    "                walk = os.walk(c_dir).next()\n",
+    "            except Exception:  # Python 3\n",
+    "                walk = os.walk(c_dir).__next__()\n",
+    "            # Add each image to the training set\n",
+    "            for sample in walk[2]:\n",
+    "                # Only keeps jpeg images\n",
+    "                if sample.endswith('.jpg') or sample.endswith('.jpeg'):\n",
+    "                    imagepaths.append(os.path.join(c_dir, sample))\n",
+    "                    labels.append(label)\n",
+    "            label += 1\n",
+    "    else:\n",
+    "        raise Exception(\"Unknown mode.\")\n",
+    "\n",
+    "    # Convert to Tensor\n",
+    "    imagepaths = tf.convert_to_tensor(imagepaths, dtype=tf.string)\n",
+    "    labels = tf.convert_to_tensor(labels, dtype=tf.int32)\n",
+    "    # Build a TF Queue, shuffle data\n",
+    "    image, label = tf.train.slice_input_producer([imagepaths, labels],\n",
+    "                                                 shuffle=True)\n",
+    "\n",
+    "    # Read images from disk\n",
+    "    image = tf.read_file(image)\n",
+    "    image = tf.image.decode_jpeg(image, channels=CHANNELS)\n",
+    "\n",
+    "    # Resize images to a common size\n",
+    "    image = tf.image.resize_images(image, [IMG_HEIGHT, IMG_WIDTH])\n",
+    "\n",
+    "    # Normalize\n",
+    "    image = image * 1.0/127.5 - 1.0\n",
+    "\n",
+    "    # Create batches\n",
+    "    X, Y = tf.train.batch([image, label], batch_size=batch_size,\n",
+    "                          capacity=batch_size * 8,\n",
+    "                          num_threads=4)\n",
+    "\n",
+    "    return X, Y"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# -----------------------------------------------\n",
+    "# THIS IS A CLASSIC CNN (see examples, section 3)\n",
+    "# -----------------------------------------------\n",
+    "# Note that a few elements have changed (usage of queues).\n",
+    "\n",
+    "# Parameters\n",
+    "learning_rate = 0.001\n",
+    "num_steps = 10000\n",
+    "batch_size = 128\n",
+    "display_step = 100\n",
+    "\n",
+    "# Network Parameters\n",
+    "dropout = 0.75 # Dropout, probability to keep units\n",
+    "\n",
+    "# Build the data input\n",
+    "X, Y = read_images(DATASET_PATH, MODE, batch_size)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Create model\n",
+    "def conv_net(x, n_classes, dropout, reuse, is_training):\n",
+    "    # Define a scope for reusing the variables\n",
+    "    with tf.variable_scope('ConvNet', reuse=reuse):\n",
+    "\n",
+    "        # Convolution Layer with 32 filters and a kernel size of 5\n",
+    "        conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)\n",
+    "        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
+    "        conv1 = tf.layers.max_pooling2d(conv1, 2, 2)\n",
+    "\n",
+    "        # Convolution Layer with 32 filters and a kernel size of 5\n",
+    "        conv2 = tf.layers.conv2d(conv1, 64, 3, activation=tf.nn.relu)\n",
+    "        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
+    "        conv2 = tf.layers.max_pooling2d(conv2, 2, 2)\n",
+    "\n",
+    "        # Flatten the data to a 1-D vector for the fully connected layer\n",
+    "        fc1 = tf.contrib.layers.flatten(conv2)\n",
+    "\n",
+    "        # Fully connected layer (in contrib folder for now)\n",
+    "        fc1 = tf.layers.dense(fc1, 1024)\n",
+    "        # Apply Dropout (if is_training is False, dropout is not applied)\n",
+    "        fc1 = tf.layers.dropout(fc1, rate=dropout, training=is_training)\n",
+    "\n",
+    "        # Output layer, class prediction\n",
+    "        out = tf.layers.dense(fc1, n_classes)\n",
+    "        # Because 'softmax_cross_entropy_with_logits' already apply softmax,\n",
+    "        # we only apply softmax to testing network\n",
+    "        out = tf.nn.softmax(out) if not is_training else out\n",
+    "\n",
+    "    return out"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Because Dropout have different behavior at training and prediction time, we\n",
+    "# need to create 2 distinct computation graphs that share the same weights.\n",
+    "\n",
+    "# Create a graph for training\n",
+    "logits_train = conv_net(X, N_CLASSES, dropout, reuse=False, is_training=True)\n",
+    "# Create another graph for testing that reuse the same weights\n",
+    "logits_test = conv_net(X, N_CLASSES, dropout, reuse=True, is_training=False)\n",
+    "\n",
+    "# Define loss and optimizer (with train logits, for dropout to take effect)\n",
+    "loss_op = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(\n",
+    "    logits=logits_train, labels=Y))\n",
+    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
+    "train_op = optimizer.minimize(loss_op)\n",
+    "\n",
+    "# Evaluate model (with test logits, for dropout to be disabled)\n",
+    "correct_pred = tf.equal(tf.argmax(logits_test, 1), tf.cast(Y, tf.int64))\n",
+    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
+    "\n",
+    "# Initialize the variables (i.e. assign their default value)\n",
+    "init = tf.global_variables_initializer()\n",
+    "\n",
+    "# Saver object\n",
+    "saver = tf.train.Saver()\n",
+    "\n",
+    "# Start training\n",
+    "with tf.Session() as sess:\n",
+    "\n",
+    "    # Run the initializer\n",
+    "    sess.run(init)\n",
+    "\n",
+    "    # Start the data queue\n",
+    "    tf.train.start_queue_runners()\n",
+    "\n",
+    "    # Training cycle\n",
+    "    for step in range(1, num_steps+1):\n",
+    "\n",
+    "        if step % display_step == 0:\n",
+    "            # Run optimization and calculate batch loss and accuracy\n",
+    "            _, loss, acc = sess.run([train_op, loss_op, accuracy])\n",
+    "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
+    "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
+    "                  \"{:.3f}\".format(acc))\n",
+    "        else:\n",
+    "            # Only run the optimization op (backprop)\n",
+    "            sess.run(train_op)\n",
+    "\n",
+    "    print(\"Optimization Finished!\")\n",
+    "\n",
+    "    # Save your model\n",
+    "    saver.save(sess, 'my_tf_model')"
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python [default]",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

notebooks/5_DataManagement/tensorflow_dataset_api.ipynb

@@ -0,0 +1,234 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": false
+   },
+   "source": [
+    "# TensorFlow Dataset API\n",
+    "\n",
+    "In this example, we will show how to load numpy array data into the new \n",
+    "TensorFlow 'Dataset' API. The Dataset API implements an optimized data pipeline\n",
+    "with queues, that make data processing and training faster (especially on GPU).\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
+     ]
+    }
+   ],
+   "source": [
+    "import tensorflow as tf\n",
+    "\n",
+    "# Import MNIST data (Numpy format)\n",
+    "from tensorflow.examples.tutorials.mnist import input_data\n",
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Parameters\n",
+    "learning_rate = 0.01\n",
+    "num_steps = 1000\n",
+    "batch_size = 128\n",
+    "display_step = 100\n",
+    "\n",
+    "# Network Parameters\n",
+    "n_input = 784 # MNIST data input (img shape: 28*28)\n",
+    "n_classes = 10 # MNIST total classes (0-9 digits)\n",
+    "dropout = 0.75 # Dropout, probability to keep units\n",
+    "\n",
+    "sess = tf.Session()\n",
+    "\n",
+    "# Create a dataset tensor from the images and the labels\n",
+    "dataset = tf.contrib.data.Dataset.from_tensor_slices(\n",
+    "    (mnist.train.images, mnist.train.labels))\n",
+    "# Create batches of data\n",
+    "dataset = dataset.batch(batch_size)\n",
+    "# Create an iterator, to go over the dataset\n",
+    "iterator = dataset.make_initializable_iterator()\n",
+    "# It is better to use 2 placeholders, to avoid to load all data into memory,\n",
+    "# and avoid the 2Gb restriction length of a tensor.\n",
+    "_data = tf.placeholder(tf.float32, [None, n_input])\n",
+    "_labels = tf.placeholder(tf.float32, [None, n_classes])\n",
+    "# Initialize the iterator\n",
+    "sess.run(iterator.initializer, feed_dict={_data: mnist.train.images,\n",
+    "                                          _labels: mnist.train.labels})\n",
+    "\n",
+    "# Neural Net Input\n",
+    "X, Y = iterator.get_next()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# -----------------------------------------------\n",
+    "# THIS IS A CLASSIC CNN (see examples, section 3)\n",
+    "# -----------------------------------------------\n",
+    "# Note that a few elements have changed (usage of sess run).\n",
+    "\n",
+    "# Create model\n",
+    "def conv_net(x, n_classes, dropout, reuse, is_training):\n",
+    "    # Define a scope for reusing the variables\n",
+    "    with tf.variable_scope('ConvNet', reuse=reuse):\n",
+    "        # MNIST data input is a 1-D vector of 784 features (28*28 pixels)\n",
+    "        # Reshape to match picture format [Height x Width x Channel]\n",
+    "        # Tensor input become 4-D: [Batch Size, Height, Width, Channel]\n",
+    "        x = tf.reshape(x, shape=[-1, 28, 28, 1])\n",
+    "\n",
+    "        # Convolution Layer with 32 filters and a kernel size of 5\n",
+    "        conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)\n",
+    "        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
+    "        conv1 = tf.layers.max_pooling2d(conv1, 2, 2)\n",
+    "\n",
+    "        # Convolution Layer with 32 filters and a kernel size of 5\n",
+    "        conv2 = tf.layers.conv2d(conv1, 64, 3, activation=tf.nn.relu)\n",
+    "        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
+    "        conv2 = tf.layers.max_pooling2d(conv2, 2, 2)\n",
+    "\n",
+    "        # Flatten the data to a 1-D vector for the fully connected layer\n",
+    "        fc1 = tf.contrib.layers.flatten(conv2)\n",
+    "\n",
+    "        # Fully connected layer (in contrib folder for now)\n",
+    "        fc1 = tf.layers.dense(fc1, 1024)\n",
+    "        # Apply Dropout (if is_training is False, dropout is not applied)\n",
+    "        fc1 = tf.layers.dropout(fc1, rate=dropout, training=is_training)\n",
+    "\n",
+    "        # Output layer, class prediction\n",
+    "        out = tf.layers.dense(fc1, n_classes)\n",
+    "        # Because 'softmax_cross_entropy_with_logits' already apply softmax,\n",
+    "        # we only apply softmax to testing network\n",
+    "        out = tf.nn.softmax(out) if not is_training else out\n",
+    "\n",
+    "    return out\n",
+    "\n",
+    "\n",
+    "# Because Dropout have different behavior at training and prediction time, we\n",
+    "# need to create 2 distinct computation graphs that share the same weights.\n",
+    "\n",
+    "# Create a graph for training\n",
+    "logits_train = conv_net(X, n_classes, dropout, reuse=False, is_training=True)\n",
+    "# Create another graph for testing that reuse the same weights, but has\n",
+    "# different behavior for 'dropout' (not applied).\n",
+    "logits_test = conv_net(X, n_classes, dropout, reuse=True, is_training=False)\n",
+    "\n",
+    "# Define loss and optimizer (with train logits, for dropout to take effect)\n",
+    "loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
+    "    logits=logits_train, labels=Y))\n",
+    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
+    "train_op = optimizer.minimize(loss_op)\n",
+    "\n",
+    "# Evaluate model (with test logits, for dropout to be disabled)\n",
+    "correct_pred = tf.equal(tf.argmax(logits_test, 1), tf.argmax(Y, 1))\n",
+    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": false,
+    "scrolled": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Step 1, Minibatch Loss= 7.9429, Training Accuracy= 0.070\n",
+      "Step 100, Minibatch Loss= 0.3491, Training Accuracy= 0.922\n",
+      "Step 200, Minibatch Loss= 0.2343, Training Accuracy= 0.922\n",
+      "Step 300, Minibatch Loss= 0.1838, Training Accuracy= 0.969\n",
+      "Step 400, Minibatch Loss= 0.1715, Training Accuracy= 0.953\n",
+      "Step 500, Minibatch Loss= 0.2730, Training Accuracy= 0.938\n",
+      "Step 600, Minibatch Loss= 0.3427, Training Accuracy= 0.953\n",
+      "Step 700, Minibatch Loss= 0.2261, Training Accuracy= 0.961\n",
+      "Step 800, Minibatch Loss= 0.1487, Training Accuracy= 0.953\n",
+      "Step 900, Minibatch Loss= 0.1438, Training Accuracy= 0.945\n",
+      "Step 1000, Minibatch Loss= 0.1786, Training Accuracy= 0.961\n",
+      "Optimization Finished!\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Initialize the variables (i.e. assign their default value)\n",
+    "init = tf.global_variables_initializer()\n",
+    "\n",
+    "# Run the initializer\n",
+    "sess.run(init)\n",
+    "\n",
+    "# Training cycle\n",
+    "for step in range(1, num_steps + 1):\n",
+    "    \n",
+    "    try:\n",
+    "        # Run optimization\n",
+    "        sess.run(train_op)\n",
+    "    except tf.errors.OutOfRangeError:\n",
+    "        # Reload the iterator when it reaches the end of the dataset\n",
+    "        sess.run(iterator.initializer, \n",
+    "                 feed_dict={_data: mnist.train.images,\n",
+    "                            _labels: mnist.train.labels})\n",
+    "        sess.run(train_op)\n",
+    "        \n",
+    "    if step % display_step == 0 or step == 1:\n",
+    "        # Calculate batch loss and accuracy\n",
+    "        # (note that this consume a new batch of data)\n",
+    "        loss, acc = sess.run([loss_op, accuracy])\n",
+    "        print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
+    "              \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
+    "              \"{:.3f}\".format(acc))\n",
+    "\n",
+    "print(\"Optimization Finished!\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python [default]",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

notebooks/5_MultiGPU/multigpu_basics.ipynb

@@ -1,23 +1,23 @@
 {
  "cells": [
   {
-   "cell_type": "code",
-   "execution_count": 1,
+   "cell_type": "markdown",
    "metadata": {
     "collapsed": true
    },
-   "outputs": [],
    "source": [
-    "# Basic Multi GPU computation example using TensorFlow library.\n",
+    "# Multi-GPU Basics\n",
+    "\n",
+    "Basic Multi-GPU computation example using TensorFlow library.\n",
     "\n",
-    "# Author: Aymeric Damien\n",
-    "# Project: https://github.com/aymericdamien/TensorFlow-Examples/\n",
+    "This tutorial requires your machine to have 2 GPUs\n",
+    "\"/cpu:0\": The CPU of your machine.\n",
+    "\"/gpu:0\": The first GPU of your machine\n",
+    "\"/gpu:1\": The second GPU of your machine\n",
+    "For this example, we are using 2 GTX-980\n",
     "\n",
-    "# This tutorial requires your machine to have 2 GPUs\n",
-    "# \"/cpu:0\": The CPU of your machine.\n",
-    "# \"/gpu:0\": The first GPU of your machine\n",
-    "# \"/gpu:1\": The second GPU of your machine\n",
-    "# For this example, we are using 2 GTX-980"
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
    ]
   },
   {
@@ -155,22 +155,23 @@
   }
  ],
  "metadata": {
+  "anaconda-cloud": {},
   "kernelspec": {
-   "display_name": "Python 2",
+   "display_name": "Python [default]",
    "language": "python",
    "name": "python2"
   },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
-    "version": 2.0
+    "version": 2
    },
    "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython2",
-   "version": "2.7.10"
+   "version": "2.7.12"
   }
  },
  "nbformat": 4,

notebooks/6_MultiGPU/multigpu_cnn.ipynb

@@ -0,0 +1,304 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Multi-GPU Training Example\n",
+    "\n",
+    "Train a convolutional neural network on multiple GPU with TensorFlow.\n",
+    "\n",
+    "This example is using TensorFlow layers, see 'convolutional_network_raw' example\n",
+    "for a raw TensorFlow implementation with variables.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Training with multiple GPU cards\n",
+    "\n",
+    "In this example, we are using data parallelism to split the training accross multiple GPUs. Each GPU has a full replica of the neural network model, and the weights (i.e. variables) are updated synchronously by waiting that each GPU process its batch of data.\n",
+    "\n",
+    "First, each GPU process a distinct batch of data and compute the corresponding gradients, then, all gradients are accumulated in the CPU and averaged. The model weights are finally updated with the gradients averaged, and the new model weights are sent back to each GPU, to repeat the training process.\n",
+    "\n",
+    "<img src=\"https://www.tensorflow.org/images/Parallelism.png\" alt=\"Parallelism\" style=\"width: 400px;\"/>\n",
+    "\n",
+    "## MNIST Dataset Overview\n",
+    "\n",
+    "This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flatten and converted to a 1-D numpy array of 784 features (28*28).\n",
+    "\n",
+    "![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n",
+    "\n",
+    "More info: http://yann.lecun.com/exdb/mnist/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
+     ]
+    }
+   ],
+   "source": [
+    "from __future__ import print_function\n",
+    "\n",
+    "import numpy as np\n",
+    "import tensorflow as tf\n",
+    "import time\n",
+    "\n",
+    "# Import MNIST data\n",
+    "from tensorflow.examples.tutorials.mnist import input_data\n",
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
+    "\n",
+    "# Parameters\n",
+    "num_gpus = 2\n",
+    "num_steps = 200\n",
+    "learning_rate = 0.001\n",
+    "batch_size = 1024\n",
+    "display_step = 10\n",
+    "\n",
+    "# Network Parameters\n",
+    "num_input = 784 # MNIST data input (img shape: 28*28)\n",
+    "num_classes = 10 # MNIST total classes (0-9 digits)\n",
+    "dropout = 0.75 # Dropout, probability to keep units"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Build a convolutional neural network\n",
+    "def conv_net(x, n_classes, dropout, reuse, is_training):\n",
+    "    # Define a scope for reusing the variables\n",
+    "    with tf.variable_scope('ConvNet', reuse=reuse):\n",
+    "        # MNIST data input is a 1-D vector of 784 features (28*28 pixels)\n",
+    "        # Reshape to match picture format [Height x Width x Channel]\n",
+    "        # Tensor input become 4-D: [Batch Size, Height, Width, Channel]\n",
+    "        x = tf.reshape(x, shape=[-1, 28, 28, 1])\n",
+    "\n",
+    "        # Convolution Layer with 64 filters and a kernel size of 5\n",
+    "        x = tf.layers.conv2d(x, 64, 5, activation=tf.nn.relu)\n",
+    "        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
+    "        x = tf.layers.max_pooling2d(x, 2, 2)\n",
+    "\n",
+    "        # Convolution Layer with 256 filters and a kernel size of 5\n",
+    "        x = tf.layers.conv2d(x, 256, 3, activation=tf.nn.relu)\n",
+    "        # Convolution Layer with 512 filters and a kernel size of 5\n",
+    "        x = tf.layers.conv2d(x, 512, 3, activation=tf.nn.relu)\n",
+    "        # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
+    "        x = tf.layers.max_pooling2d(x, 2, 2)\n",
+    "\n",
+    "        # Flatten the data to a 1-D vector for the fully connected layer\n",
+    "        x = tf.contrib.layers.flatten(x)\n",
+    "\n",
+    "        # Fully connected layer (in contrib folder for now)\n",
+    "        x = tf.layers.dense(x, 2048)\n",
+    "        # Apply Dropout (if is_training is False, dropout is not applied)\n",
+    "        x = tf.layers.dropout(x, rate=dropout, training=is_training)\n",
+    "\n",
+    "        # Fully connected layer (in contrib folder for now)\n",
+    "        x = tf.layers.dense(x, 1024)\n",
+    "        # Apply Dropout (if is_training is False, dropout is not applied)\n",
+    "        x = tf.layers.dropout(x, rate=dropout, training=is_training)\n",
+    "\n",
+    "        # Output layer, class prediction\n",
+    "        out = tf.layers.dense(x, n_classes)\n",
+    "        # Because 'softmax_cross_entropy_with_logits' loss already apply\n",
+    "        # softmax, we only apply softmax to testing network\n",
+    "        out = tf.nn.softmax(out) if not is_training else out\n",
+    "\n",
+    "    return out"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Build the function to average the gradients\n",
+    "def average_gradients(tower_grads):\n",
+    "    average_grads = []\n",
+    "    for grad_and_vars in zip(*tower_grads):\n",
+    "        # Note that each grad_and_vars looks like the following:\n",
+    "        #   ((grad0_gpu0, var0_gpu0), ... , (grad0_gpuN, var0_gpuN))\n",
+    "        grads = []\n",
+    "        for g, _ in grad_and_vars:\n",
+    "            # Add 0 dimension to the gradients to represent the tower.\n",
+    "            expanded_g = tf.expand_dims(g, 0)\n",
+    "\n",
+    "            # Append on a 'tower' dimension which we will average over below.\n",
+    "            grads.append(expanded_g)\n",
+    "\n",
+    "        # Average over the 'tower' dimension.\n",
+    "        grad = tf.concat(grads, 0)\n",
+    "        grad = tf.reduce_mean(grad, 0)\n",
+    "\n",
+    "        # Keep in mind that the Variables are redundant because they are shared\n",
+    "        # across towers. So .. we will just return the first tower's pointer to\n",
+    "        # the Variable.\n",
+    "        v = grad_and_vars[0][1]\n",
+    "        grad_and_var = (grad, v)\n",
+    "        average_grads.append(grad_and_var)\n",
+    "    return average_grads"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": false,
+    "scrolled": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Step 1: Minibatch Loss= 2.4077, Training Accuracy= 0.123, 682 Examples/sec\n",
+      "Step 10: Minibatch Loss= 1.0067, Training Accuracy= 0.765, 6528 Examples/sec\n",
+      "Step 20: Minibatch Loss= 0.2442, Training Accuracy= 0.945, 6803 Examples/sec\n",
+      "Step 30: Minibatch Loss= 0.2013, Training Accuracy= 0.951, 6741 Examples/sec\n",
+      "Step 40: Minibatch Loss= 0.1445, Training Accuracy= 0.962, 6700 Examples/sec\n",
+      "Step 50: Minibatch Loss= 0.0940, Training Accuracy= 0.971, 6746 Examples/sec\n",
+      "Step 60: Minibatch Loss= 0.0792, Training Accuracy= 0.977, 6627 Examples/sec\n",
+      "Step 70: Minibatch Loss= 0.0593, Training Accuracy= 0.979, 6749 Examples/sec\n",
+      "Step 80: Minibatch Loss= 0.0799, Training Accuracy= 0.984, 6368 Examples/sec\n",
+      "Step 90: Minibatch Loss= 0.0614, Training Accuracy= 0.988, 6762 Examples/sec\n",
+      "Step 100: Minibatch Loss= 0.0716, Training Accuracy= 0.983, 6338 Examples/sec\n",
+      "Step 110: Minibatch Loss= 0.0531, Training Accuracy= 0.986, 6504 Examples/sec\n",
+      "Step 120: Minibatch Loss= 0.0425, Training Accuracy= 0.990, 6721 Examples/sec\n",
+      "Step 130: Minibatch Loss= 0.0473, Training Accuracy= 0.986, 6735 Examples/sec\n",
+      "Step 140: Minibatch Loss= 0.0345, Training Accuracy= 0.991, 6636 Examples/sec\n",
+      "Step 150: Minibatch Loss= 0.0419, Training Accuracy= 0.993, 6777 Examples/sec\n",
+      "Step 160: Minibatch Loss= 0.0602, Training Accuracy= 0.984, 6392 Examples/sec\n",
+      "Step 170: Minibatch Loss= 0.0425, Training Accuracy= 0.990, 6855 Examples/sec\n",
+      "Step 180: Minibatch Loss= 0.0107, Training Accuracy= 0.998, 6804 Examples/sec\n",
+      "Step 190: Minibatch Loss= 0.0204, Training Accuracy= 0.995, 6645 Examples/sec\n",
+      "Step 200: Minibatch Loss= 0.0296, Training Accuracy= 0.993, 6747 Examples/sec\n",
+      "Optimization Finished!\n",
+      "Testing Accuracy: 0.990671\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Place all ops on CPU by default\n",
+    "with tf.device('/cpu:0'):\n",
+    "    tower_grads = []\n",
+    "    reuse_vars = False\n",
+    "\n",
+    "    # tf Graph input\n",
+    "    X = tf.placeholder(tf.float32, [None, num_input])\n",
+    "    Y = tf.placeholder(tf.float32, [None, num_classes])\n",
+    "\n",
+    "    # Loop over all GPUs and construct their own computation graph\n",
+    "    for i in range(num_gpus):\n",
+    "        with tf.device('/gpu:%d' % i):\n",
+    "\n",
+    "            # Split data between GPUs\n",
+    "            _x = X[i * batch_size: (i+1) * batch_size]\n",
+    "            _y = Y[i * batch_size: (i+1) * batch_size]\n",
+    "\n",
+    "            # Because Dropout have different behavior at training and prediction time, we\n",
+    "            # need to create 2 distinct computation graphs that share the same weights.\n",
+    "\n",
+    "            # Create a graph for training\n",
+    "            logits_train = conv_net(_x, num_classes, dropout,\n",
+    "                                    reuse=reuse_vars, is_training=True)\n",
+    "            # Create another graph for testing that reuse the same weights\n",
+    "            logits_test = conv_net(_x, num_classes, dropout,\n",
+    "                                   reuse=True, is_training=False)\n",
+    "\n",
+    "            # Define loss and optimizer (with train logits, for dropout to take effect)\n",
+    "            loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
+    "                logits=logits_train, labels=_y))\n",
+    "            optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
+    "            grads = optimizer.compute_gradients(loss_op)\n",
+    "\n",
+    "            # Only first GPU compute accuracy\n",
+    "            if i == 0:\n",
+    "                # Evaluate model (with test logits, for dropout to be disabled)\n",
+    "                correct_pred = tf.equal(tf.argmax(logits_test, 1), tf.argmax(_y, 1))\n",
+    "                accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
+    "\n",
+    "            reuse_vars = True\n",
+    "            tower_grads.append(grads)\n",
+    "\n",
+    "    tower_grads = average_gradients(tower_grads)\n",
+    "    train_op = optimizer.apply_gradients(tower_grads)\n",
+    "\n",
+    "    # Initializing the variables\n",
+    "    init = tf.global_variables_initializer()\n",
+    "\n",
+    "    # Launch the graph\n",
+    "    with tf.Session() as sess:\n",
+    "        sess.run(init)\n",
+    "        step = 1\n",
+    "        # Keep training until reach max iterations\n",
+    "        for step in range(1, num_steps + 1):\n",
+    "            # Get a batch for each GPU\n",
+    "            batch_x, batch_y = mnist.train.next_batch(batch_size * num_gpus)\n",
+    "            # Run optimization op (backprop)\n",
+    "            ts = time.time()\n",
+    "            sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n",
+    "            te = time.time() - ts\n",
+    "            if step % display_step == 0 or step == 1:\n",
+    "                # Calculate batch loss and accuracy\n",
+    "                loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n",
+    "                                                                     Y: batch_y})\n",
+    "                print(\"Step \" + str(step) + \": Minibatch Loss= \" + \\\n",
+    "                      \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
+    "                      \"{:.3f}\".format(acc) + \", %i Examples/sec\" % int(len(batch_x)/te))\n",
+    "            step += 1\n",
+    "        print(\"Optimization Finished!\")\n",
+    "\n",
+    "        # Calculate accuracy for 1000 mnist test images\n",
+    "        print(\"Testing Accuracy:\", \\\n",
+    "            np.mean([sess.run(accuracy, feed_dict={X: mnist.test.images[i:i+batch_size],\n",
+    "            Y: mnist.test.labels[i:i+batch_size]}) for i in range(0, len(mnist.test.images), batch_size)]))"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2.0
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}