Add TensorFlow v2 examples (#302)

* add tf v2 examples

* update README

* add README

* update doc

README.md

@@ -1,10 +1,10 @@
 # TensorFlow Examples
 
-This tutorial was designed for easily diving into TensorFlow, through examples. For readability, it includes both notebooks and source codes with explanation.
+This tutorial was designed for easily diving into TensorFlow, through examples. For readability, it includes both notebooks and source codes with explanation for both TF v1 & v2.
 
 It is suitable for beginners who want to find clear and concise examples about TensorFlow. Besides the traditional 'raw' TensorFlow implementations, you can also find the latest TensorFlow API practices (such as `layers`, `estimator`, `dataset`, ...).
 
-**Update (07/25/2018):** Add new examples (GBDT, Word2Vec) + TF1.9 compatibility! (TF v1.9+ recommended).
+**Update (04/03/2019):** Starting to add [TensorFlow v2 examples](tensorflow_v2)! (more coming soon).
 
 *If you are using older TensorFlow version (0.11 and under), please take a [look here](https://github.com/aymericdamien/TensorFlow-Examples/tree/0.11).*
 
@@ -61,11 +61,15 @@ It is suitable for beginners who want to find clear and concise examples about T
 - **Basic Operations on multi-GPU** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/6_MultiGPU/multigpu_basics.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/6_MultiGPU/multigpu_basics.py)). A simple example to introduce multi-GPU in TensorFlow.
 - **Train a Neural Network on multi-GPU** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/6_MultiGPU/multigpu_cnn.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/6_MultiGPU/multigpu_cnn.py)). A clear and simple TensorFlow implementation to train a convolutional neural network on multiple GPUs.
 
+## TensorFlow v2
+
+The tutorial index for TF v2 is available here: [TensorFlow v2 Examples](tensorflow_v2).
+
 ## Dataset
 Some examples require MNIST dataset for training and testing. Don't worry, this dataset will automatically be downloaded when running examples.
 MNIST is a database of handwritten digits, for a quick description of that dataset, you can check [this notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/0_Prerequisite/mnist_dataset_intro.ipynb).
 
-Official Website: [http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/)
+Official Website: [http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/).
 
 ## Installation
 

tensorflow_v2/README.md

@@ -0,0 +1,43 @@
+## TensorFlow v2 Examples
+
+*** More examples to be added later... *** 
+
+#### 0 - Prerequisite
+- [Introduction to Machine Learning](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/0_Prerequisite/ml_introduction.ipynb).
+- [Introduction to MNIST Dataset](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/0_Prerequisite/mnist_dataset_intro.ipynb).
+
+#### 1 - Introduction
+- **Hello World** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/1_Introduction/helloworld.ipynb)). Very simple example to learn how to print "hello world" using TensorFlow v2.
+- **Basic Operations** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/1_Introduction/basic_operations.ipynb)). A simple example that cover TensorFlow v2 basic operations.
+
+#### 2 - Basic Models
+- **Linear Regression** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb)). Implement a Linear Regression with TensorFlow v2.
+- **Logistic Regression** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/logistic_regression.ipynb)). Implement a Logistic Regression with TensorFlow v2.
+
+#### 3 - Neural Networks
+##### Supervised
+
+- **Simple Neural Network** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/3_NeuralNetworks/neural_network.ipynb)). Use TensorFlow v2 'layers' and 'model' API to build a simple neural network to classify MNIST digits dataset.
+- **Simple Neural Network (low-level)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/3_NeuralNetworks/neural_network_raw.ipynb)). Raw implementation of a simple neural network to classify MNIST digits dataset.
+- **Convolutional Neural Network** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/3_NeuralNetworks/convolutional_network.ipynb)). Use TensorFlow v2 'layers' and 'model' API to build a convolutional neural network to classify MNIST digits dataset.
+- **Convolutional Neural Network (low-level)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/3_NeuralNetworks/convolutional_network_raw.ipynb)). Raw implementation of a convolutional neural network to classify MNIST digits dataset.
+
+##### Unsupervised
+- **Auto-Encoder** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/3_NeuralNetworks/autoencoder.ipynb)). Build an auto-encoder to encode an image to a lower dimension and re-construct it.
+- **DCGAN (Deep Convolutional Generative Adversarial Networks)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/3_NeuralNetworks/tensorflow_v2/dcgan.ipynb)). Build a Deep Convolutional Generative Adversarial Network (DCGAN) to generate images from noise.
+
+#### 4 - Utilities
+- **Save and Restore a model** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/4_Utils/save_restore_model.ipynb)). Save and Restore a model with TensorFlow v2.
+- **Build Custom Layers & Modules** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/tensorflow_v2/notebooks/4_Utils/build_costum_layers.ipynb)). Learn how to build your own layers / modules and integrate them into TensorFlow v2 Models.
+
+## Installation
+
+To install TensorFlow v2, simply run:
+```
+pip install tensorflow==2.0.0a0
+```
+
+or (if you want GPU support):
+```
+pip install tensorflow_gpu==2.0.0a0
+```

tensorflow_v2/notebooks/0_Prerequisite/ml_introduction.ipynb

@@ -0,0 +1,50 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Machine Learning\n",
+    "\n",
+    "Prior to start browsing the examples, it may be useful that you get familiar with machine learning, as TensorFlow is mostly used for machine learning tasks (especially Neural Networks). You can find below a list of useful links, that can give you the basic knowledge required for this TensorFlow Tutorial.\n",
+    "\n",
+    "## Machine Learning\n",
+    "\n",
+    "- [An Introduction to Machine Learning Theory and Its Applications: A Visual Tutorial with Examples](https://www.toptal.com/machine-learning/machine-learning-theory-an-introductory-primer)\n",
+    "- [A Gentle Guide to Machine Learning](https://blog.monkeylearn.com/a-gentle-guide-to-machine-learning/)\n",
+    "- [A Visual Introduction to Machine Learning](http://www.r2d3.us/visual-intro-to-machine-learning-part-1/)\n",
+    "- [Introduction to Machine Learning](http://alex.smola.org/drafts/thebook.pdf)\n",
+    "\n",
+    "## Deep Learning & Neural Networks\n",
+    "\n",
+    "- [An Introduction to Neural Networks](http://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html)\n",
+    "- [An Introduction to Image Recognition with Deep Learning](https://medium.com/@ageitgey/machine-learning-is-fun-part-3-deep-learning-and-convolutional-neural-networks-f40359318721)\n",
+    "- [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/index.html)\n",
+    "\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "IPython (Python 2.7)",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

tensorflow_v2/notebooks/0_Prerequisite/mnist_dataset_intro.ipynb

@@ -0,0 +1,96 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "# MNIST Dataset Introduction\n",
+    "\n",
+    "Most examples are using MNIST dataset of handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flatten and converted to a 1-D numpy array of 784 features (28*28).\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "![MNIST Digits](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n",
+    "\n",
+    "## Usage\n",
+    "In our examples, we are using TensorFlow [input_data.py](https://github.com/tensorflow/tensorflow/blob/r0.7/tensorflow/examples/tutorials/mnist/input_data.py) script to load that dataset.\n",
+    "It is quite useful for managing our data, and handle:\n",
+    "\n",
+    "- Dataset downloading\n",
+    "\n",
+    "- Loading the entire dataset into numpy array: \n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Import MNIST\n",
+    "from tensorflow.examples.tutorials.mnist import input_data\n",
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
+    "\n",
+    "# Load data\n",
+    "X_train = mnist.train.images\n",
+    "Y_train = mnist.train.labels\n",
+    "X_test = mnist.test.images\n",
+    "Y_test = mnist.test.labels"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "- A `next_batch` function that can iterate over the whole dataset and return only the desired fraction of the dataset samples (in order to save memory and avoid to load the entire dataset)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Get the next 64 images array and labels\n",
+    "batch_X, batch_Y = mnist.train.next_batch(64)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Link: http://yann.lecun.com/exdb/mnist/"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

tensorflow_v2/notebooks/1_Introduction/basic_operations.ipynb

@@ -0,0 +1,172 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Basic Tensor Operations\n",
+    "\n",
+    "Basic tensor operations using TensorFlow v2.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from __future__ import print_function\n",
+    "import tensorflow as tf"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define tensor constants.\n",
+    "a = tf.constant(2)\n",
+    "b = tf.constant(3)\n",
+    "c = tf.constant(5)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "add = 5\n",
+      "sub = -1\n",
+      "mul = 6\n",
+      "div = 0.6666666666666666\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Various tensor operations.\n",
+    "# Note: Tensors also support python operators (+, *, ...)\n",
+    "add = tf.add(a, b)\n",
+    "sub = tf.subtract(a, b)\n",
+    "mul = tf.multiply(a, b)\n",
+    "div = tf.divide(a, b)\n",
+    "\n",
+    "# Access tensors value.\n",
+    "print(\"add =\", add.numpy())\n",
+    "print(\"sub =\", sub.numpy())\n",
+    "print(\"mul =\", mul.numpy())\n",
+    "print(\"div =\", div.numpy())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "mean = 3\n",
+      "sum = 10\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Some more operations.\n",
+    "mean = tf.reduce_mean([a, b, c])\n",
+    "sum = tf.reduce_sum([a, b, c])\n",
+    "\n",
+    "# Access tensors value.\n",
+    "print(\"mean =\", mean.numpy())\n",
+    "print(\"sum =\", sum.numpy())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Matrix multiplications.\n",
+    "matrix1 = tf.constant([[1., 2.], [3., 4.]])\n",
+    "matrix2 = tf.constant([[5., 6.], [7., 8.]])\n",
+    "\n",
+    "product = tf.matmul(matrix1, matrix2)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<tf.Tensor: id=31, shape=(2, 2), dtype=float32, numpy=\n",
+       "array([[19., 22.],\n",
+       "       [43., 50.]], dtype=float32)>"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Display Tensor.\n",
+    "product"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[19., 22.],\n",
+       "       [43., 50.]], dtype=float32)"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Convert Tensor to Numpy.\n",
+    "product.numpy()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

tensorflow_v2/notebooks/1_Introduction/helloworld.ipynb

@@ -0,0 +1,83 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Hello World\n",
+    "\n",
+    "A very simple \"hello world\" using TensorFlow v2 tensors.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import tensorflow as tf"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "tf.Tensor(hello world, shape=(), dtype=string)\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Create a Tensor.\n",
+    "hello = tf.constant(\"hello world\")\n",
+    "print hello"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "hello world\n"
+     ]
+    }
+   ],
+   "source": [
+    "# To access a Tensor value, call numpy().\n",
+    "print hello.numpy()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

tensorflow_v2/notebooks/4_Utils/build_custom_layers.ipynb

@@ -0,0 +1,304 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Build Custom Layers & Modules\n",
+    "\n",
+    "Build custom layers and modules with TensorFlow v2.\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from __future__ import absolute_import, division, print_function\n",
+    "\n",
+    "import tensorflow as tf\n",
+    "from tensorflow.keras import Model, layers\n",
+    "import numpy as np"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# MNIST dataset parameters.\n",
+    "num_classes = 10 # 0 to 9 digits\n",
+    "num_features = 784 # 28*28\n",
+    "\n",
+    "# Training parameters.\n",
+    "learning_rate = 0.01\n",
+    "training_steps = 500\n",
+    "batch_size = 256\n",
+    "display_step = 50"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Prepare MNIST data.\n",
+    "from tensorflow.keras.datasets import mnist\n",
+    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
+    "# Convert to float32.\n",
+    "x_train, x_test = np.array(x_train, np.float32), np.array(x_test, np.float32)\n",
+    "# Flatten images to 1-D vector of 784 features (28*28).\n",
+    "x_train, x_test = x_train.reshape([-1, num_features]), x_test.reshape([-1, num_features])\n",
+    "# Normalize images value from [0, 255] to [0, 1].\n",
+    "x_train, x_test = x_train / 255., x_test / 255."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Use tf.data API to shuffle and batch data.\n",
+    "train_data = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
+    "train_data = train_data.repeat().shuffle(5000).batch(batch_size).prefetch(1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create a custom layer\n",
+    "\n",
+    "Build a custom layer with inner-variables."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create a custom layer, extending TF 'Layer' class.\n",
+    "# Layer compute: y = relu(W * x + b)\n",
+    "class CustomLayer1(layers.Layer):\n",
+    "    \n",
+    "    # Layer arguments.\n",
+    "    def __init__(self, num_units, **kwargs):\n",
+    "        # Store the number of units (neurons).\n",
+    "        self.num_units = num_units\n",
+    "        super(CustomLayer1, self).__init__(**kwargs)\n",
+    "        \n",
+    "    def build(self, input_shape):\n",
+    "        # Note: a custom layer can also include any other TF 'layers'.\n",
+    "        shape = tf.TensorShape((input_shape[1], self.num_units))\n",
+    "        # Create weight variables for this layer.\n",
+    "        self.weight = self.add_weight(name='W',\n",
+    "                                      shape=shape,\n",
+    "                                      initializer=tf.initializers.RandomNormal,\n",
+    "                                      trainable=True)\n",
+    "        self.bias = self.add_weight(name='b',\n",
+    "                                    shape=[self.num_units])\n",
+    "        # Make sure to call the `build` method at the end\n",
+    "        super(CustomLayer1, self).build(input_shape)\n",
+    "\n",
+    "    def call(self, inputs):\n",
+    "        x = tf.matmul(inputs, self.weight)\n",
+    "        x = x + self.bias\n",
+    "        return tf.nn.relu(x)\n",
+    "\n",
+    "    def get_config(self):\n",
+    "        base_config = super(CustomLayer1, self).get_config()\n",
+    "        base_config['num_units'] = self.num_units\n",
+    "        return base_config"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create another custom layer\n",
+    "\n",
+    "Build another custom layer with inner TF 'layers'."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create a custom layer, extending TF 'Layer' class.\n",
+    "# Custom layer: 2 Fully-Connected layers with residual connection.\n",
+    "class CustomLayer2(layers.Layer):\n",
+    "    \n",
+    "    # Layer arguments.\n",
+    "    def __init__(self, num_units, **kwargs):\n",
+    "        self.num_units = num_units\n",
+    "        super(CustomLayer2, self).__init__(**kwargs)\n",
+    "        \n",
+    "    def build(self, input_shape):\n",
+    "        shape = tf.TensorShape((input_shape[1], self.num_units))\n",
+    "        \n",
+    "        self.inner_layer1 = layers.Dense(1)\n",
+    "        self.inner_layer2 = layers.Dense(self.num_units)\n",
+    "        \n",
+    "        # Make sure to call the `build` method at the end\n",
+    "        super(CustomLayer2, self).build(input_shape)\n",
+    "\n",
+    "    def call(self, inputs):\n",
+    "        x = self.inner_layer1(inputs)\n",
+    "        x = tf.nn.relu(x)\n",
+    "        x = self.inner_layer2(x)\n",
+    "        return x + inputs\n",
+    "\n",
+    "    def get_config(self):\n",
+    "        base_config = super(CustomLayer2, self).get_config()\n",
+    "        base_config['num_units'] = self.num_units\n",
+    "        return base_config"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Build Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create TF Model.\n",
+    "class CustomNet(Model):\n",
+    "    \n",
+    "    def __init__(self):\n",
+    "        super(CustomNet, self).__init__()\n",
+    "        # Use custom layers created above.\n",
+    "        self.layer1 = CustomLayer1(64)\n",
+    "        self.layer2 = CustomLayer2(64)\n",
+    "        self.out = layers.Dense(num_classes, activation=tf.nn.softmax)\n",
+    "\n",
+    "    # Set forward pass.\n",
+    "    def __call__(self, x, is_training=False):\n",
+    "        x = self.layer1(x)\n",
+    "        x = tf.nn.relu(x)\n",
+    "        x = self.layer2(x)\n",
+    "        if not is_training:\n",
+    "            # tf cross entropy expect logits without softmax, so only\n",
+    "            # apply softmax when not training.\n",
+    "            x = tf.nn.softmax(x)\n",
+    "        return x\n",
+    "\n",
+    "# Build neural network model.\n",
+    "custom_net = CustomNet()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cross-Entropy loss function.\n",
+    "def cross_entropy(y_pred, y_true):\n",
+    "    y_true = tf.cast(y_true, tf.int64)\n",
+    "    crossentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n",
+    "    return tf.reduce_mean(crossentropy)\n",
+    "\n",
+    "# Accuracy metric.\n",
+    "def accuracy(y_pred, y_true):\n",
+    "    # Predicted class is the index of highest score in prediction vector (i.e. argmax).\n",
+    "    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64))\n",
+    "    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+    "\n",
+    "# Adam optimizer.\n",
+    "optimizer = tf.optimizers.Adam(learning_rate)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Optimization process. \n",
+    "def run_optimization(x, y):\n",
+    "    # Wrap computation inside a GradientTape for automatic differentiation.\n",
+    "    with tf.GradientTape() as g:\n",
+    "        pred = custom_net(x, is_training=True)\n",
+    "        loss = cross_entropy(pred, y)\n",
+    "\n",
+    "        # Compute gradients.\n",
+    "        gradients = g.gradient(loss, custom_net.trainable_variables)\n",
+    "\n",
+    "        # Update W and b following gradients.\n",
+    "        optimizer.apply_gradients(zip(gradients, custom_net.trainable_variables))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "step: 50, loss: 3.363096, accuracy: 0.902344\n",
+      "step: 100, loss: 3.344931, accuracy: 0.910156\n",
+      "step: 150, loss: 3.336300, accuracy: 0.914062\n",
+      "step: 200, loss: 3.318396, accuracy: 0.925781\n",
+      "step: 250, loss: 3.300045, accuracy: 0.937500\n",
+      "step: 300, loss: 3.335487, accuracy: 0.898438\n",
+      "step: 350, loss: 3.330979, accuracy: 0.914062\n",
+      "step: 400, loss: 3.298509, accuracy: 0.921875\n",
+      "step: 450, loss: 3.278253, accuracy: 0.953125\n",
+      "step: 500, loss: 3.285335, accuracy: 0.945312\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Run training for the given number of steps.\n",
+    "for step, (batch_x, batch_y) in enumerate(train_data.take(training_steps), 1):\n",
+    "    # Run the optimization to update W and b values.\n",
+    "    run_optimization(batch_x, batch_y)\n",
+    "    \n",
+    "    if step % display_step == 0:\n",
+    "        pred = custom_net(batch_x, is_training=False)\n",
+    "        loss = cross_entropy(pred, batch_y)\n",
+    "        acc = accuracy(pred, batch_y)\n",
+    "        print(\"step: %i, loss: %f, accuracy: %f\" % (step, loss, acc))"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

tensorflow_v2/notebooks/4_Utils/save_restore_model.ipynb

@@ -0,0 +1,573 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Save & Restore a Model\n",
+    "\n",
+    "Save and Restore a model using TensorFlow v2. In this example, we will go over both low and high-level approaches: \n",
+    "- Low-level: TF Checkpoint.\n",
+    "- High-level: TF Module/Model saver.\n",
+    "\n",
+    "This example is using the MNIST database of handwritten digits as toy dataset\n",
+    "(http://yann.lecun.com/exdb/mnist/).\n",
+    "\n",
+    "- Author: Aymeric Damien\n",
+    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from __future__ import absolute_import, division, print_function\n",
+    "\n",
+    "import tensorflow as tf\n",
+    "import numpy as np"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# MNIST dataset parameters.\n",
+    "num_classes = 10 # 0 to 9 digits\n",
+    "num_features = 784 # 28*28\n",
+    "\n",
+    "# Training parameters.\n",
+    "learning_rate = 0.01\n",
+    "training_steps = 1000\n",
+    "batch_size = 256\n",
+    "display_step = 50"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Prepare MNIST data.\n",
+    "from tensorflow.keras.datasets import mnist\n",
+    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
+    "# Convert to float32.\n",
+    "x_train, x_test = np.array(x_train, np.float32), np.array(x_test, np.float32)\n",
+    "# Flatten images to 1-D vector of 784 features (28*28).\n",
+    "x_train, x_test = x_train.reshape([-1, num_features]), x_test.reshape([-1, num_features])\n",
+    "# Normalize images value from [0, 255] to [0, 1].\n",
+    "x_train, x_test = x_train / 255., x_test / 255."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Use tf.data API to shuffle and batch data.\n",
+    "train_data = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
+    "train_data = train_data.repeat().shuffle(5000).batch(batch_size).prefetch(1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1) TF Checkpoint\n",
+    "\n",
+    "Basic logistic regression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Weight of shape [784, 10], the 28*28 image features, and total number of classes.\n",
+    "W = tf.Variable(tf.random.normal([num_features, num_classes]), name=\"weight\")\n",
+    "# Bias of shape [10], the total number of classes.\n",
+    "b = tf.Variable(tf.zeros([num_classes]), name=\"bias\")\n",
+    "\n",
+    "# Logistic regression (Wx + b).\n",
+    "def logistic_regression(x):\n",
+    "    # Apply softmax to normalize the logits to a probability distribution.\n",
+    "    return tf.nn.softmax(tf.matmul(x, W) + b)\n",
+    "\n",
+    "# Cross-Entropy loss function.\n",
+    "def cross_entropy(y_pred, y_true):\n",
+    "    # Encode label to a one hot vector.\n",
+    "    y_true = tf.one_hot(y_true, depth=num_classes)\n",
+    "    # Clip prediction values to avoid log(0) error.\n",
+    "    y_pred = tf.clip_by_value(y_pred, 1e-9, 1.)\n",
+    "    # Compute cross-entropy.\n",
+    "    return tf.reduce_mean(-tf.reduce_sum(y_true * tf.math.log(y_pred)))\n",
+    "\n",
+    "# Accuracy metric.\n",
+    "def accuracy(y_pred, y_true):\n",
+    "    # Predicted class is the index of highest score in prediction vector (i.e. argmax).\n",
+    "    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64))\n",
+    "    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+    "\n",
+    "# Adam optimizer.\n",
+    "optimizer = tf.optimizers.Adam(learning_rate)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Optimization process. \n",
+    "def run_optimization(x, y):\n",
+    "    # Wrap computation inside a GradientTape for automatic differentiation.\n",
+    "    with tf.GradientTape() as g:\n",
+    "        pred = logistic_regression(x)\n",
+    "        loss = cross_entropy(pred, y)\n",
+    "\n",
+    "        # Compute gradients.\n",
+    "        gradients = g.gradient(loss, [W, b])\n",
+    "\n",
+    "        # Update W and b following gradients.\n",
+    "        optimizer.apply_gradients(zip(gradients, [W, b]))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "step: 50, loss: 535.380981, accuracy: 0.656250\n",
+      "step: 100, loss: 354.681152, accuracy: 0.765625\n",
+      "step: 150, loss: 225.300934, accuracy: 0.785156\n",
+      "step: 200, loss: 163.948761, accuracy: 0.859375\n",
+      "step: 250, loss: 129.653534, accuracy: 0.878906\n",
+      "step: 300, loss: 170.743576, accuracy: 0.859375\n",
+      "step: 350, loss: 97.912575, accuracy: 0.910156\n",
+      "step: 400, loss: 144.119141, accuracy: 0.906250\n",
+      "step: 450, loss: 164.991943, accuracy: 0.875000\n",
+      "step: 500, loss: 145.191666, accuracy: 0.871094\n",
+      "step: 550, loss: 82.272644, accuracy: 0.925781\n",
+      "step: 600, loss: 149.180237, accuracy: 0.878906\n",
+      "step: 650, loss: 127.171280, accuracy: 0.871094\n",
+      "step: 700, loss: 116.045761, accuracy: 0.910156\n",
+      "step: 750, loss: 92.582680, accuracy: 0.906250\n",
+      "step: 800, loss: 108.238007, accuracy: 0.894531\n",
+      "step: 850, loss: 92.755638, accuracy: 0.894531\n",
+      "step: 900, loss: 69.131119, accuracy: 0.902344\n",
+      "step: 950, loss: 67.176285, accuracy: 0.921875\n",
+      "step: 1000, loss: 104.205658, accuracy: 0.890625\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Run training for the given number of steps.\n",
+    "for step, (batch_x, batch_y) in enumerate(train_data.take(training_steps), 1):\n",
+    "    # Run the optimization to update W and b values.\n",
+    "    run_optimization(batch_x, batch_y)\n",
+    "    \n",
+    "    if step % display_step == 0:\n",
+    "        pred = logistic_regression(batch_x)\n",
+    "        loss = cross_entropy(pred, batch_y)\n",
+    "        acc = accuracy(pred, batch_y)\n",
+    "        print(\"step: %i, loss: %f, accuracy: %f\" % (step, loss, acc))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Save and Load with TF Checkpoint"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Save weights and optimizer variables.\n",
+    "# Create a dict of variables to save.\n",
+    "vars_to_save = {\"W\": W, \"b\": b, \"optimizer\": optimizer}\n",
+    "# TF Checkpoint, pass the dict as **kwargs.\n",
+    "checkpoint = tf.train.Checkpoint(**vars_to_save)\n",
+    "# TF CheckpointManager to manage saving parameters.\n",
+    "saver = tf.train.CheckpointManager(\n",
+    "      checkpoint, directory=\"./tf-example\", max_to_keep=5)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'./tf-example/ckpt-1'"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Save variables.\n",
+    "saver.save()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "-0.09673191"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Check weight value.\n",
+    "np.mean(W.numpy())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Reset variables to test restore.\n",
+    "W = tf.Variable(tf.random.normal([num_features, num_classes]), name=\"weight\")\n",
+    "b = tf.Variable(tf.zeros([num_classes]), name=\"bias\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "-0.0083419625"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Check resetted weight value.\n",
+    "np.mean(W.numpy())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x12cf965d0>"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Set checkpoint to load data.\n",
+    "vars_to_load = {\"W\": W, \"b\": b, \"optimizer\": optimizer}\n",
+    "checkpoint = tf.train.Checkpoint(**vars_to_load)\n",
+    "# Restore variables from latest checkpoint.\n",
+    "latest_ckpt = tf.train.latest_checkpoint(\"./tf-example\")\n",
+    "checkpoint.restore(latest_ckpt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "-0.09673191"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Confirm that W has been correctly restored.\n",
+    "np.mean(W.numpy())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2) TF Model\n",
+    "\n",
+    "Basic neural network with TF Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tensorflow.keras import Model, layers"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# MNIST dataset parameters.\n",
+    "num_classes = 10 # 0 to 9 digits\n",
+    "num_features = 784 # 28*28\n",
+    "\n",
+    "# Training parameters.\n",
+    "learning_rate = 0.01\n",
+    "training_steps = 1000\n",
+    "batch_size = 256\n",
+    "display_step = 100"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create TF Model.\n",
+    "class NeuralNet(Model):\n",
+    "    # Set layers.\n",
+    "    def __init__(self):\n",
+    "        super(NeuralNet, self).__init__(name=\"NeuralNet\")\n",
+    "        # First fully-connected hidden layer.\n",
+    "        self.fc1 = layers.Dense(64, activation=tf.nn.relu)\n",
+    "        # Second fully-connected hidden layer.\n",
+    "        self.fc2 = layers.Dense(128, activation=tf.nn.relu)\n",
+    "        # Third fully-connecter hidden layer.\n",
+    "        self.out = layers.Dense(num_classes, activation=tf.nn.softmax)\n",
+    "\n",
+    "    # Set forward pass.\n",
+    "    def __call__(self, x, is_training=False):\n",
+    "        x = self.fc1(x)\n",
+    "        x = self.out(x)\n",
+    "        if not is_training:\n",
+    "            # tf cross entropy expect logits without softmax, so only\n",
+    "            # apply softmax when not training.\n",
+    "            x = tf.nn.softmax(x)\n",
+    "        return x\n",
+    "\n",
+    "# Build neural network model.\n",
+    "neural_net = NeuralNet()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cross-Entropy loss function.\n",
+    "def cross_entropy(y_pred, y_true):\n",
+    "    y_true = tf.cast(y_true, tf.int64)\n",
+    "    crossentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n",
+    "    return tf.reduce_mean(crossentropy)\n",
+    "\n",
+    "# Accuracy metric.\n",
+    "def accuracy(y_pred, y_true):\n",
+    "    # Predicted class is the index of highest score in prediction vector (i.e. argmax).\n",
+    "    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64))\n",
+    "    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+    "\n",
+    "# Adam optimizer.\n",
+    "optimizer = tf.optimizers.Adam(learning_rate)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Optimization process. \n",
+    "def run_optimization(x, y):\n",
+    "    # Wrap computation inside a GradientTape for automatic differentiation.\n",
+    "    with tf.GradientTape() as g:\n",
+    "        pred = neural_net(x, is_training=True)\n",
+    "        loss = cross_entropy(pred, y)\n",
+    "\n",
+    "        # Compute gradients.\n",
+    "        gradients = g.gradient(loss, neural_net.trainable_variables)\n",
+    "\n",
+    "        # Update W and b following gradients.\n",
+    "        optimizer.apply_gradients(zip(gradients, neural_net.trainable_variables))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "step: 100, loss: 2.188605, accuracy: 0.902344\n",
+      "step: 200, loss: 2.182990, accuracy: 0.929688\n",
+      "step: 300, loss: 2.180439, accuracy: 0.945312\n",
+      "step: 400, loss: 2.178496, accuracy: 0.957031\n",
+      "step: 500, loss: 2.177517, accuracy: 0.968750\n",
+      "step: 600, loss: 2.177163, accuracy: 0.968750\n",
+      "step: 700, loss: 2.177454, accuracy: 0.960938\n",
+      "step: 800, loss: 2.177589, accuracy: 0.960938\n",
+      "step: 900, loss: 2.176507, accuracy: 0.964844\n",
+      "step: 1000, loss: 2.177557, accuracy: 0.960938\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Run training for the given number of steps.\n",
+    "for step, (batch_x, batch_y) in enumerate(train_data.take(training_steps), 1):\n",
+    "    # Run the optimization to update W and b values.\n",
+    "    run_optimization(batch_x, batch_y)\n",
+    "    \n",
+    "    if step % display_step == 0:\n",
+    "        pred = neural_net(batch_x, is_training=False)\n",
+    "        loss = cross_entropy(pred, batch_y)\n",
+    "        acc = accuracy(pred, batch_y)\n",
+    "        print(\"step: %i, loss: %f, accuracy: %f\" % (step, loss, acc))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Save and Load with TF Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Save TF model.\n",
+    "neural_net.save_weights(filepath=\"./tfmodel.ckpt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "accuracy: 0.101562\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Re-build neural network model with default values.\n",
+    "neural_net = NeuralNet()\n",
+    "# Test model performance.\n",
+    "pred = neural_net(batch_x)\n",
+    "print(\"accuracy: %f\" % accuracy(pred, batch_y))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x13118b950>"
+      ]
+     },
+     "execution_count": 23,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Load saved weights.\n",
+    "neural_net.load_weights(filepath=\"./tfmodel.ckpt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "accuracy: 0.960938\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Test that weights loaded correctly.\n",
+    "pred = neural_net(batch_x)\n",
+    "print(\"accuracy: %f\" % accuracy(pred, batch_y))"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}