minor fix

Setup_TensorFlow.md

@@ -0,0 +1,486 @@
+_From TensorFlow Official docs_
+
+# Download and Setup <a class="md-anchor" id="AUTOGENERATED-download-and-setup"></a>
+
+You can install TensorFlow using our provided binary packages or from source.
+
+## Binary Installation <a class="md-anchor" id="AUTOGENERATED-binary-installation"></a>
+
+The TensorFlow Python API currently requires Python 2.7: we are
+[working](https://github.com/tensorflow/tensorflow/issues/1) on adding support
+for Python 3.
+
+The simplest way to install TensorFlow is using
+[pip](https://pypi.python.org/pypi/pip) for both Linux and Mac.
+
+If you encounter installation errors, see
+[common problems](#common_install_problems) for some solutions. To simplify
+installation, please consider using our virtualenv-based instructions
+[here](#virtualenv_install).
+
+### Ubuntu/Linux 64-bit <a class="md-anchor" id="AUTOGENERATED-ubuntu-linux-64-bit"></a>
+
+```bash
+# For CPU-only version
+$ pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl
+
+# For GPU-enabled version (only install this version if you have the CUDA sdk installed)
+$ pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl
+```
+
+### Mac OS X <a class="md-anchor" id="AUTOGENERATED-mac-os-x"></a>
+
+On OS X, we recommend installing [homebrew](http://brew.sh) and `brew install
+python` before proceeding, or installing TensorFlow within [virtualenv](#virtualenv_install).
+
+```bash
+# Only CPU-version is available at the moment.
+$ pip install https://storage.googleapis.com/tensorflow/mac/tensorflow-0.5.0-py2-none-any.whl
+```
+
+## Docker-based installation <a class="md-anchor" id="AUTOGENERATED-docker-based-installation"></a>
+
+We also support running TensorFlow via [Docker](http://docker.com/), which lets
+you avoid worrying about setting up dependencies.
+
+First, [install Docker](http://docs.docker.com/engine/installation/). Once
+Docker is up and running, you can start a container with one command:
+
+```bash
+$ docker run -it b.gcr.io/tensorflow/tensorflow
+```
+
+This will start a container with TensorFlow and all its dependencies already
+installed.
+
+### Additional images <a class="md-anchor" id="AUTOGENERATED-additional-images"></a>
+
+The default Docker image above contains just a minimal set of libraries for
+getting up and running with TensorFlow. We also have the following container,
+which you can use in the `docker run` command above:
+
+* `b.gcr.io/tensorflow/tensorflow-full`: Contains a complete TensorFlow source
+  installation, including all utilities needed to build and run TensorFlow. This
+  makes it easy to experiment directly with the source, without needing to
+  install any of the dependencies described above.
+
+## VirtualEnv-based installation <a class="md-anchor" id="virtualenv_install"></a>
+
+We recommend using [virtualenv](https://pypi.python.org/pypi/virtualenv) to
+create an isolated container and install TensorFlow in that container -- it is
+optional but makes verifying installation issues easier.
+
+First, install all required tools:
+
+```bash
+# On Linux:
+$ sudo apt-get install python-pip python-dev python-virtualenv
+
+# On Mac:
+$ sudo easy_install pip  # If pip is not already installed
+$ sudo pip install --upgrade virtualenv
+```
+
+Next, set up a new virtualenv environment.  To set it up in the
+directory `~/tensorflow`, run:
+
+```bash
+$ virtualenv --system-site-packages ~/tensorflow
+$ cd ~/tensorflow
+```
+
+Then activate the virtualenv:
+
+```bash
+$ source bin/activate  # If using bash
+$ source bin/activate.csh  # If using csh
+(tensorflow)$  # Your prompt should change
+```
+
+Inside the virtualenv, install TensorFlow:
+
+```bash
+# For CPU-only linux x86_64 version
+(tensorflow)$ pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl
+
+# For GPU-enabled linux x86_64 version
+(tensorflow)$ pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl
+
+# For Mac CPU-only version
+(tensorflow)$ pip install --upgrade https://storage.googleapis.com/tensorflow/mac/tensorflow-0.5.0-py2-none-any.whl
+```
+
+Make sure you have downloaded the source code for TensorFlow, and then you can
+then run an example TensorFlow program like:
+
+```bash
+(tensorflow)$ cd tensorflow/models/image/mnist
+(tensorflow)$ python convolutional.py
+
+# When you are done using TensorFlow:
+(tensorflow)$ deactivate  # Deactivate the virtualenv
+
+$  # Your prompt should change back
+```
+
+## Try your first TensorFlow program <a class="md-anchor" id="AUTOGENERATED-try-your-first-tensorflow-program"></a>
+
+### (Optional) Enable GPU Support <a class="md-anchor" id="AUTOGENERATED--optional--enable-gpu-support"></a>
+
+If you installed the GPU-enabled TensorFlow pip binary, you must have the
+correct versions of the CUDA SDK and CUDNN installed on your
+system.  Please see [the CUDA installation instructions](#install_cuda).
+
+You also need to set the `LD_LIBRARY_PATH` and `CUDA_HOME` environment
+variables.  Consider adding the commands below to your `~/.bash_profile`.  These
+assume your CUDA installation is in `/usr/local/cuda`:
+
+```bash
+export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"
+export CUDA_HOME=/usr/local/cuda
+```
+
+### Run TensorFlow <a class="md-anchor" id="AUTOGENERATED-run-tensorflow"></a>
+
+Open a python terminal:
+
+```bash
+$ python
+
+>>> import tensorflow as tf
+>>> hello = tf.constant('Hello, TensorFlow!')
+>>> sess = tf.Session()
+>>> print sess.run(hello)
+Hello, TensorFlow!
+>>> a = tf.constant(10)
+>>> b = tf.constant(32)
+>>> print sess.run(a+b)
+42
+>>>
+
+```
+
+## Installing from sources <a class="md-anchor" id="source"></a>
+
+### Clone the TensorFlow repository <a class="md-anchor" id="AUTOGENERATED-clone-the-tensorflow-repository"></a>
+
+```bash
+$ git clone --recurse-submodules https://github.com/tensorflow/tensorflow
+```
+
+`--recurse-submodules` is required to fetch the protobuf library that TensorFlow
+depends on.
+
+### Installation for Linux <a class="md-anchor" id="AUTOGENERATED-installation-for-linux"></a>
+
+#### Install Bazel <a class="md-anchor" id="AUTOGENERATED-install-bazel"></a>
+
+
+Follow instructions [here](http://bazel.io/docs/install.html) to install the
+dependencies for Bazel. Then download bazel version 0.1.1 using the
+[installer for your system](https://github.com/bazelbuild/bazel/releases) and
+run the installer as mentioned there:
+
+```bash
+$ chmod +x PATH_TO_INSTALL.SH
+$ ./PATH_TO_INSTALL.SH --user
+```
+
+Remember to replace `PATH_TO_INSTALL.SH` to point to the location where you
+downloaded the installer.
+
+Finally, follow the instructions in that script to place bazel into your binary
+path.
+
+#### Install other dependencies <a class="md-anchor" id="AUTOGENERATED-install-other-dependencies"></a>
+
+```bash
+$ sudo apt-get install python-numpy swig python-dev
+```
+
+#### Optional: Install CUDA (GPUs on Linux) <a class="md-anchor" id="install_cuda"></a>
+
+In order to build or run TensorFlow with GPU support, both Cuda Toolkit 7.0 and
+CUDNN 6.5 V2 from NVIDIA need to be installed.
+
+TensorFlow GPU support requires having a GPU card with NVidia Compute Capability >= 3.5.  Supported cards include but are not limited to:
+
+* NVidia Titan
+* NVidia Titan X
+* NVidia K20
+* NVidia K40
+
+##### Download and install Cuda Toolkit 7.0 <a class="md-anchor" id="AUTOGENERATED-download-and-install-cuda-toolkit-7.0"></a>
+
+https://developer.nvidia.com/cuda-toolkit-70
+
+Install the toolkit into e.g. `/usr/local/cuda`
+
+##### Download and install CUDNN Toolkit 6.5 <a class="md-anchor" id="AUTOGENERATED-download-and-install-cudnn-toolkit-6.5"></a>
+
+https://developer.nvidia.com/rdp/cudnn-archive
+
+Uncompress and copy the cudnn files into the toolkit directory.  Assuming the
+toolkit is installed in `/usr/local/cuda`:
+
+``` bash
+tar xvzf cudnn-6.5-linux-x64-v2.tgz
+sudo cp cudnn-6.5-linux-x64-v2/cudnn.h /usr/local/cuda/include
+sudo cp cudnn-6.5-linux-x64-v2/libcudnn* /usr/local/cuda/lib64
+```
+
+##### Configure TensorFlow's canonical view of Cuda libraries <a class="md-anchor" id="AUTOGENERATED-configure-tensorflow-s-canonical-view-of-cuda-libraries"></a>
+From the root of your source tree, run:
+
+``` bash
+$ ./configure
+Do you wish to build TensorFlow with GPU support? [y/n] y
+GPU support will be enabled for TensorFlow
+
+Please specify the location where CUDA 7.0 toolkit is installed. Refer to
+README.md for more details. [default is: /usr/local/cuda]: /usr/local/cuda
+
+Please specify the location where CUDNN 6.5 V2 library is installed. Refer to
+README.md for more details. [default is: /usr/local/cuda]: /usr/local/cuda
+
+Setting up Cuda include
+Setting up Cuda lib64
+Setting up Cuda bin
+Setting up Cuda nvvm
+Configuration finished
+```
+
+This creates a canonical set of symbolic links to the Cuda libraries on your system.
+Every time you change the Cuda library paths you need to run this step again before
+you invoke the bazel build command.
+
+##### Build your target with GPU support. <a class="md-anchor" id="AUTOGENERATED-build-your-target-with-gpu-support."></a>
+From the root of your source tree, run:
+
+```bash
+$ bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
+
+$ bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu
+# Lots of output. This tutorial iteratively calculates the major eigenvalue of
+# a 2x2 matrix, on GPU. The last few lines look like this.
+000009/000005 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]
+000006/000001 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]
+000009/000009 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]
+```
+
+Note that "--config=cuda" is needed to enable the GPU support.
+
+##### Enabling Cuda 3.0. <a class="md-anchor" id="AUTOGENERATED-enabling-cuda-3.0."></a>
+TensorFlow officially supports Cuda devices with 3.5 and 5.2 compute
+capabilities. In order to enable earlier Cuda devices such as Grid K520, you
+need to target Cuda 3.0. This can be done through TensorFlow unofficial
+settings with "configure".
+
+```bash
+$ TF_UNOFFICIAL_SETTING=1 ./configure
+
+# Same as the official settings above
+
+WARNING: You are configuring unofficial settings in TensorFlow. Because some
+external libraries are not backward compatible, these settings are largely
+untested and unsupported.
+
+Please specify a list of comma-separated Cuda compute capabilities you want to
+build with. You can find the compute capability of your device at:
+https://developer.nvidia.com/cuda-gpus.
+Please note that each additional compute capability significantly increases
+your build time and binary size. [Default is: "3.5,5.2"]: 3.0
+
+Setting up Cuda include
+Setting up Cuda lib64
+Setting up Cuda bin
+Setting up Cuda nvvm
+Configuration finished
+```
+
+##### Known issues <a class="md-anchor" id="AUTOGENERATED-known-issues"></a>
+
+* Although it is possible to build both Cuda and non-Cuda configs under the same
+source tree, we recommend to run "bazel clean" when switching between these two
+configs in the same source tree.
+
+* You have to run configure before running bazel build. Otherwise, the build
+will fail with a clear error message. In the future, we might consider making
+this more conveninent by including the configure step in our build process,
+given necessary bazel new feature support.
+
+### Installation for Mac OS X <a class="md-anchor" id="AUTOGENERATED-installation-for-mac-os-x"></a>
+
+Mac needs the same set of dependencies as Linux, however their installing those
+dependencies is different. Here is a set of useful links to help with installing
+the dependencies on Mac OS X :
+
+#### Bazel <a class="md-anchor" id="AUTOGENERATED-bazel"></a>
+
+Look for installation instructions for Mac OS X on
+[this](http://bazel.io/docs/install.html) page.
+
+#### SWIG <a class="md-anchor" id="AUTOGENERATED-swig"></a>
+
+[Mac OS X installation](http://www.swig.org/Doc3.0/Preface.html#Preface_osx_installation).
+
+Notes : You need to install
+[PCRE](ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/) and *NOT* PCRE2.
+
+#### Numpy <a class="md-anchor" id="AUTOGENERATED-numpy"></a>
+
+Follow installation instructions [here](http://docs.scipy.org/doc/numpy/user/install.html).
+
+
+### Create the pip package and install <a class="md-anchor" id="create-pip"></a>
+
+```bash
+$ bazel build -c opt //tensorflow/tools/pip_package:build_pip_package
+
+# To build with GPU support:
+$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
+
+$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
+
+# The name of the .whl file will depend on your platform.
+$ pip install /tmp/tensorflow_pkg/tensorflow-0.5.0-cp27-none-linux_x86_64.whl
+```
+
+## Train your first TensorFlow neural net model <a class="md-anchor" id="AUTOGENERATED-train-your-first-tensorflow-neural-net-model"></a>
+
+Starting from the root of your source tree, run:
+
+```python
+$ cd tensorflow/models/image/mnist
+$ python convolutional.py
+Succesfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
+Succesfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
+Succesfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
+Succesfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
+Extracting data/train-images-idx3-ubyte.gz
+Extracting data/train-labels-idx1-ubyte.gz
+Extracting data/t10k-images-idx3-ubyte.gz
+Extracting data/t10k-labels-idx1-ubyte.gz
+Initialized!
+Epoch 0.00
+Minibatch loss: 12.054, learning rate: 0.010000
+Minibatch error: 90.6%
+Validation error: 84.6%
+Epoch 0.12
+Minibatch loss: 3.285, learning rate: 0.010000
+Minibatch error: 6.2%
+Validation error: 7.0%
+...
+...
+```
+
+## Common Problems <a class="md-anchor" id="common_install_problems"></a>
+
+### GPU-related issues <a class="md-anchor" id="AUTOGENERATED-gpu-related-issues"></a>
+
+If you encounter the following when trying to run a TensorFlow program:
+
+```python
+ImportError: libcudart.so.7.0: cannot open shared object file: No such file or directory
+```
+
+Make sure you followed the the GPU installation [instructions](#install_cuda).
+
+### Pip installation issues <a class="md-anchor" id="AUTOGENERATED-pip-installation-issues"></a>
+
+#### Can't find setup.py <a class="md-anchor" id="AUTOGENERATED-can-t-find-setup.py"></a>
+
+If, during `pip install`, you encounter an error like:
+
+```bash
+...
+IOError: [Errno 2] No such file or directory: '/tmp/pip-o6Tpui-build/setup.py'
+```
+
+Solution: upgrade your version of `pip`:
+
+```bash
+pip install --upgrade pip
+```
+
+This may require `sudo`, depending on how `pip` is installed.
+
+#### SSLError: SSL_VERIFY_FAILED <a class="md-anchor" id="AUTOGENERATED-sslerror--ssl_verify_failed"></a>
+
+If, during pip install from a URL, you encounter an error like:
+
+```bash
+...
+SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
+```
+
+Solution: Download the wheel manually via curl or wget, and pip install locally.
+
+### On Linux <a class="md-anchor" id="AUTOGENERATED-on-linux"></a>
+
+If you encounter:
+
+```python
+...
+ "__add__", "__radd__",
+             ^
+SyntaxError: invalid syntax
+```
+
+Solution: make sure you are using Python 2.7.
+
+### On MacOSX <a class="md-anchor" id="AUTOGENERATED-on-macosx"></a>
+
+
+If you encounter:
+
+```python
+import six.moves.copyreg as copyreg
+
+ImportError: No module named copyreg
+```
+
+Solution: TensorFlow depends on protobuf, which requires `six-1.10.0`. Apple's
+default python environment has `six-1.4.1` and may be difficult to upgrade.
+There are several ways to fix this:
+
+1. Upgrade the system-wide copy of `six`:
+
+    ```bash
+    sudo easy_install -U six
+    ```
+
+2. Install a separate copy of python via homebrew:
+
+    ```bash
+    brew install python
+    ```
+
+3. Build or use TensorFlow
+   [within `virtualenv`](#virtualenv_install).
+
+
+
+If you encounter:
+
+```
+>>> import tensorflow as tf
+Traceback (most recent call last):
+  File "<stdin>", line 1, in <module>
+  File "/usr/local/lib/python2.7/site-packages/tensorflow/__init__.py", line 4, in <module>
+    from tensorflow.python import *
+  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 13, in <module>
+    from tensorflow.core.framework.graph_pb2 import *
+...
+  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/tensor_shape_pb2.py", line 22, in <module>
+    serialized_pb=_b('\n,tensorflow/core/framework/tensor_shape.proto\x12\ntensorflow\"d\n\x10TensorShapeProto\x12-\n\x03\x64im\x18\x02 \x03(\x0b\x32 .tensorflow.TensorShapeProto.Dim\x1a!\n\x03\x44im\x12\x0c\n\x04size\x18\x01 \x01(\x03\x12\x0c\n\x04name\x18\x02 \x01(\tb\x06proto3')
+TypeError: __init__() got an unexpected keyword argument 'syntax'
+```
+
+This is due to a conflict between protobuf versions (we require protobuf 3.0.0).
+The best current solution is to make sure older versions of protobuf are not
+installed, such as:
+
+```bash
+brew reinstall --devel protobuf
+```

examples/3 - Neural Networks/alexnet.py

@@ -126,7 +126,7 @@ with tf.Session() as sess:
             acc = sess.run(accuracy, feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.})
             # Calculate batch loss
             loss = sess.run(cost, feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.})
-            print "Iter " + str(step*batch_size) + ", Loss= " + "{:.6f}".format(loss) + ", Training Accuracy= " + "{:.5f}".format(acc)
+            print "Iter " + str(step*batch_size) + ", Minibatch Loss= " + "{:.6f}".format(loss) + ", Training Accuracy= " + "{:.5f}".format(acc)
         step += 1
     print "Optimization Finished!"
     # Calculate accuracy for 256 mnist test images

examples/3 - Neural Networks/convolutional_network.py

@@ -105,7 +105,7 @@ with tf.Session() as sess:
             acc = sess.run(accuracy, feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.})
             # Calculate batch loss
             loss = sess.run(cost, feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.})
-            print "Iter " + str(step*batch_size) + ", Loss= " + "{:.6f}".format(loss) + ", Training Accuracy= " + "{:.5f}".format(acc)
+            print "Iter " + str(step*batch_size) + ", Minibatch Loss= " + "{:.6f}".format(loss) + ", Training Accuracy= " + "{:.5f}".format(acc)
         step += 1
     print "Optimization Finished!"
     # Calculate accuracy for 256 mnist test images

examples/3 - Neural Networks/multilayer_perceptron.py

@@ -18,7 +18,7 @@ training_epochs = 15
 batch_size = 100
 display_step = 1
 
-#Network Parameters
+# Network Parameters
 n_hidden_1 = 256 # 1st layer num features
 n_hidden_2 = 256 # 2nd layer num features
 n_input = 784 # MNIST data input (img shape: 28*28)

notebooks/1 - Introduction/basic_operations.ipynb

@@ -195,15 +195,6 @@
     "    print result\n",
     "    # ==> [[ 12.]]"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

notebooks/1 - Introduction/helloworld.ipynb

@@ -61,15 +61,6 @@
     "# Run graph\n",
     "print sess.run(hello)"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

notebooks/2 - Basic Classifiers/linear_regression.ipynb

@@ -225,15 +225,6 @@
     "from IPython.display import Image\n",
     "Image(filename='linearreg.png')"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

notebooks/2 - Basic Classifiers/logistic_regression.ipynb

@@ -199,15 +199,6 @@
     "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
     "    print \"Accuracy:\", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

notebooks/2 - Basic Classifiers/nearest_neighbor.ipynb

@@ -352,15 +352,6 @@
     "    print \"Done!\"\n",
     "    print \"Accuracy:\", accuracy"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

notebooks/3 - Neural Networks/convolutional_network.ipynb

@@ -0,0 +1,318 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# A Multilayer Perceptron implementation example using TensorFlow library.\n",
+    "# This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)\n",
+    "\n",
+    "# Author: Aymeric Damien\n",
+    "# Project: https://github.com/aymericdamien/TensorFlow-Examples/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
+      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Import MINST data\n",
+    "import input_data\n",
+    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "import tensorflow as tf"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Parameters\n",
+    "learning_rate = 0.001\n",
+    "training_iters = 100000\n",
+    "batch_size = 128\n",
+    "display_step = 20"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Network Parameters\n",
+    "n_input = 784 # MNIST data input (img shape: 28*28)\n",
+    "n_classes = 10 # MNIST total classes (0-9 digits)\n",
+    "dropout = 0.75 # Dropout, probability to keep units"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# tf Graph input\n",
+    "x = tf.placeholder(tf.types.float32, [None, n_input])\n",
+    "y = tf.placeholder(tf.types.float32, [None, n_classes])\n",
+    "keep_prob = tf.placeholder(tf.types.float32) #dropout (keep probability)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Create model\n",
+    "def conv2d(img, w, b):\n",
+    "    return tf.nn.relu(tf.nn.bias_add(tf.nn.conv2d(img, w, strides=[1, 1, 1, 1], padding='SAME'),b))\n",
+    "\n",
+    "def max_pool(img, k):\n",
+    "    return tf.nn.max_pool(img, ksize=[1, k, k, 1], strides=[1, k, k, 1], padding='SAME')\n",
+    "\n",
+    "def conv_net(_X, _weights, _biases, _dropout):\n",
+    "    # Reshape input picture\n",
+    "    _X = tf.reshape(_X, shape=[-1, 28, 28, 1])\n",
+    "\n",
+    "    # Convolution Layer\n",
+    "    conv1 = conv2d(_X, _weights['wc1'], _biases['bc1'])\n",
+    "    # Max Pooling (down-sampling)\n",
+    "    conv1 = max_pool(conv1, k=2)\n",
+    "    # Apply Dropout\n",
+    "    conv1 = tf.nn.dropout(conv1, _dropout)\n",
+    "\n",
+    "    # Convolution Layer\n",
+    "    conv2 = conv2d(conv1, _weights['wc2'], _biases['bc2'])\n",
+    "    # Max Pooling (down-sampling)\n",
+    "    conv2 = max_pool(conv2, k=2)\n",
+    "    # Apply Dropout\n",
+    "    conv2 = tf.nn.dropout(conv2, _dropout)\n",
+    "\n",
+    "    # Fully connected layer\n",
+    "    # Reshape conv2 output to fit dense layer input\n",
+    "    dense1 = tf.reshape(conv2, [-1, _weights['wd1'].get_shape().as_list()[0]]) \n",
+    "    # Relu activation\n",
+    "    dense1 = tf.nn.relu(tf.add(tf.matmul(dense1, _weights['wd1']), _biases['bd1']))\n",
+    "    # Apply Dropout\n",
+    "    dense1 = tf.nn.dropout(dense1, _dropout) # Apply Dropout\n",
+    "\n",
+    "    # Output, class prediction\n",
+    "    out = tf.add(tf.matmul(dense1, _weights['out']), _biases['out'])\n",
+    "    return out"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Store layers weight & bias\n",
+    "weights = {\n",
+    "    'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32])), # 5x5 conv, 1 input, 32 outputs\n",
+    "    'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])), # 5x5 conv, 32 inputs, 64 outputs\n",
+    "    'wd1': tf.Variable(tf.random_normal([7*7*64, 1024])), # fully connected, 7*7*64 inputs, 1024 outputs\n",
+    "    'out': tf.Variable(tf.random_normal([1024, n_classes])) # 1024 inputs, 10 outputs (class prediction)\n",
+    "}\n",
+    "\n",
+    "biases = {\n",
+    "    'bc1': tf.Variable(tf.random_normal([32])),\n",
+    "    'bc2': tf.Variable(tf.random_normal([64])),\n",
+    "    'bd1': tf.Variable(tf.random_normal([1024])),\n",
+    "    'out': tf.Variable(tf.random_normal([n_classes]))\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Construct model\n",
+    "pred = conv_net(x, weights, biases, keep_prob)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Define loss and optimizer\n",
+    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y))\n",
+    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Evaluate model\n",
+    "correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))\n",
+    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.types.float32))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Initializing the variables\n",
+    "init = tf.initialize_all_variables()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Iter 2560, Minibatch Loss= 26046.011719, Training Accuracy= 0.21094\n",
+      "Iter 5120, Minibatch Loss= 10456.769531, Training Accuracy= 0.52344\n",
+      "Iter 7680, Minibatch Loss= 6273.207520, Training Accuracy= 0.71875\n",
+      "Iter 10240, Minibatch Loss= 6276.231445, Training Accuracy= 0.64062\n",
+      "Iter 12800, Minibatch Loss= 4188.221680, Training Accuracy= 0.77344\n",
+      "Iter 15360, Minibatch Loss= 2717.077637, Training Accuracy= 0.80469\n",
+      "Iter 17920, Minibatch Loss= 4057.120361, Training Accuracy= 0.81250\n",
+      "Iter 20480, Minibatch Loss= 1696.550415, Training Accuracy= 0.87500\n",
+      "Iter 23040, Minibatch Loss= 2525.317627, Training Accuracy= 0.85938\n",
+      "Iter 25600, Minibatch Loss= 2341.906738, Training Accuracy= 0.87500\n",
+      "Iter 28160, Minibatch Loss= 4200.535156, Training Accuracy= 0.79688\n",
+      "Iter 30720, Minibatch Loss= 1888.964355, Training Accuracy= 0.89062\n",
+      "Iter 33280, Minibatch Loss= 2167.645996, Training Accuracy= 0.84375\n",
+      "Iter 35840, Minibatch Loss= 1932.107544, Training Accuracy= 0.89844\n",
+      "Iter 38400, Minibatch Loss= 1562.430054, Training Accuracy= 0.90625\n",
+      "Iter 40960, Minibatch Loss= 1676.755249, Training Accuracy= 0.84375\n",
+      "Iter 43520, Minibatch Loss= 1003.626099, Training Accuracy= 0.93750\n",
+      "Iter 46080, Minibatch Loss= 1176.615479, Training Accuracy= 0.86719\n",
+      "Iter 48640, Minibatch Loss= 1260.592651, Training Accuracy= 0.88281\n",
+      "Iter 51200, Minibatch Loss= 1399.667969, Training Accuracy= 0.86719\n",
+      "Iter 53760, Minibatch Loss= 1259.961426, Training Accuracy= 0.89844\n",
+      "Iter 56320, Minibatch Loss= 1415.800781, Training Accuracy= 0.89062\n",
+      "Iter 58880, Minibatch Loss= 1835.365967, Training Accuracy= 0.85156\n",
+      "Iter 61440, Minibatch Loss= 1395.168823, Training Accuracy= 0.90625\n",
+      "Iter 64000, Minibatch Loss= 973.283569, Training Accuracy= 0.88281\n",
+      "Iter 66560, Minibatch Loss= 818.093811, Training Accuracy= 0.92969\n",
+      "Iter 69120, Minibatch Loss= 1178.744263, Training Accuracy= 0.92188\n",
+      "Iter 71680, Minibatch Loss= 845.889709, Training Accuracy= 0.89844\n",
+      "Iter 74240, Minibatch Loss= 1259.505615, Training Accuracy= 0.90625\n",
+      "Iter 76800, Minibatch Loss= 738.037109, Training Accuracy= 0.89844\n",
+      "Iter 79360, Minibatch Loss= 862.499146, Training Accuracy= 0.93750\n",
+      "Iter 81920, Minibatch Loss= 739.704041, Training Accuracy= 0.90625\n",
+      "Iter 84480, Minibatch Loss= 652.880310, Training Accuracy= 0.95312\n",
+      "Iter 87040, Minibatch Loss= 635.464600, Training Accuracy= 0.92969\n",
+      "Iter 89600, Minibatch Loss= 933.166626, Training Accuracy= 0.90625\n",
+      "Iter 92160, Minibatch Loss= 213.874893, Training Accuracy= 0.96094\n",
+      "Iter 94720, Minibatch Loss= 609.575684, Training Accuracy= 0.91406\n",
+      "Iter 97280, Minibatch Loss= 560.208008, Training Accuracy= 0.93750\n",
+      "Iter 99840, Minibatch Loss= 963.577148, Training Accuracy= 0.90625\n",
+      "Optimization Finished!\n",
+      "Testing Accuracy: 0.960938\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Launch the graph\n",
+    "with tf.Session() as sess:\n",
+    "    sess.run(init)\n",
+    "    step = 1\n",
+    "    # Keep training until reach max iterations\n",
+    "    while step * batch_size < training_iters:\n",
+    "        batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
+    "        # Fit training using batch data\n",
+    "        sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys, keep_prob: dropout})\n",
+    "        if step % display_step == 0:\n",
+    "            # Calculate batch accuracy\n",
+    "            acc = sess.run(accuracy, feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.})\n",
+    "            # Calculate batch loss\n",
+    "            loss = sess.run(cost, feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.})\n",
+    "            print \"Iter \" + str(step*batch_size) + \", Minibatch Loss= \" + \\\n",
+    "                  \"{:.6f}\".format(loss) + \", Training Accuracy= \" + \"{:.5f}\".format(acc)\n",
+    "        step += 1\n",
+    "    print \"Optimization Finished!\"\n",
+    "    # Calculate accuracy for 256 mnist test images\n",
+    "    print \"Testing Accuracy:\", sess.run(accuracy, feed_dict={x: mnist.test.images[:256], \n",
+    "                                                             y: mnist.test.labels[:256], \n",
+    "                                                             keep_prob: 1.})"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "IPython (Python 2.7)",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

notebooks/3 - Neural Networks/multilayer_perceptron.ipynb

@@ -225,15 +225,6 @@
     "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
     "    print \"Accuracy:\", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {