# TensorFlow-Slim image classification library [TF-slim](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim) is a new lightweight high-level API of TensorFlow (`tensorflow.contrib.slim`) for defining, training and evaluating complex models. This directory contains code for training and evaluating several widely used Convolutional Neural Network (CNN) image classification models using TF-slim. It contains scripts that will allow you to train models from scratch or fine-tune them from pre-trained network weights. It also contains code for downloading standard image datasets, converting them to TensorFlow's native TFRecord format and reading them in using TF-Slim's data reading and queueing utilities. You can easily train any model on any of these datasets, as we demonstrate below. We've also included a [jupyter notebook](https://github.com/tensorflow/models/blob/master/slim/slim_walkthrough.ipynb), which provides working examples of how to use TF-Slim for image classification. ## Contacts Maintainers of TF-slim: * Nathan Silberman, github: [nathansilberman](https://github.com/nathansilberman) * Sergio Guadarrama, github: [sguada](https://github.com/sguada) ## Table of contents Installation and setup
Preparing the datasets
Using pre-trained models
Training from scratch
Fine tuning to a new task
Evaluating performance
# Installation In this section, we describe the steps required to install the appropriate prerequisite packages. ## Installing latest version of TF-slim TF-Slim is available as `tf.contrib.slim` via TensorFlow 1.0. To test that your installation is working, execute the following command; it should run without raising any errors. ``` python -c "import tensorflow.contrib.slim as slim; eval = slim.evaluation.evaluate_once" ``` ## Installing the TF-slim image models library To use TF-Slim for image classification, you also have to install the [TF-Slim image models library](https://github.com/tensorflow/models/tree/master/slim), which is not part of the core TF library. To do this, check out the [tensorflow/models](https://github.com/tensorflow/models/) repository as follows: ```bash cd $HOME/workspace git clone https://github.com/tensorflow/models/ ``` This will put the TF-Slim image models library in `$HOME/workspace/models/slim`. (It will also create a directory called [models/inception](https://github.com/tensorflow/models/tree/master/inception), which contains an older version of slim; you can safely ignore this.) To verify that this has worked, execute the following commands; it should run without raising any errors. ``` cd $HOME/workspace/models/slim python -c "from nets import cifarnet; mynet = cifarnet.cifarnet" ``` # Preparing the datasets As part of this library, we've included scripts to download several popular image datasets (listed below) and convert them to slim format. Dataset | Training Set Size | Testing Set Size | Number of Classes | Comments :------:|:---------------:|:---------------------:|:-----------:|:-----------: Flowers|2500 | 2500 | 5 | Various sizes (source: Flickr) [Cifar10](https://www.cs.toronto.edu/~kriz/cifar.html) | 60k| 10k | 10 |32x32 color [MNIST](http://yann.lecun.com/exdb/mnist/)| 60k | 10k | 10 | 28x28 gray [ImageNet](http://www.image-net.org/challenges/LSVRC/2012/)|1.2M| 50k | 1000 | Various sizes ## Downloading and converting to TFRecord format For each dataset, we'll need to download the raw data and convert it to TensorFlow's native [TFRecord](https://www.tensorflow.org/versions/r0.10/api_docs/python/python_io.html#tfrecords-format-details) format. Each TFRecord contains a [TF-Example](https://github.com/tensorflow/tensorflow/blob/r0.10/tensorflow/core/example/example.proto) protocol buffer. Below we demonstrate how to do this for the Flowers dataset. ```shell $ DATA_DIR=/tmp/data/flowers $ python download_and_convert_data.py \ --dataset_name=flowers \ --dataset_dir="${DATA_DIR}" ``` When the script finishes you will find several TFRecord files created: ```shell $ ls ${DATA_DIR} flowers_train-00000-of-00005.tfrecord ... flowers_train-00004-of-00005.tfrecord flowers_validation-00000-of-00005.tfrecord ... flowers_validation-00004-of-00005.tfrecord labels.txt ``` These represent the training and validation data, sharded over 5 files each. You will also find the `$DATA_DIR/labels.txt` file which contains the mapping from integer labels to class names. You can use the same script to create the mnist and cifar10 datasets. However, for ImageNet, you have to follow the instructions [here](https://github.com/tensorflow/models/blob/master/inception/README.md#getting-started). Note that you first have to sign up for an account at image-net.org. Also, the download can take several hours, and could use up to 500GB. ## Creating a TF-Slim Dataset Descriptor. Once the TFRecord files have been created, you can easily define a Slim [Dataset](https://github.com/tensorflow/tensorflow/blob/r0.10/tensorflow/contrib/slim/python/slim/data/dataset.py), which stores pointers to the data file, as well as various other pieces of metadata, such as the class labels, the train/test split, and how to parse the TFExample protos. We have included the TF-Slim Dataset descriptors for [Cifar10](https://github.com/tensorflow/models/blob/master/slim/datasets/cifar10.py), [ImageNet](https://github.com/tensorflow/models/blob/master/slim/datasets/imagenet.py), [Flowers](https://github.com/tensorflow/models/blob/master/slim/datasets/flowers.py), and [MNIST](https://github.com/tensorflow/models/blob/master/slim/datasets/mnist.py). An example of how to load data using a TF-Slim dataset descriptor using a TF-Slim [DatasetDataProvider](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/slim/python/slim/data/dataset_data_provider.py) is found below: ```python import tensorflow as tf from datasets import flowers slim = tf.contrib.slim # Selects the 'validation' dataset. dataset = flowers.get_split('validation', DATA_DIR) # Creates a TF-Slim DataProvider which reads the dataset in the background # during both training and testing. provider = slim.dataset_data_provider.DatasetDataProvider(dataset) [image, label] = provider.get(['image', 'label']) ``` # Pre-trained Models Neural nets work best when they have many parameters, making them powerful function approximators. However, this means they must be trained on very large datasets. Because training models from scratch can be a very computationally intensive process requiring days or even weeks, we provide various pre-trained models, as listed below. These CNNs have been trained on the [ILSVRC-2012-CLS](http://www.image-net.org/challenges/LSVRC/2012/) image classification dataset. In the table below, we list each model, the corresponding TensorFlow model file, the link to the model checkpoint, and the top 1 and top 5 accuracy (on the imagenet test set). Note that the VGG and ResNet parameters have been converted from their original caffe formats ([here](https://github.com/BVLC/caffe/wiki/Model-Zoo#models-used-by-the-vgg-team-in-ilsvrc-2014) and [here](https://github.com/KaimingHe/deep-residual-networks)), whereas the Inception parameters have been trained internally at Google. Also be aware that these accuracies were computed by evaluating using a single image crop. Some academic papers report higher accuracy by using multiple crops at multiple scales. Model | TF-Slim File | Checkpoint | Top-1 Accuracy| Top-5 Accuracy | :----:|:------------:|:----------:|:-------:|:--------:| [Inception V1](http://arxiv.org/abs/1409.4842v1)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/inception_v1.py)|[inception_v1_2016_08_28.tar.gz](http://download.tensorflow.org/models/inception_v1_2016_08_28.tar.gz)|69.8|89.6| [Inception V2](http://arxiv.org/abs/1502.03167)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/inception_v2.py)|[inception_v2_2016_08_28.tar.gz](http://download.tensorflow.org/models/inception_v2_2016_08_28.tar.gz)|73.9|91.8| [Inception V3](http://arxiv.org/abs/1512.00567)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/inception_v3.py)|[inception_v3_2016_08_28.tar.gz](http://download.tensorflow.org/models/inception_v3_2016_08_28.tar.gz)|78.0|93.9| [Inception V4](http://arxiv.org/abs/1602.07261)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/inception_v4.py)|[inception_v4_2016_09_09.tar.gz](http://download.tensorflow.org/models/inception_v4_2016_09_09.tar.gz)|80.2|95.2| [Inception-ResNet-v2](http://arxiv.org/abs/1602.07261)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/inception_resnet_v2.py)|[inception_resnet_v2.tar.gz](http://download.tensorflow.org/models/inception_resnet_v2_2016_08_30.tar.gz)|80.4|95.3| [ResNet 50](https://arxiv.org/abs/1512.03385)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/resnet_v1.py)|[resnet_v1_50.tar.gz](http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz)|75.2|92.2| [ResNet 101](https://arxiv.org/abs/1512.03385)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/resnet_v1.py)|[resnet_v1_101.tar.gz](http://download.tensorflow.org/models/resnet_v1_101_2016_08_28.tar.gz)|76.4|92.9| [ResNet 152](https://arxiv.org/abs/1512.03385)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/resnet_v1.py)|[resnet_v1_152.tar.gz](http://download.tensorflow.org/models/resnet_v1_152_2016_08_28.tar.gz)|76.8|93.2| [VGG 16](http://arxiv.org/abs/1409.1556.pdf)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/vgg.py)|[vgg_16.tar.gz](http://download.tensorflow.org/models/vgg_16_2016_08_28.tar.gz)|71.5|89.8| [VGG 19](http://arxiv.org/abs/1409.1556.pdf)|[Code](https://github.com/tensorflow/models/blob/master/slim/nets/vgg.py)|[vgg_19.tar.gz](http://download.tensorflow.org/models/vgg_19_2016_08_28.tar.gz)|71.1|89.8| Here is an example of how to download the Inception V3 checkpoint: ```shell $ CHECKPOINT_DIR=/tmp/checkpoints $ mkdir ${CHECKPOINT_DIR} $ wget http://download.tensorflow.org/models/inception_v3_2016_08_28.tar.gz $ tar -xvf inception_v3_2016_08_28.tar.gz $ mv inception_v3.ckpt ${CHECKPOINT_DIR} $ rm inception_v3_2016_08_28.tar.gz ``` # Training a model from scratch. We provide an easy way to train a model from scratch using any TF-Slim dataset. The following example demonstrates how to train Inception V3 using the default parameters on the ImageNet dataset. ```shell DATASET_DIR=/tmp/imagenet TRAIN_DIR=/tmp/train_logs python train_image_classifier.py \ --train_dir=${TRAIN_DIR} \ --dataset_name=imagenet \ --dataset_split_name=train \ --dataset_dir=${DATASET_DIR} \ --model_name=inception_v3 ``` This process may take several days, depending on your hardware setup. For convenience, we provide a way to train a model on multiple GPUs, and/or multiple CPUs, either synchrononously or asynchronously. See [model_deploy](https://github.com/tensorflow/models/blob/master/slim/deployment/model_deploy.py) for details. # Fine-tuning a model from an existing checkpoint Rather than training from scratch, we'll often want to start from a pre-trained model and fine-tune it. To indicate a checkpoint from which to fine-tune, we'll call training with the `--checkpoint_path` flag and assign it an absolute path to a checkpoint file. When fine-tuning a model, we need to be careful about restoring checkpoint weights. In particular, when we fine-tune a model on a new task with a different number of output labels, we wont be able restore the final logits (classifier) layer. For this, we'll use the `--checkpoint_exclude_scopes` flag. This flag hinders certain variables from being loaded. When fine-tuning on a classification task using a different number of classes than the trained model, the new model will have a final 'logits' layer whose dimensions differ from the pre-trained model. For example, if fine-tuning an ImageNet-trained model on Flowers, the pre-trained logits layer will have dimensions `[2048 x 1001]` but our new logits layer will have dimensions `[2048 x 5]`. Consequently, this flag indicates to TF-Slim to avoid loading these weights from the checkpoint. Keep in mind that warm-starting from a checkpoint affects the model's weights only during the initialization of the model. Once a model has started training, a new checkpoint will be created in `${TRAIN_DIR}`. If the fine-tuning training is stopped and restarted, this new checkpoint will be the one from which weights are restored and not the `${checkpoint_path}$`. Consequently, the flags `--checkpoint_path` and `--checkpoint_exclude_scopes` are only used during the `0-`th global step (model initialization). Typically for fine-tuning one only want train a sub-set of layers, so the flag `--trainable_scopes` allows to specify which subsets of layers should trained, the rest would remain frozen. Below we give an example of [fine-tuning inception-v3 on flowers](https://github.com/tensorflow/models/blob/master/slim/scripts/finetune_inception_v3_on_flowers.sh), inception_v3 was trained on ImageNet with 1000 class labels, but the flowers dataset only have 5 classes. Since the dataset is quite small we will only train the new layers. ```shell $ DATASET_DIR=/tmp/flowers $ TRAIN_DIR=/tmp/flowers-models/inception_v3 $ CHECKPOINT_PATH=/tmp/my_checkpoints/inception_v3.ckpt $ python train_image_classifier.py \ --train_dir=${TRAIN_DIR} \ --dataset_dir=${DATASET_DIR} \ --dataset_name=flowers \ --dataset_split_name=train \ --model_name=inception_v3 \ --checkpoint_path=${CHECKPOINT_PATH} \ --checkpoint_exclude_scopes=InceptionV3/Logits,InceptionV3/AuxLogits \ --trainable_scopes=InceptionV3/Logits,InceptionV3/AuxLogits ``` # Evaluating performance of a model To evaluate the performance of a model (whether pretrained or your own), you can use the eval_image_classifier.py script, as shown below. Below we give an example of downloading the pretrained inception model and evaluating it on the imagenet dataset. ```shell CHECKPOINT_FILE = ${CHECKPOINT_DIR}/inception_v3.ckpt # Example $ python eval_image_classifier.py \ --alsologtostderr \ --checkpoint_path=${CHECKPOINT_FILE} \ --dataset_dir=${DATASET_DIR} \ --dataset_name=imagenet \ --dataset_split_name=validation \ --model_name=inception_v3 ``` # Troubleshooting #### The model runs out of CPU memory. See [Model Runs out of CPU memory](https://github.com/tensorflow/models/tree/master/inception#the-model-runs-out-of-cpu-memory). #### The model runs out of GPU memory. See [Adjusting Memory Demands](https://github.com/tensorflow/models/tree/master/inception#adjusting-memory-demands). #### The model training results in NaN's. See [Model Resulting in NaNs](https://github.com/tensorflow/models/tree/master/inception#the-model-training-results-in-nans). #### The ResNet and VGG Models have 1000 classes but the ImageNet dataset has 1001 The ImageNet dataset provided has an empty background class which can be used to fine-tune the model to other tasks. If you try training or fine-tuning the VGG or ResNet models using the ImageNet dataset, you might encounter the following error: ```bash InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000] ``` This is due to the fact that the VGG and ResNet final layers have only 1000 outputs rather than 1001. To fix this issue, you can set the `--labels_offsets=1` flag. This results in the ImageNet labels being shifted down by one: #### I wish to train a model with a different image size. The preprocessing functions all take `height` and `width` as parameters. You can change the default values using the following snippet: ```python image_preprocessing_fn = preprocessing_factory.get_preprocessing( preprocessing_name, height=MY_NEW_HEIGHT, width=MY_NEW_WIDTH, is_training=True) ``` #### What hardware specification are these hyper-parameters targeted for? See [Hardware Specifications](https://github.com/tensorflow/models/tree/master/inception#what-hardware-specification-are-these-hyper-parameters-targeted-for).