{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c8Cx-rUMVX25"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "I9sUhVL_VZNO"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Y8E0lw5eYWm"
      },
      "source": [
        "# int16 アクティベーションによるトレーニング後の整数量子化"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CGuqeuPSVNo-"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>     <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/performance/post_training_integer_quant_16x8\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\">TensorFlow.org で表示</a>\n",
        "</td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ja/lite/performance/post_training_integer_quant_16x8.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Google Colab で実行</a>\n",
        "</td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/ja/lite/performance/post_training_integer_quant_16x8.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">GitHub でソースを表示</a>\n",
        "</td>\n",
        "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/ja/lite/performance/post_training_integer_quant_16x8.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\"> ノートブックをダウンロード</a></td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTC1rDAuei_1"
      },
      "source": [
        "## 概要\n",
        "\n",
        "現在、[TensorFlow Lite](https://www.tensorflow.org/lite/) はモデルを TensorFlow から TensorFlow Lite のフラットバッファ形式に変換する際に、アクティベーションを 16 ビット整数値に、重みを 8 ビット整数値に変換することをサポートしています。このモードは「16x8 量子化モード」と呼ばれています。このモードでは、アクティベーションが量子化の影響を受けやすい場合に、量子化モデルの精度を大幅に向上させ、モデルサイズを約 1/3〜1/4 に縮小することができます。また、この完全に量子化されたモデルは、整数のみのハードウェアアクセラレータで使用できます。\n",
        "\n",
        "次のようなモデルでは、このモードのトレーニング後の量子化が有用です。\n",
        "\n",
        "- 超解像\n",
        "- ノイズキャンセリングやビームフォーミングなどのオーディオ信号処理\n",
        "- 画像ノイズ除去\n",
        "- 単一の画像からの HDR 再構成\n",
        "\n",
        "このチュートリアルでは、MNIST モデルを新規にトレーニングし、TensorFlow でその精度を確認してから、このモードを使用してモデルを Tensorflow Lite フラットバッファに変換します。最後に、変換されたモデルの精度を確認し、元の float32 モデルと比較します。この例は、このモードの使用法を示しており、TensorFlow Liteで利用可能な他の量子化手法と比較した場合の利点は示していません。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2XsEP17Zelz9"
      },
      "source": [
        "## MNIST モデルの構築"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDqqUIZjZjac"
      },
      "source": [
        "### セットアップ"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gyqAw1M9lyab"
      },
      "outputs": [],
      "source": [
        "import logging\n",
        "logging.getLogger(\"tensorflow\").setLevel(logging.DEBUG)\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "import numpy as np\n",
        "import pathlib"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "srTSFKjn1tMp"
      },
      "source": [
        "16x8 量子化モードが使用可能であることを確認します "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c6nb7OPlXs_3"
      },
      "outputs": [],
      "source": [
        "tf.lite.OpsSet.EXPERIMENTAL_TFLITE_BUILTINS_ACTIVATIONS_INT16_WEIGHTS_INT8"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eQ6Q0qqKZogR"
      },
      "source": [
        "### モデルをトレーニングしてエクスポートする"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hWSAjQWagIHl"
      },
      "outputs": [],
      "source": [
        "# Load MNIST dataset\n",
        "mnist = keras.datasets.mnist\n",
        "(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
        "\n",
        "# Normalize the input image so that each pixel value is between 0 to 1.\n",
        "train_images = train_images / 255.0\n",
        "test_images = test_images / 255.0\n",
        "\n",
        "# Define the model architecture\n",
        "model = keras.Sequential([\n",
        "  keras.layers.InputLayer(input_shape=(28, 28)),\n",
        "  keras.layers.Reshape(target_shape=(28, 28, 1)),\n",
        "  keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation=tf.nn.relu),\n",
        "  keras.layers.MaxPooling2D(pool_size=(2, 2)),\n",
        "  keras.layers.Flatten(),\n",
        "  keras.layers.Dense(10)\n",
        "])\n",
        "\n",
        "# Train the digit classification model\n",
        "model.compile(optimizer='adam',\n",
        "              loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "              metrics=['accuracy'])\n",
        "model.fit(\n",
        "  train_images,\n",
        "  train_labels,\n",
        "  epochs=1,\n",
        "  validation_data=(test_images, test_labels)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5NMaNZQCkW9X"
      },
      "source": [
        "この例では、モデルを 1 エポックでトレーニングしたので、トレーニングの精度は 96％ 以下になります。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xl8_fzVAZwOh"
      },
      "source": [
        "### TensorFlow Lite モデルに変換する\n",
        "\n",
        "TensorFlow Lite [Converter](https://www.tensorflow.org/lite/models/convert) を使用して、トレーニング済みモデルを TensorFlow Lite モデルに変換できるようになりました。\n",
        "\n",
        "次に、`TFliteConverter`を使用してモデルをデフォルトの float32 形式に変換します。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_i8B2nDZmAgQ"
      },
      "outputs": [],
      "source": [
        "converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
        "tflite_model = converter.convert()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F2o2ZfF0aiCx"
      },
      "source": [
        "`.tflite`ファイルに書き込みます。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vptWZq2xnclo"
      },
      "outputs": [],
      "source": [
        "tflite_models_dir = pathlib.Path(\"/tmp/mnist_tflite_models/\")\n",
        "tflite_models_dir.mkdir(exist_ok=True, parents=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ie9pQaQrn5ue"
      },
      "outputs": [],
      "source": [
        "tflite_model_file = tflite_models_dir/\"mnist_model.tflite\"\n",
        "tflite_model_file.write_bytes(tflite_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BONhYtYocQY"
      },
      "source": [
        "モデルを 16x8 量子化モードに量子化するには、最初に`optimizations`フラグを設定してデフォルトの最適化を使用します。次に、16x8 量子化モードがターゲット仕様でサポートされる必要な演算であることを指定します。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HEZ6ET1AHAS3"
      },
      "outputs": [],
      "source": [
        "converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
        "converter.target_spec.supported_ops = [tf.lite.OpsSet.EXPERIMENTAL_TFLITE_BUILTINS_ACTIVATIONS_INT16_WEIGHTS_INT8]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zLxQwZq9CpN7"
      },
      "source": [
        "int8 トレーニング後の量子化の場合と同様に、コンバーターオプション`inference_input(output)_type`を tf.int16 に設定することで、完全整数量子化モデルを生成できます。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yZekFJC5-fOG"
      },
      "source": [
        "キャリブレーションデータを設定します。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y3a6XFqvHbYM"
      },
      "outputs": [],
      "source": [
        "mnist_train, _ = tf.keras.datasets.mnist.load_data()\n",
        "images = tf.cast(mnist_train[0], tf.float32) / 255.0\n",
        "mnist_ds = tf.data.Dataset.from_tensor_slices((images)).batch(1)\n",
        "def representative_data_gen():\n",
        "  for input_value in mnist_ds.take(100):\n",
        "    # Model has only one input so each data point has one element.\n",
        "    yield [input_value]\n",
        "converter.representative_dataset = representative_data_gen"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xW84iMYjHd9t"
      },
      "source": [
        "最後に、通常どおりにモデルを変換します。デフォルトでは、変換されたモデルは呼び出しの便宜上、浮動小数点の入力と出力を引き続き使用します。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yuNfl3CoHNK3"
      },
      "outputs": [],
      "source": [
        "tflite_16x8_model = converter.convert()\n",
        "tflite_model_16x8_file = tflite_models_dir/\"mnist_model_quant_16x8.tflite\"\n",
        "tflite_model_16x8_file.write_bytes(tflite_16x8_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PhMmUTl4sbkz"
      },
      "source": [
        "生成されるファイルのサイズが約`1/3`であることに注目してください。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JExfcfLDscu4"
      },
      "outputs": [],
      "source": [
        "!ls -lh {tflite_models_dir}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L8lQHMp_asCq"
      },
      "source": [
        "## TensorFlow Lite モデルを実行する"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-5l6-ciItvX6"
      },
      "source": [
        "Python TensorFlow Lite インタープリタを使用して TensorFlow Lite モデルを実行します。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ap_jE7QRvhPf"
      },
      "source": [
        "### モデルをインタープリタに読み込む"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jn16Rc23zTss"
      },
      "outputs": [],
      "source": [
        "interpreter = tf.lite.Interpreter(model_path=str(tflite_model_file))\n",
        "interpreter.allocate_tensors()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J8Pztk1mvNVL"
      },
      "outputs": [],
      "source": [
        "interpreter_16x8 = tf.lite.Interpreter(model_path=str(tflite_model_16x8_file))\n",
        "interpreter_16x8.allocate_tensors()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2opUt_JTdyEu"
      },
      "source": [
        "### 1 つの画像でモデルをテストする"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AKslvo2kwWac"
      },
      "outputs": [],
      "source": [
        "test_image = np.expand_dims(test_images[0], axis=0).astype(np.float32)\n",
        "\n",
        "input_index = interpreter.get_input_details()[0][\"index\"]\n",
        "output_index = interpreter.get_output_details()[0][\"index\"]\n",
        "\n",
        "interpreter.set_tensor(input_index, test_image)\n",
        "interpreter.invoke()\n",
        "predictions = interpreter.get_tensor(output_index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XZClM2vo3_bm"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pylab as plt\n",
        "\n",
        "plt.imshow(test_images[0])\n",
        "template = \"True:{true}, predicted:{predict}\"\n",
        "_ = plt.title(template.format(true= str(test_labels[0]),\n",
        "                              predict=str(np.argmax(predictions[0]))))\n",
        "plt.grid(False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3gwhv4lKbYZ4"
      },
      "outputs": [],
      "source": [
        "test_image = np.expand_dims(test_images[0], axis=0).astype(np.float32)\n",
        "\n",
        "input_index = interpreter_16x8.get_input_details()[0][\"index\"]\n",
        "output_index = interpreter_16x8.get_output_details()[0][\"index\"]\n",
        "\n",
        "interpreter_16x8.set_tensor(input_index, test_image)\n",
        "interpreter_16x8.invoke()\n",
        "predictions = interpreter_16x8.get_tensor(output_index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CIH7G_MwbY2x"
      },
      "outputs": [],
      "source": [
        "plt.imshow(test_images[0])\n",
        "template = \"True:{true}, predicted:{predict}\"\n",
        "_ = plt.title(template.format(true= str(test_labels[0]),\n",
        "                              predict=str(np.argmax(predictions[0]))))\n",
        "plt.grid(False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LwN7uIdCd8Gw"
      },
      "source": [
        "### モデルを評価する"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "05aeAuWjvjPx"
      },
      "outputs": [],
      "source": [
        "# A helper function to evaluate the TF Lite model using \"test\" dataset.\n",
        "def evaluate_model(interpreter):\n",
        "  input_index = interpreter.get_input_details()[0][\"index\"]\n",
        "  output_index = interpreter.get_output_details()[0][\"index\"]\n",
        "\n",
        "  # Run predictions on every image in the \"test\" dataset.\n",
        "  prediction_digits = []\n",
        "  for test_image in test_images:\n",
        "    # Pre-processing: add batch dimension and convert to float32 to match with\n",
        "    # the model's input data format.\n",
        "    test_image = np.expand_dims(test_image, axis=0).astype(np.float32)\n",
        "    interpreter.set_tensor(input_index, test_image)\n",
        "\n",
        "    # Run inference.\n",
        "    interpreter.invoke()\n",
        "\n",
        "    # Post-processing: remove batch dimension and find the digit with highest\n",
        "    # probability.\n",
        "    output = interpreter.tensor(output_index)\n",
        "    digit = np.argmax(output()[0])\n",
        "    prediction_digits.append(digit)\n",
        "\n",
        "  # Compare prediction results with ground truth labels to calculate accuracy.\n",
        "  accurate_count = 0\n",
        "  for index in range(len(prediction_digits)):\n",
        "    if prediction_digits[index] == test_labels[index]:\n",
        "      accurate_count += 1\n",
        "  accuracy = accurate_count * 1.0 / len(prediction_digits)\n",
        "\n",
        "  return accuracy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T5mWkSbMcU5z"
      },
      "outputs": [],
      "source": [
        "print(evaluate_model(interpreter))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Km3cY9ry8ZlG"
      },
      "source": [
        "16x8 量子化モデルで評価を繰り返します。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-9cnwiPp6EGm"
      },
      "outputs": [],
      "source": [
        "# NOTE: This quantization mode is an experimental post-training mode,\n",
        "# it does not have any optimized kernels implementations or\n",
        "# specialized machine learning hardware accelerators. Therefore,\n",
        "# it could be slower than the float interpreter.\n",
        "print(evaluate_model(interpreter_16x8))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L7lfxkor8pgv"
      },
      "source": [
        "この例では、モデルの精度を低下することなく、16x8 に量子化しました。サイズは 1/3 に縮小されました。\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "post_training_integer_quant_16x8.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}