{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Tce3stUlHN0L"
   },
   "source": [
    "##### Copyright 2018 The TensorFlow Authors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:34.465574Z",
     "iopub.status.busy": "2022-12-14T22:56:34.465060Z",
     "iopub.status.idle": "2022-12-14T22:56:34.468499Z",
     "shell.execute_reply": "2022-12-14T22:56:34.467986Z"
    },
    "id": "tuOe1ymfHZPu"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MfBg1C5NB3X0"
   },
   "source": [
    "# GPU を使用する\n",
    "\n",
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td><a target=\"_blank\" href=\"https://www.tensorflow.org/guide/gpu\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\">TensorFlow.org で表示</a></td>\n",
    "  <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ja/guide/gpu.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Google Colab で実行</a></td>\n",
    "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/ja/guide/gpu.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">GitHub でソースを表示</a></td>\n",
    "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/ja/guide/gpu.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\">ノートブックをダウンロード</a></td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SoYIwe40vEPI"
   },
   "source": [
    "TensorFlow のコードと`tf.keras`モデルは、コードを変更することなく単一の GPU で透過的に実行されます。\n",
    "\n",
    "注意: `tf.config.list_physical_devices('GPU')`を使用して、TensorFlow が GPU を使用していることを確認してください。\n",
    "\n",
    "単一または複数のマシンで複数の GPU を実行する最も簡単な方法は、[分散ストラテジー](distributed_training.ipynb)の使用です。\n",
    "\n",
    "このガイドは、これらのアプローチを試し、TensorFlow の GPU 使用方法を細かく制御する必要があることに気づいたユーザーを対象としています。シングル GPU とマルチ GPU のシナリオにおいてパフォーマンスをデバッグする方法については、「[TensorFlow GPU パフォーマンスを最適化する](gpu_performance_analysis.md)」ガイドをご覧ください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MUXex9ctTuDB"
   },
   "source": [
    "## セットアップ\n",
    "\n",
    "最新リリースの TensorFlow GPU がインストールされていることを確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:34.471785Z",
     "iopub.status.busy": "2022-12-14T22:56:34.471259Z",
     "iopub.status.idle": "2022-12-14T22:56:36.511020Z",
     "shell.execute_reply": "2022-12-14T22:56:36.510292Z"
    },
    "id": "IqR2PQG4ZaZ0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-12-14 22:56:35.420719: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
      "2022-12-14 22:56:35.420819: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
      "2022-12-14 22:56:35.420829: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num GPUs Available:  4\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(\"Num GPUs Available: \", len(tf.config.list_physical_devices('GPU')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZELutYNetv-v"
   },
   "source": [
    "## 概要\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xHxb-dlhMIzW"
   },
   "source": [
    "TensorFlow は、CPU や GPU など、さまざまなタイプのデバイスでの計算の実行をサポートしています。それらは、例えば文字列の識別子で表されています。\n",
    "\n",
    "- `\"/device:CPU:0\"`: マシンの CPU。\n",
    "- `\"/GPU:0\"`: TensorFlow が認識するマシンの最初の GPU の簡略表記。\n",
    "- `\"/job:localhost/replica:0/task:0/device:GPU:1\"`: TensorFlow が認識するマシンの 2 番目の GPU の完全修飾名。\n",
    "\n",
    "TensorFlow 演算に CPU と GPU 実装の両方を持つ場合、演算がデバイスに割り当てられるときに、デフォルトでは GPU デバイスが優先されます。たとえば、`tf.matmul` は CPU と GPU カーネルの両方を持ちます。デバイス `CPU:0` と `GPU:0` を持つシステム上では、それを他のデバイス上で実行することを明示的に要求しない限りは、`GPU:0` デバイスが `tf.matmul` の実行に選択されます。\n",
    "\n",
    "TensorFlow 演算に対応する GPU 実装がない場合は、演算は CPU デバイスにフォールバックします。たとえば、`tf.cast` には CPU カーネルしかないため、デバイス `CPU:0` と `GPU:0` のあるシステムでは、`CPU:0` デバイスが `tf.cast` の実行に選択され、`GPU:0` デバイスでの実行をリクエストされても無視されます。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UhNtHfuxCGVy"
   },
   "source": [
    "## デバイスの配置をログに記録する\n",
    "\n",
    "演算と tensor がどのデバイスに割り当てられたかを確認するには、`tf.debugging.set_log_device_placement(True)`をプログラムの最初のステートメントとして置きます。デバイス配置ログを有効にすると、テンソルの割り当てや演算が出力されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:36.515230Z",
     "iopub.status.busy": "2022-12-14T22:56:36.514806Z",
     "iopub.status.idle": "2022-12-14T22:56:40.078734Z",
     "shell.execute_reply": "2022-12-14T22:56:40.078008Z"
    },
    "id": "2Dbw0tpEirCd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[22. 28.]\n",
      " [49. 64.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "# Create some tensors\n",
    "a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "c = tf.matmul(a, b)\n",
    "\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kKhmFeraTdEI"
   },
   "source": [
    "上記のコードの出力は、`GPU:0`で`MatMul`演算が実行されたことを示します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U88FspwGjB7W"
   },
   "source": [
    "## 手動でデバイスを配置する\n",
    "\n",
    "自動的に選択されたものの代わりに、自分が選択したデバイスで特定の演算を実行する場合は、デバイスコンテキストの作成に`with tf.device`を使用すると、そのコンテキスト内のすべての演算は同じ指定されたデバイス上で実行されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:40.082148Z",
     "iopub.status.busy": "2022-12-14T22:56:40.081841Z",
     "iopub.status.idle": "2022-12-14T22:56:40.088143Z",
     "shell.execute_reply": "2022-12-14T22:56:40.087533Z"
    },
    "id": "8wqaQfEhjHit"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[22. 28.]\n",
      " [49. 64.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "# Place tensors on the CPU\n",
    "with tf.device('/CPU:0'):\n",
    "  a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "  b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "\n",
    "c = tf.matmul(a, b)\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8ixO89gRjJUu"
   },
   "source": [
    "これで`a`と`b`は`CPU:0`に割り当てられたことが分かります。`MatMul`演算に対してデバイスが明示的に指定されていないため、TensorFlow ランタイムは演算と利用可能なデバイスに基づいてデバイスを選択し（この例では`GPU:0`）、必要に応じて自動的にデバイス間でテンソルをコピーします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ARrRhwqijPzN"
   },
   "source": [
    "## GPU のメモリ増加を制限する\n",
    "\n",
    "デフォルトでは、TensorFlow は（[`CUDA_VISIBLE_DEVICES`](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars)に従い）プロセスが認識する全 GPU の ほぼ全てのGPU メモリをマップします。これはメモリの断片化を減らしてデバイス上のかなり貴重な GPU メモリリソースをより効率的に使用するために行われます。TensorFlow を特定の GPU セットに制限するには、`tf.config.set_visible_devices`メソッドを使用します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:40.091445Z",
     "iopub.status.busy": "2022-12-14T22:56:40.090952Z",
     "iopub.status.idle": "2022-12-14T22:56:40.095526Z",
     "shell.execute_reply": "2022-12-14T22:56:40.094780Z"
    },
    "id": "hPI--n_jhZhv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Visible devices cannot be modified after being initialized\n"
     ]
    }
   ],
   "source": [
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "  # Restrict TensorFlow to only use the first GPU\n",
    "  try:\n",
    "    tf.config.set_visible_devices(gpus[0], 'GPU')\n",
    "    logical_gpus = tf.config.list_logical_devices('GPU')\n",
    "    print(len(gpus), \"Physical GPUs,\", len(logical_gpus), \"Logical GPU\")\n",
    "  except RuntimeError as e:\n",
    "    # Visible devices must be set before GPUs have been initialized\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N3x4M55DhYk9"
   },
   "source": [
    "場合によりますが、プロセスが使用可能なメモリのサブセットのみを割り当てるか、またはプロセスに必要とされるメモリ使用量のみを増やすことが望ましいです。TensorFlow は、これを制御する 2 つのメソッドを提供します。\n",
    "\n",
    "最初のオプションは、`tf.config.experimental.set_memory_growth` を呼び出してメモリ増大を有効にすることです。これはランタイムの割り当てに必要な GPU メモリだけを割り当てようと試みます。非常に小さいメモリの割り当てから始め、プログラムが実行されてより多くの GPU メモリが必要になってくるにつれて、TensorFlow プロセスに割り当てられる GPU メモリ領域を拡張します。メモリの断面化につながる可能性があるため、メモリは解放されません。特定の GPU のメモリ増大を有効にするためには、テンソルの割り当てや演算の実行の前に次のコードを使用してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:40.098562Z",
     "iopub.status.busy": "2022-12-14T22:56:40.098146Z",
     "iopub.status.idle": "2022-12-14T22:56:40.102839Z",
     "shell.execute_reply": "2022-12-14T22:56:40.102182Z"
    },
    "id": "jr3Kf1boFnCO"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Physical devices cannot be modified after being initialized\n"
     ]
    }
   ],
   "source": [
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "  try:\n",
    "    # Currently, memory growth needs to be the same across GPUs\n",
    "    for gpu in gpus:\n",
    "      tf.config.experimental.set_memory_growth(gpu, True)\n",
    "    logical_gpus = tf.config.list_logical_devices('GPU')\n",
    "    print(len(gpus), \"Physical GPUs,\", len(logical_gpus), \"Logical GPUs\")\n",
    "  except RuntimeError as e:\n",
    "    # Memory growth must be set before GPUs have been initialized\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I1o8t51QFnmv"
   },
   "source": [
    "このオプションを有効にするもう 1 つの方法は、環境変数`TF_FORCE_GPU_ALLOW_GROWTH`を`true`に設定するというものです。この構成はプラットフォーム固有です。\n",
    "\n",
    "2 番目のメソッドは、`tf.config.set_logical_device_configurartion`で仮想 GPU デバイスを構成し、GPU に割り当てられるメモリ総量を固定することです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:40.106011Z",
     "iopub.status.busy": "2022-12-14T22:56:40.105454Z",
     "iopub.status.idle": "2022-12-14T22:56:40.110102Z",
     "shell.execute_reply": "2022-12-14T22:56:40.109551Z"
    },
    "id": "2qO2cS9QFn42"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Virtual devices cannot be modified after being initialized\n"
     ]
    }
   ],
   "source": [
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "  # Restrict TensorFlow to only allocate 1GB of memory on the first GPU\n",
    "  try:\n",
    "    tf.config.set_logical_device_configuration(\n",
    "        gpus[0],\n",
    "        [tf.config.LogicalDeviceConfiguration(memory_limit=1024)])\n",
    "    logical_gpus = tf.config.list_logical_devices('GPU')\n",
    "    print(len(gpus), \"Physical GPUs,\", len(logical_gpus), \"Logical GPUs\")\n",
    "  except RuntimeError as e:\n",
    "    # Virtual devices must be set before GPUs have been initialized\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bsg1iLuHFoLW"
   },
   "source": [
    "これは TensorFlow プロセスに利用可能な GPU メモリの量を正確に抑制する場合に有用です。これは GPU がワークステーション GUI などの他のアプリケーションと共有されている場合のローカル開発では一般的な方法です。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B27_-1gyjf-t"
   },
   "source": [
    "## マルチ GPU システムで単一 GPU を使用する\n",
    "\n",
    "システムに 2 つ以上の GPU が搭載されている場合、デフォルトでは最小の ID を持つ GPU が選択されます。異なる GPU 上で実行する場合には、その選択を明示的に指定する必要があります。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:40.113619Z",
     "iopub.status.busy": "2022-12-14T22:56:40.112964Z",
     "iopub.status.idle": "2022-12-14T22:56:40.698544Z",
     "shell.execute_reply": "2022-12-14T22:56:40.697792Z"
    },
    "id": "wep4iteljjG1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "try:\n",
    "  # Specify an invalid GPU device\n",
    "  with tf.device('/device:GPU:2'):\n",
    "    a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "    b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "    c = tf.matmul(a, b)\n",
    "except RuntimeError as e:\n",
    "  print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jy-4cCO_jn4G"
   },
   "source": [
    "指定したデバイスが存在しない場合には、`RuntimeError`: `.../device:GPU:2 unknown device`が表示されます。\n",
    "\n",
    "指定されたデバイスが存在しない場合に TensorFlow に既に演算の実行をサポートしているデバイスを自動的に選択させたければ、`tf.config.set_soft_device_placement(True)`を呼び出すことができます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:40.702721Z",
     "iopub.status.busy": "2022-12-14T22:56:40.702074Z",
     "iopub.status.idle": "2022-12-14T22:56:40.710261Z",
     "shell.execute_reply": "2022-12-14T22:56:40.709589Z"
    },
    "id": "sut_UHlkjvWd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[22. 28.]\n",
      " [49. 64.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.config.set_soft_device_placement(True)\n",
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "# Creates some tensors\n",
    "a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "c = tf.matmul(a, b)\n",
    "\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sYTYPrQZj2d9"
   },
   "source": [
    "## マルチ GPU を使用する\n",
    "\n",
    "マルチ GPU 向けの開発は、追加リソースでのモデルのスケーリングを可能にします。単一 GPU システム上で開発している場合は、仮想デバイスで複数の GPU をシミュレートできます。これにより、追加のリソースがなくてもマルチ GPU セットアップの簡単なテストが可能になります。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:40.713458Z",
     "iopub.status.busy": "2022-12-14T22:56:40.712880Z",
     "iopub.status.idle": "2022-12-14T22:56:40.717447Z",
     "shell.execute_reply": "2022-12-14T22:56:40.716881Z"
    },
    "id": "8EMGuGKbNkc6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Virtual devices cannot be modified after being initialized\n"
     ]
    }
   ],
   "source": [
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "  # Create 2 virtual GPUs with 1GB memory each\n",
    "  try:\n",
    "    tf.config.set_logical_device_configuration(\n",
    "        gpus[0],\n",
    "        [tf.config.LogicalDeviceConfiguration(memory_limit=1024),\n",
    "         tf.config.LogicalDeviceConfiguration(memory_limit=1024)])\n",
    "    logical_gpus = tf.config.list_logical_devices('GPU')\n",
    "    print(len(gpus), \"Physical GPU,\", len(logical_gpus), \"Logical GPUs\")\n",
    "  except RuntimeError as e:\n",
    "    # Virtual devices must be set before GPUs have been initialized\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xmNzO0FxNkol"
   },
   "source": [
    "ランタイムで利用可能な複数の論理 GPU を取得したら、`tf.distribute.Strategy` または手動配置でマルチ GPU を利用できます。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IDZmEGq4j6kG"
   },
   "source": [
    "#### `tf.distribute.Strategy`を使用する\n",
    "\n",
    "マルチ GPU を使用するベストプラクティスは、`tf.distribute.Strategy`を使用することです。次に単純な例を示します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:40.720786Z",
     "iopub.status.busy": "2022-12-14T22:56:40.720218Z",
     "iopub.status.idle": "2022-12-14T22:56:41.674913Z",
     "shell.execute_reply": "2022-12-14T22:56:41.674238Z"
    },
    "id": "1KgzY8V2AvRv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op StatelessRandomGetKeyCounter in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op StatelessRandomUniformV2 in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "seed: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "StatelessRandomGetKeyCounter: (StatelessRandomGetKeyCounter): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "key_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "counter_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "key: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "counter: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "alg: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "StatelessRandomUniformV2: (StatelessRandomUniformV2): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Sub: (Sub): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "z_RetVal: (_Retval): /job:localhost/replica:Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddV2 in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0/task:0/device:GPU:0\n",
      "x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AddV2: (AddV2): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "NoOp: (NoOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "dims: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Fill: (Fill): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/taskExecuting op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ":0/device:GPU:2\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "inputs_0: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "inputs_1: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "inputs_2: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "inputs_3: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "AddN: (AddN): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "sum_RetVExecuting op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "gpus = tf.config.list_logical_devices('GPU')\n",
    "strategy = tf.distribute.MirroredStrategy(gpus)\n",
    "with strategy.scope():\n",
    "  inputs = tf.keras.layers.Input(shape=(1,))\n",
    "  predictions = tf.keras.layers.Dense(1)(inputs)\n",
    "  model = tf.keras.models.Model(inputs=inputs, outputs=predictions)\n",
    "  model.compile(loss='mse',\n",
    "                optimizer=tf.keras.optimizers.SGD(learning_rate=0.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dy7nxlKsAxkK"
   },
   "source": [
    "このプログラムは各 GPU でモデルのコピーを実行し、それらの間で入力データを分割します。これは、「[データの並列処理](https://en.wikipedia.org/wiki/Data_parallelism)」としても知られています。\n",
    "\n",
    "分散ストラテジーの詳細については、[こちら](./distributed_training.ipynb)のガイドをご覧ください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8phxM5TVkAY_"
   },
   "source": [
    "#### 手動で配置する\n",
    "\n",
    "`tf.distribute.Strategy`は、内部的には複数のデバイスに渡って計算を複製することによって動作しています。各 GPU 上でモデルを構築することにより、複製を手動で実装することができます。例を示します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T22:56:41.678822Z",
     "iopub.status.busy": "2022-12-14T22:56:41.678159Z",
     "iopub.status.idle": "2022-12-14T22:56:42.139196Z",
     "shell.execute_reply": "2022-12-14T22:56:42.138542Z"
    },
    "id": "AqPo9ltUA_EY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 88. 112.]\n",
      " [196. 256.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "gpus = tf.config.list_logical_devices('GPU')\n",
    "if gpus:\n",
    "  # Replicate your computation on multiple GPUs\n",
    "  c = []\n",
    "  for gpu in gpus:\n",
    "    with tf.device(gpu.name):\n",
    "      a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "      b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "      c.append(tf.matmul(a, b))\n",
    "\n",
    "  with tf.device('/CPU:0'):\n",
    "    matmul_sum = tf.add_n(c)\n",
    "\n",
    "  print(matmul_sum)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "gpu.ipynb",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}