{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Tce3stUlHN0L"
   },
   "source": [
    "##### Copyright 2018 The TensorFlow Authors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:26.536722Z",
     "iopub.status.busy": "2022-12-14T21:22:26.536187Z",
     "iopub.status.idle": "2022-12-14T21:22:26.540139Z",
     "shell.execute_reply": "2022-12-14T21:22:26.539549Z"
    },
    "id": "tuOe1ymfHZPu"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MfBg1C5NB3X0"
   },
   "source": [
    "# GPU 사용하기\n",
    "\n",
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>     <a target=\"_blank\" href=\"https://www.tensorflow.org/guide/gpu\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\">TensorFlow.org에서 보기</a>   </td>\n",
    "  <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ko/guide/gpu.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Google Colab에서 실행</a></td>\n",
    "  <td>GitHub에서 소스 보기</td>\n",
    "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/ko/guide/gpu.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\">노트북 다운로드</a></td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SoYIwe40vEPI"
   },
   "source": [
    "TensorFlow 코드 및 `tf.keras` 모델은 코드를 변경할 필요 없이 단일 GPU에서 투명하게 실행됩니다.\n",
    "\n",
    "참고: `tf.config.list_physical_devices('GPU')`를 사용하여 TensorFlow가 GPU를 사용하고 있는지 확인하세요.\n",
    "\n",
    "하나 또는 여러 시스템의 여러 GPU에서 실행하는 가장 간단한 방법은 [배포 전략](distributed_training.ipynb)을 이용하는 것입니다.\n",
    "\n",
    "이 가이드는 이러한 접근 방식을 시도해 보고 TensorFlow가 GPU를 사용하는 방식을 세밀한 제어해야 할 필요성을 느낀 사용자를 대상으로 합니다. 단일 및 다중 GPU 시나리오에서 성능 문제를 디버깅하는 방법을 알아보려면 [TensorFlow GPU 성능 최적화](gpu_performance_analysis.md) 가이드를 참조하세요."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MUXex9ctTuDB"
   },
   "source": [
    "## 설정\n",
    "\n",
    "최신 버전의 텐서플로가 설치되어있는지 확인하세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:26.543797Z",
     "iopub.status.busy": "2022-12-14T21:22:26.543255Z",
     "iopub.status.idle": "2022-12-14T21:22:28.469302Z",
     "shell.execute_reply": "2022-12-14T21:22:28.468621Z"
    },
    "id": "IqR2PQG4ZaZ0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-12-14 21:22:27.507083: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
      "2022-12-14 21:22:27.507177: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
      "2022-12-14 21:22:27.507186: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZELutYNetv-v"
   },
   "source": [
    "## 개요\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xHxb-dlhMIzW"
   },
   "source": [
    "Note: 이 문서는 텐서플로 커뮤니티에서 번역했습니다. 커뮤니티 번역 활동의 특성상 정확한 번역과 최신 내용을 반영하기 위해 노력함에도 불구하고 [공식 영문 문서](https://www.tensorflow.org/?hl=en)의 내용과 일치하지 않을 수 있습니다. 이 번역에 개선할 부분이 있다면 [tensorflow/docs-l10n](https://github.com/tensorflow/docs-l10n/) 깃헙 저장소로 풀 리퀘스트를 보내주시기 바랍니다. 문서 번역이나 리뷰에 참여하려면 [docs-ko@tensorflow.org](https://groups.google.com/a/tensorflow.org/forum/#!forum/docs-ko)로 메일을 보내주시기 바랍니다.\n",
    "\n",
    "- `\"/device:CPU:0\"`: 컴퓨터의 CPU입니다.\n",
    "- `\"/GPU:0\"`: TensorFlow에 인식되는 시스템의 첫 번째 GPU에 대한 약식 표기입니다.\n",
    "- `\"/job:localhost/replica:0/task:0/device:GPU:1\"`: TensorFlow에 인식되는 시스템의 두 번째 GPU에 대한 정규화된 이름입니다.\n",
    "\n",
    "TensorFlow 연산에 CPU와 GPU 구현이 모두 있는 경우, 기본적으로 연산이 할당될 때 GPU 장치에 우선 순위가 지정됩니다. 예를 들어, `tf.matmul`에는 CPU 및 GPU 커널이 모두 있으며 `CPU:0` 및 `GPU:0` 장치가 있는 시스템에서는 다른 장치에서 실행하도록 명시적으로 요청하지 않는 한 `GPU:0` 장치가 `tf.matmul`을 실행하도록 선택됩니다.\n",
    "\n",
    "TensorFlow 작업에 해당 GPU 구현이 없는 경우 연산은 CPU 장치로 대체됩니다. 예를 들어 `tf.cast`에는 CPU 커널만 있기 때문에 `CPU:0` 및 `GPU:0` 장치가 있는 시스템에서는 `GPU:0` 장치에서 실행하도록 요청된 경우라도 `CPU:0` 장치가 `tf.cast`를 실행하도록 선택됩니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UhNtHfuxCGVy"
   },
   "source": [
    "## 장치 할당 로깅\n",
    "\n",
    "연산과 텐서가 어떤 장치에 할당되었는지 확인하려면 `tf.debugging.set_log_device_placement(True)`를 프로그램의 가장 처음에 선언하세요. 장치 할당 로깅을 활성화하면 모든 텐서나 연산 할당이 출력됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:28.473803Z",
     "iopub.status.busy": "2022-12-14T21:22:28.473367Z",
     "iopub.status.idle": "2022-12-14T21:22:32.171650Z",
     "shell.execute_reply": "2022-12-14T21:22:32.170621Z"
    },
    "id": "2Dbw0tpEirCd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[22. 28.]\n",
      " [49. 64.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "# 텐서 생성\n",
    "a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "c = tf.matmul(a, b)\n",
    "\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kKhmFeraTdEI"
   },
   "source": [
    "위 코드는 `MatMul` 연산이 `GPU:0`에서 수행되었다고 보여줄 것입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U88FspwGjB7W"
   },
   "source": [
    "## 장치 수동 할당\n",
    "\n",
    "특정 연산을 수행할 장치를 직접 선택하고 싶다면, `with tf.device`로 장치 컨텍스트를 생성할 수 있고 해당 컨텍스트에서의 모든 연산은 지정된 장치에서 수행됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:32.175777Z",
     "iopub.status.busy": "2022-12-14T21:22:32.175376Z",
     "iopub.status.idle": "2022-12-14T21:22:32.182671Z",
     "shell.execute_reply": "2022-12-14T21:22:32.181856Z"
    },
    "id": "8wqaQfEhjHit"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[22. 28.]\n",
      " [49. 64.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "# 텐서를 CPU에 할당\n",
    "with tf.device('/CPU:0'):\n",
    "  a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "  b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "\n",
    "c = tf.matmul(a, b)\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8ixO89gRjJUu"
   },
   "source": [
    "`a`와 `b`가 `CPU:0`에 할당되었습니다. `MatMul` 연산은 수행할 장치가 명시적으로 할당되어 있지 않기 때문에 텐서플로 런타임(runtime)은 연산과 가용한 장치들(이 예제에서는 `GPU:0`)을 기반으로 하나를 고를 것이고 필요하다면 장치들간에 텐서를 자동으로 복사할 것입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ARrRhwqijPzN"
   },
   "source": [
    "## GPU 메모리 제한하기\n",
    "\n",
    "기본적으로 텐서플로는 모든 GPU의 거의 모든 메모리를 프로세스가 볼 수 있도록 매핑합니다([`CUDA_VISIBLE_DEVICES`](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars)에 포함되었다고 가정합니다). 이는 메모리 단편화를 줄여서 상대적으로 귀한 GPU 메모리 리소스를 장치에서 보다 효율적으로 사용할 수 있게 합니다. `tf.config.set_visible_devices` 메서드를 사용하여 텐서플로에서 접근할 수 있는 GPU를 조정할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:32.186343Z",
     "iopub.status.busy": "2022-12-14T21:22:32.185718Z",
     "iopub.status.idle": "2022-12-14T21:22:32.190548Z",
     "shell.execute_reply": "2022-12-14T21:22:32.189675Z"
    },
    "id": "hPI--n_jhZhv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Visible devices cannot be modified after being initialized\n"
     ]
    }
   ],
   "source": [
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "  # 텐서플로가 첫 번째 GPU만 사용하도록 제한\n",
    "  try:\n",
    "    tf.config.set_visible_devices(gpus[0], 'GPU')\n",
    "  except RuntimeError as e:\n",
    "    # 프로그램 시작시에 접근 가능한 장치가 설정되어야만 합니다\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N3x4M55DhYk9"
   },
   "source": [
    "어떤 경우에는 프로세스가 가용한 메모리의 일부에만 할당되도록 하거나 프로세스의 요구량만큼 메모리 사용이 가능할 필요가 있습니다. 텐서플로에서는 이를 위해 두 가지 방법을 제공합니다.\n",
    "\n",
    "런타임 할당에 필요한 만큼의 GPU 메모리만 할당하려고 시도하는 `tf.config.experimental.set_memory_growth`를 호출하여 메모리 증가가 이루어지도록 하는 것이 첫 번째 옵션입니다. 처음에는 매우 적은 메모리만 할당하고, 프로그램이 실행되어 더 많은 GPU 메모리가 필요해짐에 따라 GPU 메모리 영역이 TensorFlow 프로세스를 위해 확장됩니다. 메모리 조각화로 이어질 수 있으므로 메모리 할당이 해제되지 않습니다. 특정 GPU에 대한 메모리 증가를 활성화하려면 텐서를 할당하거나 연산을 실행하기 전에 다음 코드를 사용하세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:32.193804Z",
     "iopub.status.busy": "2022-12-14T21:22:32.193168Z",
     "iopub.status.idle": "2022-12-14T21:22:32.197824Z",
     "shell.execute_reply": "2022-12-14T21:22:32.197058Z"
    },
    "id": "jr3Kf1boFnCO"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Physical devices cannot be modified after being initialized\n"
     ]
    }
   ],
   "source": [
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "  try:\n",
    "    tf.config.experimental.set_memory_growth(gpus[0], True)\n",
    "  except RuntimeError as e:\n",
    "    # 프로그램 시작시에 메모리 증가가 설정되어야만 합니다\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I1o8t51QFnmv"
   },
   "source": [
    "또 다른 방법은 `TF_FORCE_GPU_ALLOW_GROWTH` 환경변수를 `true`로 설정하는 것입니다. 이 설정은 플랫폼 종속적입니다.\n",
    "\n",
    "두 번째 방법은 `tf.config.set_logical_device_configuration`으로 가상 GPU 장치를 설정하고 GPU에 할당될 전체 메모리를 제한하는 것입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:32.201191Z",
     "iopub.status.busy": "2022-12-14T21:22:32.200581Z",
     "iopub.status.idle": "2022-12-14T21:22:32.205288Z",
     "shell.execute_reply": "2022-12-14T21:22:32.204437Z"
    },
    "id": "2qO2cS9QFn42"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Virtual devices cannot be modified after being initialized\n"
     ]
    }
   ],
   "source": [
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "  # 텐서플로가 첫 번째 GPU에 1GB 메모리만 할당하도록 제한\n",
    "  try:\n",
    "    tf.config.set_logical_device_configuration(\n",
    "        gpus[0],\n",
    "        [tf.config.LogicalDeviceConfiguration(memory_limit=1024)])\n",
    "  except RuntimeError as e:\n",
    "    # 프로그램 시작시에 가상 장치가 설정되어야만 합니다\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bsg1iLuHFoLW"
   },
   "source": [
    "이는 텐서플로 프로세스에서 사용가능한 GPU 메모리량을 제한하는데 유용합니다. 워크스테이션 GUI같이 GPU가 다른 어플리케이션들에 공유되는 로컬 개발환경에서 보통 사용되는 방법입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B27_-1gyjf-t"
   },
   "source": [
    "## 멀티 GPU 시스템에서 하나의 GPU만 사용하기\n",
    "\n",
    "시스템에 두 개 이상의 GPU가 있다면 낮은 ID의 GPU가 기본으로 선택됩니다. 다른 GPU에서 실행하고 싶으면 명시적으로 표시해야 합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:32.208735Z",
     "iopub.status.busy": "2022-12-14T21:22:32.208043Z",
     "iopub.status.idle": "2022-12-14T21:22:32.783693Z",
     "shell.execute_reply": "2022-12-14T21:22:32.782652Z"
    },
    "id": "wep4iteljjG1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "try:\n",
    "  # 유효하지 않은 GPU 장치를 명시\n",
    "  with tf.device('/device:GPU:2'):\n",
    "    a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "    b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "    c = tf.matmul(a, b)\n",
    "except RuntimeError as e:\n",
    "  print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jy-4cCO_jn4G"
   },
   "source": [
    "명시한 장치가 존재하지 않으면 `RuntimeError`가 나옵니다:\n",
    "\n",
    "명시한 장치가 존재하지 않을 때 텐서플로가 자동으로 현재 지원하는 장치를 선택하게 하려면 `tf.config.set_soft_device_placement(True)`를 호출하세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:32.787933Z",
     "iopub.status.busy": "2022-12-14T21:22:32.787595Z",
     "iopub.status.idle": "2022-12-14T21:22:32.797377Z",
     "shell.execute_reply": "2022-12-14T21:22:32.796486Z"
    },
    "id": "sut_UHlkjvWd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[22. 28.]\n",
      " [49. 64.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.config.set_soft_device_placement(True)\n",
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "# 텐서 생성\n",
    "a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "c = tf.matmul(a, b)\n",
    "\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sYTYPrQZj2d9"
   },
   "source": [
    "## 멀티 GPU 사용하기\n",
    "\n",
    "여러 GPU용으로 개발하면 추가 리소스를 사용하여 모델을 확장할 수 있습니다. 단일 GPU가 있는 시스템에서 개발하는 경우, 가상 기기로 여러 GPU를 시뮬레이션할 수 있습니다. 이를 통해 추가 리소스 없이도 다중 GPU 설정을 쉽게 테스트할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:32.801093Z",
     "iopub.status.busy": "2022-12-14T21:22:32.800413Z",
     "iopub.status.idle": "2022-12-14T21:22:32.805759Z",
     "shell.execute_reply": "2022-12-14T21:22:32.804911Z"
    },
    "id": "8EMGuGKbNkc6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Virtual devices cannot be modified after being initialized\n"
     ]
    }
   ],
   "source": [
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "  # Create 2 virtual GPUs with 1GB memory each\n",
    "  try:\n",
    "    tf.config.set_logical_device_configuration(\n",
    "        gpus[0],\n",
    "        [tf.config.LogicalDeviceConfiguration(memory_limit=1024),\n",
    "         tf.config.LogicalDeviceConfiguration(memory_limit=1024)])\n",
    "    logical_gpus = tf.config.list_logical_devices('GPU')\n",
    "    print(len(gpus), \"Physical GPU,\", len(logical_gpus), \"Logical GPUs\")\n",
    "  except RuntimeError as e:\n",
    "    # Virtual devices must be set before GPUs have been initialized\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xmNzO0FxNkol"
   },
   "source": [
    "런타임에 여러 논리 GPU를 사용할 수 있게 되면 `tf.distribute.Strategy` 또는 수동 배치를 통해 여러 GPU를 활용할 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IDZmEGq4j6kG"
   },
   "source": [
    "#### `tf.distribute.Strategy` 사용\n",
    "\n",
    "멀티 GPU를 사용하는 가장 좋은 방법은 `tf.distriute.Strategy`를 사용하는 것입니다. 간단한 예제를 살펴봅시다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:32.809053Z",
     "iopub.status.busy": "2022-12-14T21:22:32.808568Z",
     "iopub.status.idle": "2022-12-14T21:22:33.833533Z",
     "shell.execute_reply": "2022-12-14T21:22:33.832682Z"
    },
    "id": "1KgzY8V2AvRv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op StatelessRandomGetKeyCounter in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op StatelessRandomUniformV2 in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "seed: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "StatelessRandomGetKeyCounter: (StatelessRandomGetKeyCounter): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "key_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "counter_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "key: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "counter: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "alg: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "StatelessRandomUniformV2: (StatelessRandomUniformV2): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Sub: (Sub): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "z_RetVal: (_Retval): /job:localhost/replica:Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddV2 in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0/task:0/device:GPU:0\n",
      "x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AddV2: (AddV2): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "NoOp: (NoOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "dims: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Fill: (Fill): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/taskExecuting op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ":0/device:GPU:2\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2\n",
      "resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3\n",
      "inputs_0: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "inputs_1: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "inputs_2: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "inputs_3: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "AddN: (AddN): /job:localhost/replica:0/task:0/device:CPU:0\n",
      "sum_RetVExecuting op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "gpus = tf.config.list_logical_devices('GPU')\n",
    "strategy = tf.distribute.MirroredStrategy(gpus)\n",
    "with strategy.scope():\n",
    "  inputs = tf.keras.layers.Input(shape=(1,))\n",
    "  predictions = tf.keras.layers.Dense(1)(inputs)\n",
    "  model = tf.keras.models.Model(inputs=inputs, outputs=predictions)\n",
    "  model.compile(loss='mse',\n",
    "                optimizer=tf.keras.optimizers.SGD(learning_rate=0.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dy7nxlKsAxkK"
   },
   "source": [
    "이 프로그램은 입력 데이터를 나누고 모델의 복사본을 각 GPU에서 실행할 것입니다. 이는 \"[데이터 병렬처리](https://en.wikipedia.org/wiki/Data_parallelism)\"라고도 합니다.\n",
    "\n",
    "병렬화 전략에 대해 더 알고 싶으시면 [가이드](./distributed_training.ipynb)를 참조하세요."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8phxM5TVkAY_"
   },
   "source": [
    "#### `tf.distribute.Strategy` 미사용\n",
    "\n",
    "`tf.distribute.Strategy`는 여러 장치에 걸쳐 계산을 복제해서 동작합니다. 모델을 각 GPU에 구성하여 수동으로 이를 구현할 수 있습니다. 예를 들면:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-14T21:22:33.837498Z",
     "iopub.status.busy": "2022-12-14T21:22:33.836841Z",
     "iopub.status.idle": "2022-12-14T21:22:34.306734Z",
     "shell.execute_reply": "2022-12-14T21:22:34.305861Z"
    },
    "id": "AqPo9ltUA_EY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 88. 112.]\n",
      " [196. 256.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "gpus = tf.config.list_logical_devices('GPU')\n",
    "if gpus:\n",
    "  # Replicate your computation on multiple GPUs\n",
    "  c = []\n",
    "  for gpu in gpus:\n",
    "    with tf.device(gpu.name):\n",
    "      a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "      b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "      c.append(tf.matmul(a, b))\n",
    "\n",
    "  with tf.device('/CPU:0'):\n",
    "    matmul_sum = tf.add_n(c)\n",
    "\n",
    "  print(matmul_sum)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "gpu.ipynb",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}