{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "CCQY7jpBfMur" }, "source": [ "##### Copyright 2018 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2022-12-14T21:09:17.913760Z", "iopub.status.busy": "2022-12-14T21:09:17.913212Z", "iopub.status.idle": "2022-12-14T21:09:17.916922Z", "shell.execute_reply": "2022-12-14T21:09:17.916419Z" }, "id": "z6X9omPnfO_h" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "2QQJJyDzqGRb" }, "source": [ "# Eager execution\n" ] }, { "cell_type": "markdown", "metadata": { "id": "B1xdylywqUSX" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " TensorFlow.org에서 보기\n", " \n", " 구글 코랩(Colab)에서 실행하기\n", " \n", " 깃허브(GitHub)에서 소스 보기\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "-agGVYp_4GWZ" }, "source": [ "Note: 이 문서는 텐서플로 커뮤니티에서 번역했습니다. 커뮤니티 번역 활동의 특성상 정확한 번역과 최신 내용을 반영하기 위해 노력함에도\n", "불구하고 [공식 영문 문서](https://www.tensorflow.org/?hl=en)의 내용과 일치하지 않을 수 있습니다.\n", "이 번역에 개선할 부분이 있다면\n", "[tensorflow/docs-l10n](https://github.com/tensorflow/docs-l10n/) 깃헙 저장소로 풀 리퀘스트를 보내주시기 바랍니다.\n", "문서 번역이나 리뷰에 참여하려면\n", "[docs-ko@tensorflow.org](https://groups.google.com/a/tensorflow.org/forum/#!forum/docs-ko)로\n", "메일을 보내주시기 바랍니다." ] }, { "cell_type": "markdown", "metadata": { "id": "EGjDcGxIqEfX" }, "source": [ "텐서플로의 즉시 실행은 그래프를 생성하지 않고 함수를 바로 실행하는 명령형 프로그래밍 환경입니다.\n", "나중에 실행하기 위해 계산가능한 그래프를 생성하는 대신에 계산값을 즉시 알려주는 연산입니다.\n", "이러한 기능은 텐서플로를 시작하고 모델을 디버깅하는 것을 더욱 쉽게 만들고 불필요한 상용구 코드(boilerplate code) 작성을 줄여줍니다.\n", "가이드를 따라하려면, 대화형 `파이썬` 해석기(interpreter)를 이용해 아래에 있는 코드 샘플을 실행하세요.\n", "\n", "즉시 실행은 연구와 실험을 위한 유연한 기계학습 플랫폼으로 다음과 같은 기능을 제공합니다:\n", "\n", "* *직관적인 인터페이스*-코드를 자연스럽게 구조화하고 파이썬의 데이터 구조를 활용. 작은 모델과 작은 데이터를 빠르게 반복\n", "* *손쉬운 디버깅*-실행중인 모델을 검토하거나 변경 사항을 테스트해보기 위해서 연산을 직접 호출. 에러 확인을 위해서 표준 파이썬 디버깅 툴을 사용\n", "* *자연스런 흐름 제어*-그래프 제어 흐름 대신에 파이썬 제어 흐름을 사용함으로써 동적인 모델 구조의 단순화\n", "\n", "즉시 실행은 대부분의 텐서플로 연산과 GPU 가속을 지원합니다.\n", "\n", "Note: 일부 모델은 즉시 실행을 활성화한 경우 추가연산비용(overhead)이 증가한 경우도 있습니다.성능을 향상시키려는 노력은 계속 진행 중이지만 만약에 문제점을 찾거나 관련된 벤치마크를 공유하고 싶다면 [버그를 기록해주세요](https://github.com/tensorflow/tensorflow/issues)" ] }, { "cell_type": "markdown", "metadata": { "id": "RBAeIwOMrYk8" }, "source": [ "## 설치와 기본 사용법" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:17.920548Z", "iopub.status.busy": "2022-12-14T21:09:17.919998Z", "iopub.status.idle": "2022-12-14T21:09:19.823633Z", "shell.execute_reply": "2022-12-14T21:09:19.822940Z" }, "id": "ByNsp4VqqEfa" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-12-14 21:09:18.853444: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n", "2022-12-14 21:09:18.853541: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n", "2022-12-14 21:09:18.853550: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n" ] } ], "source": [ "import tensorflow as tf\n", "\n", "import cProfile" ] }, { "cell_type": "markdown", "metadata": { "id": "48P3-8q4qEfe" }, "source": [ "텐서플로 2.0에서 즉시 실행은 기본으로 활성화되어 있습니다." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:19.827785Z", "iopub.status.busy": "2022-12-14T21:09:19.827369Z", "iopub.status.idle": "2022-12-14T21:09:19.833659Z", "shell.execute_reply": "2022-12-14T21:09:19.833114Z" }, "id": "7aFsD8csqEff" }, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tf.executing_eagerly()" ] }, { "cell_type": "markdown", "metadata": { "id": "x_G1zZT5qEfh" }, "source": [ "이제부터는 텐서플로 연산을 바로 실행할 수 있고 결과를 즉시 확인할 수 있습니다:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:19.837138Z", "iopub.status.busy": "2022-12-14T21:09:19.836592Z", "iopub.status.idle": "2022-12-14T21:09:23.498597Z", "shell.execute_reply": "2022-12-14T21:09:23.497823Z" }, "id": "9gsI54pbqEfj" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hello, [[4.]]\n" ] } ], "source": [ "x = [[2.]]\n", "m = tf.matmul(x, x)\n", "print(\"hello, {}\".format(m))" ] }, { "cell_type": "markdown", "metadata": { "id": "ajFn6qsdqEfl" }, "source": [ "즉시 실행 활성화는 텐서플로 연산을 바로 평가하고 그 결과를 파이썬에게 알려주는 방식으로 동작을 변경합니다.\n", "`tf.Tensor` 객체는 계산 그래프에 있는 노드를 가르키는 간접 핸들(symbolic handle) 대신에 구체적인 값을 참조합니다.\n", "나중에 실행하기 위해서 생성된 계산 그래프가 없기 때문에, `print()`나 디버거를 통해서 결과를 검토하기 쉽습니다.\n", "텐서값을 평가, 출력하거나 확인하는 것이 그래디언트(gradients)를 계산하는 흐름을 방해하지 않습니다.\n", "\n", "즉시 실행은 [NumPy](http://www.numpy.org/)와 같이 잘 작동됩니다.\n", "NumPy 연산에 `tf.Tensor`를 매개변수로 사용가능합니다.\n", "텐서플로 [수학 연산](https://www.tensorflow.org/api_guides/python/math_ops)은 파이썬 객체와 NumPy 배열을 `tf.Tensor` 객체로 변환합니다.\n", "`tf.Tensor.numpy` 메서드는 객체 값을 NumPy `ndarray`로 반환합니다." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:23.502042Z", "iopub.status.busy": "2022-12-14T21:09:23.501800Z", "iopub.status.idle": "2022-12-14T21:09:23.507437Z", "shell.execute_reply": "2022-12-14T21:09:23.506835Z" }, "id": "sTO0_5TYqz1n" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(\n", "[[1 2]\n", " [3 4]], shape=(2, 2), dtype=int32)\n" ] } ], "source": [ "a = tf.constant([[1, 2],\n", " [3, 4]])\n", "print(a)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:23.510727Z", "iopub.status.busy": "2022-12-14T21:09:23.510298Z", "iopub.status.idle": "2022-12-14T21:09:23.516077Z", "shell.execute_reply": "2022-12-14T21:09:23.515502Z" }, "id": "Dp14YT8Gq4r1" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(\n", "[[2 3]\n", " [4 5]], shape=(2, 2), dtype=int32)\n" ] } ], "source": [ "# 브로드캐스팅(Broadcasting) 지원\n", "b = tf.add(a, 1)\n", "print(b)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:23.519253Z", "iopub.status.busy": "2022-12-14T21:09:23.518789Z", "iopub.status.idle": "2022-12-14T21:09:23.523414Z", "shell.execute_reply": "2022-12-14T21:09:23.522837Z" }, "id": "69p3waMfq8cQ" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(\n", "[[ 2 6]\n", " [12 20]], shape=(2, 2), dtype=int32)\n" ] } ], "source": [ "# 연산자 오버로딩 지원\n", "print(a * b)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:23.526708Z", "iopub.status.busy": "2022-12-14T21:09:23.526102Z", "iopub.status.idle": "2022-12-14T21:09:23.530020Z", "shell.execute_reply": "2022-12-14T21:09:23.529434Z" }, "id": "Ui025t1qqEfm" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[ 2 6]\n", " [12 20]]\n" ] } ], "source": [ "# NumPy값 사용\n", "import numpy as np\n", "\n", "c = np.multiply(a, b)\n", "print(c)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:23.533023Z", "iopub.status.busy": "2022-12-14T21:09:23.532603Z", "iopub.status.idle": "2022-12-14T21:09:23.536195Z", "shell.execute_reply": "2022-12-14T21:09:23.535582Z" }, "id": "Tq_aFRzWrCua" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2]\n", " [3 4]]\n" ] } ], "source": [ "# 텐서로부터 numpy 값 얻기:\n", "print(a.numpy())\n", "# => [[1 2]\n", "# [3 4]]" ] }, { "cell_type": "markdown", "metadata": { "id": "H08f9ss9qEft" }, "source": [ "## 동적인 제어 흐름\n", "\n", "즉시 실행의 가장 큰 이점은 모델을 실행하는 동안에도 호스트 언어의 모든 기능을 활용할 수 있다는 것입니다.\n", "그래서 다음과 같이 [fizzbuzz](https://en.wikipedia.org/wiki/Fizz_buzz)를 손쉽게 작성할 수 있습니다:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:23.539449Z", "iopub.status.busy": "2022-12-14T21:09:23.538939Z", "iopub.status.idle": "2022-12-14T21:09:23.543537Z", "shell.execute_reply": "2022-12-14T21:09:23.542893Z" }, "id": "0fudRMeUqEfu" }, "outputs": [], "source": [ "def fizzbuzz(max_num):\n", " counter = tf.constant(0)\n", " max_num = tf.convert_to_tensor(max_num)\n", " for num in range(1, max_num.numpy()+1):\n", " num = tf.constant(num)\n", " if int(num % 3) == 0 and int(num % 5) == 0:\n", " print('FizzBuzz')\n", " elif int(num % 3) == 0:\n", " print('Fizz')\n", " elif int(num % 5) == 0:\n", " print('Buzz')\n", " else:\n", " print(num.numpy())\n", " counter += 1" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:23.546766Z", "iopub.status.busy": "2022-12-14T21:09:23.546191Z", "iopub.status.idle": "2022-12-14T21:09:23.555385Z", "shell.execute_reply": "2022-12-14T21:09:23.554811Z" }, "id": "P2cKknQWrJLB" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1\n", "2\n", "Fizz\n", "4\n", "Buzz\n", "Fizz\n", "7\n", "8\n", "Fizz\n", "Buzz\n", "11\n", "Fizz\n", "13\n", "14\n", "FizzBuzz\n" ] } ], "source": [ "fizzbuzz(15)" ] }, { "cell_type": "markdown", "metadata": { "id": "7kA-aC3BqEfy" }, "source": [ "여기에 텐서값에 따른 조건절이 있고 실행중에 그 결과를 출력합니다." ] }, { "cell_type": "markdown", "metadata": { "id": "8huKpuuAwICq" }, "source": [ "## 즉시 훈련" ] }, { "cell_type": "markdown", "metadata": { "id": "mp2lCCZYrxHd" }, "source": [ "### 그래디언트 계산하기\n", "\n", "[자동 미분](https://en.wikipedia.org/wiki/Automatic_differentiation)은 인공 신경망 훈련을 위한\n", "[역전파](https://en.wikipedia.org/wiki/Backpropagation)와 같은 기계학습 알고리즘을 구현하는데 유용합니다.\n", "즉시 실행을 사용하는 동안에는, 나중에 그래디언트를 계산하는 연산을 추적하기 위해 `tf.GradientTape`을 사용하세요.\n", "\n", "즉시 실행 중에 그래디언트를 계산하고 모델 훈련에 이용하기 위해서 `tf.GradientTape`을 사용할 수 있습니다.\n", "특히 복잡하고 반복적인 훈련인 경우에 더 유용합니다.\n", "\n", "매번 실행될 때 서로 다른 연산이 수행될 수 있기 때문에 모든 정방향(forward-pass) 연산은 \"tape\"에 기록됩니다.\n", "그다음 tape를 거꾸로 돌려 그래디언트를 계산한 후 tape를 폐기합니다.\n", "특정한 `tf.GradientTape`는 오직 하나의 그래디언트만을 계산할 수 있고 부가적인 호출은 실행중 에러(runtime error)를 발생시킵니다." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:23.558823Z", "iopub.status.busy": "2022-12-14T21:09:23.558286Z", "iopub.status.idle": "2022-12-14T21:09:23.572182Z", "shell.execute_reply": "2022-12-14T21:09:23.571599Z" }, "id": "7g1yWiSXqEf-" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor([[2.]], shape=(1, 1), dtype=float32)\n" ] } ], "source": [ "w = tf.Variable([[1.0]])\n", "with tf.GradientTape() as tape:\n", " loss = w * w\n", "\n", "grad = tape.gradient(loss, w)\n", "print(grad) # => tf.Tensor([[ 2.]], shape=(1, 1), dtype=float32)" ] }, { "cell_type": "markdown", "metadata": { "id": "vkHs32GqweYS" }, "source": [ "### 모델 훈련\n", "\n", "다음 예는 표준 MNIST 손글씨 분류를 위한 다층 모델을 생성합니다.\n", "즉시 실행 환경에서 훈련가능한 그래프를 생성하기 위한 옵티마이저(optimizer)와 층 API를 보여줍니다." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:23.575600Z", "iopub.status.busy": "2022-12-14T21:09:23.575120Z", "iopub.status.idle": "2022-12-14T21:09:24.554935Z", "shell.execute_reply": "2022-12-14T21:09:24.554213Z" }, "id": "38kymXZowhhz" }, "outputs": [], "source": [ "# mnist 데이터 가져오기 및 포맷 맞추기\n", "(mnist_images, mnist_labels), _ = tf.keras.datasets.mnist.load_data()\n", "\n", "dataset = tf.data.Dataset.from_tensor_slices(\n", " (tf.cast(mnist_images[...,tf.newaxis]/255, tf.float32),\n", " tf.cast(mnist_labels,tf.int64)))\n", "dataset = dataset.shuffle(1000).batch(32)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:24.558786Z", "iopub.status.busy": "2022-12-14T21:09:24.558509Z", "iopub.status.idle": "2022-12-14T21:09:24.613746Z", "shell.execute_reply": "2022-12-14T21:09:24.613154Z" }, "id": "rl1K8rOowmwT" }, "outputs": [], "source": [ "# 모델 생성\n", "mnist_model = tf.keras.Sequential([\n", " tf.keras.layers.Conv2D(16,[3,3], activation='relu',\n", " input_shape=(None, None, 1)),\n", " tf.keras.layers.Conv2D(16,[3,3], activation='relu'),\n", " tf.keras.layers.GlobalAveragePooling2D(),\n", " tf.keras.layers.Dense(10)\n", "])" ] }, { "cell_type": "markdown", "metadata": { "id": "fvyk-HgGwxwl" }, "source": [ "즉시 실행에서는 훈련을 하지 않아도 모델을 사용하고 결과를 점검할 수 있습니다:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:24.617283Z", "iopub.status.busy": "2022-12-14T21:09:24.616824Z", "iopub.status.idle": "2022-12-14T21:09:25.683219Z", "shell.execute_reply": "2022-12-14T21:09:25.682488Z" }, "id": "BsxystjBwxLS" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "로짓: [[ 0.00129889 0.01228362 -0.0372225 0.01005802 -0.00351836 0.0153954\n", " -0.01946059 -0.00020366 -0.03208666 -0.06244648]]\n" ] } ], "source": [ "for images,labels in dataset.take(1):\n", " print(\"로짓: \", mnist_model(images[0:1]).numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "Y3PGa8G7qEgB" }, "source": [ "케라스 모델은 자체적인 훈련 메서드(fit)을 포함하고 있지만 때로는 좀 더 수정할 필요가 있습니다.\n", "다음은 즉시 실행을 활용한 반복적인 훈련의 예입니다:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:25.687090Z", "iopub.status.busy": "2022-12-14T21:09:25.686448Z", "iopub.status.idle": "2022-12-14T21:09:25.693305Z", "shell.execute_reply": "2022-12-14T21:09:25.692767Z" }, "id": "bzRhM7JDnaEG" }, "outputs": [], "source": [ "optimizer = tf.keras.optimizers.Adam()\n", "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n", "\n", "loss_history = []" ] }, { "cell_type": "markdown", "metadata": { "id": "tXaupYXRI2YM" }, "source": [ "Note: 조건을 만족했는지 확인하기 위해서 `tf.debugging`에 있는 단언문(assert) 함수를 사용하세요. 이것은 즉시 실행과 그래프 실행 모두에서 동작합니다." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:25.696745Z", "iopub.status.busy": "2022-12-14T21:09:25.696189Z", "iopub.status.idle": "2022-12-14T21:09:25.700612Z", "shell.execute_reply": "2022-12-14T21:09:25.700073Z" }, "id": "DDHrigtiCIA4" }, "outputs": [], "source": [ "def train_step(images, labels):\n", " with tf.GradientTape() as tape:\n", " logits = mnist_model(images, training=True)\n", " \n", " # 결과의 형태를 확인하기 위해서 단언문 추가\n", " tf.debugging.assert_equal(logits.shape, (32, 10))\n", " \n", " loss_value = loss_object(labels, logits)\n", "\n", " loss_history.append(loss_value.numpy().mean())\n", " grads = tape.gradient(loss_value, mnist_model.trainable_variables)\n", " optimizer.apply_gradients(zip(grads, mnist_model.trainable_variables))" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:25.703813Z", "iopub.status.busy": "2022-12-14T21:09:25.703298Z", "iopub.status.idle": "2022-12-14T21:09:25.706921Z", "shell.execute_reply": "2022-12-14T21:09:25.706345Z" }, "id": "0m1xAXrmqEgJ" }, "outputs": [], "source": [ "def train():\n", " for epoch in range(3):\n", " for (batch, (images, labels)) in enumerate(dataset):\n", " train_step(images, labels)\n", " print ('에포크 {} 종료'.format(epoch))" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:09:25.710118Z", "iopub.status.busy": "2022-12-14T21:09:25.709575Z", "iopub.status.idle": "2022-12-14T21:10:44.323513Z", "shell.execute_reply": "2022-12-14T21:10:44.322716Z" }, "id": "C5dGz0p_nf4W" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 5 calls to triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:6 out of the last 6 calls to triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "에포크 0 종료\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "에포크 1 종료\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "에포크 2 종료\n" ] } ], "source": [ "train()" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:44.327042Z", "iopub.status.busy": "2022-12-14T21:10:44.326795Z", "iopub.status.idle": "2022-12-14T21:10:45.141136Z", "shell.execute_reply": "2022-12-14T21:10:45.140371Z" }, "id": "5vG5ql_2vYB5" }, "outputs": [ { "data": { "text/plain": [ "Text(0, 0.5, 'Loss [entropy]')" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABw+UlEQVR4nO3dd1QUZ9sG8GvpIE1FKYoiYkVRY0HsRqKo0ZhijDGxpGo0zSR+McWoKeZNNcVoEgupGvMm0bxqbNgVJaLYe0UFrHSl7Xx/qCsLW2Z2Z3dmd6/fOZwDu888c+8AO/c+VSMIggAiIiIiF+KmdABERERE9sYEiIiIiFwOEyAiIiJyOUyAiIiIyOUwASIiIiKXwwSIiIiIXA4TICIiInI5HkoHoEZarRYXLlxAQEAANBqN0uEQERGRCIIgoKCgABEREXBzM93GwwTIgAsXLiAyMlLpMIiIiMgCmZmZqF+/vskyTIAMCAgIAHDzAgYGBiocDREREYmRn5+PyMhI3X3cFCZABtzu9goMDGQCRERE5GDEDF/hIGgiIiJyOUyAiIiIyOUwASIiIiKXwwSIiIiIXA4TICIiInI5TICIiIjI5TABIiIiIpfDBIiIiIhcDhMgIiIicjlMgIiIiMjlMAEiIiIil8MEiIiIiFwOEyAVEwQBN8oqlA6DiIjI6TABUrGnf9yJ5m+vRFbedaVDISIicipMgFRkT2YuzufeSXbWHroIAPgj/ZxSIRERETklJkAqceJSIe6btRVdP1xX7bkdp67imw3HodUKCkRGRETkfJgAKeiDFYcwekEaKrQC9p/P0z2ed71Mr9zmY5fx0coj+N/eC/YOkYiIyClpBEFgs0IV+fn5CAoKQl5eHgIDA2WtO+3UVTz8bSpGJTTED6lnJB07oHUYPn6oDWp4e8gaExERkTOQcv9mC5CdPfxtKgBITn4AYMW+bPT4aL3cIREREbkcJkB2tOnoJavruFJUioe/TcWitLNImrkJucWlMkRGRETkWpgA2dH3m0/KUk/aqat4/c99OJxdgMfm7ZClTiIiIlfCBMiOSsq1ste5/3w+Z4cRERFJxATIjjpF1bJJvY/PZysQERGRFEyA7Gh87xib1Lv1+BWb1EtEROSsmADZka+XO74c3k7pMIiIiFweEyA7GxQXjsQWdWWvVxAEvL/8IH7eLn16PRERkavhinp2ptFoMHdURwBA5tVidJdpXZ/dmbn4fvMpAMBjnRvKUicREZGzYguQgiJr+clW1/O/7patLiIiImenaAI0Y8YMdOzYEQEBAahbty6GDBmCI0eOmDzm+++/R/fu3VGzZk3UrFkTiYmJSEtL0yszevRoaDQava+kpCRbvhSLTbynqSz1VN5FnoiIiExTNAHauHEjxo8fj+3bt2PNmjUoKytD3759UVRUZPSYDRs2YPjw4Vi/fj1SU1MRGRmJvn374vz583rlkpKSkJWVpftauHChrV+ORV7o0wQ730rEGwOao1FIDTSuU0PpkIiIiJyeqjZDvXTpEurWrYuNGzeiR48eoo6pqKhAzZo18fXXX2PkyJEAbrYA5ebmYsmSJRbFYcvNUM0puFGG1lNXW1XH6Q8HyhQNERGR43DYzVDz8vIAALVqiV8wsLi4GGVlZdWO2bBhA+rWrYtmzZph3LhxuHLF+Fo5JSUlyM/P1/tSioeb9b8SFeW0REREqqSaBEir1eKll15C165d0apVK9HH/d///R8iIiKQmJioeywpKQk//vgjUlJS8J///AcbN25E//79UVFRYbCOGTNmICgoSPcVGRlp9euxlAz5D7gzBhERkWmqmQY/fvx47N+/H1u2bBF9zIcffohFixZhw4YN8PHx0T3+yCOP6L5v3bo14uLi0LhxY2zYsAF9+vSpVs/kyZMxceJE3c/5+fmKJUFe7ncyoM2Tels0Tb5CK8DdTYM3/toHNw3w3pDWcoZIRETk8FSRAE2YMAHLli3Dpk2bUL9+fVHHfPLJJ/jwww+xdu1axMXFmSwbHR2NkJAQHD9+3GAC5O3tDW9vb4til5tGo8HeqX1RUSHAz9vdojp+2n4G97WNwK87zgIAJiU1R6CPp5xhEhEROTRFEyBBEPD888/jr7/+woYNG9CoUSNRx3300Ud4//33sWrVKnTo0MFs+XPnzuHKlSsIDw+3NmS7qJysbHv9bgBAlw/XiT7+3WUH0aSuv+5nDgkiIiLSp+gYoPHjx+Pnn3/Gr7/+ioCAAGRnZyM7OxvXr99Z02bkyJGYPHmy7uf//Oc/ePvttzF//nxERUXpjiksLAQAFBYW4rXXXsP27dtx+vRppKSk4L777kNMTAz69etn99dorYhgX0QE+0o+buT8O2sjaTRyRkREROT4FE2AZs+ejby8PPTq1Qvh4eG6r99++01X5uzZs8jKytI7prS0FA899JDeMZ988gkAwN3dHXv37sXgwYPRtGlTPPnkk2jfvj02b96smm4uS6x4oTsAwMNNejbD/IeIiEifqtYBUgsl1wEyRRAEaDQaRL2+XNJx+6b2RcCtbjVBEFBQUs4xQURE5HQcdh0gMk1jYV9W5Qz3nb8PIG7qany6+gjXCyIiIpfFBMgFVM5zfkw9AwD4at1x/LDttDIBERERKYwJkCsw0tAz9X8Hsf2k8RWyiYiInBUTIBcgGMuAAPx0q0WIiIjIlTABcmB9W4aKKicIgNbI/himkiMiIiJnpYqVoMky97aJwKwRd6HJm/+YLHf/N1vh7qbB4Db17BQZERGRujEBcmDuGg083c034p2+UgwA+Hzt0WrPcSIYERG5InaBOaCRCQ3Rql4gElvWtbouJkBEROSK2ALkgKbf10q2ujgGiIiIXBFbgIiIiMjlMAFyAg1q+Vl8LLvAiIjIFTEBcgLLXuhm8bFGZscTERE5NSZATiDQxxMfPRSndBhEREQOgwmQk3i4Q6SFRxpvAiopr7CwTiIiInVjAuREPragFcjQGCBBELBk93k0e2slpizdL0NkRERE6sIEyIkM7RCJxBbitse4Lf9GWbXH5m4+hZd+ywBwZ/d4IiIiZ8IEyMlIHQt0/tr1ao/N3nhCrnCIiIhUiQmQk6lVw0tS+bzrZTh1uQgnLhXaKCIi9Uo/cw2z1h9HBadDErkcrgTt4opKK9D7kw0AgIPT+8HPi38S5DoenL0NwM0PDsM7NVA4GiKyJ7YAkc6VwlK7nGfl/iys3J9tl3MRiXHiIltAiVwNEyAn1KZ+kEXHae2wLHTBjTKM/XkXxv6cjuLScpufj4iIyBAmQE4o0NfTouPKTYyDOHOlCON/2YV95/IsDQsAcL30ztpCJWVaq+oiIiKyFBMg0tGaSICe/Skdy/dlYdDXW2Q7H4edEhGRUpgAkU6FiS6wk5eK5DmJRp5qiOTEZJzI9TABckIajWVZxsEL+RC4Pbwiyiq0mPDrLvy0nQtPEhHZA+c8OyFLG1kmLt4Df2/+SSjh74wLWLY3C8v2ZuHxzg2VDsflsGGSyPWwBYj0/LnrvOEnbHCHYGvTHQUGtiQhIiLbYQLkhCzsAdMda8vp8Bp+1iYiIhVgfwfp+YcLFJILYlskkethCxCZVV7B9XpszdKB60REZBkmQE5I7ltpiykrUVp+Jwn6e88FjJqfhtxi67bO4KduIiJSChMgJ+Qmc2tCWYV+qvLCwt3YePQSPl9zVHJdbOggpRy/WIhz14oNPsc/SyLXwzFAZLHc65y5RI4ht7gUiZ9tBACc/nCgwtEQkRqwBYgstjTjAvZk5iodBpFZmVevm3ye3bFErocJkBOyZzfTfbO2SirPrgbD2DVIRGRfTICIiIjI5TABckqO0ZzAhaCJiEgpTICc0KPxkUqHQEREpGpMgJxQaKCP0iEYJWbBv8KScny6+ggOZeXbISKyhpT93M5dK8b+83k2jIaISDwmQE7I19PdrudbtveCrPV9tPIwvlp3HP2/2CxrvSSv33dmou30NUg/c01U+W7/WY97v9qCs1cMr8VDRGRPiiZAM2bMQMeOHREQEIC6detiyJAhOHLkiNnjfv/9dzRv3hw+Pj5o3bo1VqxYofe8IAiYMmUKwsPD4evri8TERBw7dsxWL0N1ouv4Y3SXKLyc2NQu55vw626UWbBdhmBk8vE+F2wlcIxRW/pe++9e5F0vw7if0yUddzibLXtEpDxFE6CNGzdi/Pjx2L59O9asWYOysjL07dsXRUVFRo/Ztm0bhg8fjieffBK7d+/GkCFDMGTIEOzfv19X5qOPPsKXX36JOXPmYMeOHahRowb69euHGzdu2ONlqcLUwbF4MbGJ3c4X/0EKSsu1yCsusygZcnWOPB6cU/iJyBEpmgCtXLkSo0ePRmxsLNq0aYPk5GScPXsW6enGP1F+8cUXSEpKwmuvvYYWLVrg3XffxV133YWvv/4awM3Wn5kzZ+Ktt97Cfffdh7i4OPz444+4cOEClixZYqdXpk5rXu5hs7qvFpUi9eQVtJm+WrfiriFi7pW8nzoWzuYjIkekqjFAeXk3uz5q1apltExqaioSExP1HuvXrx9SU1MBAKdOnUJ2drZemaCgIMTHx+vKVFVSUoL8/Hy9L2fUJDTApvVvOnoJAHBG7BgP3jh1mPQREdmXahIgrVaLl156CV27dkWrVq2MlsvOzkZoaKjeY6GhocjOztY9f/sxY2WqmjFjBoKCgnRfkZHOM438q+HtAAAfPxRn83PxJu6amMeSIcWl5Xhs7g78mHpa6VCIDFJNAjR+/Hjs378fixYtsvu5J0+ejLy8PN1XZmam3WOwlUFtInDkvSQM7eA4SZ2hqfIl5RUY9NUWTP37gAIREZFUP6aewZbjlzFlKf9nSZ1UkQBNmDABy5Ytw/r161G/fn2TZcPCwpCTk6P3WE5ODsLCwnTP337MWJmqvL29ERgYqPflTLw97DMt3tRg2IsFN/DmX/tEre1jqJpVB3Kw73wekredtjg+IrKfopJypUMgMknRBEgQBEyYMAF//fUX1q1bh0aNGpk9JiEhASkpKXqPrVmzBgkJCQCARo0aISwsTK9Mfn4+duzYoStDtmFqkcNXFu/BLzvO4tG5O3SPSek60Wodu6Pl41WH2RWgYhzITeR6PJQ8+fjx4/Hrr79i6dKlCAgI0I3RCQoKgq+vLwBg5MiRqFevHmbMmAEAePHFF9GzZ098+umnGDhwIBYtWoSdO3fiu+++A3DzJvzSSy/hvffeQ5MmTdCoUSO8/fbbiIiIwJAhQxR5na4it7jU6HOHswusqtvYmkGO4GhOAWatPwEAGJkQZbiQHeaSn75chPBgH7u1CKoJp+oTUVWKJkCzZ88GAPTq1Uvv8QULFmD06NEAgLNnz8LN7U5DVZcuXfDrr7/irbfewhtvvIEmTZpgyZIlegOnJ02ahKKiIjzzzDPIzc1Ft27dsHLlSvj4qHeLCGeweOc5WepxtptVwQ3luwK2HLuMx+btQGxEIJa/0F3Wup2h9cTZ/uaIyDxFEyAx+wht2LCh2mNDhw7F0KFDjR6j0Wgwffp0TJ8+3ZrwiJzG7+k3B/YfuKA/BksQBBSXVqCGt6JvBTbnDEkaEclLFYOgyXkJgoBZ64/jUkGJgecUCEillGqAmPDrbsS+swrHcizvorxcWIIPVhxy+HFaRORamACRTa05mIOPV5nf383VKZU6LN+XBQBWz677btNJ/LPf8DpbjoDJuDhHcwpwxMrxfERqwQSIbOrcteuSj9FwSUWHdKnAdfbac0U3yirQ9/NN6DdzE26UVSgaS4VWEDWEQq5zKf16yTaYAJHN/LLjDNYfuSj9QBfMf5zhJbMRxblVXtdHyTV+Ssor0P0/6/DYvB3mC8vg3q+2IG7qaq5r5ISYALmYtDf72OU8h7Pz8eZf+7H52GWjZapObc/Ou4HjF12zeX1PZq5N62cXDzmL9DPXcCHvBrYev2KX8x3KykdphRa7zl6zy/nIfpx76gdVUzfAB6GB3sjJrz4oWU4HL0jfULbzjBSjz0m9gWfn3cAXKccwqktDNA9TdmVvMVOsf0+XZwkBJam5Fcvc74DT4IlcD1uAXFBYoO3XQ5q4eI/ZMmKSmpLyCrz6+x6suDVYV6wXFu3GwrSz6P/FZknHOSt73OAduZHJ2N9iyqEcfLr6iN3GmzgKU1fjRlkFLnI8GDkAtgCRqv264yz+a0HryKFbLVC8b5E1nvxhJwAgNiIQSa3CFY5GWaa2uqms84wU5BaX4aH2pvd1JFIaW4BckFpyAjFxXC6s3lV3rcj4lhtU3YELeViacUHpMBRlbSKcnccWDbFyi8sAAKkn7DNGx174Ycr5MAFyQTW8HKfhz9CU+Nf+u1fMgXTLwC+3KB0CEZHqMAFyQTMeaI0W4coODAbEbYViyI5Txj9ZFpaUY9neCygu5bodRM6I64SRXJgAuaCokBr450V5N8S0FYPDDm7lTSXl1ZOcFxfuxoRfd6Oi0rYMcg1gdcWtHib/uReDv96Csgqt0qGQg+HMOlI7JkAurF6wr6LntzQv0QoCFqWdRbO3VuLvPfpjW1IOV1948a5312D+llPIyTc9jkMQBBQaWezsf3suoM301dhiYl0jKdLPXMWOk8qOkRCTGC5My8Tec3nYfOySHSKyHWtvxq6X+pom5n/XVglQ1fXDiCzFBMiFTR0cq3QIZhl6Dy0qrcDrf+4DALywcLfZOq4Vl2H6soOI/yAFi//NNFpu8p/70OqdVdh5+mq1555fuBsFN8qtXn32naX7UVquxYOzUzHsu+3Iv1FmVX3WkLKQnJYNQC5Paj7jDIOGD1zIUzoEsiEmQORSPvjnkNHnFt1Kjr5IOSa6vov5NzBjxSGcuVIkqvwPqWf0uu7yr5tOgCps2O12IU/6Pm1yEDudmkzLv1GGjUcvoZzdkzbDCQTOjQkQKabyp6sCYy0hKr9ZTli4G99uOokHZ28TfUzllMZUMrA04zyav/0PVh9Qfpd1OdMwR1xUUI1/hcO/245R89Pw7aaTdjmfWn5rHARNcmECRIoZ+/MuADfXWGk9dbXC0VjmdnfZ5ULjaxOZers29dyLizJQViHgmZ/SLQsO6kg28opvtlQoSQWXQXYHbi32uWT3eYUjMUzln12ImACRsm6UVWDCr7usqmPE3O2iu6DEKCwpx7+nr4pKHizpzrHXzXji4gz0m7lJlrqsuZfdP3srRs1PkyUOOViSFDph/iRZ1b+B7zedxNA524zuku6MSSc5FyZALszDTfmPaAO+3Kz7JGuprcev4IVFGfIEBGD32VwMnZNqdIPSzKvFupuomCtY7T5Q6QFrPiWXlmuxJzPX6PT8P3edx9GcQstPUImYe5mxG97JS/Ilp6Qe7684hH9PX8NP288oHYpdMJ9zPkyAXFi3JiFKh4CTl4pMTmsVmx9cLpB/d/u/jWwf0f2j9Xh/ufHB1FJYM57h+YW7cN+srZi98YQssVRWXFpu0wHY5uQWl2LT0UuyxcDuGHlV/p+9UWZ40VFec1I7JkAuzNPdDYktQpUOw65u71NkrblbTgEQ9yZftYhc65isOpADAJh3KxapjIV+ragULaeswr1fSZsBI+cN775ZWzFyfhp+tkHrQv8vNuMrCTP91M5eiYbU83CwMqkdEyBSnKk3yn8NrMmjJpa8yQsydYHZyu0By4eypHVNVu0Cu1hwA2MWVB/7I2bc1JkrxQCA5fuyJMUgxuHsAny65qjeY468uB7H2hBZhgkQKe66kSZ0ANim4I7S9khOVJj/yGba/w5i/ZHqs79uj59Swwy1ylJPXMFDs7fhcLZ1Y9KcldRflyMnleQamAC5POd4kzqfex3HL0of8Lvp6CUcyS6wQUR3VJ0iL/cVlzuJkivxu2RiXNZHKw8j/oMUXCwwvT2JvWigwfDvt2PnmWt4MnmnXc+9NOM83vhrn8ULGqqxFZHIETABIqfx+07j21wYciS7ACPnp1k3VdzMzefkpUI8/aP+DVWv5cNBbl4WtdaYOOSbDSdwsaAEczaYXsRPictzuVD+AfWmvLgoA7/uOIu/VLqez21MtMjZMAEip5EqcXPRoznWt/yYuyf8s9/0Ks6OMlD0sIhWsvSz1/R+VlMXiLn8TQ2xXi0yvpimI3KUv21yXUyAXJzKhmFYRepr2V9pK46PVx22+vxT/z6AmWuPmi3niJf8szXmX9fyvfIPWN5xyv6D4B3x92N3Cl4ktkSRXDyUDoBILqcui19wLyvvOr7deKf7ZdZ642vpmOr+qfxmnLztNADgpcSm+Pf0Vaw9mIOS8urjOuSeBSbnDaGwxPbr/1SeBWavlhfeNK0ntUXn7NViSeUv5t9A2umrSIoNg4c7P5uT7TEBIqdRaGRJfkOkrE781A+GB8Weu1Zs8KZwMf8Ghs5JFVW3Wu7L/56+ig7vrbXL+Be1zf6qytDvxB4hq/uq2D5ZTfxsI/JvlGNy/+Z4tmdjm57LEmr/uyXpmGa7uPBgH6VDUL2UwxcNPr5yf7bBloWTZlqi5L+RWJ9GHc0plD35EXO/UMs4EbXE4cryb9z8ALP+iOH/NyK5MQFyca/1ba50CIowtoFjVeeuSWvGF0X2D5KO+8n0djJ4Mf8GEj/biLmbTc8Ks3UcN7+vTs1rQtkreWOSSM6GCZCLC/LzRLsGwUqHYXfP/JQuqlz/mZuNPqfROPYtwZKd7G3l87VHcfxiId6TaY81sh/H/i8gV8YEiMiISwUlKJAwrkgswcj3lrt5Ayqr0EqaSq2mMQ2GBosrRanbuXp+G0SugQkQuRxju1dXZW7tm/zrZSgqrV6X2TVnbHSnG/jlZtz17hqcvWKDbrtKcvJv4Jcd5jcpNfYyLW15Kq/QYt+5PEV2qVdRrqgK+km8a18crVZQ1YcJEo8JELmclxZlyFLP7I3Gp86bopX5zfJ2PnE05+ZWIKsOmF588c5xliUiD3yzDW/+td9sOblvCm/+tR+Dvt6Cj1cdkbXeqngrM4I9XdWUV2jRd+YmPPLddqVDIQswASKXs1JkgmBOqZFuG3OfiCsnQFJzhB0nr2DonG0md2o3d/5z14rR59MN+FVEK44h53OvW3TcbZYmRr/d2upkjoWJpzU2H7uEd5buF916aAnV5xd2ygwdaUzR0ZxCHL9YqMiCnWQ9rgNEqOPvrXQILkVrxXCXYbc+aY5Z8K/Fdby37BBOSFgHydUYuv3e3tU+NMgHz/WKQYVWwKnLhWhcx190S1qFVsD2k1fQql4Qgnw9qz3Plici+2ILEOHdIa3Qq1kdzB/dQelQXEJFpRaQCq2Anaelf3q0Zs2eknLbtWIo6XB2Pp77JR3HL1q/x5sx56/dbP1q/MYKJH62CXM3nxJ97E+ppzFi7g4MnbPNVuHZlkoaZlQSBjkBtgARQgN9kDymk9JhOA8zH+Urd4ENmbUV2fk3ZD2FWsZj2juMh2anorCkHOlnrmHHG4kAgMyrxfD3lvY2J+DmwFZjDTuVuz7fX3EIT/eIFlXvXxkXANwZq2XIjbIKHM4uQFy9ILi5OcatXu4uK7UOqlZnVGQNJkCkJzTQGzn5tt8OwZmZ2wNJW2kWkyXJDwC9mVBVbz9qf6PW2wtMxmBvb4Vy++/3Yv4NdP9oPQBg2fPdRNdToRXQb+YmRNbyM/i8pYPYxYx9euandGw6eglvDWyBp7qLS6yqyrxajBreHqhVw8ui49VO7X/f5DgU7QLbtGkTBg0ahIiICGg0GixZssRk+dGjR99cfK7KV2xsrK7M1KlTqz3fvLlrrnZsifWv9sLaiT0MPlfTr/q4Baru9T/3mXxegVncdmePqermEor9F/IsqrdCK+DYxUKsM7IFiqXEJE6bjt4ca3R7Y10xKrdUXSooQfeP1uOud9dIDU+Syi9FrS02ROYomgAVFRWhTZs2mDVrlqjyX3zxBbKysnRfmZmZqFWrFoYOHapXLjY2Vq/cli1bbBG+U/Lz8kCjEH+Dz7k7SJO82smdHFwsKDE6I00JszecQOw7K3HwgvGZarfpbt6VLompGW63Xcy/gfgPUvDxqsOiYsq/UWby+Y23Eg9bqrDDr+hwtvlrZ4ogCMa3ieEsMJO0WgEXrJwhSfalaBdY//790b9/f9Hlg4KCEBQUpPt5yZIluHbtGsaMGaNXzsPDA2FhYbLFSbc55huT2tjiE/Pgr+8k+UqPAfrPStNJiWBmGYD+X+hvP1KhFfDP/iy9xzp9kAIAmLX+BF7rZ7iFt/KN9LXf95qM6aTIWXECLL++Uqb/K7VLyZM/7MS6wxex4dVeiAqpYbSckruoKHVqc+d9ftFuLN+bhdkj7kL/1uF2iYms49CzwObNm4fExEQ0bNhQ7/Fjx44hIiIC0dHRGDFiBM6ePWuynpKSEuTn5+t9uTKmObZliwTF3KrVlVm6AOKnq224AKGJkBamncWEX3dbVb21axdJVVhSjpwq47vkXgDTFm53+91ec8kYB3gpsjP3kpfvvZmkW7pAKtmfwyZAFy5cwD///IOnnnpK7/H4+HgkJydj5cqVmD17Nk6dOoXu3bujoMD4DWLGjBm61qWgoCBERkbaOnxyYVOWml9FuSope3yZY+lChF+tO45z10wP8BbT7VUusQtwy7HLksrr2CCT10Bc68dd09cg/oMUvSTIWZMGR+2yIhLVBfb3339Lrviee+6Br6+v5OPE+uGHHxAcHIwhQ4boPV65Sy0uLg7x8fFo2LAhFi9ejCeffNJgXZMnT8bEiRN1P+fn57t0EqSiTcKd0q6zuZKPefm3DPzwhLilCsx1sV2yYg2hG2WmB7IM+HKzyecB4Llfdll8fkdRemvAz64z13TdIRUyZEA3yirw2Zqj6NO8rtV1meKsyRpRZaISoKpJhjkajQbHjh1DdLRl0zjNEQQB8+fPx+OPPw4vL9NTPYODg9G0aVMcP37caBlvb294e3M1ZFKv7SevSCpvaqD1/vMq7OI1ccN15IRcb9NQCUmFsVaVeVtO4btNJ/HdppPWBaaQw9n5WLDlNF5MbIKIYNt9QFYSk0fHIboLLDs7G1qtVtSXn5/h9TPksnHjRhw/ftxoi05lhYWFOHHiBMLDOShNLEvHiJDtSPmVCAJwzIarIZNpxaWGZ1HJsTnsqcv22cLE3N+bpQP5+3+xGb/tzMT4Xy1vBaz8/mSP5Rbo5pi2tFNX9dYwcwaiEqBRo0ZJ6s567LHHEBgYaLZcYWEhMjIykJGRAQA4deoUMjIydIOWJ0+ejJEjR1Y7bt68eYiPj0erVq2qPffqq69i48aNOH36NLZt24b7778f7u7uGD58uOj4yTDmRcox1/VkP/K/ARYam3ZtJVv8uYp59fG3ZqhZcqyaGUt6pCRDt3PAw1mWJ+iVE8lmb/1jlyUMbp7YPqdRo+HfbcfD36bip+2WbaCsVqISoAULFiAgIEB0pbNnz0ZISIjZcjt37kS7du3Qrl07AMDEiRPRrl07TJkyBQCQlZVVbQZXXl4e/vjjD6OtP+fOncPw4cPRrFkzPPzww6hduza2b9+OOnXqiI6fSEnGPtWqYQ8vuZv3D2blo9U7q/Dn7vPyVqygghu2SeiMqbwGlFy/H3P1qGXgc7lWwNif0pUOQ7LCknKM/Skdy/be3B7l1OUiDP9uO7YeNzzgv0IrICMzF2X2WEzKgH3nby4q+seuc4qc31YkrwO0YMECDBs2TJZurl69eplsFk5OTq72WFBQEIqLjc9EWbRokdVxESnplx1nMDIhqtrj+dfte2O1h7RT5jeCNdfyuPvsNTQNrf4BzZG7cqWEfvJyEcoqtPB0t9+kXmvXsnL11aNnbziOlQeysfJANu6Ni8CEX3fhwIV8pJ68gtMfDqxW/j8rD+O7TScxtH19fDy0jQIROyfJ/zGvv/46wsLC8OSTT2LbNgfd1ZhIxaYsPWDwk57Ym8YBC7eAcFT3f7MND3xjv/ciY5/Z0s9cw/T/HaxWNvNqMVJPSBvELtXt6fZy5XxS6lG6NchuyZSML/NKof6yFhcLTM/MvD3o/fd052qBUZrkFqDz58/jf//7H5KTk9GrVy9ER0djzJgxGDVqFFdfdnKO+3na8Vgzy2fFvmwMsNFKtGr93H4kR/qYEksGJWsATF92wOBzD842nITd3pDVkXAmk+VcvXXLkUhuAfLw8MD999+PpUuXIjMzE08//TR++eUXNGjQAIMHD8bSpUuh1aplwCaRY1p1INuq43dbsNaQq/l+s/QkMye/BAvTTK+S7ArskSBptQKe+yUdn1VZgVyxrk0ZX7MD9846Fas6jUNDQ9GtWzckJCTAzc0N+/btw6hRo9C4cWNs2LBBphCJXI+12ybM23JKpkj0KdEyYGkXi7mjvl5nfG0wY2w5CLXyqtGufH/UaIAzV4oQ/cYKrNiXjS8t+D0pyZK/V1f+fSvJogQoJycHn3zyCWJjY9GrVy/k5+dj2bJlOHXqFM6fP4+HH34Yo0aNkjtWIpdxMd/AmAC2rCtOamIqpTtkwdbTEqMxLf9Gmaz1GXvptujyeX/5IdnrtBd2gTkOyQnQoEGDEBkZieTkZDz99NM4f/48Fi5ciMTERABAjRo18MorryAzk83EzuCpbo0AAG/f21LhSFyLuUGRSnGkN3fzi/k5D0PdQiv3W9eNavp8NqsaggAotd7etaJSnLlij8UmHbPNx9nGhkkeBF23bl1s3LgRCQkJRsvUqVMHp07Zpgme7GtgXDhe6dsMvl7u+Ja7HJOLk3oDUPSGUeXcZ68UQ4CAhrVrWF91pbot6fIxu86QQvlBu3fXAAC2vn436lXZqsNeyX9puRbnrhUjuo6/Xc5nToHMLYlqIjkBmjdvntkyGo0GDRs2tCggUhdPdzf4erkD4MA9Au79cov9T+qCf3fGBvpaeilKyivQ4+Obs9EOv5sEH093C2tSH7mSzH9P31mTam9mbrUEyF7xPDZ3B9JOX8W3j7dHv1jlZ1aPnJ+m+97Z7gEWjQFKSUnBvffei8aNG6Nx48a49957sXbtWrljIwWN69UY97erh9gI81uakH2oofW5XIG+CTW95yp9A7D06hdWWp3a3itVW0KJyzy60o3+4yozz2yt8t9V2q1E7NcdZ42Uti9nnlEqOQH65ptvkJSUhICAALz44ot48cUXERgYiAEDBmDWrFm2iJEU8H9JzfH5sLYOvZqus8m77rxN0bZgiwX6DH26V2p7gsoMvVI59u4yxpZpsEZj6zFGAk5cKqy2sWfl5P7kJXHjgAxdS2d+y3T5MUAffPABPv/8c0yYMEH32AsvvICuXbvigw8+wPjx42UNkIhucsQ9j5yNoVlgJeXyb0NhbNd3Je+tar33SY1r9sYT+GjlETzWuQHeG9Ja9HFyJtTOnCQ5Esn/tbm5uUhKSqr2eN++fZGX51pL8BPZ00kjN0UyzOxNxoI7utKfgKWcXo5YC0vKUVhSvcvM1vdvW26v8dHKm91bP2+X1sVksLXHQJyCAMzdfBIfrzqM7zed1Nus1tE5W+ImOQEaPHgw/vrrr2qPL126FPfee68sQZFjeL1/c6VDIBfAbljxjCU9llzDsgotWr2zCq3eWYVyFXTzrTucg1d/34PiUv2ETK6/jqqXrshA4ifGsYuFeG/5IcxafwLvrziEBVs5I1qtJHeBtWzZEu+//z42bNigmwq/fft2bN26Fa+88gq+/PJLXdkXXnhBvkhJcVU/7TzcIRIf/nNYoWiI7M9wK4Cp8vKyVyooQNDbsLOopKLK89KVlN+pw9TxgmC4peGJ5J0AqncPGqpLEASsOZiDVvWCEGHhbK6nfthp0XFVW3z2X8ivVkbM75F5v+1ZNA2+Zs2aOHjwIA4evLPzcXBwsN4UeY1GwwSIiFSrwMJP+FVZkgzsycxFm8hgWc5fWeWbprXJl5zr3uTfKMN3G8XtvZZ68orJ59PPXDNbx9KMC3jptwwAwOkPB4o6b9V8w1wcgOXXqOpRxrrSyLYkJ0Bc4JCIHIEtPkBLXwjR8AH3zdpq9sYsCILorisprQX7zuWhTwsf0/VVvXoaWJVRvbBwN7LzbLvXWebVYpRWaNG4jj+2HL9sgzPYjiOtsG6NbScu4/tNJzH9vlaIrOWndDjWbYYqCILRf3AiIjmoqSfA0LudmuIT40kRXTvmbshS3/c3HLmkl8zJete4VVn3j9ajz6cbqy0XIQgC/t5zAScuFVp3GgNB22qjXkuVVWix6kA2rhWVmi8sws5Ki0PK4dHvd2D9kUuYuDhD1notZVEC9OOPP6J169bw9fWFr68v4uLi8NNPP8kdG6kM+6TJoajg77Vyq4dUt2+4xtatMXWMLVg6KNjesvKu6/28+mAOXli4G30+3WjyOCU/yhtKpCx5v52z4QSe/SkdD87ZJkNUwMGs6uOX5JBlxf+FnCQnQJ999hnGjRuHAQMGYPHixVi8eDGSkpIwduxYfP7557aIkVRiwt0xej+r4P5CZFc5+dLeuGfIMEngi5Rj6PPpRrwncYd0a5OhqsdPX3ZnzKcls8ps9n5RpeKqcWdk5trqzBbTG6slY9a6bG8WAPELObo6yWOAvvrqK8yePRsjR47UPTZ48GDExsZi6tSpePnll2UNkNRjRHxDpJ++hj93nweg3oXRiGzlzJXiao8t+jdTb6sJudz+/5q59hgAYP7WUxjavr7JY4zlJdYmHxoNsPHopTuxGblpnxTbzSTnm0eVuqqGJlfiZejayjF2Z/MxxxqvdFt5hRajFqShRVgg3rq3pdLhWERyC1BWVha6dOlS7fEuXbogKytLlqBIvYa0q2f0ueg61u8yTVSVJV0BN8oqsP7wRfmDMeDdZQfx+dqjdjmX2j33yy5FzitHK4pSLdqXC0tkGV4gCAKO5BRYX5FIm49dxtbjVzB3i+NOjJKcAMXExGDx4sXVHv/tt9/QpEkTWYIi9erRtA6+e7w91r/aq9obhpe7G+aO7IBPh7ZRJDZyTpbc25q/vRLfb1b3G7O5MTWGbuq/p58zWHbl/mxczL+BNQdz7hxvZevE3nO5VQIyf0y2iS5Ce44hrHyqyi1X1pB3ELT8F2PVgRzzhWSkhj3wrCW5C2zatGkYNmwYNm3ahK5duwIAtm7dipSUFIOJETmfvrFhAICrBmYaJLYMRVFJOV75fY+9wyIntfmYPDcwtYl9ZxUOTa++rZAlpv3vIKb976D5giJptQLG/nynNUf227WMFRpK9Co/csDAQoSG67ENKRvVWsOR/k/UMqFGcgvQgw8+iLS0NISEhGDJkiVYsmQJQkJCkJaWhvvvv98WMRKRC7tWXGa+kINqMWWl0efkvEVKveGUm5lxZnTLDbEnkPjiHpu7w3R1Kh6QaCi0ywXyTFNXkjNsUSOpBaisrAzPPvss3n77bfz888+2iokchOP/+ROp14i5O9DOitWiBeFmN4VWEGRZGNDaVovK98tSid0nUuIXIKj6valCK2DlgWylw7CaM6wBKKkFyNPTE3/88YetYiEiolvSTl3Ft5vEbR9hiACg23/Woe20NVhya+amLVRObKS0Cny2xjUHjlfeEw24tfeZDPU6QYOM3UnuAhsyZAiWLFlig1CIiEgugiAgJ78E18sqsPaQdTPi5OjuqDrw98uUY1bXKSdzr1CuBg+x9agxn6kcu8t1gQFAkyZNMH36dGzduhXt27dHjRr6U5+5AarrqPp/fPufwwn+L4hcwk+pp/Fj6hn8+GQnhAdZtmt6Vt4NvLfsIB5PaGiy3L7zeRbVL4YcuYk9B0HrPW+kgON3MKmfRbvBBwcHIz09Henp6XrPcQd4AtQ9IJGI7nh76QEAwEcrj+DzYW11j5v7Hy4pvzOGZ2HaWQDA//ZekD0+MarGaqv3n+tlFZI2qBVLbe+X/+zLwhcpx/D1o+2qPedsH265GzxZrOr/wsS+TRWJg4iqk3JjrTouxdyNbt6W6mOTcvJLUKuGl/iT2pAtbtTPL9yNrccv48MH4yQdp7L8xqxxtxazfGFhBh7pFGmTc9hiHSRLSB4DNH36dBQXV18O/vr165g+fbosQZHj+efF7uh3a30gZ/uUQOTsxNyQKidUR3MMb3mh1L9+1RlJtmpVWfRvpv55RKQ35mZLyfV+KXdSUVxafaFOUy+lvEIremaYLdZBsoTkBGjatGkoLKz+x19cXIxp06bJEhQ5ntoq+eRHRDdZM025ereSOm5YjsjcpRMEwwOKjaUzTyb/i9xiedcRyr9RhjUHc1BabtnqzgU3ytDpgxSM+1mZrVAsJTkBMtYHumfPHtSqVUuWoMix+Xq6o0/zutUej2/Evw8iVZKp8aBCJYmSvVqhxbS6LN9nfo9MKQlmyuGLsi8h8MSCf/H0jzvx8arDuscMJmWVlzyo9PiqAzm4WlQqen0jtXSBiR4DVLNmTWg0Gmg0GjRt2lTv4lRUVKCwsBBjx461SZCkTt6ed/JnHy933fcajQbzRnfEZ2uO6k119fF0BxGpT0mZFscv3tlIs+q97/jFQlGdFrkKrdptTdpVoRUwc+1Ri1s/xNh3Lg+t6wcBkGdM0BUD2xBZY+eZawCAP3bZbr0oNRKdAM2cOROCIOCJJ57AtGnTEBQUpHvOy8sLUVFRSEhIsEmQpE5+Xh74ZsRdqNAKCPTxVDocIqrE1I3209VH4FvpQ8vaQzlYe6jSRqpVDl68U38TVjV1iQkAfth2+s7PEkP7e895fLXuuKwxVVVQYjoxdJQ1dYxdWzX9PUghOgEaNWoUAKBRo0bo0qULPD15wyNgQOtw409W+adwkP9xIqdg6p4k9Yav1epvL6Gm212FVsB7yw9ZfHzm1esWHyt6MK8dLpgt3l+d/S1b8jT4nj17QqvV4ujRo7h48SK0Wv1mwx49esgWHDmvSUnN8NHKI0qHQeS05Lzn/rZTf/bT7rO5MtauHEEQnHpLDkEQsOdcHqLr1BDVSq8x8r3uMSMZkaO0YFUlOQHavn07Hn30UZw5c6Zas5dGo0FFRYWRI8nVNQsLwIYjlwCob/EvInIOUqZYWzOW5q0l+7Dz9DVxhWXODwwmJwYeW30wB8/+lI56wb7Y+vrdZuuV8rbsoDmPHskJ0NixY9GhQwcsX74c4eHhDpv5kX1NHdQSj3VuiC3HLqtmsTQicm2WfhArLdfi5+1nJZyo8jn1T5qVdx3nc6V1w524VGTqFDorbs1AM1T/T9vPIK+4FBPubmL4JAZu7S47Bui2Y8eO4b///S9iYmJsEQ85kcr/EqO7NgIALHu+GwBg9sYTCkRE5Doc4aaUcigHSzMu4P37WylyfksX5JNzIb9PVhvugjPVuHAoKx+pJ64goXFto2Vy8m+YPO/bS/YDAAa1ibhzTpNH3LFyfzZeWLhbZOnq1NJuIjkBio+Px/Hjx5kAkUXYYkhkH4eyCswXUtiTP+wEANQN8Ja1XluvM2NNbin2UHMJ7LK9F/QSoKqv+HORY5uKSsQPW7n99j325+r7gEqhltxc8kKIzz//PF555RUkJycjPT0de/fu1fuSYtOmTRg0aBAiIiKg0WiwZMkSk+U3bNigW4uo8ld2tv7iS7NmzUJUVBR8fHwQHx+PtLQ0qS+TiMih/bHrnPlCRiRXmlZuD9lmWiukEATgmtiVku10I/7PqiMY+OVmXC+Vb4ysudDLtfbLMhyhtdEQyS1ADz74IADgiSee0D2m0Wh0K0RLGQRdVFSENm3a4IknnsADDzwg+rgjR44gMDBQ93PdundWHf7tt98wceJEzJkzB/Hx8Zg5cyb69euHI0eO6JUjIiLnM/TbVJsuamiJPZm5AID/7jqHwZW6nORUtRXG2pzEUJuOXHmOWjoCFN0Nvn///ujfv7/k4+rWrYvg4GCDz3322Wd4+umnMWbMGADAnDlzsHz5csyfPx+vv/66wWNKSkpQUlKi+zk/P19yTFSdqX+We1qEcho8EQGQtyFGbclPZeUVWnUtouTiJCdADRs2tEUckrRt2xYlJSVo1aoVpk6diq5duwIASktLkZ6ejsmTJ+vKurm5ITExEampqUbrmzFjBjdytbO6AT5Kh0BEpFrWjpcUO1BbLTuzK0HyGCAA+Omnn9C1a1dERETgzJkzAG5ulbF06VJZg6sqPDwcc+bMwR9//IE//vgDkZGR6NWrF3bturkD7eXLl1FRUYHQ0FC940JDQ6uNE6ps8uTJyMvL031lZmYaLUtERDJT6B5sx2EyAIATlwptknBotQJOXq4+NV5uaum6kovkBGj27NmYOHEiBgwYgNzcXN2Yn+DgYMycOVPu+PQ0a9YMzz77LNq3b48uXbpg/vz56NKlCz7//HOr6vX29kZgYKDeFxEROTcxO7XL6eftZ1Fhg6zr3eUHsenoJZNlen68HpcLS0yW0dvt3UC246BjnY2SnAB99dVX+P777/Hmm2/C3f3OZnodOnTAvn37ZA1OjE6dOuH48Zv72oSEhMDd3R05OTl6ZXJychAWFmb32FydKzetEpH6/Xvqqt3PaYvZWQu2njZb5syVYszeYHr9NXslOGppSJKcAJ06dQrt2rWr9ri3tzeKimzfBFdVRkYGwsNvbsjp5eWF9u3bIyUlRfe8VqtFSkoKd6pXGSZHROSK7NaKYuA8tmh9MqW4tBzrD1/EjTJ1bpEleRB0o0aNkJGRUW0w9MqVK9GiRQtJdRUWFupab4CbyVVGRgZq1aqFBg0aYPLkyTh//jx+/PFHADfHGTVq1AixsbG4ceMG5s6di3Xr1mH16tW6OiZOnIhRo0ahQ4cO6NSpE2bOnImioiLdrDAiIlIXR/tAZN1CiOp9rebG+EgdA/TCwgysPZSDRzpG4sMH43SPn75SbEF08pOcAE2cOBHjx4/HjRs3IAgC0tLSsHDhQsyYMQNz586VVNfOnTvRu3dvvboBYNSoUUhOTkZWVhbOnr2z30ppaSleeeUVnD9/Hn5+foiLi8PatWv16hg2bBguXbqEKVOmIDs7G23btsXKlSurDYwmIiLXZmky8sGKQxafU0ojzKWCEoT4G9478XaritZIhdeKS7H+iOFxQZYuXLj3XJ6k8msP3RyOsujfTL0ECADyrpchyNf8DvW2JDkBeuqpp+Dr64u33noLxcXFePTRRxEREYEvvvgCjzzyiKS6evXqZfIXkZycrPfzpEmTMGnSJLP1TpgwARMmTJAUC8nP31vZP24icgxKDa615LyXCkrw0/Yz8gdTxbrDF9Hx/bXo1KiWwef/3HUeH9zf2uh6asaSH7FsPU7nUsENx0uAAGDEiBEYMWIEiouLUVhYyBWWyaBRXRpi24nL6NuSrW9E5BzKKqxbaFFq60uaiYHaT/2wE1uOXxZdV1FJuYF47nx/uVDkFiJOwqJ1gG7z8/Nj8kNG+Xl54Kcn4/F4QlS155xtOiURWU6p9wN7rJ1TlZyvVUryAwC/p58THYOU8T6WvCQ13ANEJUB33XUXrl27JrrSbt264fz58xYHRa4nPIgrQxORfR2/WKh0CIpztsUNpRDVBZaRkYE9e/agVi3DfZGGylfeW4vInAAfD6x44R60e3eN7rFOjWqZbP4lIueg5plRVd0e2GspNbR8VGYsnqIS8VPXLcmh1JB4iR4D1KdPH9F9l9buYUKuqWYNw7MdiIjUYsrSA1Yd7yjJ3vnc68i7XiaqrKN2gYlKgCzZAb5+/fqSjyHXIeZvn2k0ETkbNdz4xYbw72nxQ1/MeeCbrbLVJRdRCZAadoAnIiKyxoXc60qHAK0aMiAF7Dqbq/ezGjqKrJoFRiQXjYXtPYPbRBh8fMcbfawJh4jsyF45wZgF/9rnRCYonf4s3pmJxm+ssPl5SstNLxeghjyQCRCpipeHtD9JY/9DoYGcVUZE+o7kFCgdguIm/Xev6LLW5CifrDa8QKOaMAEiRRgbUL/77Xtscr4+zbleFZFaqaAxwG7U0PIh1qaj4laTXrU/u9pjf+46J3c4smMCRKpwuz+4hrfxYWlrJ/bAq32b6h8nou6vH22HeaM7WhEdEdmSIyUF1nO+F5ty+KLSIVhEcgKUmZmJc+fuZHZpaWl46aWX8N1338kaGFFVMXUDMOHuJpKPc603VyJSM1d5P3KEbTUkJ0CPPvoo1q9fDwDIzs7GPffcg7S0NLz55puYPn267AESERE5Cym7wZNtSU6A9u/fj06dOgEAFi9ejFatWmHbtm345Zdfqu3eTmSMmxrmQFohOqSG0iEQORHXyQocZSFEVyA5ASorK4O3tzcAYO3atRg8eDAAoHnz5sjKypI3OnJaNWt44d64cLucyxZvN2sm9rRBrUTk7LTWbSZPMpKcAMXGxmLOnDnYvHkz1qxZg6SkJADAhQsXULt2bdkDJOf19aN36b53tO1T3N0cK14iNXOVcTEAMODLzUqHoGd3Zq7SIShGcgL0n//8B99++y169eqF4cOHo02bNgCAv//+W9c1RkREROr39pL9SoegGNGbod7Wq1cvXL58Gfn5+ahZs6bu8WeeeQZ+fn6yBkeuTa5GITGb+NYL9sV5A8vk16rhhatF0mcz9IsNxaoD1u0aTUSkNnucqMVIcgvQ9evXUVJSokt+zpw5g5kzZ+LIkSOoW5eLzZFlDK0AHRHkCx/Pm483qetvcd2e7vp13xsXjhB//Z3nPdwNZ1ufPdxG8vkCvD0Q4OMp+TgiV+Wo68i4ovtmqW9TU0tJToDuu+8+/PjjjwCA3NxcxMfH49NPP8WQIUMwe/Zs2QMk5/bxQ3GoX9MXnzwUp3vs5yfjcW9cON4c2AJLx3fDA3fVw7xR1RcyfKh9fb2fH41voPfzc70aIyG6Nu5pGar3eHiQD9a92kvvsciahlsvfT3dpbwcm1oyvqvSIRARyUINw74kd4Ht2rULn3/+OQDgv//9L0JDQ7F792788ccfmDJlCsaNGyd7kOS8hnaIxNAOkXqPdWsSgm5NQgAAtf298dnDbQ0eW7Vny69KsjIpqbnB4zQaDQIrtdA82yMaBy7kGy0rla3+sQN9JP+7EhGREZJbgIqLixEQEAAAWL16NR544AG4ubmhc+fOOHPmjOwBElX1Yp8mqF3DCy/f00RvnJCliYep4+LqB6FJXX/0bFpH8katpjQPC5CtLiIitZj2vwPIzrthtpwa5tFKfkePiYnBkiVLkJmZiVWrVqFv374AgIsXLyIwMFD2AImqevmepvj3zUTUr+kn2/TZtpHBBh/3cnfDqpd6IHlMRwT5yjeuZ1CbCNnqIiJSiwVbT6PzjBSz5dTQBSY5AZoyZQpeffVVREVFoVOnTkhISABwszWoXbt2sgdIZIibgXV4rPlEMeHuGEzur99l9se4LnBz08DNTWOwK+yZHtEG6xIz68zTyKBrIiKyD8kJ0EMPPYSzZ89i586dWLVqle7xPn366MYGEdlL4zrSZ4dVTT00AHw83fFsz8Z6j7dvWLNaucreGNACf4zrUq3+pFb2WeGaiIgsZ9GoyrCwMISFhel2ha9fvz4XQSRFPNszGkWl5binZShW7c+2qA5rmmJjDEzPn35fLN75+4AVtVb31XC2rhIRyUlyC5BWq8X06dMRFBSEhg0bomHDhggODsa7774LLTc5ITvz8XTHGwNaoGNULXSOFrkVi4W9T4YSparjglqEB6KGt4fZsUkaiUEMaM1WJSIiOUluAXrzzTcxb948fPjhh+ja9ea6JFu2bMHUqVNx48YNvP/++7IHSSRGnxZ1sWB0RzS10QwrMSlL91vT983WJTEJczcyDomIiCwjuQXohx9+wNy5czFu3DjExcUhLi4Ozz33HL7//nskJyfbIEQicTQaDXo3r4t6wb4W1/HmgBYAbk61r16/xdWa9cMT5ruQxQyuJiIicSS3AF29ehXNm1dfYK558+a4evWqLEERKeXpHtEY3DYCdQO8qz0nZ/6R1CoM7y0/pPvZVG6V2CLUxLNERGQJyS1Abdq0wddff13t8a+//lq3MzyRIwsN9LF5d1P9mn7ImHKPqLIf3domhF1gtlV1yxQish01NGhLbgH66KOPMHDgQKxdu1a3BlBqaioyMzOxYsUK2QMkkpvUAchS1PGv3nJkTLDfnQ1ZTb0X3I6WXWC2FSLhd0dEjk9yC1DPnj1x9OhR3H///cjNzUVubi4eeOABHDlyBN27d7dFjESysrQhRcxxjyc0BAAIEifXM7lRA/4OiOxFDQ3aFq0DFBERUW2217lz5/DMM8/gu+++kyUwIltpVLuGTeoN8PaAj4W7xwf4mN9mg11gtsVVPIjsRw2f+WTb3fHKlSuYN2+eXNURyW7xswl45Z6meLB9fb3HPQxsq2EvHz7QGs/2iMZdDYIVi+HeOK4xBEhvtSMixybf9tZEKtepUS0836cJ3G8lPK/1a4amof5G9/SqSvRCixI80qkBJg9oIap1x9/bogZbsz4ZyskLgDo+kRK5CjU0aDMBIpc1vncMVr/cU28wsik+Hoa7t+5vVw8AMK53Y4PPj7w1LshadQxMzZeDu5sGNf3k2+neUTH/IbIfNXzgsM1HSiIX8vFDcXi2ZzSahcqzAnXdAG9cLCiRpS4xVPBBTBXU8IZMRPYjOgF64IEHTD6fm5trbSxEDsnD3Q3NwwKNPi/1xipX0/BbA1voLbZYVcPafph4T1N4uLvJ0vqx552+aDNttQw1KYMz8Yhci+gusKCgIJNfDRs2xMiRI20ZKxGJ9FjnBmgZYTwpA4Ch7evjvrY3u+/6tQyz+pxBvp6YlNTM6nqUElnLT+kQiMiORLcALViwQPaTb9q0CR9//DHS09ORlZWFv/76C0OGDDFa/s8//8Ts2bORkZGBkpISxMbGYurUqejXr5+uzNSpUzFt2jS945o1a4bDhw/LHj+RGFJnF8mxUGMNiQOm3xncEnc1DMb//bHPqvM+1ysGI+IbOmRL0NiejfFFyjGlwyAiO1F0EHRRURHatGmDWbNmiSq/adMm3HPPPVixYgXS09PRu3dvDBo0CLt379YrFxsbi6ysLN3Xli1bbBE+uRhJXVOVch6xPSuv9bvZevLBA60knMiw2wOzTakcl5+XB4Z1bGD1eYGbLUHdYkJkqcteAn084Otl2RpOROSYFB0E3b9/f/Tv3190+ZkzZ+r9/MEHH2Dp0qX43//+h3bt2uke9/DwQFiY+Cb9kpISlJTcGXSan58v+lhyHXINEenRtI7Bx8f3jsFT3RtZ1QI0qE0E3hnUEiH+3th24rLJsrYc8fLDE51w/GIh+s3cpHusQS0/nL1abHXdj8Y3wK87zlpdT2VcZJLI9Tj0NHitVouCggLUqlVL7/Fjx44hIiIC0dHRGDFiBM6eNf1mOWPGDL3xTJGRkbYMm1xM5USjQ8Oa+H5ke6NlvY1Mta+qexPDLSwebhpV7Gnl7qZBoK/+5ytvD3nebgJ85P/cxvyHyL6KS8uVDsGxE6BPPvkEhYWFePjhh3WPxcfHIzk5GStXrsTs2bNx6tQpdO/eHQUFBUbrmTx5MvLy8nRfmZmZ9gifHIwcN8lW9YJEJzmmBBrZOkPN93G5kgxbTNZS83Ujckb3f7MNZ64UKRqDwyZAv/76K6ZNm4bFixejbt26usf79++PoUOHIi4uDv369cOKFSuQm5uLxYsXG63L29sbgYGBel9E1qh8j5bzhv3B/a3Rp3ldjOhs/Xgde8/6FtO1J6aVKMiXizYSOYOFaco2NjhkArRo0SI89dRTWLx4MRITE02WDQ4ORtOmTXH8+HE7RUdUlXyZxqPxDTBvdEf4Gtt01cGbMtLeMP3/DABdY0IwpmsUhnVgVzWRIzP6PmYnDpcALVy4EGPGjMHChQsxcOBAs+ULCwtx4sQJhIdzw0eyztiejaHR3Fxjx5ZsNR7llXuaYngneZKGsEAfWeqpSsySARoA7wyKxX8eipPtvBwETWR/Sv/bKToLrLCwUK9l5tSpU8jIyECtWrXQoEEDTJ48GefPn8ePP/4I4Ga316hRo/DFF18gPj4e2dnZAABfX18EBQUBAF599VUMGjQIDRs2xIULF/DOO+/A3d0dw4cPt/8LJKcSFVIDR9/rD093aZ8b1LLAcLOwADzfp4mu2dma3c+9PaV/dqr8Zuft4YaScq3F55cb0x8i16NoC9DOnTvRrl073RT2iRMnol27dpgyZQoAICsrS28G13fffYfy8nKMHz8e4eHhuq8XX3xRV+bcuXMYPnw4mjVrhocffhi1a9fG9u3bUaeO4anHRFJITX4A9SRAVcOwZjC2ta/J2OFi6rXF5VT6kygR2Z+iLUC9evUyuf9OcnKy3s8bNmwwW+eiRYusjIpI/Zoa2Xi18kDj2jUMT4ef3L85Vh3ItmqXemtaj0zRqiVbJCKn53BjgIgcjfStMMyr4e2BD+5vbbJMs7AAvH1vy2qPP9uzMf58rqvk7TLklDymoyz1vJzYVJZ62AlG5HqYABHZwIt9mtj8HI/GN0CjkBp6j1XtynmyWyObx2FM1cacylPcE6JrGz5G4jleTJTnOrMLjMj1MAEisoGoSomJoLcvmG27eOx1H7fkZUy7rxUa1PLDe0OM73WmVA8Y8x8i18MEiMjGbHlPV+rGLSZRqdqqElXbD5sm9cZjnY2PPRIg4Jen4q2MznLTBscqdm4isi8mQER2JGa9GWvWpNEq1IJiqKutapJUeYC2RqNB+lvVFz30cndD15gQnP7Q+BpftmxFq1nDy2Z1E5G6MAEisjF7dutUaI2vrWPLOHzErAtUJa+rXWXT1jcGNEewnzIJCMcAEbkeJkBEdmTrMUBlFfbJth64q56s9Q3rEIlnejSWtU4pxOxTRkTOhQkQkY1Zu2aOh7vxm/NT3aP1fm5dP8iqc932TI9oJETXNrrlxQt9mmD+6A6S6jTVylKhwOjn6Dp3Bqrfjs1eadDnw9rY6UxEZIyiCyESuQSJ9/bKN+HHOjdAgI/x3c+Hd4pEx6iaKNcK2HbiCh43McBYioTo2nhjQAs8+v12ZOffqPa8p7sb7m4eWinm6qlD1YTHVHIh9wKIL/Rpgi9Tjul+rlXDC1eLSvXKVN5V3t7tP2xxIlIeW4CIVGx0F9Pr+Gg0GjQJDUCL8EA82a0RvDyq/0vH1Q+CRgN0iTG89o4hDWr7SY61qmqDoE00AXlZsMWIKe5VzvXWwBaijrPlWKCH2tc3+lxXCb8bIpIHW4CIbExq24bcnUFLnuuK0gotfDzN7/217PluuFRQgsZ1/AHYb3BwLQmzr8QMlBa7q7zuezu80MpJWdX4/i+pOQZ/vdXmMRDRHWwBIrIxawY+y3FfdnPTiEp+AKBVvSD0bl5X9/O0wbGoVcPLbAuKmDgNFfng/tZIiK6Nsb3ED4Cuuvp1y/BA0cfayzgJrwdglxiREpgAEamMW6V7obFByPYSUzcA6W8lVhtsbQlDSdKj8Q2w8JnOCDQxzsmchU93xpiuUSbLuLuJSzDkSkReuNv0Fh3c85VIeUyAiGxM6r1Oo9Eg7c0+2Pb63YpuWFo5HlnqkZhcPNBO3FT7ID9PJMWGVXu8XrCv7vukVmFoExmMpyot2CgAGNwmAgAw4e4YSbGZY+iSWTsbkMjZKP1BQPl3VyIXIvb/vW6Asi0/atA+qib+3H3e4uMjgn1wPvc6AMDbwx1Lx3cFAMzdckpX5vNhbTHxnqZ6e7fJwVzOmNCYg56JlMYWICIbU/pTjj2IadtR42rL7m4a2ZMfwHxrV3iQr8nnicj2mAAR2Vjl/EeFOYDN1Pa3blsLWw8MNlS7XEmaRgNM7t9c77HuTeoAED8eie7oFhOidAjkhJgAEdmYGmcpyc5A5uDt4W6uiOgqh3WINFO2euXmWt5s2TDnptEgqZX+uKR748KxYExHpL5+t97jnRrVUmXrmJrw+pAtMAEispHlL3TD/yU119sp3QV6w2RT+Z7nbmI7EAAIqdLa9LCZhMnWDLcuadC7WV3UrTKzr1HtGi7RTWoNe6zTRK6Hg6CJbCQ2IgixEfLszeUMbNmlFV3HHx89GIfa/l7oGhMCH093i5LNyhHundoXXWesQ0FJufR6eL8mUj22ABGRXVjTBSbGwx0j0adFqOhFH80J9PE0Omjr+btjTI7lkdJiodHImzA93d309imOiPkk2QITICKyC6k3MVu1GPVsenMw8uguUeYLG2lGCvbz0k2rN3qoQt1adzWoqcyJbaiGtzxJLVFl7AIjIqvZJFWx0cf+uaM6IPNqMaJv7XdmCVPbm9QWua+Zv7cHCkvK0atZXfOFXVigD29TZBv8yyIiu5A6kNXa/MdYkuLp7mY0+ZGjK0psHRtf64WjOYXoHF0LBy7kW39iJ+Xmpqm2/xuRHJgAEZHVLN0M1VFpNBo0vpVE+Xm5o7i0QnIdtf29keDvLXdoTsddo8H43jG4XqrF/K2nzB9AJBLHABGRzbSNDNZ9L30QtBIpk7hzCoIAXy93HJjWD7vevsdgHZzZLg83Nw38vDwwZVBLpUMhJ8MEiMiOnG29l9sbjvZtWX0zUgB4c2ALe4ZjtY5RNwcQ1/QTtzt9DW8PUbPONk/qbVVcUjjZnxi4cDbZChMgIrJYyis9sXlSb7SMMLzatZ/XneTAmjFA9roH1vb3RsaUe5A6uY+s9YYG2nZz29uJmzMKE7lvmpeHG759vL3u5y7ccJbMYAJERKLcvqH0r7TFg4+nOyJr+Rk9xpoWr57Nbk5Xj65Tw6JWDUtPHeznJdtaQvaS1Cpc6RBsZmyPaFHlEqL1E543BjhW6yPZHxMgIjty5BWCZz/WHp8ObYOPh7axy/lC/L2xd2pfrHqph0XH307YPM1soyGH5++O0X1/+3dceRaaI//elTC2Z2Pd9z5eliWjztbdTPLjLDAiO3LkN+UgX0882L6+pGNahAeiRXhgtb26xAr0ETcWx5Dn726CsCBf9Lq18KGcmoQG6P38St9m+GrdcdnPI0aYjbvXlJDUKgxzNp6QfJyXBz/Tk3hMgIjIZtzdNFj+fDdFWkB8PN3xeOeGstY5rldjRIfUQI8mIUbLyPVS2zesifQz18yWWzqhK5bvzZLprLbVvUkINh+7bJO6Pdw06B4Tgm4xIWgRHsBWNwcgKDxknwkQEdmUmwzTeNRyL4urF4T+reUfb2PoZh1Z01dUAmTrAdZyWPRMZ2Rk5sLf20NyAiT2d+/upoGHuxt+fioeALD/fJ7EKMnVsL2QiEgkUQs+3h4DVPkxM8dY2zU6uG0EAKB3M/m7++TQObq23rgecyxJeD3sMNaL5GWr/f7EYgJEZEfx0bWUDsEhKdVQPndUB729qCxdnNHWizqG+HvjyHtJmD+6o14yFSxyPSM1E3vtvNz1b2eOPN6O7IMJEJEdbPm/3pjzWHsMtEH3CdlOfHRtZEzpq/tZShrjXunGbWrzVEDaLLGHjAxE9/Zwr5YsuNs48bqnZahN6m1U587eX63rBYk6ZlJSc5vEQs6LY4CI7KB+TT/Ur2l8vRxSr8pjmKS05DSs7YcBrcMQ4O0JD3f5Pms+2yMaJy4VYvfZXNnqtJTU9Epso0ygjyc2vtYLFVoBtWqIm0EYESxuwUSi25gAEZHqqWV0h5g4bo9r0Gg0+GZEezOlpUmKDUNMXcM72RuMRS0X7hYpqzM3rC1+B/gODZ13JWyyHXaBERGJ5CbiHdNWSUd0SA3Meby9QpvEGiZ1mE3jOv74fmQHWWNY/GyCbuYXkRRMgIiIRLLnrJXKiY6vpzuWTOhqSS3yBWSAJQONo2rL2xXcJjLI6q1L6gZ4yxQNORJFE6BNmzZh0KBBiIiIgEajwZIlS8wes2HDBtx1113w9vZGTEwMkpOTq5WZNWsWoqKi4OPjg/j4eKSlpckfPBHZjaeMY2isImYavA1O2yGqpuhVsZVeXK6q94a0smn9Sk+lJsel6LtKUVER2rRpg1mzZokqf+rUKQwcOBC9e/dGRkYGXnrpJTz11FNYtWqVrsxvv/2GiRMn4p133sGuXbvQpk0b9OvXDxcvXrTVyyAiG5twdwwa16mB1/srO9PHVrdae9/EW9ULtPjYwW0iRJdNHtMRj8m8GndVcvQIqqhXkexI0QSof//+eO+993D//feLKj9nzhw0atQIn376KVq0aIEJEybgoYcewueff64r89lnn+Hpp5/GmDFj0LJlS8yZMwd+fn6YP3++rV4GEdlYiL83Ul7pJWkxPVtQ0/gbS3WLCcGy57tbfHxSq7BKP0lvbbJX+5TaWsKoOqV/RyppVxYnNTUViYmJeo/169cPqampAIDS0lKkp6frlXFzc0NiYqKujCElJSXIz8/X+yIiqsrf2zYTZ6NvrXtTW+SUb3t510D3lZQU0NDtzdfK8TpVOUJKumS8JeO3yNYcKgHKzs5GaKj+wluhoaHIz8/H9evXcfnyZVRUVBgsk52dbbTeGTNmICgoSPcVGRlpk/iJyDFNHdQSo7tE4a4GwWbLWtJK5OPpjsPvJiF1ch9R5SOCxK15Y22D1aC46gt3SqkzysBU9shalg+CNrS5rdytcvVryr+eUNvIYKuO7xojfvkAR5J/vVzR8ztUAmQrkydPRl5enu4rMzNT6ZCISEVGd22EqYNjbdoF5uPpDi8PcW/JUwfHIik2DD8+0clm8QCGk4smoQGijv35yXg0ChG/lo8Yhrb2MPYbkTKuSg0DqYcaWeEbUEd8tjB/6ylFz+9QCyGGhYUhJydH77GcnBwEBgbC19cX7u7ucHd3N1gmLCwMxnh7e8Pbm9Mgicgx1AnwxpzHDS+yWHlquqHbppQWGD+vO91VPZvWwbhejdG4zp2FGE1Ng+/WJET0ecRy5v293ru/FX5PP2fwOScYeqZKDtUClJCQgJSUFL3H1qxZg4SEBACAl5cX2rdvr1dGq9UiJSVFV4aIyBUteqYzHukYidcl7Jnl6e6Gda/0xOv9m2P2Y3ehc7S8XTFPdmtkdR3GkgNLB9gqkWQF+Xo6bSuPmimaABUWFiIjIwMZGRkAbk5zz8jIwNmzZwHc7JoaOXKkrvzYsWNx8uRJTJo0CYcPH8Y333yDxYsX4+WXX9aVmThxIr7//nv88MMPOHToEMaNG4eioiKMGTPGrq+NiFyTnJ/WX+jTBO5uGkzu38LqujpH18aHD8YhyMwO8bc3OO3TvC4AILqOP8b2bAw/L/k7DAJ8pNVp6NrK3S1Zw1veQdqkXop2ge3cuRO9e/fW/Txx4kQAwKhRo5CcnIysrCxdMgQAjRo1wvLly/Hyyy/jiy++QP369TF37lz069dPV2bYsGG4dOkSpkyZguzsbLRt2xYrV66sNjCaiEhOY3s2xpyNJ/DWwJay1KcBMPGepnjh7hhZN1M15/NhbZFyKAd330qATLFFY4nYzU8nJTVDgMjFIc2pnEMlRNdG63rBuJB7Haknr8hSvzmCM/ftqZiiCVCvXr1M/uINrfLcq1cv7N6922S9EyZMwIQJE6wNj4hItNf7N8e4Xo0R5CvPTfk2qclPr2Z1UMPLHW0bBONYTqHk8/l7e+C+tvUkHycXse05z/WKke2ceuOmNBp8+nAbrD980W4JkDnOsP6UGjnUGCAiIjWTO/mxRICPJ3ZP6Yufn4y3+eDZyh9gH41vYNuTQZ1r/sixNtTEe5qa/F2xhcg2mAARETkZLw83u7ca1PCSPnZG6sBfW6UBPZpWn7EmZhD16pd7IHXy3QZfe2IL812It43uanoweGm5VnRdJJ5DTYMnIiL1sEVCokRvz5RBsVi88+YU9EAJA7Ob3loTadXLPfDF2mN4NL4Bcq+XISG6Nrw93DD253TUr3lz2YEvh7fDCwuND9/wcLvzwpvU9cexi3e6Lw2tf+QMlO7ZYwJERESqYaq3x1b3S39vD3z0YBz+t/cCnrm135yU1qn6Nf3w8dA21R7/9vEOuu8Ht4kwmQBpNBrsn9YPFVoBlwtL0OfTjbrnwgJ9RMfiSJTu0mQXGBERWaRysiL3Hl/Wkjps5uGOkfjpyXjdmB4lNur09/ZAkK+n3mKTADC0g3Nuz6T04G4mQERETsqei+tZcjMzvK6PDMEYkNjCMZdCefe+WETWtHz/NKW7mUxROjQmQERE5BjUfDe3EXMLUL5/fyuTz3u5u2H1yz1QL1j+TV4dHRMgIiIn5YL5glGOei3MdcSNiG9oto6moQFoGupvthwANBO52a0clP6dMAEiIiKL2GaUjPG7Yr/Ym91YYgYFW3tztdfSO4ufte0+lbevg5iXM390B6Ob7NqC0vufMQEiInJSIxOiAAA9mtaxSf09K9VrSb4g9fYXGxGEzZN6Y/2rvcyWVePagX8+16XaY63qBZo8Rq5FEMVUc3dzO4+TYgsQERHZwrM9ovHHuC74zkaf6ns00V9AcP7oDqhVwwsLRncUXcebA1qgZqV1bsy13ETW8oOvBYsuSmWL7pm7GtSs9piHm21vw1JbWbR2zByV7pVkAkRE5KTc3DRo37AmfExMUffykH4bmD+6Az58oDWaVBkvcnfzUKS/lYjeIjZSve3pHtFIf+seyTFIJfVmWzkP+OfF7rLGcltYoI+46y9DpvBSYhNR5Sq3ONUWuTGtpTgGiIiI7O6v57qgQ8Oa+N2CMSh3Nw/FI50M7/1lyXR4NzcNokNqAAD6trRNN4yflztmDmuLeyyov0V4IF7r1wzxjWrJGlPv5ua7Jq1tj7n962jXoCYOTu+Hd4eYnjVWOfHb/kYfK89umtJjgLgSNBGRGtn43tCuQU38d1z1MSmWEjvLqLLKudJvzyZg/ZGLGBQXIUs8VROHyQNaIDTQB0Pa1cPpy0XYdfYaJi7eI7q+8b1jML53DHaevop3/j6AqYNjZYlTbrVqeOFqUanB5/y8PPB454Z4e8l+o8dXvm5S/gSjQ2rg5OUiCUewBYiIiBzY/yZ0w7v3xWJg63DJx8ZGBOm+rxPgjYc7RNpkfE+tGl4IrTRzLCqkhsUrV3eIqoXlL3RHxyjrW4NEDbex82DuyjG5SchQ1okYmF6V0mOA2AJERKRC9lyPxRqt6wehdf0g8wUrWf1yDxzKykevZraZnQaYv7mqcJKY3bzevzk+/OewwecqD4K2dQuN0lthMAEiIlKRvyd0xcYjlzCmayOlQ7GZmwvzKZvgqXGavCEChGqJSP2avni2Z2N0bVwbwM2Esu/nm26Wr/LCRneJqlZn1b3GKtNPgJRuo7EtJkBERCoSVz8YcfWDlQ7D4ZnLb5TY7LQqS9dn8nJ3w+Od76wAbSqZnHhPU0l11zCz9YacxM5MsxUmQERE5HISW4QiOqQG2jYINvi8rVuIXuvXDP1bhZktZ00ckbV84eEubahvVEgNvNCnid7aTLZibCahvTABIiIip/bFI22rPebj6Y6UV3oq1s3zcIdIUeeWmv9ULm8seTJ3VqmtRpaSa5VrSzEBIiIip9a9ieGuJiWSn9Uv90BRSTnqBHhbXom1+5xV+v65Xo3xzYYTGNq+vnWVWhmHEpgAERER2YnUwd+WNJK8cHcMvlx3HNPvM79W0at9m2FA63A0D7P/oHRBa/dT6uE6QERERCpWtcHHbBdW32Y4OL2f0c1NKx/v5qZBq3pBkscKmWNo9llV9tx3zBAmQERELu6tgS0A3Nw8lW5qVU/a2kZqcTun8LPjbC5DnuxmfhkHD3euA0RERAp6qns0+rcOR0SQj/nCLiIsyAcbX+uFAB/bz4YyxdB0fWdYn2fa4FjFry0TICIiQr1gX6VDUJ2GtWsodm4vdzeUVmiREF1bsRhsaZSILjJbYwJEREROR+kp1tba+XYirhaWIiqkBvJvlCkdjlPiGCAiIiKVCfTxRFSIci1Qlri7eV2lQ5CECRAREZGKSZkFZquWr9iIQGx6rTee7m58cLOhBSfVjAkQERE5HWcYKHybGjrz6gR4o0FtPwT7eRkt42jXnAkQERE5HUcfA2SKqTxD6kKL1pjcv7nez6bSnxB/L3z2cBvbBiQRB0ETERE5uOUvdEPy1tN4WcQ+Xh0b1QIARNX2E12/oXyyQ1RN0cfXq+kHH0930eXtgQkQERGRA9EYaGuJjQjCx0PFtbAE+XriwLR+8PaQtxPIwXrA2AVGRETkSLo3CbG6jhreHqK2v/D3vtlO0ruZ4Q1lHRlbgIiIiBzEC32a4Lleje12vnWv9ET6mWvoGxtm4FnxTT4B3h4WbexqS2wBIiIip9O6XhBahAcisYVjrU1jzqOdGth1LE3dQB/0bx0Odzfp/VuLnukMAAjx98aMB1rLHZrV2AJEREROx8PdDSte6KZ0GLJQW8vJbebGEHWOro3THw7U/bzvfJ6tQ5KECRARETklR1uXxtHERgTi/nb1EBHsmJvoMgEiIiIiyTQaDT4f1lZ0ebW1ZHEMEBERkYqxIcs2VJEAzZo1C1FRUfDx8UF8fDzS0tKMlu3Vqxc0Gk21r4ED7/Qzjh49utrzSUlJ9ngpREREslJby4mzULwL7LfffsPEiRMxZ84cxMfHY+bMmejXrx+OHDmCunWrj97/888/UVpaqvv5ypUraNOmDYYOHapXLikpCQsWLND97O3tbbsXQUREZAdsDZKP4i1An332GZ5++mmMGTMGLVu2xJw5c+Dn54f58+cbLF+rVi2EhYXpvtasWQM/P79qCZC3t7deuZo1xS/ZTURERNUF+noqHYJsFE2ASktLkZ6ejsTERN1jbm5uSExMRGpqqqg65s2bh0ceeQQ1atTQe3zDhg2oW7cumjVrhnHjxuHKlStG6ygpKUF+fr7eFxERkdoo3R32cIf6SIoNwwf3V1/Xp/IWHW4WrBtkb4p2gV2+fBkVFRUIDQ3Vezw0NBSHDx82e3xaWhr279+PefPm6T2elJSEBx54AI0aNcKJEyfwxhtvoH///khNTYW7e/UFpGbMmIFp06ZZ92KIiIicnLeHO+Y83t7gc75e7rg3Lhw3yioQEaT+qfGKjwGyxrx589C6dWt06tRJ7/FHHnlE933r1q0RFxeHxo0bY8OGDejTp0+1eiZPnoyJEyfqfs7Pz0dkZKTtAiciIrKA2scAff3oXUafaxLqb8dIzFM0AQoJCYG7uztycnL0Hs/JyUFYmKF9R+4oKirCokWLMH36dLPniY6ORkhICI4fP24wAfL29uYgaSIiIhtqGhqAn57shNBAdbQOKToGyMvLC+3bt0dKSoruMa1Wi5SUFCQkJJg89vfff0dJSQkee+wxs+c5d+4crly5gvDwcKtjJiIisisnmgbfvUkdNA0NUDoMACqYBTZx4kR8//33+OGHH3Do0CGMGzcORUVFGDNmDABg5MiRmDx5crXj5s2bhyFDhqB27dp6jxcWFuK1117D9u3bcfr0aaSkpOC+++5DTEwM+vXrZ5fXREREZAsq7wFzKIqPARo2bBguXbqEKVOmIDs7G23btsXKlSt1A6PPnj0LNzf9PO3IkSPYsmULVq9eXa0+d3d37N27Fz/88ANyc3MRERGBvn374t1332U3FxEROTQnagxSnEYQlJ5Upz75+fkICgpCXl4eAgMDlQ6HiIhcWN71MrSZdvMD/443+qhmDI0aSbl/K94FRkRERCawmcImmAARERE5CI4Bkg8TICIiInI5TICIiIjUjM0+NqH4LDAiIiIyLsjXE/1iQ1FeIaBOAGczy4UJEBERkcp9+3gHpUNwOuwCIyIiIpfDBIiIiIhcDhMgIiIicjlMgIiIiMjlMAEiIiIil8MEiIiIiFwOEyAiIiJyOUyAiIiIyOUwASIiIiKXwwSIiIiIXA4TICIiInI5TICIiIjI5TABIiIiIpfDBIiIiIhcjofSAaiRIAgAgPz8fIUjISIiIrFu37dv38dNYQJkQEFBAQAgMjJS4UiIiIhIqoKCAgQFBZksoxHEpEkuRqvV4sKFCwgICIBGo5G17vz8fERGRiIzMxOBgYGy1u0KeP2sw+tnHV4/6/D6WYfXzzxBEFBQUICIiAi4uZke5cMWIAPc3NxQv359m54jMDCQf8BW4PWzDq+fdXj9rMPrZx1eP9PMtfzcxkHQRERE5HKYABEREZHLYQJkZ97e3njnnXfg7e2tdCgOidfPOrx+1uH1sw6vn3V4/eTFQdBERETkctgCRERERC6HCRARERG5HCZARERE5HKYABEREZHLYQJkR7NmzUJUVBR8fHwQHx+PtLQ0pUNSxKZNmzBo0CBERERAo9FgyZIles8LgoApU6YgPDwcvr6+SExMxLFjx/TKXL16FSNGjEBgYCCCg4Px5JNPorCwUK/M3r170b17d/j4+CAyMhIfffSRrV+azc2YMQMdO3ZEQEAA6tatiyFDhuDIkSN6ZW7cuIHx48ejdu3a8Pf3x4MPPoicnBy9MmfPnsXAgQPh5+eHunXr4rXXXkN5eblemQ0bNuCuu+6Ct7c3YmJikJycbOuXZxezZ89GXFycbjG5hIQE/PPPP7rnef3E+/DDD6HRaPDSSy/pHuP1M23q1KnQaDR6X82bN9c9z+tnRwLZxaJFiwQvLy9h/vz5woEDB4Snn35aCA4OFnJycpQOze5WrFghvPnmm8Kff/4pABD++usvvec//PBDISgoSFiyZImwZ88eYfDgwUKjRo2E69ev68okJSUJbdq0EbZv3y5s3rxZiImJEYYPH657Pi8vTwgNDRVGjBgh7N+/X1i4cKHg6+srfPvtt/Z6mTbRr18/YcGCBcL+/fuFjIwMYcCAAUKDBg2EwsJCXZmxY8cKkZGRQkpKirBz506hc+fOQpcuXXTPl5eXC61atRISExOF3bt3CytWrBBCQkKEyZMn68qcPHlS8PPzEyZOnCgcPHhQ+OqrrwR3d3dh5cqVdn29tvD3338Ly5cvF44ePSocOXJEeOONNwRPT09h//79giDw+omVlpYmREVFCXFxccKLL76oe5zXz7R33nlHiI2NFbKysnRfly5d0j3P62c/TIDspFOnTsL48eN1P1dUVAgRERHCjBkzFIxKeVUTIK1WK4SFhQkff/yx7rHc3FzB29tbWLhwoSAIgnDw4EEBgPDvv//qyvzzzz+CRqMRzp8/LwiCIHzzzTdCzZo1hZKSEl2Z//u//xOaNWtm41dkXxcvXhQACBs3bhQE4ea18vT0FH7//XddmUOHDgkAhNTUVEEQbiagbm5uQnZ2tq7M7NmzhcDAQN31mjRpkhAbG6t3rmHDhgn9+vWz9UtSRM2aNYW5c+fy+olUUFAgNGnSRFizZo3Qs2dPXQLE62feO++8I7Rp08bgc7x+9sUuMDsoLS1Feno6EhMTdY+5ubkhMTERqampCkamPqdOnUJ2drbetQoKCkJ8fLzuWqWmpiI4OBgdOnTQlUlMTISbmxt27NihK9OjRw94eXnpyvTr1w9HjhzBtWvX7PRqbC8vLw8AUKtWLQBAeno6ysrK9K5f8+bN0aBBA73r17p1a4SGhurK9OvXD/n5+Thw4ICuTOU6bpdxtr/XiooKLFq0CEVFRUhISOD1E2n8+PEYOHBgtdfI6yfOsWPHEBERgejoaIwYMQJnz54FwOtnb0yA7ODy5cuoqKjQ+4MFgNDQUGRnZysUlTrdvh6mrlV2djbq1q2r97yHhwdq1aqlV8ZQHZXP4ei0Wi1eeukldO3aFa1atQJw87V5eXkhODhYr2zV62fu2hgrk5+fj+vXr9vi5djVvn374O/vD29vb4wdOxZ//fUXWrZsyesnwqJFi7Br1y7MmDGj2nO8fubFx8cjOTkZK1euxOzZs3Hq1Cl0794dBQUFvH52xt3giRzU+PHjsX//fmzZskXpUBxOs2bNkJGRgby8PPz3v//FqFGjsHHjRqXDUr3MzEy8+OKLWLNmDXx8fJQOxyH1799f931cXBzi4+PRsGFDLF68GL6+vgpG5nrYAmQHISEhcHd3rzaSPycnB2FhYQpFpU63r4epaxUWFoaLFy/qPV9eXo6rV6/qlTFUR+VzOLIJEyZg2bJlWL9+PerXr697PCwsDKWlpcjNzdUrX/X6mbs2xsoEBgY6xZu0l5cXYmJi0L59e8yYMQNt2rTBF198wetnRnp6Oi5evIi77roLHh4e8PDwwMaNG/Hll1/Cw8MDoaGhvH4SBQcHo2nTpjh+/Dj//uyMCZAdeHl5oX379khJSdE9ptVqkZKSgoSEBAUjU59GjRohLCxM71rl5+djx44dumuVkJCA3NxcpKen68qsW7cOWq0W8fHxujKbNm1CWVmZrsyaNWvQrFkz1KxZ006vRn6CIGDChAn466+/sG7dOjRq1Ejv+fbt28PT01Pv+h05cgRnz57Vu3779u3TSyLXrFmDwMBAtGzZUlemch23yzjr36tWq0VJSQmvnxl9+vTBvn37kJGRofvq0KEDRowYofue10+awsJCnDhxAuHh4fz7szelR2G7ikWLFgne3t5CcnKycPDgQeGZZ54RgoOD9Ubyu4qCggJh9+7dwu7duwUAwmeffSbs3r1bOHPmjCAIN6fBBwcHC0uXLhX27t0r3HfffQanwbdr107YsWOHsGXLFqFJkyZ60+Bzc3OF0NBQ4fHHHxf2798vLFq0SPDz83P4afDjxo0TgoKChA0bNuhNoy0uLtaVGTt2rNCgQQNh3bp1ws6dO4WEhAQhISFB9/ztabR9+/YVMjIyhJUrVwp16tQxOI32tddeEw4dOiTMmjXLaabRvv7668LGjRuFU6dOCXv37hVef/11QaPRCKtXrxYEgddPqsqzwASB18+cV155RdiwYYNw6tQpYevWrUJiYqIQEhIiXLx4URAEXj97YgJkR1999ZXQoEEDwcvLS+jUqZOwfft2pUNSxPr16wUA1b5GjRolCMLNqfBvv/22EBoaKnh7ewt9+vQRjhw5olfHlStXhOHDhwv+/v5CYGCgMGbMGKGgoECvzJ49e4Ru3boJ3t7eQr169YQPP/zQXi/RZgxdNwDCggULdGWuX78uPPfcc0LNmjUFPz8/4f777xeysrL06jl9+rTQv39/wdfXVwgJCRFeeeUVoaysTK/M+vXrhbZt2wpeXl5CdHS03jkc2RNPPCE0bNhQ8PLyEurUqSP06dNHl/wIAq+fVFUTIF4/04YNGyaEh4cLXl5eQr169YRhw4YJx48f1z3P62c/GkEQBGXanoiIiIiUwTFARERE5HKYABEREZHLYQJERERELocJEBEREbkcJkBERETkcpgAERERkcthAkREREQuhwkQERERuRwmQETk8pKTkxEcHKx0GERkR0yAiEgVRo8eDY1Go/uqXbs2kpKSsHfvXkn1TJ06FW3btrVNkCaMHz8eb7zxBgDggw8+wBNPPGH3GIhIPCZARKQaSUlJyMrKQlZWFlJSUuDh4YF7771X6bBESU1NRdeuXQEAmzdv1n1PROrEBIiIVMPb2xthYWEICwtD27Zt8frrryMzMxOXLl3Slfm///s/NG3aFH5+foiOjsbbb7+NsrIyADe7sqZNm4Y9e/boWpKSk5MBALm5uXj22WcRGhoKHx8ftGrVCsuWLdM7/6pVq9CiRQv4+/vrkjExioqKsH//fnTp0gVarVYvGSIidfJQOgAiIkMKCwvx888/IyYmBrVr19Y9HhAQgOTkZERERGDfvn14+umnERAQgEmTJmHYsGHYv38/Vq5cibVr1wIAgoKCoNVq0b9/fxQUFODnn39G48aNcfDgQbi7u+vqLS4uxieffIKffvoJbm5ueOyxx/Dqq6/il19+MRrjc889h19//RVarRZlZWVo1KgRBEFAfn4+OnfuDADYu3cvGjRoYKOrRESWYgJERKqxbNky+Pv7A7jZqhIeHo5ly5bBze1OY/Vbb72l+z4qKgqvvvoqFi1ahEmTJsHX1xf+/v7w8PBAWFiYrtzq1auRlpaGQ4cOoWnTpgCA6OhovXOXlZVhzpw5aNy4MQBgwoQJmD59usl4p0+fjkmTJuG9997Txfbdd9/h8OHD+OyzzwAAERERll4OIrIhJkBEpBq9e/fG7NmzAQDXrl3DN998g/79+yMtLQ0NGzYEAPz222/48ssvceLECRQWFqK8vByBgYEm683IyED9+vV1yY8hfn5+uuQHAMLDw3Hx4kWT9YaEhCAkJATbtm3DF198gaioKPz7778YNWoUoqKiRL5qIlICxwARkWrUqFEDMTExiImJQceOHTF37lwUFRXh+++/B3BzoPGIESMwYMAALFu2DLt378abb76J0tJSk/X6+vqaPbenp6fezxqNBoIgGC3/yy+/wN/fH/7+/jh06BCGDBkCf39/pKSk4JlnnoG/v7/J7jMiUhZbgIhItTQaDdzc3HD9+nUAwLZt29CwYUO8+eabujJnzpzRO8bLywsVFRV6j8XFxeHcuXM4evSoyVYgKQYPHoz4+HgsXboUf/75J3744Qds27YN7733HlasWAEACA0NleVcRCQ/JkBEpBolJSXIzs4GcLML7Ouvv0ZhYSEGDRoEAGjSpAnOnj2LRYsWoWPHjli+fDn++usvvTqioqJw6tQpXbdXQEAAevbsiR49euDBBx/EZ599hpiYGBw+fBgajQZJSUkWxRoQEICAgAAcO3YMiYmJiImJwa+//orevXsjJibGugtBRDbHLjAiUo2VK1ciPDwc4eHhiI+Px7///ovff/8dvXr1AnCz1eXll1/GhAkT0LZtW2zbtg1vv/22Xh0PPvggkpKS0Lt3b9SpUwcLFy4EAPzxxx/o2LEjhg8fjpYtW2LSpEnVWoossWHDBvTo0QMAsHHjRt33RKRuGsFUJzcRERGRE2ILEBEREbkcJkBERETkcpgAERERkcthAkREREQuhwkQERERuRwmQERERORymAARERGRy2ECRERERC6HCRARERG5HCZARERE5HKYABEREZHL+X+66UMd0vdPpQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "plt.plot(loss_history)\n", "plt.xlabel('Batch #')\n", "plt.ylabel('Loss [entropy]')" ] }, { "cell_type": "markdown", "metadata": { "id": "kKpOlHPLqEgl" }, "source": [ "### 변수와 옵티마이저\n", "\n", "`tf.Variable` 객체는 자동 미분을 쉽게 하기 위해서 학습동안 변경된 `tf.Tensor` 값을 저장합니다.\n", "모델 파라미터는 클래스 인스턴스 변수로 캡슐화될 수 있습니다.\n", "\n", "효과적으로 모델 파라미터를 캡슐화하려면 `tf.Variable`을 `tf.GradientTape`과 함께 사용합니다.\n", "예를 들어, 위의 자동 미분은 다음과 같이 재작성 가능합니다:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:45.144874Z", "iopub.status.busy": "2022-12-14T21:10:45.144591Z", "iopub.status.idle": "2022-12-14T21:10:46.412916Z", "shell.execute_reply": "2022-12-14T21:10:46.412227Z" }, "id": "nnQLBYmEqEgm" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "초기 손실: 68.819\n", "스텝 000에서 손실: 66.154\n", "스텝 020에서 손실: 30.232\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "스텝 040에서 손실: 14.127\n", "스텝 060에서 손실: 6.906\n", "스텝 080에서 손실: 3.667\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "스텝 100에서 손실: 2.214\n", "스텝 120에서 손실: 1.562\n", "스텝 140에서 손실: 1.270\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "스텝 160에서 손실: 1.139\n", "스텝 180에서 손실: 1.080\n", "스텝 200에서 손실: 1.053\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "스텝 220에서 손실: 1.041\n", "스텝 240에서 손실: 1.036\n", "스텝 260에서 손실: 1.034\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "스텝 280에서 손실: 1.033\n", "최종 손실: 1.032\n", "W = 3.00956130027771, B = 1.9996976852416992\n" ] } ], "source": [ "class Model(tf.keras.Model):\n", " def __init__(self):\n", " super(Model, self).__init__()\n", " self.W = tf.Variable(5., name='weight')\n", " self.B = tf.Variable(10., name='bias')\n", " def call(self, inputs):\n", " return inputs * self.W + self.B\n", "\n", "# 약 3 * x + 2개의 점으로 구성된 실험 데이터\n", "NUM_EXAMPLES = 2000\n", "training_inputs = tf.random.normal([NUM_EXAMPLES])\n", "noise = tf.random.normal([NUM_EXAMPLES])\n", "training_outputs = training_inputs * 3 + 2 + noise\n", "\n", "# 최적화할 손실함수\n", "def loss(model, inputs, targets):\n", " error = model(inputs) - targets\n", " return tf.reduce_mean(tf.square(error))\n", "\n", "def grad(model, inputs, targets):\n", " with tf.GradientTape() as tape:\n", " loss_value = loss(model, inputs, targets)\n", " return tape.gradient(loss_value, [model.W, model.B])\n", "\n", "# 정의:\n", "# 1. 모델\n", "# 2. 모델 파라미터에 대한 손실 함수의 미분\n", "# 3. 미분에 기초한 변수 업데이트 전략\n", "model = Model()\n", "optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)\n", "\n", "print(\"초기 손실: {:.3f}\".format(loss(model, training_inputs, training_outputs)))\n", "\n", "# 반복 훈련\n", "for i in range(300):\n", " grads = grad(model, training_inputs, training_outputs)\n", " optimizer.apply_gradients(zip(grads, [model.W, model.B]))\n", " if i % 20 == 0:\n", " print(\"스텝 {:03d}에서 손실: {:.3f}\".format(i, loss(model, training_inputs, training_outputs)))\n", "\n", "print(\"최종 손실: {:.3f}\".format(loss(model, training_inputs, training_outputs)))\n", "print(\"W = {}, B = {}\".format(model.W.numpy(), model.B.numpy()))" ] }, { "cell_type": "markdown", "metadata": { "id": "rPjb8nRWqEgr" }, "source": [ "## 즉시 실행에서 상태를 위한 객체 사용\n", "\n", "텐서플로 1.x 그래프 실행에서, 프로그램 상태(예: 변수)는 전역 컬렉션에 저장되고 그 수명은 `tf.Session` 객체에 의해서 관리됩니다.\n", "반면에 즉시 실행에서 상태 객체 수명은 그와 관련된 파이썬 객체 수명에 의해서 결정됩니다.\n", "\n", "### 변수는 객체입니다\n", "\n", "즉시 실행에서 변수는 그 객체의 마지막 참조가 제거될 때까지 유지되고 그 이후 삭제됩니다." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.416307Z", "iopub.status.busy": "2022-12-14T21:10:46.416072Z", "iopub.status.idle": "2022-12-14T21:10:46.423647Z", "shell.execute_reply": "2022-12-14T21:10:46.423055Z" }, "id": "A2boS674qEgs" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GPU 사용 가능\n" ] } ], "source": [ "if tf.config.experimental.list_physical_devices(\"GPU\"):\n", " with tf.device(\"gpu:0\"):\n", " print(\"GPU 사용 가능\")\n", " v = tf.Variable(tf.random.normal([1000, 1000]))\n", " v = None # v는 더이상 GPU 메모리를 사용하지 않음" ] }, { "cell_type": "markdown", "metadata": { "id": "scMjg6L6qEgv" }, "source": [ "### 객체 기반의 저장\n", "\n", "이번 장은 [훈련 체크포인트 가이드](./checkpoint.ipynb) 요약버전입니다.\n", "\n", "`tf.train.Checkpoint`는 `tf.Variable`을 체크포인트 파일로 저장하거나 체크포인트 파일에서 복구할 수 있습니다:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.426849Z", "iopub.status.busy": "2022-12-14T21:10:46.426631Z", "iopub.status.idle": "2022-12-14T21:10:46.430347Z", "shell.execute_reply": "2022-12-14T21:10:46.429795Z" }, "id": "7z5xRfdHzZOQ" }, "outputs": [], "source": [ "x = tf.Variable(10.)\n", "checkpoint = tf.train.Checkpoint(x=x)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.433079Z", "iopub.status.busy": "2022-12-14T21:10:46.432869Z", "iopub.status.idle": "2022-12-14T21:10:46.457847Z", "shell.execute_reply": "2022-12-14T21:10:46.457310Z" }, "id": "IffrUVG7zyVb" }, "outputs": [ { "data": { "text/plain": [ "'./ckpt/-1'" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x.assign(2.) # 변수에 새로운 값을 할당하고 저장\n", "checkpoint_path = './ckpt/'\n", "checkpoint.save('./ckpt/')" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.460729Z", "iopub.status.busy": "2022-12-14T21:10:46.460504Z", "iopub.status.idle": "2022-12-14T21:10:46.468469Z", "shell.execute_reply": "2022-12-14T21:10:46.467908Z" }, "id": "eMT9koCoqEgw" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "x.assign(11.) # 저장한 후에 변수 변경\n", "\n", "# 체크포인트로부터 값을 복구\n", "checkpoint.restore(tf.train.latest_checkpoint(checkpoint_path))\n", "\n", "print(x) # => 2.0" ] }, { "cell_type": "markdown", "metadata": { "id": "vbFnP-yLqEgx" }, "source": [ "모델을 저장하거나 읽어들이기 위해서, `tf.train.Checkpoint`는 숨겨진 변수를 요구하지 않고 객체 내부 상태를 저장합니다. \n", "`옵티마이저`와 `모델`, 전역 단계 상태를 기록하려면 `tf.train.Checkpoint`에 전달하면 됩니다:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.471210Z", "iopub.status.busy": "2022-12-14T21:10:46.470999Z", "iopub.status.idle": "2022-12-14T21:10:46.498721Z", "shell.execute_reply": "2022-12-14T21:10:46.498131Z" }, "id": "hWZHyAXMqEg0" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import os\n", "\n", "model = tf.keras.Sequential([\n", " tf.keras.layers.Conv2D(16,[3,3], activation='relu'),\n", " tf.keras.layers.GlobalAveragePooling2D(),\n", " tf.keras.layers.Dense(10)\n", "])\n", "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n", "checkpoint_dir = 'path/to/model_dir'\n", "if not os.path.exists(checkpoint_dir):\n", " os.makedirs(checkpoint_dir)\n", "checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n", "root = tf.train.Checkpoint(optimizer=optimizer,\n", " model=model)\n", "\n", "root.save(checkpoint_prefix)\n", "root.restore(tf.train.latest_checkpoint(checkpoint_dir))" ] }, { "cell_type": "markdown", "metadata": { "id": "R-ITwkBCF6GJ" }, "source": [ "Note: 대부분의 반복 훈련 과정에서 변수는 `tf.train.Checkpoint.restore`가 호출된 이후에 생성됩니다.\n", "이러한 변수는 생성되자마자 복원될 것이므로 단언문을 통해 체크포인트가 완벽히 적재되었다는 것을 보장받을 수 있습니다.\n", "자세한 내용은 [훈련 체크포인트 가이드](./checkpoint.ipynb)를 참고하세요." ] }, { "cell_type": "markdown", "metadata": { "id": "3yoD0VJ7qEg3" }, "source": [ "### 객체 지향형 지표\n", "\n", "`tf.keras.metrics`는 객체로 저장됩니다.\n", "새로운 데이터를 이 객체에 전달하여 지표를 수정하고 `tf.keras.metrics.result` 메서드를 사용해 그 결과를 얻습니다.\n", "예를 들어:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.501642Z", "iopub.status.busy": "2022-12-14T21:10:46.501425Z", "iopub.status.idle": "2022-12-14T21:10:46.518614Z", "shell.execute_reply": "2022-12-14T21:10:46.518003Z" }, "id": "9ccu0iAaqEg5" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "m = tf.keras.metrics.Mean(\"loss\")\n", "m(0)\n", "m(5)\n", "m.result() # => 2.5\n", "m([8, 9])\n", "m.result() # => 5.5" ] }, { "cell_type": "markdown", "metadata": { "id": "aB8qWtT955pI" }, "source": [ "### 서머리(summary)와 텐서보드\n", "\n", "[텐서보드](https://tensorflow.org/tensorboard)는 훈련과정에서 모델을 파악하거나 디버깅하고 최적화하기 위해 사용하는 시각화 도구입니다.\n", "텐서보드는 프로그램이 실행되는 동안 작성된 서머리 이벤트를 사용합니다.\n", "\n", "즉시 실행에서 변수의 서머리 정보를 기록하기 위해서 `tf.summary`를 사용합니다.\n", "예를 들어, 다음은 매 100번째 훈련마다 `loss`의 서머리 정보를 기록합니다:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.521508Z", "iopub.status.busy": "2022-12-14T21:10:46.521299Z", "iopub.status.idle": "2022-12-14T21:10:46.539734Z", "shell.execute_reply": "2022-12-14T21:10:46.539158Z" }, "id": "z6VInqhA6RH4" }, "outputs": [], "source": [ "logdir = \"./tb/\"\n", "writer = tf.summary.create_file_writer(logdir)\n", "\n", "with writer.as_default(): # 또는 반복 전에 writer.set_as_default()를 호출\n", " for i in range(1000):\n", " step = i + 1\n", " # 실제 훈련 함수로 손실을 계산\n", " loss = 1 - 0.001 * step\n", " if step % 100 == 0:\n", " tf.summary.scalar('손실', loss, step=step)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.542632Z", "iopub.status.busy": "2022-12-14T21:10:46.542404Z", "iopub.status.idle": "2022-12-14T21:10:46.730922Z", "shell.execute_reply": "2022-12-14T21:10:46.730076Z" }, "id": "08QQD2j36TaI" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "events.out.tfevents.1671052246.kokoro-gcp-ubuntu-prod-1438429585.86417.0.v2\r\n" ] } ], "source": [ "!ls tb/" ] }, { "cell_type": "markdown", "metadata": { "id": "xEL4yJe5qEhD" }, "source": [ "## 자동 미분 관련 고급편\n", "\n", "### 동적 모델\n", "\n", "`tf.GradientTape`는 또한 동적인 모델에서도 사용가능합니다.\n", "아래 예는 [역추적 길찾기](https://wikipedia.org/wiki/Backtracking_line_search) 알고리즘의 복잡한 제어 흐름에도 불구하고,\n", "그래디언트가 있으며 미분 가능이 하다는 것을 제외하면 일반적인 NumPy으로 작성한 코드처럼 보입니다:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.734967Z", "iopub.status.busy": "2022-12-14T21:10:46.734672Z", "iopub.status.idle": "2022-12-14T21:10:46.739892Z", "shell.execute_reply": "2022-12-14T21:10:46.739276Z" }, "id": "L518n5dkqEhE" }, "outputs": [], "source": [ "def line_search_step(fn, init_x, rate=1.0):\n", " with tf.GradientTape() as tape:\n", " # 변수는 자동적으로 기록되지만 텐서는 사용자가 스스로 확인해야 함\n", " tape.watch(init_x)\n", " value = fn(init_x)\n", " grad = tape.gradient(value, init_x)\n", " grad_norm = tf.reduce_sum(grad * grad)\n", " init_value = value\n", " while value > init_value - rate * grad_norm:\n", " x = init_x - rate * grad\n", " value = fn(x)\n", " rate /= 2.0\n", " return x, value" ] }, { "cell_type": "markdown", "metadata": { "id": "gieGOf_DqEhK" }, "source": [ "### 사용자 정의 그래디언트\n", "\n", "사용자 정의 그래디언트는 그래디언트를 재정의(override)하는 가장 쉬운 방법입니다.\n", "정방향 함수안에서 입력값 또는 출력값, 중간값과 관련된 그래디언트를 정의해야 합니다. \n", "예를 들어 다음은 역전파 과정에서 그래디언트의 놈(norm)을 클리핑(clip)하는 가장 쉬운 방법입니다:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.742973Z", "iopub.status.busy": "2022-12-14T21:10:46.742757Z", "iopub.status.idle": "2022-12-14T21:10:46.746423Z", "shell.execute_reply": "2022-12-14T21:10:46.745825Z" }, "id": "-OwwsWUAqEhK" }, "outputs": [], "source": [ "@tf.custom_gradient\n", "def clip_gradient_by_norm(x, norm):\n", " y = tf.identity(x)\n", " def grad_fn(dresult):\n", " return [tf.clip_by_norm(dresult, norm), None]\n", " return y, grad_fn" ] }, { "cell_type": "markdown", "metadata": { "id": "JPLDHkF_qEhN" }, "source": [ "사용자 정의 그래디언트는 일반적으로 연산에 대해 수치적으로(numerically) 안정된 그래디언트를 제공하기 위해 사용됩니다:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.749663Z", "iopub.status.busy": "2022-12-14T21:10:46.749261Z", "iopub.status.idle": "2022-12-14T21:10:46.752826Z", "shell.execute_reply": "2022-12-14T21:10:46.752271Z" }, "id": "24WiLROnqEhO" }, "outputs": [], "source": [ "def log1pexp(x):\n", " return tf.math.log(1 + tf.exp(x))\n", "\n", "def grad_log1pexp(x):\n", " with tf.GradientTape() as tape:\n", " tape.watch(x)\n", " value = log1pexp(x)\n", " return tape.gradient(value, x)\n" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.755783Z", "iopub.status.busy": "2022-12-14T21:10:46.755276Z", "iopub.status.idle": "2022-12-14T21:10:46.765481Z", "shell.execute_reply": "2022-12-14T21:10:46.764880Z" }, "id": "n8fq69r9-B-c" }, "outputs": [ { "data": { "text/plain": [ "0.5" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 그래디언트 계산은 x = 0일 때 잘 동작\n", "grad_log1pexp(tf.constant(0.)).numpy()" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.768188Z", "iopub.status.busy": "2022-12-14T21:10:46.767976Z", "iopub.status.idle": "2022-12-14T21:10:46.773615Z", "shell.execute_reply": "2022-12-14T21:10:46.773080Z" }, "id": "_VFSU0mG-FSp" }, "outputs": [ { "data": { "text/plain": [ "nan" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 그러나, x = 100일 때 수치적으로 불안정하기 때문에 실패\n", "grad_log1pexp(tf.constant(100.)).numpy()" ] }, { "cell_type": "markdown", "metadata": { "id": "-VcTR34rqEhQ" }, "source": [ "여기 `log1pexp` 함수는 이론적으로 사용자 정의 그래디언트를 활용해 간결해 질 수 있습니다.\n", "아래 구현은 불필요한 계산을 제거함으로써 계산을 좀 더 효율적으로 하기 위해 정방향 경로안에서 계산된 `tf.exp(x)`값을 재사용합니다:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.776464Z", "iopub.status.busy": "2022-12-14T21:10:46.776255Z", "iopub.status.idle": "2022-12-14T21:10:46.780190Z", "shell.execute_reply": "2022-12-14T21:10:46.779639Z" }, "id": "Q7nvfx_-qEhS" }, "outputs": [], "source": [ "@tf.custom_gradient\n", "def log1pexp(x):\n", " e = tf.exp(x)\n", " def grad(dy):\n", " return dy * (1 - 1 / (1 + e))\n", " return tf.math.log(1 + e), grad\n", "\n", "def grad_log1pexp(x):\n", " with tf.GradientTape() as tape:\n", " tape.watch(x)\n", " value = log1pexp(x)\n", " return tape.gradient(value, x)\n" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.782782Z", "iopub.status.busy": "2022-12-14T21:10:46.782571Z", "iopub.status.idle": "2022-12-14T21:10:46.788267Z", "shell.execute_reply": "2022-12-14T21:10:46.787765Z" }, "id": "5gHPKMfl-Kge" }, "outputs": [ { "data": { "text/plain": [ "0.5" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 전처럼, 그래디언트 계산은 x = 0일 때 잘 동작\n", "grad_log1pexp(tf.constant(0.)).numpy()" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.790886Z", "iopub.status.busy": "2022-12-14T21:10:46.790677Z", "iopub.status.idle": "2022-12-14T21:10:46.796209Z", "shell.execute_reply": "2022-12-14T21:10:46.795656Z" }, "id": "u38MOfz3-MDE" }, "outputs": [ { "data": { "text/plain": [ "1.0" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 그래디언트 계산은 x = 100일 때 역시 잘 동작\n", "grad_log1pexp(tf.constant(100.)).numpy()" ] }, { "cell_type": "markdown", "metadata": { "id": "rnZXjfQzqEhV" }, "source": [ "## 성능\n", "\n", "즉시 실행에서 계산은 자동으로 GPU로 분배됩니다.\n", "만약 계산 분배를 사용자가 제어하고 싶다면 그 부분을 `tf.device('/gpu:0')` 블록 (또는 CPU도 동일)으로 감싸서 실행하세요:" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:46.799197Z", "iopub.status.busy": "2022-12-14T21:10:46.798768Z", "iopub.status.idle": "2022-12-14T21:10:47.385820Z", "shell.execute_reply": "2022-12-14T21:10:47.385061Z" }, "id": "Ac9Y64H-qEhX" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(1000, 1000) 크기 행렬을 자기 자신과 200번 곱했을 때 걸리는 시간:\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "CPU: 0.4814612865447998 초\n", "GPU: 0.0684516429901123 초\n" ] } ], "source": [ "import time\n", "\n", "def measure(x, steps):\n", " # 텐서플로는 처음 사용할 때 GPU를 초기화, 시간계산에서 제외\n", " tf.matmul(x, x)\n", " start = time.time()\n", " for i in range(steps):\n", " x = tf.matmul(x, x)\n", " # tf.matmul는 행렬 곱셈을 완료하기 전에 결과를 반환할 수 있습니다\n", " # (예, CUDA 스트림 대기열에 연산을 추가한 후에 결과를 반환할 수 있다).\n", " # 아래 x.numpy() 호출은 대기열에 추가된 모든 연산이 완료될 것임을 보장합니다\n", " # (그리고 그 결과가 호스트 메모리에 복사될 것이고,\n", " # 그래서 matnul 연산시간보다는 조금 많은 연산시간이\n", " # 포함됩니다).\n", " _ = x.numpy()\n", " end = time.time()\n", " return end - start\n", "\n", "shape = (1000, 1000)\n", "steps = 200\n", "print(\"{} 크기 행렬을 자기 자신과 {}번 곱했을 때 걸리는 시간:\".format(shape, steps))\n", "\n", "# CPU에서 실행:\n", "with tf.device(\"/cpu:0\"):\n", " print(\"CPU: {} 초\".format(measure(tf.random.normal(shape), steps)))\n", "\n", "# GPU에서 실행, 가능하다면:\n", "if tf.config.experimental.list_physical_devices(\"GPU\"):\n", " with tf.device(\"/gpu:0\"):\n", " print(\"GPU: {} 초\".format(measure(tf.random.normal(shape), steps)))\n", "else:\n", " print(\"GPU: 없음\")" ] }, { "cell_type": "markdown", "metadata": { "id": "RLw3IS7UqEhe" }, "source": [ "`tf.Tensor` 객체는 실제로 그 연산을 수행할 다른 디바이스로 복사될 수 있습니다:" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T21:10:47.389142Z", "iopub.status.busy": "2022-12-14T21:10:47.388876Z", "iopub.status.idle": "2022-12-14T21:10:47.395677Z", "shell.execute_reply": "2022-12-14T21:10:47.395023Z" }, "id": "ny6LX2BVqEhf" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_86417/2531695931.py:4: _EagerTensorBase.gpu (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.identity instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_86417/2531695931.py:5: _EagerTensorBase.cpu (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.identity instead.\n" ] } ], "source": [ "if tf.config.experimental.list_physical_devices(\"GPU\"):\n", " x = tf.random.normal([10, 10])\n", "\n", " x_gpu0 = x.gpu()\n", " x_cpu = x.cpu()\n", "\n", " _ = tf.matmul(x_cpu, x_cpu) # CPU에서 실행\n", " _ = tf.matmul(x_gpu0, x_gpu0) # GPU:0에서 실행" ] }, { "cell_type": "markdown", "metadata": { "id": "oA_qaII3-p6c" }, "source": [ "### 벤치마크\n", "\n", "GPU에서 학습을 하는 [ResNet50](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/resnet50) 같은 계산량이 많은 모델에서,\n", "즉시 실행 성능은 `tf.function` 실행과 비교될 수 있습니다.\n", "그러나 이러한 차이는 계산량이 작은 모델인 경우 더 커지고, 수많은 작은 연산으로 구성된 모델은 자주 반복되는 부분을 최적화하는 사례도 있습니다.\n", "\n", "## 함수를 활용\n", "\n", "즉시 실행이 개발과 디버깅 과정을 좀 더 대화형(interactive)으로 만들어 주지만\n", "텐서플로 1.x 형태 그래프 실행은 학습의 분산과 성능, 운영 배포에 장점을 가지고 있습니다.\n", "이러한 차이를 해소하기 위해서, 텐서플로 2.0에서는 `tf.function` API를 도입했습니다.\n", "자세한 내용은 [tf.function](./function.ipynb) 가이드를 참고하세요." ] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "name": "eager.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" } }, "nbformat": 4, "nbformat_minor": 0 }