{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "CCQY7jpBfMur" }, "source": [ "##### Copyright 2018 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2022-12-14T22:20:45.818776Z", "iopub.status.busy": "2022-12-14T22:20:45.818569Z", "iopub.status.idle": "2022-12-14T22:20:45.822390Z", "shell.execute_reply": "2022-12-14T22:20:45.821853Z" }, "id": "z6X9omPnfO_h" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "2QQJJyDzqGRb" }, "source": [ "# Eager Execution\n" ] }, { "cell_type": "markdown", "metadata": { "id": "B1xdylywqUSX" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
在 TensorFlow.org 上查看 在 Google Colab 中运行 在 GitHub 上查看源代码 下载笔记本
" ] }, { "cell_type": "markdown", "metadata": { "id": "EGjDcGxIqEfX" }, "source": [ "TensorFlow 的 Eager Execution 是一种命令式编程环境，可立即评估运算，无需构建计算图：运算会返回具体的值，而非构建供稍后运行的计算图。这样能使您轻松入门 TensorFlow 并调试模型，同时也减少了样板代码。要跟随本指南进行学习，请在交互式 `python` 解释器中运行以下代码示例。\n", "\n", "Eager Execution 是用于研究和实验的灵活机器学习平台，具备以下特性：\n", "\n", "- *直观的界面* - 自然地组织代码结构并使用 Python 数据结构。快速迭代小模型和小数据。\n", "- *更方便的调试功能* - 直接调用运算以检查正在运行的模型并测试更改。使用标准 Python 调试工具立即报告错误。\n", "- *自然的控制流* - 使用 Python 而非计算图控制流，简化了动态模型的规范。\n", "\n", "Eager Execution 支持大部分 TensorFlow 运算和 GPU 加速。\n", "\n", "注：启用 Eager Execution 后可能会增加某些模型的开销。我们正在持续改进其性能，如果您遇到问题，请[提交错误报告](https://github.com/tensorflow/tensorflow/issues)并分享您的基准。" ] }, { "cell_type": "markdown", "metadata": { "id": "RBAeIwOMrYk8" }, "source": [ "## 设置和基本用法" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:45.826073Z", "iopub.status.busy": "2022-12-14T22:20:45.825549Z", "iopub.status.idle": "2022-12-14T22:20:47.766558Z", "shell.execute_reply": "2022-12-14T22:20:47.765787Z" }, "id": "ByNsp4VqqEfa" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-12-14 22:20:46.774124: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n", "2022-12-14 22:20:46.774214: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n", "2022-12-14 22:20:46.774224: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n" ] } ], "source": [ "import os\n", "\n", "import tensorflow as tf\n", "\n", "import cProfile" ] }, { "cell_type": "markdown", "metadata": { "id": "48P3-8q4qEfe" }, "source": [ "在 Tensorflow 2.0 中，默认启用 Eager Execution。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:47.771006Z", "iopub.status.busy": "2022-12-14T22:20:47.770347Z", "iopub.status.idle": "2022-12-14T22:20:47.776739Z", "shell.execute_reply": "2022-12-14T22:20:47.776134Z" }, "id": "7aFsD8csqEff" }, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tf.executing_eagerly()" ] }, { "cell_type": "markdown", "metadata": { "id": "x_G1zZT5qEfh" }, "source": [ "现在您可以运行 TensorFlow 运算，结果将立即返回：" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:47.780511Z", "iopub.status.busy": "2022-12-14T22:20:47.780043Z", "iopub.status.idle": "2022-12-14T22:20:51.511365Z", "shell.execute_reply": "2022-12-14T22:20:51.510613Z" }, "id": "9gsI54pbqEfj" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hello, [[4.]]\n" ] } ], "source": [ "x = [[2.]]\n", "m = tf.matmul(x, x)\n", "print(\"hello, {}\".format(m))" ] }, { "cell_type": "markdown", "metadata": { "id": "ajFn6qsdqEfl" }, "source": [ "启用 Eager Execution 会改变 TensorFlow 运算的行为方式 - 现在它们会立即评估并将值返回给 Python。`tf.Tensor` 对象会引用具体值，而非指向计算图中节点的符号句柄。由于无需构建计算图并稍后在会话中运行，可以轻松使用 `print()` 或调试程序检查结果。评估、输出和检查张量值不会中断计算梯度的流程。\n", "\n", "Eager Execution 可以很好地配合 [NumPy](http://www.numpy.org/) 使用。NumPy 运算接受 `tf.Tensor` 参数。TensorFlow `tf.math` 运算会将 Python 对象和 NumPy 数组转换为 `tf.Tensor` 对象。`tf.Tensor.numpy` 方法会以 NumPy `ndarray` 的形式返回该对象的值。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:51.515143Z", "iopub.status.busy": "2022-12-14T22:20:51.514482Z", "iopub.status.idle": "2022-12-14T22:20:51.520357Z", "shell.execute_reply": "2022-12-14T22:20:51.519692Z" }, "id": "sTO0_5TYqz1n" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(\n", "[[1 2]\n", " [3 4]], shape=(2, 2), dtype=int32)\n" ] } ], "source": [ "a = tf.constant([[1, 2],\n", " [3, 4]])\n", "print(a)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:51.523758Z", "iopub.status.busy": "2022-12-14T22:20:51.523167Z", "iopub.status.idle": "2022-12-14T22:20:51.529202Z", "shell.execute_reply": "2022-12-14T22:20:51.528592Z" }, "id": "Dp14YT8Gq4r1" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(\n", "[[2 3]\n", " [4 5]], shape=(2, 2), dtype=int32)\n" ] } ], "source": [ "# Broadcasting support\n", "b = tf.add(a, 1)\n", "print(b)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:51.532235Z", "iopub.status.busy": "2022-12-14T22:20:51.531776Z", "iopub.status.idle": "2022-12-14T22:20:51.536307Z", "shell.execute_reply": "2022-12-14T22:20:51.535735Z" }, "id": "69p3waMfq8cQ" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(\n", "[[ 2 6]\n", " [12 20]], shape=(2, 2), dtype=int32)\n" ] } ], "source": [ "# Operator overloading is supported\n", "print(a * b)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:51.539225Z", "iopub.status.busy": "2022-12-14T22:20:51.538766Z", "iopub.status.idle": "2022-12-14T22:20:51.542274Z", "shell.execute_reply": "2022-12-14T22:20:51.541714Z" }, "id": "Ui025t1qqEfm" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[ 2 6]\n", " [12 20]]\n" ] } ], "source": [ "# Use NumPy values\n", "import numpy as np\n", "\n", "c = np.multiply(a, b)\n", "print(c)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:51.545209Z", "iopub.status.busy": "2022-12-14T22:20:51.544749Z", "iopub.status.idle": "2022-12-14T22:20:51.548248Z", "shell.execute_reply": "2022-12-14T22:20:51.547634Z" }, "id": "Tq_aFRzWrCua" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2]\n", " [3 4]]\n" ] } ], "source": [ "# Obtain numpy value from a tensor:\n", "print(a.numpy())\n", "# => [[1 2]\n", "# [3 4]]" ] }, { "cell_type": "markdown", "metadata": { "id": "H08f9ss9qEft" }, "source": [ "## 动态控制流\n", "\n", "Eager Execution 的一个主要优势是，在执行模型时，主机语言的所有功能均可用。因此，编写 [fizzbuzz](https://en.wikipedia.org/wiki/Fizz_buzz) 之类的代码会很容易：" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:51.551238Z", "iopub.status.busy": "2022-12-14T22:20:51.550795Z", "iopub.status.idle": "2022-12-14T22:20:51.555143Z", "shell.execute_reply": "2022-12-14T22:20:51.554584Z" }, "id": "0fudRMeUqEfu" }, "outputs": [], "source": [ "def fizzbuzz(max_num):\n", " counter = tf.constant(0)\n", " max_num = tf.convert_to_tensor(max_num)\n", " for num in range(1, max_num.numpy()+1):\n", " num = tf.constant(num)\n", " if int(num % 3) == 0 and int(num % 5) == 0:\n", " print('FizzBuzz')\n", " elif int(num % 3) == 0:\n", " print('Fizz')\n", " elif int(num % 5) == 0:\n", " print('Buzz')\n", " else:\n", " print(num.numpy())\n", " counter += 1" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:51.558218Z", "iopub.status.busy": "2022-12-14T22:20:51.557725Z", "iopub.status.idle": "2022-12-14T22:20:51.566244Z", "shell.execute_reply": "2022-12-14T22:20:51.565641Z" }, "id": "P2cKknQWrJLB" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1\n", "2\n", "Fizz\n", "4\n", "Buzz\n", "Fizz\n", "7\n", "8\n", "Fizz\n", "Buzz\n", "11\n", "Fizz\n", "13\n", "14\n", "FizzBuzz\n" ] } ], "source": [ "fizzbuzz(15)" ] }, { "cell_type": "markdown", "metadata": { "id": "7kA-aC3BqEfy" }, "source": [ "这段代码具有依赖于张量值的条件语句并会在运行时输出这些值。" ] }, { "cell_type": "markdown", "metadata": { "id": "8huKpuuAwICq" }, "source": [ "## Eager 训练" ] }, { "cell_type": "markdown", "metadata": { "id": "mp2lCCZYrxHd" }, "source": [ "### 计算梯度\n", "\n", "[自动微分](https://en.wikipedia.org/wiki/Automatic_differentiation)对实现机器学习算法（例如用于训练神经网络的[反向传播](https://en.wikipedia.org/wiki/Backpropagation)）十分有用。在 Eager Execution 期间，请使用 `tf.GradientTape` 跟踪运算以便稍后计算梯度。\n", "\n", "您可以在 Eager Execution 中使用 `tf.GradientTape` 来训练和/或计算梯度。这对复杂的训练循环特别有用。\n", "\n", "由于在每次调用期间都可能进行不同运算，所有前向传递的运算都会记录到“条带”中。要计算梯度，请反向播放条带，然后丢弃。特定 `tf.GradientTape` 只能计算一个梯度；后续调用会引发运行时错误。" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:51.569557Z", "iopub.status.busy": "2022-12-14T22:20:51.569119Z", "iopub.status.idle": "2022-12-14T22:20:51.582072Z", "shell.execute_reply": "2022-12-14T22:20:51.581501Z" }, "id": "7g1yWiSXqEf-" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor([[2.]], shape=(1, 1), dtype=float32)\n" ] } ], "source": [ "w = tf.Variable([[1.0]])\n", "with tf.GradientTape() as tape:\n", " loss = w * w\n", "\n", "grad = tape.gradient(loss, w)\n", "print(grad) # => tf.Tensor([[ 2.]], shape=(1, 1), dtype=float32)" ] }, { "cell_type": "markdown", "metadata": { "id": "vkHs32GqweYS" }, "source": [ "### 训练模型\n", "\n", "以下示例创建了一个多层模型，该模型会对标准 MNIST 手写数字进行分类。示例演示了在 Eager Execution 环境中构建可训练计算图的优化器和层 API。" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:51.585315Z", "iopub.status.busy": "2022-12-14T22:20:51.584736Z", "iopub.status.idle": "2022-12-14T22:20:52.578372Z", "shell.execute_reply": "2022-12-14T22:20:52.577604Z" }, "id": "38kymXZowhhz" }, "outputs": [], "source": [ "# Fetch and format the mnist data\n", "(mnist_images, mnist_labels), _ = tf.keras.datasets.mnist.load_data()\n", "\n", "dataset = tf.data.Dataset.from_tensor_slices(\n", " (tf.cast(mnist_images[...,tf.newaxis]/255, tf.float32),\n", " tf.cast(mnist_labels,tf.int64)))\n", "dataset = dataset.shuffle(1000).batch(32)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:52.582608Z", "iopub.status.busy": "2022-12-14T22:20:52.582078Z", "iopub.status.idle": "2022-12-14T22:20:52.645312Z", "shell.execute_reply": "2022-12-14T22:20:52.644642Z" }, "id": "rl1K8rOowmwT" }, "outputs": [], "source": [ "# Build the model\n", "mnist_model = tf.keras.Sequential([\n", " tf.keras.layers.Conv2D(16,[3,3], activation='relu',\n", " input_shape=(None, None, 1)),\n", " tf.keras.layers.Conv2D(16,[3,3], activation='relu'),\n", " tf.keras.layers.GlobalAveragePooling2D(),\n", " tf.keras.layers.Dense(10)\n", "])" ] }, { "cell_type": "markdown", "metadata": { "id": "fvyk-HgGwxwl" }, "source": [ "即使没有训练，也可以在 Eager Execution 中调用模型并检查输出：" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:52.649484Z", "iopub.status.busy": "2022-12-14T22:20:52.648968Z", "iopub.status.idle": "2022-12-14T22:20:53.758160Z", "shell.execute_reply": "2022-12-14T22:20:53.757411Z" }, "id": "BsxystjBwxLS" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Logits: [[ 0.0270367 -0.01142959 -0.04025798 -0.04152567 0.06224848 0.04886491\n", " 0.03862026 -0.06363453 -0.04545979 0.04428613]]\n" ] } ], "source": [ "for images,labels in dataset.take(1):\n", " print(\"Logits: \", mnist_model(images[0:1]).numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "Y3PGa8G7qEgB" }, "source": [ "虽然 Keras 模型有内置训练循环（使用 `fit` 方法），但有时您需要进行更多自定义。下面是一个使用 Eager Execution 实现训练循环的示例：" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:53.762163Z", "iopub.status.busy": "2022-12-14T22:20:53.761641Z", "iopub.status.idle": "2022-12-14T22:20:53.768659Z", "shell.execute_reply": "2022-12-14T22:20:53.768045Z" }, "id": "bzRhM7JDnaEG" }, "outputs": [], "source": [ "optimizer = tf.keras.optimizers.Adam()\n", "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n", "\n", "loss_history = []" ] }, { "cell_type": "markdown", "metadata": { "id": "tXaupYXRI2YM" }, "source": [ "注：请在 `tf.debugging` 中使用断言函数检查条件是否成立。这在 Eager Execution 和计算图执行中均有效。" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:53.772064Z", "iopub.status.busy": "2022-12-14T22:20:53.771596Z", "iopub.status.idle": "2022-12-14T22:20:53.775869Z", "shell.execute_reply": "2022-12-14T22:20:53.775262Z" }, "id": "DDHrigtiCIA4" }, "outputs": [], "source": [ "def train_step(images, labels):\n", " with tf.GradientTape() as tape:\n", " logits = mnist_model(images, training=True)\n", " \n", " # Add asserts to check the shape of the output.\n", " tf.debugging.assert_equal(logits.shape, (32, 10))\n", " \n", " loss_value = loss_object(labels, logits)\n", "\n", " loss_history.append(loss_value.numpy().mean())\n", " grads = tape.gradient(loss_value, mnist_model.trainable_variables)\n", " optimizer.apply_gradients(zip(grads, mnist_model.trainable_variables))" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:53.778995Z", "iopub.status.busy": "2022-12-14T22:20:53.778761Z", "iopub.status.idle": "2022-12-14T22:20:53.782267Z", "shell.execute_reply": "2022-12-14T22:20:53.781667Z" }, "id": "0m1xAXrmqEgJ" }, "outputs": [], "source": [ "def train(epochs):\n", " for epoch in range(epochs):\n", " for (batch, (images, labels)) in enumerate(dataset):\n", " train_step(images, labels)\n", " print ('Epoch {} finished'.format(epoch))" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:20:53.785510Z", "iopub.status.busy": "2022-12-14T22:20:53.785048Z", "iopub.status.idle": "2022-12-14T22:22:12.948361Z", "shell.execute_reply": "2022-12-14T22:22:12.947266Z" }, "id": "C5dGz0p_nf4W" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 5 calls to triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:6 out of the last 6 calls to triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 0 finished\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1 finished\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 2 finished\n" ] } ], "source": [ "train(epochs = 3)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:12.951952Z", "iopub.status.busy": "2022-12-14T22:22:12.951696Z", "iopub.status.idle": "2022-12-14T22:22:13.387890Z", "shell.execute_reply": "2022-12-14T22:22:13.386993Z" }, "id": "5vG5ql_2vYB5" }, "outputs": [ { "data": { "text/plain": [ "Text(0, 0.5, 'Loss [entropy]')" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGyCAYAAAAYveVYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABw50lEQVR4nO3dd3wT9f8H8Fe6W7oo0AWFAmWPsqFspDJFceBABcSFFhVxfEUEFJXiQkURf4paB1NlKCCyNwgUClT2bCktuxtK29zvj0po2oy75C53SV/Px6MPaPK5u3euyd07n6kTBEEAERERkYtwUzsAIiIiIjkxuSEiIiKXwuSGiIiIXAqTGyIiInIpTG6IiIjIpTC5ISIiIpfC5IaIiIhcCpMbIiIicilMboiIiMileKgdgKPp9XqcP38eAQEB0Ol0aodDREREIgiCgLy8PERGRsLNzUrdjKCiadOmCR06dBD8/f2FWrVqCffcc49w5MgR0dvPnz9fACDcc889ordJT08XAPCHP/zhD3/4wx8n/ElPT7d6r1e15mbTpk1ISEhAx44dUVJSgjfffBP9+vXDoUOHUK1aNYvbnjlzBq+++ip69Ogh6ZgBAQEAgPT0dAQGBtocOxERETlObm4uoqKiDPdxS3SCoJ2FMy9duoTQ0FBs2rQJPXv2NFuutLQUPXv2xOjRo7FlyxZkZ2dj6dKloo6Rm5uLoKAg5OTkMLkhIiJyElLu35rqUJyTkwMACAkJsVhu6tSpCA0NxZNPPml1n0VFRcjNzTX6ISIiItelmeRGr9dj3Lhx6NatG1q2bGm23NatW/Hdd9/h22+/FbXfxMREBAUFGX6ioqLkCpmIiIg0SDPJTUJCAlJTU7FgwQKzZfLy8vD444/j22+/Rc2aNUXtd8KECcjJyTH8pKenyxUyERERaZAmhoKPHTsWy5cvx+bNm1GnTh2z5U6ePIkzZ85gyJAhhsf0ej0AwMPDA0ePHkXDhg2NtvH29oa3t7cygRMREZHmqJrcCIKAF154AUuWLMHGjRtRv359i+WbNm2KgwcPGj321ltvIS8vD59//jmbnIiIiEjd5CYhIQHz5s3DsmXLEBAQgKysLABAUFAQfH19AQAjRoxA7dq1kZiYCB8fn0r9cYKDgwHAYj8dIiIiqjpUTW5mz54NAOjdu7fR4z/88ANGjRoFAEhLS7M+EyERERHRfzQ1z40jcJ4bIiIi5+O089wQERER2YvJDREREbkUJjdERETkUpjcEBERkUthcqNRN4pLoddXqb7eREREsmByo0E514vRfPIqDPu/HWqHQkRE5HSY3MjocGYulu7LgL2j6zcevQi9ACSfvSZTZGAtEBERVRlMbmRSeLMEAz/fgnELU7Dl+GWLZa8V3MSN4lKjxwRBwKHzuZUel0PujWJ0TlyHV3/db7HcxbwbGPb1dizdlyF7DERERI7C5EYm+TdKDP//93wuAKBUL6C4VG9U7mrBTbR9dw3iEtch53oxikrKkpmlKRkYNHMLHpvzj+yxLd2XgUt5Rfgt+ZzFcokrj2D3mWsYtzBF9hiIiIgchcmNTDzdb5/KD1YdgSAIuHPGJsQlrkdG9nWcu1YI4HZT07XCYsS+sxrdP9gAAJj3TxoAYM/Za1Brzujc68XqHJiIiEhGqq4t5Uo8PYzzxN+Sz+HU5QIAQLfp6wEAqe/0r9TsdCmvCACgg84BURIREbk+1tzIxNPdODl57bcDlcos2XsOL8zfZ3oH5TYXIG/VTdVaPYyIiKo6Jjcy8RSxcvmkZf+afDz6jRXYdfqq4ffyyYi5kVc5hcUYvzAFW45fkhaoBcyBiIjIFTC5kYmbmzLNSol/HTH5+Eerj2Dxvgw8/t0uq/vQscWLiIiqECY3GnQ++7rh/99sPmWyzLlr100+borYZinmQERE5AqY3GjQx6uPGf2eU1g2iun05QLkcEQTERGRRRwtJaO4BjWw49QV2fc7+Y9UpF0txL60bADA1HtaoKT0dnWMIAhYuDsdzSICERsVbPNx2OeGiIhcAWtuZPTRsNaK7HdZynlDYgMAk5f9i60nbs+CvPHoJbyx+CDumbVNkeMTERE5EyY3MqpT3Q+7J8bj/nZ1MKprNLa/cYdDjnvsQp5DjkNEROQM2Cwls1oB3vjkwVjVji8IAlLSsxET6o8AH0/V4iAiIlILa24UlnhfK3SPqYmDb/dT7Bjlh3qvOJiJe7/ajru/NN1EdTm/yObjKLGoJxERkdyY3CjskU518ctTnRHg44l372mh+PH+SDkPoGxklSkd3ltrWPJBisS/DqPppFVGkw0SkeMIgoDswptqh0HkFJjcOFCrOsFqhwAA2Jd2TfI2/7epbL6dxL8Oyx0OEYnw8sIUtJm6BjtOyj8ik8jVMLlxoFa1g9CubjCGtonEfW1rqx0OETmRpf/Vyn618YTKkRBpHzsUO5C7mw6Ln+8GAHhl0X5FjiFmqQXOZ0NERK6MNTcqGdU1WrZ96Ry4cAJXGCciIq1jcqOSVnWCFNmvIxMdIiKlTVh8EE/9uBsCv1mRBExuVHR3bKRDjsOLAhE5q/m70rD28EUcysx16HE3HLmI77eedugxST7sc6Oiat7uRr/7ebkjPMgHpy6ZHsZtzvsrpY1gsifXEdOnh4hIbnq9Y4/3RNJuAEBsVBDa1wtx7MHJbqy5UZG72+1MYcnzXbF7Yjz0evlrWXQiM5LyNTwf/33UTBlZQiIiG4n9PJM8snJsn/iU1MPkRkX+3reXR4itE4xq3h4osTO5KX/du+uLLdifnm1Ts9SXG05wRmIiDWIzM5F1TG40wu2/Whw5a25SM3Lx2Hf/iC7Pb4REpFUCJ7EgCVRNbhITE9GxY0cEBAQgNDQUQ4cOxdGjpptDbvn222/Ro0cPVK9eHdWrV0d8fDx27drloIjlZerDKmfNDQDk3SgxeWQx+AWRiIickarJzaZNm5CQkICdO3dizZo1KC4uRr9+/VBQYL5D7caNG/HII49gw4YN2LFjB6KiotCvXz9kZGQ4MHJ5DGsfBQDoFH27s9qAluF27fNirvX24bWHL5p8nNXdRETGWGPknFQdLbVq1Sqj35OSkhAaGork5GT07NnT5DZz5841+n3OnDn4/fffsW7dOowYMUKxWJUQE+qPlMl3IsDndt+bCQOb4acdZ23e556z1teN+i35HD4eFmu1HFupiIjIGWlqKHhOTg4AICRE/LC7wsJCFBcXm92mqKgIRUW3azNycx07V4I1wX5eRr/7ermbKel4rMgh0h72jSOyTjMdivV6PcaNG4du3bqhZcuWorf73//+h8jISMTHx5t8PjExEUFBQYafqKgouUJ2OWIumsx3iIhI6zST3CQkJCA1NRULFiwQvc306dOxYMECLFmyBD4+PibLTJgwATk5OYaf9PR0uUJ2OexzQ6R9/JwSWaeJ5Gbs2LFYvnw5NmzYgDp16oja5uOPP8b06dOxevVqtG7d2mw5b29vBAYGGv04i2d7Nqj0WFigt8OOz450RETkjFRNbgRBwNixY7FkyRKsX78e9evXF7Xdhx9+iHfffRerVq1Chw4dFI5SPaaaifq3sG80lRT8gkhERM5I1Q7FCQkJmDdvHpYtW4aAgABkZWUBAIKCguDr6wsAGDFiBGrXro3ExEQAwAcffIDJkydj3rx5iI6ONmzj7+8Pf39/dV6IQnw9K3cutqUroa05CnMbIu2pqh2K+WWLpFC15mb27NnIyclB7969ERERYfhZuHChoUxaWhoyMzONtrl58yYeeOABo20+/vhjNV6CIt4e0hwdo6tjdPdobHi1Nz5/uI3hObkubJk512XZDxERkdaoWnMjpmPcxo0bjX4/c+aMMsFoyKhu9TGqW1kTXYCPJ+rXrIZWtYMQ7OeF+bvSJO/PVDqUdqUQEUG+Frdjx0UiInJGmuhQTNY1qOWPkGpeeLK7uH5J5ZlKUcSkLUxtiIjIGTG5cTI+nu7o0aim3fsRUynDihsi7WGNqmPxdDsnJjdOaOo94ic5BABTa3HaO8w7u/AmPll9FCcv5du1HyIiIrlpavkFEiciyPSEhea8u/xQ5QdtbJfan56NT1YfxcLd6biYV4T/23QKx94fKCkeIrJdVR0tRSQFkxsnJMe1rXze8sGqI9ifno2SClU85mp3vlh/wvD/m6V6+4MhIiKSEZMbJ+QmQ3ZTvh159saTVssQEVVFrChzTuxz44Tk+KyJ6XPD3IaISkr1mPdPGk6p3L9OresRv+Q5J9bcOCE52tz5gSUiMX7ccdbQb+/M9MEqR0MkDmtunJCbzH1uzJaRmAEVlZTaFgwRaVby2atqh0AkGZMbJ+So0RJSUputxy+jyVur8NXGE9YLExERKYjJTRUlplZGSsXNG4sPAAA+XHXU1pCISIPYhE3OiMmNkxoX38iu7XnBIiKyjpdK58TkxkmFB0qbyM8W9s5iTESuZfpfR3DHJxuRe6NY7VCILGJyU0WJSlz+K2KtCWvJvnPQm1rjgTQpp7AYq//Nws0STsBI1pX/+H+96SROXSrA3J1p6gVEJAKHgldRYpqlVh7MxJ6z17D95BWL5V5euF+mqMgRhs/ZiX/P52JMr4Z4Y2BTtcMhibQwp5ye7dqkcay5qaLEXJve/vMQlh/IxNWCm8oHRA7z7/lcAMCylAyVIyEST63V0LWQTJJ0TG6qKH7vInJOjv7sVvW+d1X71TsvJjdOqm4NP7u2V+tbEBERkdLY58ZJxTWogdrBvsjIvm7T9lOXH8LivRloHhkoc2RE5Ep0bJghJ8TkxknpdDpse+MOlOoFvLRgH5YfyJS0/blr13Hu2nWs+jdLoQiJXMv1m6UY+f0u9GpSCwl9YtQOx2HUbJZiDTPZis1STs7dTQd3ORaboiqH9w1pFu1Jx64zV/HR3+rOws1PO5F1TG5cgLNd7PR6AVfyi9QOg0iSG8VVb2FYvV5A8tlraoehCEEQUHizRFQ5cj5MbsjhRiXtRvv31iIlPVvtUKo0B62/Sk5s4Z50XM53zakgXl6YguaT/8bRrDy1QyEFMLkhReUXleDrTSdx9kqB4bHNxy4BAH7ZeVatsByqqKSU3/5INo58Jy3Zp525kOR+3UtTzgMAvtt6SuY9kxYwuXEBOg1/BX9/xWFM/+sI7vx0syz7+/vfLLy0YB8KiqxXJ2vBtYKbaDVlNZ5I2q12KCTByUv5SL9aqHYYRGQjjpYiRe06XbZ0g1zrGD37czIAoG6IH17p10SWfSppZWombpbqsfHoJbVDqYSVSabl3ihG3082AQBOJw7S3JcHbUXj+rT29ydxWHPjArT60Zu/Kw0nL91ujjp1KV+2fV/KK8LZKwWYteGE5lYoLinVY+T3u/DJanVH1ZBtsnJuqB0CEdmJyY0r0GB2c6O4FBMWHzR67I7/vg3Lpd+nm/HR30fxzh+HZN2vvdYfuYhNxy7hi/UnOAEakUzU+iSxv5xzYnJDiijRK39BKPqvqWv3mauKH0uKIpma4EhbeItzvPJ5Bc8/ScHkxgVorXbg4Lkc0RHtOXPVpjlv+GXKfuxKYF2Vf59V9ddPTovJjQvQ2k1qyJdbca3Q9NwYxaW3azU2HbuEB77egR4fbpB8jPL70drrdxZV/sZthrm3kxpvs4zs69A7oBaUyNUwuSFFmOuU+euec4b/X8orq7EpvCl95tfF5ebf0PJNmokX2erP/efRbfp6vLQwRe1QiJwOkxsXoMX7p7l845vNJxU/dkp6Nr7edBKlVewb7+X8IjzyzU4sS9HOxGtku1kbTgAoS3JUo6GLi5a/xJD2qJrcJCYmomPHjggICEBoaCiGDh2Ko0etD5/99ddf0bRpU/j4+KBVq1ZYuXKlA6LVruia1dQOoRJzicWZK5YnRssuvIlH5+zE78nnLJYrr2LtyNBZ2zD9ryNYuDtd9D5cwYerjmDHqSt4aUGK2qG4DN5PiZyTqsnNpk2bkJCQgJ07d2LNmjUoLi5Gv379UFBQYHab7du345FHHsGTTz6Jffv2YejQoRg6dChSU1MdGLm2PNm9PkZ3q4++TUPVDsVAb+PXrM/XHce2E1fwyq/77Y7h2AX114xx5Bff7EJtzfdDRKQWVZObVatWYdSoUWjRogViY2ORlJSEtLQ0JCcnm93m888/x4ABA/Daa6+hWbNmePfdd9GuXTt8+eWXDoxcW3w83TF5SHPENayhdii32fiVN+c6b9BEVBn7r5EUmupzk5OTAwAICQkxW2bHjh2Ij483eqx///7YsWOHyfJFRUXIzc01+iHlldraQC5jOwAvhkR2YrscOSnNJDd6vR7jxo1Dt27d0LJlS7PlsrKyEBYWZvRYWFgYsrKyTJZPTExEUFCQ4ScqKkrWuMm05LPXJJW/UVw2YsqWa+mtbU25kl+EkzIu+6BlWkrmbhSX4sGvdxg6xTqr8rPTOvo+r+U1jRwVWvlzzg7FJIVmkpuEhASkpqZiwYIFsu53woQJyMnJMfykp1etTqZq+WztcUnlh87aZvOxLuSangRQBx3av7cWfT/ZhKl/HsKZy+b7cinFkfcnLV38f92Tjl1nruKjv51vfS21c4r8ohIMnbUNhzO1W8uspfcakSmaSG7Gjh2L5cuXY8OGDahTp47FsuHh4bhw4YLRYxcuXEB4eLjJ8t7e3ggMDDT6Ie05kmVf599sM5MG3vL9ttPo88lGu45xy9x/zuLL9ZWTt0Pnc7Eq1XQNYlVzo9g5l6AoKCrBTzvOmnzOUTnPTzvOICU920FHI3JNqiY3giBg7NixWLJkCdavX4/69etb3SYuLg7r1q0zemzNmjWIi4tTKkxykLNXCmweZXVNxEghub5tTlySio9XH6tUEzRo5haM+SVZtRuT2jUOrmDqn4fMJjeO4qyJIZGWeKh58ISEBMybNw/Lli1DQECAod9MUFAQfH19AQAjRoxA7dq1kZiYCAB46aWX0KtXL3zyyScYPHgwFixYgD179uCbb75R7XVohbNXFff6aKOs+1PiZn+z3KKY+UUlJsscv3i7j4/W1v0qT5C5F8nVgpuYue44HuwQ5bSJ1sZjF41+d/KPlP009XdU5q/h7NdNMk3VmpvZs2cjJycHvXv3RkREhOFn4cKFhjJpaWnIzMw0/N61a1fMmzcP33zzDWJjY/Hbb79h6dKlFjshk3P693yO6LKOmo1428nLDjmOM3pz8UEkbT+DQTO3qB2KS3PWxNFZMflxTqrW3Agi3jUbN26s9NiwYcMwbNgwBSIiLVlz6AJaRAaJKvvj9jN4d6jyCe63m09ZLeMs9578G6Zrnmx1SOEOsIU3S+DupoO3h7uix6FyqsCNXc5kccfJK1hz6AJeH9AEPp58n6pJEx2KiUyR0qTz5wHx6++cz76OI1mVb8R6vYCLuaYX/Lxl+8krVvfvLPeDgpulOHiucu3Yv+dzMGfLKZSUaqfvx/WbpWg++W90nrbOemEZ8Vu7aerUHqnztUHKa33k2534fttp/N8m61+CSFlMbkiz3Oy8lpnbvOv09Rjw2Racz75u9PjLi1LQado6rP5X3IinUr1gtt+N1SA0Ys7WyhfhwTO34r0VhzFfQ2tz3VpKg0tMMOFyBmevOn7aCTLG5MaFtK4jrgnHWeh0QFFJKUYn7cYP205bLgsg7UohDp0X3zRytMLaU8tSymp/Zm0Ut3L5PbO2oeWUv3E533ienfL5jCMnEJS787KUcwnI30FZrI1HLyL9quUFWeXgqFen8XwYgOMSLOOuC8oeNCvnBqb/dQTnrin/XiLlMblxIZ0b1MB3Iztg/Su91A5FNr/uOYf1Ry7inT8PWS3b86MN0jqzmrtWSrxybz52yej38tXYSlZPZ+XcqJRYmTJj9VF8tvaYyeec4UZqydbjlzHqh93o8eEGtUOxaNOxS+g2fT22n7DeIZ0VM+p46qfd+HrTSTw25x+jx1lT5pyY3LiYvs3C0KCWv9phyEKn01lv9ilXVipzNQ0VH9XrBSSfvWpxmQej7R1wMSy8WYIuievQ4b210FsYKXYlvwgz15/AZ2uPI+9G5SYdOaf4L19zJPfSAeZ2t+fsVVmPU1H594g9r2jk97uQkX0dwyvcOG3B0VLKSM0oq6k8c8VyzY1eL2DPmau4flPc9YDUoepoKSJL7L2Iy3UT+HbLKST+dQTdY2rKs0MZlF9yolQQ4Gbm1ltcevvmrNdO/2AixQmCoMj6XN9vO433VhxG5/ohWPgsJ4/VKtbckGbpoLOrFkSuC9vPO8tmrN0qokmh7LiyHNYh5AxVrloOU7Q8GSI5hpRrwaW8InSetg7T/zoiexzz/kkDAPxzWtlaQ7IPkxvSrIpJwjaRyYVY5i6WFR9Xopnpr4OZuOuLLTht42KeYuaIUpOp6Jbuy8Cwr7fjYp7l4fZE9vpm80lczCvC15usDw5Q5KOk7Y9nlcDkhjRLB+PagEdl6K8wc53p1crLJwsHM3KwXcJMxGJrasov7vnc3L1IzcjFq7/uN1u+qKQUaVba/6XEIXU0k9w1UOMWpmD3mWuYvlL+b9NKUSOHtHbatZDXqlE7KeWYip4jViI6BSY3pFlLU84jT+ZZdGesuT1qKGn7GSzdl4FZG06g2/T1RuWGf2t7ImXu2tfx/bWVHrM0S/D9s7ej50cbRCdaNl3QLVyo7WoStPBcroXXLAgCjl3Is9hJmkhOztSMTOKxQ7GL8vZwQ1GJc/cgPZyZi8Mip/S/WnCz0mPWrllbjl/GluOOWyuqfOfeWyxN9nVr9MZvyefQtaH9nZmdod/Kp2uOYeb6E6gdXLZw7sxH2qB9vRCVo9IWLdyMtVB7pGka+BtVday5IbJTcamASUtTDb9Lue7fKBaRgIrcoS03PUsJj05XVpNyJCvXaDV0e2Ox9NzM9ScAABnZ15GRfR0jvttldRtHcdT9nHnDbUZT+GnkxIh6K2ok1qqMyQ25LgfdEBftTjeMqFLDrZdpy8XfWtKwYHc6Bny2Bc/+vEf6zmXg7LWPRKQOJjdEVlgbmZRVYbFNNSoZlu7LwCqRa2JJ8f3WsmUvNhy9ZKWk+ORKyvkxtUu1RoppoPKIiERicuOiTF3+n+xe3+FxVAUV77XHLii/nlTFFXfGLUwxen7OlqqxKrFWmirkpFQSlZF9HYNnbsFvyecUOoJrulFcipcW7MOf+8+rHQpJwOTGVZm46E+6q7nj41DR6UuOWZm3Yk1CRoXVxss7cTEPE5ccVDokvLficKU5dEwNBZfzRlq+ievbzfIkV6wtkc/UP//Fv+ctTz+gZWrlsUnbz2BZynm8MH8fAPmXFiFlcLQUuazVhy7Ish9rF1UpF90hX2zDdZFrVFki5vJ6rfAmfIN87T6WLc7nyDNRn+lmqduJlNL3GXtqhs5cLsCaQxfwaJe68gVkh4Ii+dZC0vr9Xcqfzdrf+IqJkZikfUxuiOwk5QYoR2IDiLt4i5krRs6blOg+Nxq/McqlzycbIQjA+RzzNXkknlpvm4rv6yry9nV6TG6qgDG9GqK6n6faYbgsqTP/ynJMEYcUMw+enHPflCow8Z6p6LQ4PNiUW7HtPsM1iByNCQixz42LKn/DfWNgUzzbqyEA4PfnuIqtszF1/16y73anUHM3eH2FJ5SexC+/SNxs0lLiuPUKnKm2RxAEl+98qkZSKampSUJZZ3pvkXhMbqoYzvYqnbULuRoX+p92WJ9XR68XjC7catQwKU3xPjflzpnYs/fngUxD51Nnp/Z7Rhs1cxW+JDAZcgpMbojsZG9rjFLztpQKgtWbg6wXagVehs7w7+1Atb4ierLCzVC8t1rnzOfoq41la91lydQpv6picuOiNH79dypKf3tV6m/l6LUnlTic1ZFqDnyNWrlh8qNtXcVzVFKqR/LZa5KXESmj/F/+3LVCJJ+9BgD4cNVRZGRfx6flFvkl6ZjcENlNm7cbMTUcFteAgjKvzN7aIm2ebfOc+YuGuf5RztY08+HfR3H/7O343+8HbFhxXvk/YPcPNuD+2dtxNCvP8FipM79xNIDJjYvix8J+F3JvYNdp600M9l6D7P1b2VezZP4upYX3kLV7qJbnuXEWer2A/enZuCHTNAX2xKFUk+M3/00quWRfBvrO2ITiUvE1OJWHgiv3pjuYkaPYvqsaJjdV0IoXu6sdglPoPG0dHvy/HbiQW2SxnN3JiYbvoJYu46V6AeMXpmDuP9IWDdXpyqa0v5BrvU+BlpqlbKX1Wo4ftp/BPbO24Zmfk1WL4UZxKXp9vAEJ8/YqfqzTlwtwJPN2DYnVAQM2HEOOt6UzvLe1jMlNFdQiMkjtEFyK0smJqf3bcr80FaY9zVIrD2Zi8b4MTFyS+t/+xZ+HTu+vRedp63Diorh1uIxGfbnoRf/W+XN0MvTj9jMAgM3HrC+OqpSNRy8h/ep1rDwoffHXopJSJMzdi0W70xWIrDKtJ6tUhsmNi9JybYCrsb9ZSTlSL8TXREw1n114E+lXC5F7o9jocbGv4/rNUuTeKJsTZ41MS2Q4u6l/HkKfjzci70ax9RFujgkJgPkmTy1dXhbuTseKg5l4/fcDordRe4i7Obxuy4czFBPZye4+NzZsr9Ql8Pttp41+N3UjbTN1DQBgbJ8Y45hEBnX2SqEtod0+TrlXbyl5++tgJo5fzMcLd8SIXuyw4muw5TxXPJaY83LrvC90UO2DMzJ3HrMLi00/IRNbkklW7qiPyY2LYv7vOPZ+27L3W6TYwwsAftlp3D+m4kW4uNR4Z+V/S83IMeoTsf9ctugYzcckLnhzuYml1/7c3LJYO0aHIK5hDamhVfLh30ft3oc1VfeLu5kaIgdcyazlvWr1uSH7MLkhspPSF7Ly+5+28jDOXimweV9vLU01+l1Ks9Xzc/ci7ap9tS5quJxvuUN4eebOx9GsPNFrZ9mT7Ir5e/DGaUzpWhJHNhWJrWFUyu4zV3HmcgGGdYhSNQ45MLkhspeDmqVKSvWGIa22Hcf6gSp+Uy5/qa24orlaF2Kp9xp7/jy3zpnYdbOqCnaqVUb5z6ga/YKGfb0DANCglj/a16vu8OPLSdUOxZs3b8aQIUMQGRkJnU6HpUuXWt1m7ty5iI2NhZ+fHyIiIjB69GhcuXJF+WCdTNWt3nZuqRk5SDdROzJj9VE0mbTKrn2/+ut+ydtIW6zQ8W86Ld5kKyZ9csS4LCXj9v7s351oSi+26qqc/ayZugY5G1WTm4KCAsTGxmLWrFmiym/btg0jRozAk08+iX///Re//vordu3ahaefflrhSInMK7JpSvfKzmdfx11fbEWPDzdUem7m+hOim0XM2XBUxFBfOw4hOqGW8cov5pjONgLFVDI0/a8jVrfbm3bNaIZbkkfFd4+YGktz77iTl/Lx886zkiYRJNuo2iw1cOBADBw4UHT5HTt2IDo6Gi+++CIAoH79+nj22WfxwQcfKBWi0+rZuBY2H7uEejX81A7F5d2080J169777/lcGaKRxtI386ycG0bPKpUjXM4vwpM/7sHDHc2385uL03Fpi+1HknLe3ltxGOPiG0nex6W8Itz31XYAwJnpg6WEZ5HctXGCIODf87moX7MaqnnLc/uxpWZMrXy37yebAAA3S/R4snv9Ss87IqysnBuoFeANdzdLs5M71xcCU5xqnpu4uDikp6dj5cqVEAQBFy5cwG+//YZBgwaZ3aaoqAi5ublGP1XBZw+1wWv9m2DBM11MPr/8he548Y4Yk8+VZ+mGQ/J4+qc9yCksxtM/7ZF931JXBS9ffN2RixYvcTZXwJTbaWmpgE9WH8P+9GxMWHxQ0m5mrjuOmeuO2xqFKGpc4k39zazdbM5nX1coGtNsTQ5WH7qAu77YiiFfbpU3IDtIfS1yVDzuTbsmw16k23j0IrokrsMzClxrtMapkptu3bph7ty5eOihh+Dl5YXw8HAEBQVZbNZKTExEUFCQ4ScqqmrcrEOqeSGhTwwignxNPt+ydhDG92vi4KjIlK0nLmPFwUyzz284ctGm/b72m/hJzW45c9n2kVhi7xHly32y5pioZRhMzVA8Q4FVk83f6Jy9F4U49jSXXL9ZisKbljte3+o7dOqS7e+zipTu2K5ILY+ZfSr9Lvtua9l8SutsvKY4E6dKbg4dOoSXXnoJkydPRnJyMlatWoUzZ85gzJgxZreZMGECcnJyDD/p6Zwky5IOJnrIj+oa7fhAqhhL38xvzeYr1aW8Iqvf+CteTFc7YMbg0xUSqPUauNCWlOqx4kAmLuaJHzaulIrnB3BcM8q4BSk2bVeqF9B8yio0n/y3TQmSs/admrIsFa//Jr2jPinPqZKbxMREdOvWDa+99hpat26N/v3746uvvsL333+PzEzT33y9vb0RGBho9EPmmWqHffvuFipEUrXcWp9Jy6x+QVb53pN+tRB3fLIR8/5Jk7xt0vYzVhZtdNyL+2P/eYvPXyssxp/7z5tdxbukVI+VBzNxMc96jVhFlmoQLcm/UWJIUMwt4WHLquPXbyq/Urk9FT8/7jiLRXvOIUNis6DZZS3M/kJSierR9ccff0je8Z133glfX9NNIrYqLCyEh4dxyO7u7gC0mdUTOYKlt/7es9l27dsRHYqlMndjmLr8EE5dKsCbSw5ieOe6hsd/2HYGbaKCUa9GNbP7NFd7pJXXXF5KejZemL8Po7vVx+QhzSs9//2205i28ghCqnlh76Q7FYlBakIw95+zkhP4TccuYeT3u6QdSCV6MyMZS0r1mLXhJLrF1ECH6BAHR1W1iUpuhg4dKmmnOp0Ox48fR4MGDSyWy8/Px4kTJwy/nz59GikpKQgJCUHdunUxYcIEZGRk4KeffgIADBkyBE8//TRmz56N/v37IzMzE+PGjUOnTp0QGRkpKUaiqsBybYT1/grHLazarbU5ZszVDKSkZ6P3xxtxOtGeUUTqvlhTt84VB8+bTG7WHi5L1K6KWATVUSwlNssPnMdXG05WenzKMuVqM6XkrPoKGa6Y9/2tmbwX7E7Hp2uP4dO1xqPYtJg0uxrRzVJZWVnQ6/Wifvz8xA0/3rNnD9q2bYu2bdsCAMaPH4+2bdti8uTJAIDMzEykpd2uYh41ahRmzJiBL7/8Ei1btsSwYcPQpEkTLF68WMprpnIGtgxXOwRSkT03wPIX6E9WH7XYvyf57FWbjwOIq0GylKgJArBk3zm7YpDLlfwijPx+F/6S0AQkddSbMxk7bx8OZWp3FGueDX3eks+WjYayp+O0mvmPKyRfompuRo4cKamJ6bHHHhPVt6V3794Wm5OSkpIqPfbCCy/ghRdeEB0LWfbVo+1QeLMULab8rXYopII/9p/HzEfa2rTtzXKTF36x/gQ83c3fYe+fvcOmY0hh7f7+8sL9uLdtHdmOJwgCCm6Wwl/ifC3T/zqCTccuYdOxS7LOSVNVlU/sSkr18HC3/p3dnlxQbCLpCrP8OjNRn8offvhB0k5nz55tUzDkeDqdTrbJtKhq2XFK3LIn4xemKBvIf8rfdF6cv0/StiWlZr5kCZX+Y5Awby9WHszCqnE90DRc/EAFLTUXuZq2U9dgQMtwBPp6WizniIqJHh9uQLMI2wewcBZj+0geLfXDDz+gsJAZqaty5uptV/KnlREz5TlqNlFrq2ubq4RdvC/D9BMilOoFCIJgPM+NiO2sjTgqb/pfR7DrjPRms5UHswAASdvOSN5WOm21E1hqtlAz0ryiEvyafA6L9oif8kPJJpjDZprbxBxz+YFMZBeKT4TlHFTjCvcBycnNG2+8gfDwcDz55JPYvn27EjGRBg1uHaF2CFXKCxJrHuyxaLc8cz8pdY+Y8se/osrZej3+elPlzqzy7Z3UYEs/GaWV2FAT02bqGvyebL6v2K19frL6KDq+vxaZOZaHpJfvl7b8wHmzX6Jcoc+N5OQmIyMDP/74Iy5fvozevXujadOm+OCDD5CVlaVEfKSyW5+FED8vdQMhxbz+u/SZjB3ppx1njX439w1V6Zlq5WLLfUNrNxtLp3rOllM4dkH+BTzNnQJHr1xu6/Hu+G9dKcBCbauJh1/51fQkgRdyb6D9e2vx5pKD+GL9CVzOv2lyOZIzlwvw886zuFmiN/rsjJ23Dy/M34fcG8XSXoiTkJzceHh44N5778WyZcuQnp6Op59+GnPnzkXdunVx9913Y9myZdDr2VboalxhITUyLzPnul1LL2iBEre42+97dd//ah3dlqTq2y2n0e/TzXYfW8lkVe550cTsLs1EB2O9XkDnaWttOubcf9KQc724wqSVlc9Z7483YtLSVHy75ZTJ/dxwwESJarBrhuKwsDB0794dcXFxcHNzw8GDBzFy5Eg0bNgQGzdulClEIrIk97r9VfBxievR++ONdu2jqk6kWWnxUSvnQa5b9oVcZZeKuH6ztNIcL45U8TzmXjddw2BvDiR1ezlyrlsv7d/zucZ/Rwn79vV0N7Vns+V3S+hX5gofZZuSmwsXLuDjjz9GixYt0Lt3b+Tm5mL58uU4ffo0MjIy8OCDD2LkyJFyx0oO4FlhGOWtN7krvNldVZfEdWqHAEDp2oXbV32zzRMqtUrd+my8tfQgZqw5hh8rNKM5owu5N9Bs8irsTcsWVT4zV/lVyZ/5Odmm7czNHiwnW997lZJHCaHackxnabqVg+TkZsiQIYiKikJSUhKefvppZGRkYP78+YiPjwcAVKtWDa+88goXqHQy79/bEnVD/PDuPS1NPn9H01AHR0QklZIXbsv7PnExD7/sTDPZ50GOLwbWa4Pkfe1SRusBwIDPtsh27Nd/24+l+zJkuxFPW3lYlv3ccvyC+Vm7HanqpCm2kTzBSWhoKDZt2oS4uDizZWrVqoXTp0/bFRg51qOd6+HRzvUqPX7r+nJH01AsfKYLHvpmp4MjIzL+lmp+hmL5j3v7WJaTixvF4vsZ2tShWLHCps+bI77hr0rNwgATM6Qv2nMOi/acQ/2a5tcCk2LOVsv3otSMXElDx6/bsACoo2QXFuNoVh6ahAdUes7c58bUw65QwSO55ua7776zmNgAZR+MevUq3yjJeel0OnRuUEPtMEjDqmrTpbPfCGz5u8nxtx7zS7Isq37be/oPZebi9d/EjRgsPyt3eVLPhxwfFVPvu79Ss9D/s804dN7c/DrijuwKn2Wb+tysW7cOd911Fxo2bIiGDRvirrvuwtq1tvX4JiKSgyNzjK82nrBe6D8Vb0K2xOnom42jzqW5ZMGc4xfyVO24/smao6oduyJLTZHbTlyu9Jj5ofSuSXJy89VXX2HAgAEICAjASy+9hJdeegmBgYEYNGgQZs2apUSMRES3mblKrz50QbFD7k/PMfr9w1W3b3KCAOwUuRSF3CYsPoh3lx8yekyOaRus1UbJVlslcT93froZM9cZJ5a2xGLrGVq81/Rs21JjUCM/EwQB+UWVR1a6QCWNSZL73EybNg2ffvopxo4da3jsxRdfRLdu3TBt2jQkJCTIGiARkVpuXfinVkggKnpvhbydVsWav6tsjpNBreSdQdxR3+anLEuVvM2na4/hpfhGCkRjXY6Z4eiOIAgCnv05GXoB+HZEe8tlzaQsRzLFTa547GIeTlzMQ4lekLRumpZIrrnJzs7GgAEDKj3er18/5OTkmNiCXFV0DT+1Q6AqovzNVmsTSlr71m7HaN9y+7C2lbznxFFDhpemSBuVpTapzWjmSf975VwvxupDF7D28AVczld28dX/23QK8TM2Y8BnW0zW9jgDycnN3XffjSVLllR6fNmyZbjrrrtkCYq064H2dQz/fzwuWr1AiKqQQgkdb+UYTSalrLnE60LuDfE7cXJyNTNZStzLH0OAYPFvJGez1zUnXcVecrNU8+bN8f7772Pjxo2GUVM7d+7Etm3b8Morr2DmzJmGsi+++KJ8kZLD/P5cHO6fvcPkcx890Bq//beQm6t2RCPtkbMmITUjB3vTruGxzvXg5mZ5v2I6rxZJGAYO2Pa5KZEwEZ25ko7u5zE6aTdWvNhD8eM4em0ptdj7EXCFEVBSSE5uvvvuO1SvXh2HDh3CoUO326GDg4Px3XffGX7X6XRMbpxU+3ohZp+rSjNcknZMWnq7b4a9F+m7vtgKoKyJoWXtIPt2BmDxPtOdTG85cdF40rebNqwObZ28n0s59vavmeHIjozBZVn5DJi7TFely7fk5IaT8xGRo201MbTVXmI6AZ+5Uoga/t52HadiMrPl+O3XMuCzzVia0M2u/ZeR+Wu5lbvgh6uOomN0CJqGB6Cal+TbiKy0cMOWa7SUlFooS180Te3f1r5qzlrjY9fCmYIgVNnF8sg2tYN91Q6BnJwjrziPKDwj95GsPDSdtErRY9hCzC122Nc70Ort1YrFUHBTuY6s1m5bFWvbHEVKAqKBnE7TbEpufvrpJ7Rq1Qq+vr7w9fVF69at8fPPP8sdG2mC5Y/QB/e3clAcRI6nTBOSEsotLGrmzq1Uh+Lf954TX1gCpVc9t2TsvL2SyjtihmKdiMVjrT1vS0KkhZoxW0hObmbMmIHnnnsOgwYNwqJFi7Bo0SIMGDAAY8aMwaeffqpEjKRROh3wUMe6OPJu5akBLG1jSo9GNWWKiqhqu5hnf1IgpXnkNZFLF2jJXwczLT6fd8N0rZGqN3qVOhRvOnbJvgOrRHJj6RdffIHZs2djxIgRhsfuvvtutGjRAm+//TZefvllWQMk7fN0t6t1EwDw3ciOaPzWXzJEQ67ucn4RQqp52bTtsK+3yxyNNpS/6Z67dt3ouTOXC/DYd/9UetzUtpYecyXWFtM0Rwd5mkWV7s5h7s9n6qjWQnlraSoe6+J8a0VKvitlZmaia9eulR7v2rUrMjMtZ8PkjOSeHMz04x5WhuQS3fLMT3ts3nb3mWuStzmaJW5WVzVZukFN+eNfs4mNtW2rKnPJh9nmHgdcvv6psMSHxXluTD0mVK1+OpKTm5iYGCxatKjS4wsXLkSjRupMiU3Oz9W/KZJ8zlwpdOjxEiT2v9CaYhP9hi5ZabqS8+P44/YzMu7ttnWHL+CnHWcUmUH3fI7pCQjVTASf+TnZKA7Zlvdy0Wuv5Gapd955Bw899BA2b96Mbt3KhjBu27YN69atM5n0kLMz/8635TPh5qqfJHJZ+Wb6X2iJ1I9Vx/fXYvydjfFi30ZIu1o5WZTrY5p2pRBT/vhXnp2Vk3z2Gp780fYaPLk5OumxNqrK1BpYZbMaV/7D/rzjLJ7oFm33lAdaI7nm5v7778euXbtQs2ZNLF26FEuXLkXNmjWxa9cu3HvvvUrESBpX/uPycMcoTL/P/Agqc9dMTg5IZDtLnx5zH60Za46ZfPyxOf/INuvvtUJlpu4/eC5bkf3aI8tMbY8pBWaW05CSJFm6Zs7eeLJyeTN/0y83nJAtUdx95irSTSTLapBUc1NcXIxnn30WkyZNwi+//KJUTKQhXRvWkFR+VLdoNA0PxBuLDyoUEZFjaW2hTlMsRWjphjlny6lKj209cRkdo83PUi7FMz9rp3ZFSW//+S/WH7kouvyu01ftPqac3wdT0rPt3seRrFwM+7ps2Z4z0wfbvT97Saq58fT0xO+//65ULKQhOybcgTkjOuCu1hGStrP2jY81NETys7VZxNwszXJ9TNWcq8aRpCQ2rurAuRy1QzAiuVlq6NChWLp0qQKhkJZEBPkivnmYxWTE1HP2fMvdMeEOm7clqsrMfUxLSvU2JSpa/wqi/bo025SKzFJt7VBsqfP1zRJnmbBSHMkdihs1aoSpU6di27ZtaN++PapVq2b0PBfLrHokzXxq4bmIIC7NQNrjzEOlu05fj/o1q1kvWAErWNUxcUmq9UL/OXmpQNK+rf1N3/7TfMfvnMJiBPl5Sjqe2mxaFTw4OBjJyclITk42eo4rgVctNl0AedEkJ+MMuY25BOxiXpFNk2xqvflY29GJU1KqxzebK/d5EitJ4hB7a3/Sef+kmX0udupqbH6tD+rW8DO/f0nRKI+rgpNDae0DQOQKChVcZFKLnCHhtObnnWexwsoyEOao8fr/Ss3Es70amn1ea38TySn91KlTUVhYeajX9evXMXXqVFmCItdk65T5RGrKKaw8Z4jWPP7dLrPP6Z25Xc2Msw6eyFEJhzNzRZfdL8NopqpGcnLzzjvvID+/8nLwhYWFeOeddyTta/PmzRgyZAgiIyOh0+lEdVQuKirCxIkTUa9ePXh7eyM6Ohrff/+9pOOSvKxVYVf388SS57tiwyu9OYkfyWLFAcct9eI8K4ObZktyo/WPqdQmGS3KvS6+tu2eWduMfrdlbaptJ65YL2QHrb1lJDdLCYLpWQ7379+PkBBpcyMUFBQgNjYWo0ePxn333SdqmwcffBAXLlzAd999h5iYGGRmZkKvd+6Lj7OS8mZuW7d62TYyfgL+N6ApPlh1RL4dktNw9iURHMmWihu5JvEj81b9m6X4MRbsMt+PxtWJTm6qV68OnU4HnU6Hxo0bGyU4paWlyM/Px5gxYyQdfODAgRg4cKDo8qtWrcKmTZtw6tQpQyIVHR0t6ZjkeEp1TqxTnaOriKyxpeZm41HO2+LsSkr1VXoyVdHJzWeffQZBEDB69Gi88847CAoKMjzn5eWF6OhoxMXFKRLkLX/88Qc6dOiADz/8ED///DOqVauGu+++G++++y58fU3f6IqKilBUdHsiqdxc8e2cJF0tE+uTNIsIMPx/QMsIHLtwHFEhvki/an6lYgAY3DoC0+9rhdh3VkNf4fr8cnxjWeIlcnW21Nz8I8MMuqQcMUs9VLxm2kvrTZUViU5uRo4cCQCoX78+unbtCk9Px495P3XqFLZu3QofHx8sWbIEly9fxvPPP48rV67ghx9+MLlNYmKi5L5AJN3yF7qjqKTU5OJrnz7YxvD/sX1i0DjMH10a1ECH99ZWKntfu9pYvDej7BcBCPDxRHTNajhVYU6H+rWqIU7i0hBERK7gxfn7HH5MZ+uXLrnPTa9evaDX63Hs2DFcvHixUn+Xnj17yhZcRXq9HjqdDnPnzjXUHM2YMQMPPPAAvvrqK5O1NxMmTMD48eMNv+fm5iIqKkqxGKuUcql8y9pBZouFBvoY/u/l4Ya7WkeaLdsoNMDscxXV9PfG2vE9ET9js+htiKoaVxwtVdWdl7BIZ1UlObnZuXMnhg8fjrNnz1bqsa3T6VBaanq1UzlERESgdu3aRk1izZo1gyAIOHfuHBo1alRpG29vb3h7u9ZS7lXGrdzJxLX51lOBvs41ayaRo5XK3T5BVdL/bT5lcZ4brZE8FHzMmDHo0KEDUlNTcfXqVVy7ds3wc/Wqsu203bp1w/nz542Goh87dgxubm6oU6eOoscm9QSYSGCcrf2XSC25N6rWBH9U5ve952Td39WCm7LuT2mSk5vjx49j2rRpaNasGYKDgxEUFGT0I0V+fj5SUlKQkpICoGz245SUFKSllQ1fmzBhAkaMGGEoP3z4cNSoUQNPPPEEDh06hM2bN+O1117D6NGjzXYoJuc1tE1tAMCnD8ZWeo5DVYmIzJugwEip35PlTZiUJDm56dy5M06cOCHLwffs2YO2bduibdu2AIDx48ejbdu2mDx5MgAgMzPTkOgAgL+/P9asWYPs7Gx06NABjz76KIYMGYKZM2fKEg9Jo3R6Ed8sFADQoJY/Hupgpp8Ua9yJiBzilV/341JekfWCGiC5z80LL7yAV155BVlZWWjVqlWlUVOtW7cWva/evXtbnGkxKSmp0mNNmzbFmjVrRB+DnItQLluxND8Om6WIiBwvv6gEtQIq92PV2mKrkpOb+++/HwAwevRow2M6nc4wc7GSHYrJeXi661BcKl+1SsXPjbY+RkREpCVcFZwUUae6H05fLrBeUKTGYcZDxDX2JYGIqEowd+nNva6tBWYlJzf16tVTIg5yQpYSDFsWdrPk8bh6uF5cio/+Pmr0uIe75G5jRERkI1NX9tSMHExdfsjhsVhi053h559/Rrdu3RAZGYmzZ88CKFueYdmyZbIGR1WPuZzI090NCX1iKj0eUs1L4YiIiOiWv00s+Pn9Nu216EhObmbPno3x48dj0KBByM7ONvSxCQ4OxmeffSZ3fKRhloZjK9+5TJ791w7mFAJERGKdvWK9u8Gq1EwHRGKZ5OTmiy++wLfffouJEyfC3d3d8HiHDh1w8GDVXYGUjMndLFVR+dypYh71WJe6ovfz17geMkVERFQ1GdYD/M+YX/YiNSNHpWjKSE5uTp8+bZiXpjxvb28UFMjXgZSqjmd7NTD8v2N0iKhtLNXbSFkx3MONPZOJiOR28lK+9UIKktyhuH79+khJSanUsXjVqlVo1qyZbIFR1dGubnXD/zvVD8HcpzqjXg0/0dvrYNzJzdTK5EREVHVITm7Gjx+PhIQE3LhxA4IgYNeuXZg/fz4SExMxZ84cJWIkjVKqW023mJoijn374PY0gHEZByIi1yM5uXnqqafg6+uLt956C4WFhRg+fDgiIyPx+eef4+GHH1YiRtIoNVt0yh/aTadDqQ19fL4f1UG+gIiISDNsGgr+6KOP4vjx48jPz0dWVhbOnTuHJ598Uu7YSKOe6BaNpuEBuDu2tmoxlK81Snqio037aBwWwMkAiYgkMb5olpTqVYrDMsk1N+X5+fnBz0983whyDVOGtJB1f/bmF10a1JAlDiIiskynK0toLuQV4ZtNJ7Foj+mVwktkXH7HFqJqbtq1a4dr166J3mn37t2RkZFhvSARlFnY+wcRtTmm5uL5YZRttUBERFXFkz/uQbfp6/HjjrO4Xmx6PclXft3v4KiMiaq5SUlJwf79+xESIm6YbkpKCoqKnGNZdHJORvPcmHi+T5NQk9v9OiYOw77eAaDyXDxrx/dCTKi/XCESEbmkTccuqR2CVaKbpfr27St6YjatLX1OrkeJUU6c8oaIyLKUtGy1QxBFVHJjy0rgderUkbwNVU225BR1qnPZBCIiRzuUmat2CKKISm64EjgpSUqfm9/GxOFCbhEahQXYdCwfD3d4ubvhZqkeoQE+EModnTWORESuwa7RUkSO1sHK8gw1/S2vEq7TAQfe7gdBALw83FBUYrozHBEROS8mN6Q6e+tLyte4zHy48rpnFfl43l7wlTMUExG5Hpsm8SPSqloBt9eVio0KlrQt0xwiItfA5IZcSvluMwuf6YL5T3dR7Fg1qlluAiMiInVITm7S09Nx7tztGQl37dqFcePG4ZtvvpE1MKo6WtQOUmS/Pp7uaB4ZKHm7UV2jRZVLnnSn5H0TEZHyJCc3w4cPx4YNGwAAWVlZuPPOO7Fr1y5MnDgRU6dOlT1Acl27J8Zj7fheqB2s3LDuQB8P9GhkfZXx8hL6xJh8PCqEw8+JiJyB5OQmNTUVnTp1AgAsWrQILVu2xPbt2zF37lwkJSXJHR85qeia1ayWqRXgLcuMwJb6yuh0OiQ90cns84Iiiz8QEZGaJCc3xcXF8PYu67S5du1a3H333QCApk2bIjMzU97oyGl9eH9r3Nu2Nn5/rquDj2x7t+Bb/XXMTXcjcoJuIiJSmeTkpkWLFvj666+xZcsWrFmzBgMGDAAAnD9/HjVqcHVmKhMa6INPH2qD9vWqK34sazmHpXTHvVwm4+flYbU8ERFpn+R5bj744APce++9+OijjzBy5EjExsYCAP744w9DcxWRWkzVuliaeNjD3Q2fDIvFjZJSo2HkYvdNRETaIzm56d27Ny5fvozc3FxUr377W/kzzzwDPz8/WYMjEqP8gpe2dE6+v73xOmhchoGIyLlJTm6uX78OQRAMic3Zs2exZMkSNGvWDP3795c9QCJrdDodUt/pj9JSwWj24fLPSyGY6Vxjb5+bHo1qYsvxy/bthIiIrJLc5+aee+7BTz/9BADIzs5G586d8cknn2Do0KGYPXu27AESieHv7YEgP09Fj2FvchPXkH3SiIgcQXJys3fvXvTo0QMA8NtvvyEsLAxnz57FTz/9hJkzZ8oeIJFWsLWKiMg5SE5uCgsLERAQAABYvXo17rvvPri5uaFLly44e/as7AESaYW9c/JwKDkRkWNITm5iYmKwdOlSpKen4++//0a/fv0AABcvXkRgoLSp7jdv3owhQ4YgMjISOp0OS5cuFb3ttm3b4OHhgTZt2kg6JlVtQb62N129c3cLGSMhIiKlSE5uJk+ejFdffRXR0dHo1KkT4uLiAJTV4rRt21bSvgoKChAbG4tZs2ZJ2i47OxsjRoxA3759JW1HVdenD8ViypDmiAqxPqLPXAVLdS6USUTkFCSPlnrggQfQvXt3ZGZmGua4AYC+ffvi3nvvlbSvgQMHYuDAgVJDwJgxYzB8+HC4u7tLqu2hquvetnWsF5LZrOHtkDBvr+F3dzd22iEicgTJNTcAEB4ejrZt2+L8+fOGFcI7deqEpk2byhqcKT/88ANOnTqFKVOmiCpfVFSE3Nxcox8iKYbERuKjB1pbnbk4tMIkgINbRxj97q7BHsl1RdRkERE5G8nJjV6vx9SpUxEUFIR69eqhXr16CA4Oxrvvvgu9Xq9EjAbHjx/HG2+8gV9++QUeHuIqnRITExEUFGT4iYqKUjRGcj0zH26DYR2irC7zcG+72haf12LNjQZDIiKym+TkZuLEifjyyy8xffp07Nu3D/v27cO0adPwxRdfYNKkSUrECAAoLS3F8OHD8c4776Bx48ait5swYQJycnIMP+np6YrFSFVb7WBf9GlSy+zz7m46rB3f04ERWeemwdokIiJ7Se5z8+OPP2LOnDmG1cABoHXr1qhduzaef/55vP/++7IGeEteXh727NmDffv2YezYsQDKapEEQYCHhwdWr16NO+64o9J23t7ehlXMicQoP2R70bNxomc4fqRTXeh0Omw4esnk894ebogJDcBdrSOw/ECmHKHaj7kNEbkgycnN1atXTfatadq0Ka5evSpLUKYEBgbi4MGDRo999dVXWL9+PX777TfUr19fsWNT1dWwVjVR5Sbd1Rye7m4WJ7MZ2ras2crWtatqVPPClYKbNm1rDmtuiMgVSW6Wio2NxZdfflnp8S+//NJo9JQY+fn5SElJQUpKCgDg9OnTSElJQVpaGoCyJqURI0aUBermhpYtWxr9hIaGwsfHBy1btkS1auJuQkTWlO+HInldqgq/d4+pafj/rXWvKu4xvlmoqH2/2LdRpcemDGkuJbxKmNoQkSuSXHPz4YcfYvDgwVi7dq1hjpsdO3YgPT0dK1eulLSvPXv2oE+fPobfx48fDwAYOXIkkpKSkJmZaUh0iBylhr83BreOgA5AiJm5bQK8jT865hbbNJUblX+sUag//u/xDmj4pvXPzkMdozDlj3+N92V1KyKiqkdyzU2vXr1w7Ngx3HvvvcjOzkZ2djbuu+8+HD161LDmlFi9e/eGIAiVfpKSkgAASUlJ2Lhxo9nt3377bUOtD5GcZg1vhy+HtzP7fNLojiYf79c8HADQPML8bN3lExIB4kdRmVrx3F6u3CrVIlLajOlE5Dok19wAQGRkZKWOw+fOncMzzzyDb775RpbAiLQsMtjX5OPhQT448HY/VPMq+2h5uVf+/mBrnxtT7F2uSufCdT9T72mB+2fvUDsMIlKBTZP4mXLlyhV89913cu2OyKmUT1gCfTwNtTGThzRHVIgvpt7TolzZ29vp7VxN094Owa5cc1PN26bvbkTkAvjpJ7KB2BqPejWqYcvrxlMUGG1rIbcJD/TBnJEd8OaSg3i9v/Kzf7uShD4N0TSczVJEVRWTGyIZmOtQbIrY2pKwIB+0rB2EP8Z2N1vGx9O+ylc5m8i0ZFCrCOuFiMhlydYsRUTS2dtnJjTQx/B/dzcdEu9rJXrbYD9PfDystZ0REBFpj+iam/vuu8/i89nZ2fbGQqRp3h63vwv4+8hT6VmnuumOyYC4Yd4VyzzSqS4mLD5osmxF+ybd6bI1N3Z2ZSIiJyf6Ch0UFGT1+VsT7hG5Im8Pdyx6Ng6legH+FTqr2pokzHiwjQyRlZHSNAa4bpMUEZHo5OaHH35QMg4ip9CpfojJx6UmFrfUCjC/7pmY/jTlE5SQalxDjYgIYJ8bIs2adq+4/jMLnumCtnWD8aOZiQWJiKoaJjdEGvT7c13RoJa/1XI6AF0a1MCS57uhRaTlpmNXYa72jIjoFiY3RORU2FOIiKxhckOkQWL7+mqxT3DH6Opqh8DRUkRVHJMbIrLqiW7Rosu2q3s7uVn0bJzssWgxoSMibWFyQ+TC5j7VWZb9TBnSwnohExrUqibL8csTs/SFYPf0iETkzJjcEDkxazf6bjE1se6VXnikU10HRaQ81twQkTVMbog0SOz9W8yNvmEtf0nLMtjjh1HKD0eXI7mp7udp/04quL9dHdn3SUS2YXJDpEHO2qgSGijvRILubpUzGbErskvdr70GtQqXfZ9EZBsmN0Qa0yIyEK1rO27OGjk7/Vaca8dSCiGmBsbWmZ+tb8a2LSJXxuSGSEM+e6gNlr/QHR7u4j6actyiO9UPQe8mtWTYkzTH3hvo8GPeokDFDRFpCJMbIge7t21tAEBMqOkZiCUtaKnyTfrZXg1s3tZTZAJXkagaHxn2QUTOi8kNkYPdGsG0/IXudu+rWXig6LKT7mpu9/HKGxIbibgGNSyW0erK43L023GU35+Tf64gIlcnelVwIjJP6rpODc2sGyVlfpYfRnVE9Wpeoss/2b0+3HXA238eEr2NNXoVpgKWI2HSaM5lkr+3/CO7iFwda26I7LB2fE98/Vh7xDW0XIOhhFoB0kcmie3LU17Pxqb74+gA6PWSd2c3OfKSwa0iZNiLMaUSJmdKxIi0gjU3RHaICQ1ATGiA2mHYzdz986MHWqNfC/NDnEut1NyodV+2Nsrq8bh62H3mKvafy3FQRLarX1P+WZ6JXB1rbojIrGEdohDka75ZpHmE+D4/Yr0c39jsc7WDfWWpyXDT6dDLTI2U0r4f1UFSeU93Nyx5vqtC0RC5JiY3RBoipQuL2s0VOh0QFeJn4gn79uvjefuyVPF03Nu2tqjdizmNcvcWEvu361RfehNm23KLkRKRdUxuiMgu0TUqJDgK9jGWc0HM2sG+lR4z179IDLHJpr+3B0Z1jcbwznUt1ooRke3Y54bISTnDcGYlapdsGS3VKToEu85cNXrsgfZ1cPpKAXafvoq9adkAbJ8RGZD293j77rJV1lcezLT5eERkHmtuiKoQOZMNpVIrS+mFzsaUrm294EqPebi7YcLAZnbV1pQnZ60SGYut47jlSMg1MLkh0hBn6ltxa46dSrd0BSuUxCYQKky/YxNniVNt4yx0MicyhckNkQYkvxWPNS/3tDrs9+GOUYb/21IL07dpGADzSz+I1btJLYzra/2GY0s9S/kbvqmbv021TxaSCCYY2sdaMZKKfW6INKCGvzdq+FuflC/xvlZYsDsdgG03+fAgH+yf0g/VvNzNlokI8kFmzg2zz7euE4SkJzpJP7hs5BovpT61R7wRKSXYT93O8qy5IXIiOp0OEUE+8PZws3lytyBfz0ozFZfvpBtpYhSRJHbmFf1alNUumRrNJDoE58htSCRn6DxPxvo0CVX1+KomN5s3b8aQIUMQGRkJnU6HpUuXWiy/ePFi3HnnnahVqxYCAwMRFxeHv//+2zHBEmnE5tf74MDb/eDtYb72Raryo4Q61Q+xWLbibUZKIvFA+zpWyzSs5Y+dE/pi3Su9TD7vxvsckeap/TFVNbkpKChAbGwsZs2aJar85s2bceedd2LlypVITk5Gnz59MGTIEOzbt0/hSIm0w9PdTdbEpqKX+jbC5LuaY8OrvW3bQbmrWsWFNV+4IwZ1qvti/J2W++uEB/nAx9P0a7SlKadi/qWV5iDWMBEpQ9U+NwMHDsTAgQNFl//ss8+Mfp82bRqWLVuGP//8E23btpU5OqKqo3yzlI+nO0Z3ry/LfotLjVfWrBXgja3/u8Nk2ce61MUdTS1XZXu5y5PUGXVaNvM4ETkvp+5zo9frkZeXh5AQ89XoRUVFyM3NNfohIsfwsdBxuaL3hrbCHf+N5jKlZe1APNE9WlT/i0o1NaKjUFaXBsbXKq3UIBG5GqdObj7++GPk5+fjwQcfNFsmMTERQUFBhp+oqCizZYlIBAl35EAfT7w+oMntTe1IM5a/0AOBPraNwBDbLFW+v1HK5DvhYWMHn24xNfDJsFi0iDReWHTOyI6S9lN+eYZnezZAi8hA1Kle1tG6VW1ObEcapnLi7rTJzbx58/DOO+9g0aJFCA01X5U9YcIE5OTkGH7S09MdGCWR66k4jLziHCS9GpXN+Ov+X2LQ10JtjC3E5FYVm5fELqvwbK8GeHdoS2x8tTeC/bzw23PiV+MOqXZ7KH9Cnxjc375OpVj9vY17AlgL65NhsYb/TxjUDCte7IFfx8ThxTti8N1IaauLE1UlTjnPzYIFC/DUU0/h119/RXx8vMWy3t7e8Pa2Pn8IEVn2+cNt8PWmU0i8r5XFcl1jauK3MXGINjFUXY5mmFFdo/FXapb9O/pPTX8vw/+9PdzxeJd6ht/bRAVb3X7W8HY4e7XAqOytGqpWtYOQmmF7U7ibia+fEUG+GN+vSeUniLRE5f5rTpfczJ8/H6NHj8aCBQswePBgtcMhcglico572tTGPW1qV3rcVO1Dh+jbzTtyzy7bKCxA1v093LEuDp7LsXmNqcGtI8w+9+agZpi/y3xtsdx9buqG+CHtaqG8O3WghzpEYeEe1q67hKrcLJWfn4+UlBSkpKQAAE6fPo2UlBSkpaUBKGtSGjFihKH8vHnzMGLECHzyySfo3LkzsrKykJWVhZycHDXCJ3IZan3Jalm7rE9KvRp+Dj1uZNDtCQK9PNzw0bBYDImNlG3/t5KWAB9PjO4mz8gzMcr3b3JG0+83UyvIjtdOR+2Rh6omN3v27EHbtm0Nw7jHjx+Ptm3bYvLkyQCAzMxMQ6IDAN988w1KSkqQkJCAiIgIw89LL72kSvxEJK32oWLZOSM6Ykyvhpj7VGdZY7LWx8ZN4ZkAxV7YrZWTeoNQ+4ZiLx2Hj7mMEr26b0ZVm6V69+5t8SKUlJRk9PvGjRuVDYioirLnlmLPDTo8yAdvDGwq6Xi2xCrHpIddG9bA9pNX7N6PnOaM6ICnftoDAAjwdrpeBqJ1jLY8azZRRU47WoqItKFhLfErjMuxRpCY74O3yoyLb4R72kSiXb1gu4/7wh2NRJctXwHRJNz8+bG3oiK+eRhWvNgdfZrUwsJn4+zbmYb5e3vgA3NNVmSTuiHKNgWrXQfH5IaI0DBUfIJS0UcPtMb97epgWUI3GSMyT+ywbgAYF98Ynz/cVqakyrZq9gfaR+HNQU2xVKHz0yIyCD880QnNK8yp42rK175ZWqMsLND66Ng/xjrmvVqVqd1C6rr1mEQk2kt9G6FUL2BQq3DJ24YG+uCTB2OtF4TjZuStFaDu9A/lX6a7mw7P9GzokOOqfUORg7ubDqVW+mt8PCwWvyWfq/T4+DsbI+1qocnnyqvu52Xx+apgQMtwfLP5lNphKIY1N0SEat4emHRXc7SvJ3/fBkd3cv1kWKykpjLSlvlPd0HdED/cbcPoNbWbQpzFkXcHIKSasgleXyvrxCmNNTdE5DBy3HzM5Uo/jS5rmqnpX7nWRo65dqJrVJ6UUGnSR0s5f91Np/oh2Px6H/yefA5/7D8vaVudzvlHjDmCj6c8C9Ba0rqOusuDMLkhIpdg6yR81qx/pRdyrhcjMtjXemGSTcUcpWmEPJM3PtKJ6wtWBUxuiFxUsJ9ti0wqSa15TLzcbf+m2oBNXLJ5rX8TbDhyEXvOXrNatmItVNPwQPzyZGeEB/lY3I5T5YijdA2X2hVo7HND5GIWPtMFbesG45cn5Z0Yz1blm4RkaZYqd9W8VVszqmu0xW3iGtaQ4ciOF+KvnY6vbw1uhi8eaStpm/KrmgNAneq+ohcj9XCv/G7p3qgmYqyM7LN+02b2U5GjRjo6EmtuiFxM5wY1sOR57Vys5P6GWD5Z+vD+1jh7pQDt6lW3uI27wjMSV2RvDdV97WqjYS1/tKtr+XU50lM9GuBmiV7SNslvxaPTtHW4WnBT8vEGtozA91vPoFN98Z3cxZz3mhpKGLUiVsQCsVKp3feJNTdE5DByNxl4ebihc4Ma8HR3/kvZe0NbGv4/uFUEEvrESN6H0jcULw9p59nDjr+Lj6c7/nyhOybd1VzSdp4manwA4KtH22FwqwiM6SV+WL6pzum2aqHwPEThgZab6xxP3ezG+a8IRFS1lLtmulIDw2Nd6hn+r0S/EUd2pA3y9cQ8mdcLEyvKzMy7g1pFYNaj7VDN20N0Erj1f30sPv/5w21E7Sf5rXjFm35mPSqtyVBprLkhoiqDCyMqy1QNVt+moWgU6o/3hxovX/DmoKbwUqjGa8EzXdA1pqYi+7amYS3rQ/bLN21ued18AuNhoTmzVe0g0ZMBBvt52VWLJUbjsAAMa18HT3V33Cr0WsbkhoicitqjMOQ0aXBZk8vTPeS5Id3ZPAwdo4376Xw3qiNWv9yz0kroz/RsiAmDpC1aKpaa39r7t7A+y3b5+KpLmMyu/MzXcsydZI4tM4XrdDp8NCwWrw1oIqq8kvFrAZMbInJaWq0IEhvX/e3rYM9b8XhzUDOjx21NDrw83PDrmK64o8LssOZqzEbERUs+xsvxjRFdQ/5FFy3Vkkghpnaw/OkVe9QmYQGY/3QX22KSWH7mw22xdnwvm46lFWqnTkxuiMipqN2WL7ea/t6qNde5u+mw+Pmu6N1E/ASIL8U3wsbXLPdFsfZyKiZHY/vE4O+Xe4qOQaomYcYTAOrLvYncLARb/u/yxfC2RsPQlXwferi7WR3ybo7YuORYTFaOOJTC5IaInJbSF2hbaTOq2/MBJfS5PWKoXd3qSHqik0PjWDXOOJF5tX8TxdYD++bx9lj4rHGNS/kbr9i8suLNWu2bt72UbpZSu9mL89wQkaKc/SZgC3tfslIVOZPvao5HOtVF4zBlZ1029ze/9biPpztCqnnZNP+N5eNWPnA/k31wRNbcyBEUHNN8qrWEWu3PPZMbIlKUp4e8l11BA2PBvTzcJE9opwVubjo0CZe2RtPPT3ZSZAVpNW/GejM1NwHeHsgrKjG5jakERez9WxC02z/MVbFZiogU1SQsAPe0ibRpiGqPRmXDiR3V10Gsl+Mb4+GOUZhrZi4XV7qP9WhUCy0i1V3h2ZTQANsn2Cv/Hipfc/Pm4GYmSlfeBhCf2DQJC6g0Uu39e1uaKU1yYc0NESlKp9Ph84dtm2Ds84fbYuHudNzXrrbhsfKjahy8qoKBv7c7pt/fWrH925vAqZ1c1VVgNFVFf4/riYMZOegeUxM9PtyAjOzrZpqgKhOMmqVuP17xvFmrbRFznqfcfXuG5W1v3IGTF/PRs3EtTFySKmJr8bRWM6T2lxAmN0SkWSHVvPBcb+Pp8kMDffBA+zrwdHdDgI86K59roPJI0/y9Td9ajBZRtfNmXL2al2Hh1HWv9MLVgpuIDPYVta2+XIti+RFR9WpYnwDwFkGQ3mW2drAvav8XY/t61ZEsYnV0qbSyFAk7FBMRSfTxsFi1Q7BI7W/R9t7gRnWNRt0QP3Rp4Byrqft4uotObADjoeAAsOjZOBy/mKfI6vHmRvR1a1gDyWevwcvDDUuf74b7Zm/DjWL7+3G5u+mw7Y07sHRfBj76+6jZckrXrLDmhoiIjNibHDUK88eqf23btkHNanitfxNUM1P7Yg9H3fAe7hiFBbvT0b9FmMnnY0L94aaDoaN0p/ohhtXHR8bVw487zgJQdrmQhDtiEB7kix6NaiIqxA/VvDxwo9j20WPlk6jawb5m19hyFEuj0ByByQ0RkURqfyu1xp7byrg7GyuS2DjS23e3QP8W4WZrnnw83fHvOwPgbqLTVsXOv+YIAmyeaA8AvD3cMbxzXZu3t5eY3CO+WSjWHr5o0/6bRUgblSc3bTTOEREp7NYstX0rLE1AxkzNFWOPl+9sLOv+xPDxdEefpqHw9XI3W8bXyx1eHuJvgab6kNQO9sUfY7vhiW7RtoRpREpFR60Ab8TWsW8Em7g/s+1pstqL5DK5IaIq4ecnO+Gtwc0w48E2du9LK502zdFSxdLjXeqZeUZjw3v+IzW3a10nGPVUaAKSsuCnGK1NJEtS85Nh7evIFI39tP0JJSKSSWigD57q0QBBfvaPsPJ0t3bV1+aNW21ab86TQuxoIEdVYFQ8jtQauJ9Hd8bsR9sZPVbLjrmE1MbkhohIoroqd9ZUktrNCVrTNDwAoQHelda+kjtRk7I/Of5CFRP0ID9PDGwVYfTYa/2aWNzHvkl3Gv0+7s7GqOnvhRfviJEhQvs4d68xIiIHSnqiI05dKkBnMx1VB7QIR9rVQrv7Q9jLno6ucve5MUerOdT97eogafsZtIgMBACsfLEH9IIADw01RbauE4wSveVh49aS1OGd62Hx3gzENzM9ouytwc2sNn1VfL52sC92T4zXRILM5IaISKTeTULR28KX2a8fbw9BEFS/uA9pHYkr+TfRrl51xY6x8JkueOibnYrtXy2t6gThnzf7GoaJu7np4GairkTuFFDMW+b7UR2w/cQVPNe7IV79db/FstaSVH9vj0qrs8tB7ff+LUxuiIhkpIWLu5ubDqNtWMtLis4NamB0t/r4fttp0ds4S5ebsEAftUMwqVFoAO5oarqmhYxpp56NiIhUp8QK4K5IbPOd+qmuMsyPgtMGVZObzZs3Y8iQIYiMjIROp8PSpUutbrNx40a0a9cO3t7eiImJQVJSkuJxEhE5kreH+flZlPL5w23wfO+G6B5T06btH+kUBQBmZwUGHNefxxFio4LVDsEipacr6Pbf+6TRf/27qsswClFOqiY3BQUFiI2NxaxZs0SVP336NAYPHow+ffogJSUF48aNw1NPPYW///5b4UiJiJT3yp2NcVfrCMSpsKbTPW1q4/UBTW1uVmseEYgDb/fD14+1r/TcA+3rICrEF4Nb3x6N46w1GmvH98QLd8RgypAWDj92+T+Ntb9TfLMwRIWUrbeV0Of24rP1a4pfHNQSj/9mcv5+VEc80ikKv47pKst+5aJqn5uBAwdi4MCBost//fXXqF+/Pj755BMAQLNmzbB161Z8+umn6N+/v1JhEhE5xAt9G6kdgl0CzazS/vGwWE10tJZDTGgAXrEyRNoWUiu1rNWCeXm4YcvrdxjO+6wNJwEAT/Ww3BerT5Na2HD0Eu5uE2ny+egafggP8kHvJmUrskeF+CHxvtbSgncAp+pQvGPHDsTHxxs91r9/f4wbN87sNkVFRSgqKjL8npubq1R4RERkhiskNmI0DQ9Aw1r+WHEwU+1QAFQ+7+ZWKb/l+1EdUVSih4+n6abRNwc1Q78W4bLFpxSn6lCclZWFsDDj9tywsDDk5ubi+vXrJrdJTExEUFCQ4ScqKsoRoRIRuTxrtQCWuGqu89dLPTCrwky/Yog5H+UTFaWSRZ1OZzaxcSZOldzYYsKECcjJyTH8pKenqx0SEZFLiAz2tXnbjx6IBQC8MbCpXOFoQqWaEo1lcRoLRzFO1SwVHh6OCxcuGD124cIFBAYGwtfX9IfM29sb3t7Ouz4GEZEr6tm4Fo6+N0CVkWFVUfOIQBzKzEXfpqFqh+IQTpXcxMXFYeXKlUaPrVmzBnFxcSpFREREtqoKiU2wRoZI//lCdxTeLEGAmU7frkbV5CY/Px8nTpww/H769GmkpKQgJCQEdevWxYQJE5CRkYGffvoJADBmzBh8+eWXeP311zF69GisX78eixYtwooVK9R6CURERJV89lAbXMi9gcZhAYrsX2rrkrubrsokNoDKyc2ePXvQp08fw+/jx48HAIwcORJJSUnIzMxEWlqa4fn69etjxYoVePnll/H555+jTp06mDNnDoeBExGRpgxtW1vtEKo0VZOb3r17Wxyrb2r24d69e2Pfvn0KRkVERKQ95WtrvDyMxwMpPSOxs3GqPjdEREQETBzcDEey8tCveRhGdo2Gu1sVGQYlEpMbIiKyX1UZY6yQu1pHImn7GTSPCBRVvk51P2x4tbeyQTkxJjdEREQqe2NgU7SvV93iwqXMH8VjIx0REdnPhVb8VoOPpzuGxEaiejUvs2XUOMWzbZhtWQuY3BAREZFJA1tFYNNrvdUOQzImN0REZL8q3Gbi6eGYW6lap7hejWrqHNgO7HNDRERkh/va1sGiPefQs5H5/jLkWExuiIiI7ODr5Y5lCd3UDoPKYbMUERHZzOO/+VXiGtRQORLXp5O86ELVxZobIiKyWfKkO3ElvwgNavmrHYrLCQ/0QVbuDbXDcEqsuSEiIpsF+XoysVHIwme74L52XKPKFkxuiIiINKhejWqYMqSF4fcqPCBNMiY3RERETkAL8yS2qB2kdgiisM8NERERWbRrYl/kFBajdrCv2qGIwuSGiIjICajZLBUa4IPQAB/1ApCIzVJERETkUpjcEBERaZUG+tk4IyY3REREToCDpcRjckNEREQuhckNERGRRglsl7IJkxsiIiJyKUxuiIiINIqLZdqGyQ0REZFGsVnKNkxuiIiInAErcURjckNEREQuhckNERGRRmlhsUxnxOSGiIjICbBzsXhMboiIiMilMLkhIiLSKLZK2YbJDREREbkUJjdEREQaxV42tmFyQ0REpFFslrINkxsiIiInoGM1jmiaSG5mzZqF6Oho+Pj4oHPnzti1a5fF8p999hmaNGkCX19fREVF4eWXX8aNGzccFC0RERFpmerJzcKFCzF+/HhMmTIFe/fuRWxsLPr374+LFy+aLD9v3jy88cYbmDJlCg4fPozvvvsOCxcuxJtvvungyImIiJQlcBY/m6ie3MyYMQNPP/00nnjiCTRv3hxff/01/Pz88P3335ssv337dnTr1g3Dhw9HdHQ0+vXrh0ceecRqbQ8REZEzY6uUeKomNzdv3kRycjLi4+MNj7m5uSE+Ph47duwwuU3Xrl2RnJxsSGZOnTqFlStXYtCgQSbLFxUVITc31+iHiIiIXJeHmge/fPkySktLERYWZvR4WFgYjhw5YnKb4cOH4/Lly+jevTsEQUBJSQnGjBljtlkqMTER77zzjuyxExERKU3HXsQ2Ub1ZSqqNGzdi2rRp+Oqrr7B3714sXrwYK1aswLvvvmuy/IQJE5CTk2P4SU9Pd3DEREREtqnu54k7m4chvlkYQqp5qR2O01C15qZmzZpwd3fHhQsXjB6/cOECwsPDTW4zadIkPP7443jqqacAAK1atUJBQQGeeeYZTJw4EW5uxvmat7c3vL29lXkBRERECtLpdPh2RAe1w3A6qtbceHl5oX379li3bp3hMb1ej3Xr1iEuLs7kNoWFhZUSGHd3dwDsVU5EREQq19wAwPjx4zFy5Eh06NABnTp1wmeffYaCggI88cQTAIARI0agdu3aSExMBAAMGTIEM2bMQNu2bdG5c2ecOHECkyZNwpAhQwxJDhEREVVdqic3Dz30EC5duoTJkycjKysLbdq0wapVqwydjNPS0oxqat566y3odDq89dZbyMjIQK1atTBkyBC8//77ar0EIiIi0hCdUMXacnJzcxEUFIScnBwEBgaqHQ4RERGJIOX+7XSjpYiIiIgsYXJDRERELoXJDREREbkUJjdERETkUpjcEBERkUthckNEREQuhckNERERuRQmN0RERORSmNwQERGRS2FyQ0RERC5F9bWlHO3WahO5ubkqR0JERERi3bpvi1k1qsolN3l5eQCAqKgolSMhIiIiqfLy8hAUFGSxTJVbOFOv1+P8+fMICAiATqeTdd+5ubmIiopCeno6F+W0Ac+ffXj+7MPzZz+eQ/vw/FkmCALy8vIQGRkJNzfLvWqqXM2Nm5sb6tSpo+gxAgMD+ca0A8+ffXj+7MPzZz+eQ/vw/JlnrcbmFnYoJiIiIpfC5IaIiIhcCpMbGXl7e2PKlCnw9vZWOxSnxPNnH54/+/D82Y/n0D48f/Kpch2KiYiIyLWx5oaIiIhcCpMbIiIicilMboiIiMilMLkhIiIil8LkRiazZs1CdHQ0fHx80LlzZ+zatUvtkFSxefNmDBkyBJGRkdDpdFi6dKnR84IgYPLkyYiIiICvry/i4+Nx/PhxozJXr17Fo48+isDAQAQHB+PJJ59Efn6+UZkDBw6gR48e8PHxQVRUFD788EOlX5pDJCYmomPHjggICEBoaCiGDh2Ko0ePGpW5ceMGEhISUKNGDfj7++P+++/HhQsXjMqkpaVh8ODB8PPzQ2hoKF577TWUlJQYldm4cSPatWsHb29vxMTEICkpSemXp7jZs2ejdevWhknQ4uLi8Ndffxme57mTZvr06dDpdBg3bpzhMZ5D895++23odDqjn6ZNmxqe57lzIIHstmDBAsHLy0v4/vvvhX///Vd4+umnheDgYOHChQtqh+ZwK1euFCZOnCgsXrxYACAsWbLE6Pnp06cLQUFBwtKlS4X9+/cLd999t1C/fn3h+vXrhjIDBgwQYmNjhZ07dwpbtmwRYmJihEceecTwfE5OjhAWFiY8+uijQmpqqjB//nzB19dX+L//+z9HvUzF9O/fX/jhhx+E1NRUISUlRRg0aJBQt25dIT8/31BmzJgxQlRUlLBu3Tphz549QpcuXYSuXbsani8pKRFatmwpxMfHC/v27RNWrlwp1KxZU5gwYYKhzKlTpwQ/Pz9h/PjxwqFDh4QvvvhCcHd3F1atWuXQ1yu3P/74Q1ixYoVw7Ngx4ejRo8Kbb74peHp6CqmpqYIg8NxJsWvXLiE6Olpo3bq18NJLLxke5zk0b8qUKUKLFi2EzMxMw8+lS5cMz/PcOQ6TGxl06tRJSEhIMPxeWloqREZGComJiSpGpb6KyY1erxfCw8OFjz76yPBYdna24O3tLcyfP18QBEE4dOiQAEDYvXu3ocxff/0l6HQ6ISMjQxAEQfjqq6+E6tWrC0VFRYYy//vf/4QmTZoo/Ioc7+LFiwIAYdOmTYIglJ0vT09P4ddffzWUOXz4sABA2LFjhyAIZQmmm5ubkJWVZSgze/ZsITAw0HDOXn/9daFFixZGx3rooYeE/v37K/2SHK569erCnDlzeO4kyMvLExo1aiSsWbNG6NWrlyG54Tm0bMqUKUJsbKzJ53juHIvNUna6efMmkpOTER8fb3jMzc0N8fHx2LFjh4qRac/p06eRlZVldK6CgoLQuXNnw7nasWMHgoOD0aFDB0OZ+Ph4uLm54Z9//jGU6dmzJ7y8vAxl+vfvj6NHj+LatWsOejWOkZOTAwAICQkBACQnJ6O4uNjoHDZt2hR169Y1OoetWrVCWFiYoUz//v2Rm5uLf//911Cm/D5ulXGl92xpaSkWLFiAgoICxMXF8dxJkJCQgMGDB1d6nTyH1h0/fhyRkZFo0KABHn30UaSlpQHguXM0Jjd2unz5MkpLS43ejAAQFhaGrKwslaLSplvnw9K5ysrKQmhoqNHzHh4eCAkJMSpjah/lj+EK9Ho9xo0bh27duqFly5YAyl6fl5cXgoODjcpWPIfWzo+5Mrm5ubh+/boSL8dhDh48CH9/f3h7e2PMmDFYsmQJmjdvznMn0oIFC7B3714kJiZWeo7n0LLOnTsjKSkJq1atwuzZs3H69Gn06NEDeXl5PHcOVuVWBSdyFgkJCUhNTcXWrVvVDsWpNGnSBCkpKcjJycFvv/2GkSNHYtOmTWqH5RTS09Px0ksvYc2aNfDx8VE7HKczcOBAw/9bt26Nzp07o169eli0aBF8fX1VjKzqYc2NnWrWrAl3d/dKPd4vXLiA8PBwlaLSplvnw9K5Cg8Px8WLF42eLykpwdWrV43KmNpH+WM4u7Fjx2L58uXYsGED6tSpY3g8PDwcN2/eRHZ2tlH5iufQ2vkxVyYwMNDpL8JeXl6IiYlB+/btkZiYiNjYWHz++ec8dyIkJyfj4sWLaNeuHTw8PODh4YFNmzZh5syZ8PDwQFhYGM+hBMHBwWjcuDFOnDjB95+DMbmxk5eXF9q3b49169YZHtPr9Vi3bh3i4uJUjEx76tevj/DwcKNzlZubi3/++cdwruLi4pCdnY3k5GRDmfXr10Ov16Nz586GMps3b0ZxcbGhzJo1a9CkSRNUr17dQa9GGYIgYOzYsViyZAnWr1+P+vXrGz3fvn17eHp6Gp3Do0ePIi0tzegcHjx40ChJXLNmDQIDA9G8eXNDmfL7uFXGFd+zer0eRUVFPHci9O3bFwcPHkRKSorhp0OHDnj00UcN/+c5FC8/Px8nT55EREQE33+OpnaPZlewYMECwdvbW0hKShIOHTokPPPMM0JwcLBRj/eqIi8vT9i3b5+wb98+AYAwY8YMYd++fcLZs2cFQSgbCh4cHCwsW7ZMOHDggHDPPfeYHAretm1b4Z9//hG2bt0qNGrUyGgoeHZ2thAWFiY8/vjjQmpqqrBgwQLBz8/PJYaCP/fcc0JQUJCwceNGo+GkhYWFhjJjxowR6tatK6xfv17Ys2ePEBcXJ8TFxRmevzWctF+/fkJKSoqwatUqoVatWiaHk7722mvC4cOHhVmzZrnEcNI33nhD2LRpk3D69GnhwIEDwhtvvCHodDph9erVgiDw3Nmi/GgpQeA5tOSVV14RNm7cKJw+fVrYtm2bEB8fL9SsWVO4ePGiIAg8d47E5EYmX3zxhVC3bl3By8tL6NSpk7Bz5061Q1LFhg0bBACVfkaOHCkIQtlw8EmTJglhYWGCt7e30LdvX+Ho0aNG+7hy5YrwyCOPCP7+/kJgYKDwxBNPCHl5eUZl9u/fL3Tv3l3w9vYWateuLUyfPt1RL1FRps4dAOGHH34wlLl+/brw/PPPC9WrVxf8/PyEe++9V8jMzDTaz5kzZ4SBAwcKvr6+Qs2aNYVXXnlFKC4uNiqzYcMGoU2bNoKXl5fQoEEDo2M4q9GjRwv16tUTvLy8hFq1agl9+/Y1JDaCwHNni4rJDc+heQ899JAQEREheHl5CbVr1xYeeugh4cSJE4bnee4cRycIgqBOnRERERGR/NjnhoiIiFwKkxsiIiJyKUxuiIiIyKUwuSEiIiKXwuSGiIiIXAqTGyIiInIpTG6IiIjIpTC5ISIiIpfC5IaIXFpSUhKCg4PVDoOIHIjJDREpbtSoUdDpdIafGjVqYMCAAThw4ICk/bz99tto06aNMkFakJCQgDfffBMAMG3aNIwePdrhMRCReExuiMghBgwYgMzMTGRmZmLdunXw8PDAXXfdpXZYouzYsQPdunUDAGzZssXwfyLSJiY3ROQQ3t7eCA8PR3h4ONq0aYM33ngD6enpuHTpkqHM//73PzRu3Bh+fn5o0KABJk2ahOLiYgBlzUvvvPMO9u/fb6gBSkpKAgBkZ2fj2WefRVhYGHx8fNCyZUssX77c6Ph///03mjVrBn9/f0OiJUZBQQFSU1PRtWtX6PV6o0SHiLTJQ+0AiKjqyc/Pxy+//IKYmBjUqFHD8HhAQACSkpIQGRmJgwcP4umnn0ZAQABef/11PPTQQ0hNTcWqVauwdu1aAEBQUBD0ej0GDhyIvLw8/PLLL2jYsCEOHToEd3d3w34LCwvx8ccf4+eff4abmxsee+wxvPrqq5g7d67ZGJ9//nnMmzcPer0excXFqF+/PgRBQG5uLrp06QIAOHDgAOrWravQWSIiWzG5ISKHWL58Ofz9/QGU1YZERERg+fLlcHO7XYH81ltvGf4fHR2NV199FQsWLMDrr78OX19f+Pv7w8PDA+Hh4YZyq1evxq5du3D48GE0btwYANCgQQOjYxcXF+Prr79Gw4YNAQBjx47F1KlTLcY7depUvP7663jvvfcMsX3zzTc4cuQIZsyYAQCIjIy09XQQkYKY3BCRQ/Tp0wezZ88GAFy7dg1fffUVBg4ciF27dqFevXoAgIULF2LmzJk4efIk8vPzUVJSgsDAQIv7TUlJQZ06dQyJjSl+fn6GxAYAIiIicPHiRYv7rVmzJmrWrInt27fj888/R3R0NHbv3o2RI0ciOjpa5KsmIjWwzw0ROUS1atUQExODmJgYdOzYEXPmzEFBQQG+/fZbAGWddh999FEMGjQIy5cvx759+zBx4kTcvHnT4n59fX2tHtvT09Pod51OB0EQzJafO3cu/P394e/vj8OHD2Po0KHw9/fHunXr8Mwzz8Df399ikxYRqYs1N0SkCp1OBzc3N1y/fh0AsH37dtSrVw8TJ040lDl79qzRNl5eXigtLTV6rHXr1jh37hyOHTtmsfZGirvvvhudO3fGsmXLsHjxYvz444/Yvn073nvvPaxcuRIAEBYWJsuxiEh+TG6IyCGKioqQlZUFoKxZ6ssvv0R+fj6GDBkCAGjUqBHS0tKwYMECdOzYEStWrMCSJUuM9hEdHY3Tp08bmqICAgLQq1cv9OzZE/fffz9mzJiBmJgYHDlyBDqdDgMGDLAp1oCAAAQEBOD48eOIj49HTEwM5s2bhz59+iAmJsa+E0FEimOzFBE5xKpVqxAREYGIiAh07twZu3fvxq+//orevXsDKKstefnllzF27Fi0adMG27dvx6RJk4z2cf/992PAgAHo06cPatWqhfnz5wMAfv/9d3Ts2BGPPPIImjdvjtdff71SDY8tNm7ciJ49ewIANm3aZPg/EWmbTrDU8ExERETkZFhzQ0RERC6FyQ0RERG5FCY3RERE5FKY3BAREZFLYXJDRERELoXJDREREbkUJjdERETkUpjcEBERkUthckNEREQuhckNERERuRQmN0RERORS/h9/6awQMchsiQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "plt.plot(loss_history)\n", "plt.xlabel('Batch #')\n", "plt.ylabel('Loss [entropy]')" ] }, { "cell_type": "markdown", "metadata": { "id": "kKpOlHPLqEgl" }, "source": [ "### 变量和优化器\n", "\n", "`tf.Variable` 对象会存储在训练期间访问的可变、类似于 `tf.Tensor` 的值，以更简单地实现自动微分。\n", "\n", "变量的集合及其运算方法可以封装到层或模型中。有关详细信息，请参阅[自定义 Keras 层和模型](https://render.githubusercontent.com/view/keras/custom_layers_and_models.ipynb)。层和模型之间的主要区别在于模型添加了如下方法：`Model.fit`、`Model.evaluate` 和 `Model.save`。\n", "\n", "例如，上面的自动微分示例可以改写为：" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:13.392032Z", "iopub.status.busy": "2022-12-14T22:22:13.391779Z", "iopub.status.idle": "2022-12-14T22:22:13.397105Z", "shell.execute_reply": "2022-12-14T22:22:13.396344Z" }, "id": "2qXcPngYk8dN" }, "outputs": [], "source": [ "class Linear(tf.keras.Model):\n", " def __init__(self):\n", " super(Linear, self).__init__()\n", " self.W = tf.Variable(5., name='weight')\n", " self.B = tf.Variable(10., name='bias')\n", " def call(self, inputs):\n", " return inputs * self.W + self.B" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:13.400366Z", "iopub.status.busy": "2022-12-14T22:22:13.399941Z", "iopub.status.idle": "2022-12-14T22:22:13.409541Z", "shell.execute_reply": "2022-12-14T22:22:13.408962Z" }, "id": "nnQLBYmEqEgm" }, "outputs": [], "source": [ "# A toy dataset of points around 3 * x + 2\n", "NUM_EXAMPLES = 2000\n", "training_inputs = tf.random.normal([NUM_EXAMPLES])\n", "noise = tf.random.normal([NUM_EXAMPLES])\n", "training_outputs = training_inputs * 3 + 2 + noise\n", "\n", "# The loss function to be optimized\n", "def loss(model, inputs, targets):\n", " error = model(inputs) - targets\n", " return tf.reduce_mean(tf.square(error))\n", "\n", "def grad(model, inputs, targets):\n", " with tf.GradientTape() as tape:\n", " loss_value = loss(model, inputs, targets)\n", " return tape.gradient(loss_value, [model.W, model.B])" ] }, { "cell_type": "markdown", "metadata": { "id": "Q7x1CDurl3IG" }, "source": [ "下一步：\n", "\n", "1. 创建模型。\n", "2. 损失函数对模型参数的导数。\n", "3. 基于导数的变量更新策略。" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:13.413051Z", "iopub.status.busy": "2022-12-14T22:22:13.412500Z", "iopub.status.idle": "2022-12-14T22:22:14.684156Z", "shell.execute_reply": "2022-12-14T22:22:14.683211Z" }, "id": "SbXJk0f2lztg" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initial loss: 68.058\n", "Loss at step 000: 65.445\n", "Loss at step 020: 30.118\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Loss at step 040: 14.165\n", "Loss at step 060: 6.954\n", "Loss at step 080: 3.691\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Loss at step 100: 2.213\n", "Loss at step 120: 1.542\n", "Loss at step 140: 1.238\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Loss at step 160: 1.100\n", "Loss at step 180: 1.037\n", "Loss at step 200: 1.009\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Loss at step 220: 0.996\n", "Loss at step 240: 0.990\n", "Loss at step 260: 0.987\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Loss at step 280: 0.986\n" ] } ], "source": [ "model = Linear()\n", "optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)\n", "\n", "print(\"Initial loss: {:.3f}\".format(loss(model, training_inputs, training_outputs)))\n", "\n", "steps = 300\n", "for i in range(steps):\n", " grads = grad(model, training_inputs, training_outputs)\n", " optimizer.apply_gradients(zip(grads, [model.W, model.B]))\n", " if i % 20 == 0:\n", " print(\"Loss at step {:03d}: {:.3f}\".format(i, loss(model, training_inputs, training_outputs)))" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.687562Z", "iopub.status.busy": "2022-12-14T22:22:14.687251Z", "iopub.status.idle": "2022-12-14T22:22:14.692625Z", "shell.execute_reply": "2022-12-14T22:22:14.691890Z" }, "id": "PV_dqer7pzSH" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Final loss: 0.985\n" ] } ], "source": [ "print(\"Final loss: {:.3f}\".format(loss(model, training_inputs, training_outputs)))" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.695976Z", "iopub.status.busy": "2022-12-14T22:22:14.695443Z", "iopub.status.idle": "2022-12-14T22:22:14.700594Z", "shell.execute_reply": "2022-12-14T22:22:14.699791Z" }, "id": "rvt_Wj3Tp0hm" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "W = 3.010180711746216, B = 2.00154972076416\n" ] } ], "source": [ "print(\"W = {}, B = {}\".format(model.W.numpy(), model.B.numpy()))" ] }, { "cell_type": "markdown", "metadata": { "id": "rPjb8nRWqEgr" }, "source": [ "注：变量将一直存在，直至删除对 Python 对象的最后一个引用，并删除该变量。" ] }, { "cell_type": "markdown", "metadata": { "id": "scMjg6L6qEgv" }, "source": [ "### 基于对象的保存\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Y-0ZcCcjwkux" }, "source": [ "`tf.keras.Model` 包括一个方便的 `save_weights` 方法，您可以通过该方法轻松创建检查点： " ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.704919Z", "iopub.status.busy": "2022-12-14T22:22:14.704122Z", "iopub.status.idle": "2022-12-14T22:22:14.726777Z", "shell.execute_reply": "2022-12-14T22:22:14.726220Z" }, "id": "oJrMX94PwD9s" }, "outputs": [], "source": [ "model.save_weights('weights')\n", "status = model.load_weights('weights')" ] }, { "cell_type": "markdown", "metadata": { "id": "2EfTjWV_wEng" }, "source": [ "您可以使用 `tf.train.Checkpoint` 完全控制此过程。\n", "\n", "本部分是[检查点训练指南](https://render.githubusercontent.com/view/checkpoint.ipynb)的缩略版。\n" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.730354Z", "iopub.status.busy": "2022-12-14T22:22:14.729891Z", "iopub.status.idle": "2022-12-14T22:22:14.733925Z", "shell.execute_reply": "2022-12-14T22:22:14.733184Z" }, "id": "7z5xRfdHzZOQ" }, "outputs": [], "source": [ "x = tf.Variable(10.)\n", "checkpoint = tf.train.Checkpoint(x=x)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.737210Z", "iopub.status.busy": "2022-12-14T22:22:14.736741Z", "iopub.status.idle": "2022-12-14T22:22:14.753677Z", "shell.execute_reply": "2022-12-14T22:22:14.752921Z" }, "id": "IffrUVG7zyVb" }, "outputs": [ { "data": { "text/plain": [ "'./ckpt/-1'" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x.assign(2.) # Assign a new value to the variables and save.\n", "checkpoint_path = './ckpt/'\n", "checkpoint.save('./ckpt/')" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.757192Z", "iopub.status.busy": "2022-12-14T22:22:14.756630Z", "iopub.status.idle": "2022-12-14T22:22:14.765231Z", "shell.execute_reply": "2022-12-14T22:22:14.764431Z" }, "id": "eMT9koCoqEgw" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "x.assign(11.) # Change the variable after saving.\n", "\n", "# Restore values from the checkpoint\n", "checkpoint.restore(tf.train.latest_checkpoint(checkpoint_path))\n", "\n", "print(x) # => 2.0" ] }, { "cell_type": "markdown", "metadata": { "id": "vbFnP-yLqEgx" }, "source": [ "要保存和加载模型，`tf.train.Checkpoint` 会存储对象的内部状态，而无需隐藏变量。要记录 `model`、`optimizer` 和全局步骤的状态，请将它们传递到 `tf.train.Checkpoint`：" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.768713Z", "iopub.status.busy": "2022-12-14T22:22:14.768245Z", "iopub.status.idle": "2022-12-14T22:22:14.796516Z", "shell.execute_reply": "2022-12-14T22:22:14.795746Z" }, "id": "hWZHyAXMqEg0" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = tf.keras.Sequential([\n", " tf.keras.layers.Conv2D(16,[3,3], activation='relu'),\n", " tf.keras.layers.GlobalAveragePooling2D(),\n", " tf.keras.layers.Dense(10)\n", "])\n", "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n", "checkpoint_dir = 'path/to/model_dir'\n", "if not os.path.exists(checkpoint_dir):\n", " os.makedirs(checkpoint_dir)\n", "checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n", "root = tf.train.Checkpoint(optimizer=optimizer,\n", " model=model)\n", "\n", "root.save(checkpoint_prefix)\n", "root.restore(tf.train.latest_checkpoint(checkpoint_dir))" ] }, { "cell_type": "markdown", "metadata": { "id": "R-ITwkBCF6GJ" }, "source": [ "注：在许多训练循环中，会在调用 `tf.train.Checkpoint.restore` 后创建变量。这些变量将在创建后立即恢复，并且可以使用断言来确保检查点已完全加载。有关详细信息，请参阅[检查点训练指南](https://render.githubusercontent.com/view/checkpoint.ipynb)。" ] }, { "cell_type": "markdown", "metadata": { "id": "3yoD0VJ7qEg3" }, "source": [ "### 面向对象的指标\n", "\n", "`tf.keras.metrics` 会被存储为对象。可以通过将新数据传递给可调用对象来更新指标，并使用 `tf.keras.metrics.result` 方法检索结果，例如：" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.799986Z", "iopub.status.busy": "2022-12-14T22:22:14.799537Z", "iopub.status.idle": "2022-12-14T22:22:14.817229Z", "shell.execute_reply": "2022-12-14T22:22:14.816481Z" }, "id": "9ccu0iAaqEg5" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "m = tf.keras.metrics.Mean(\"loss\")\n", "m(0)\n", "m(5)\n", "m.result() # => 2.5\n", "m([8, 9])\n", "m.result() # => 5.5" ] }, { "cell_type": "markdown", "metadata": { "id": "aB8qWtT955pI" }, "source": [ "### 摘要和 TensorBoard\n", "\n", "[TensorBoard](https://tensorflow.google.cn/tensorboard) 是一种可视化工具，用于了解、调试和优化模型训练过程。它使用在执行程序时编写的摘要事件。\n", "\n", "您可以在 Eager Execution 中使用 `tf.summary` 记录变量摘要。例如，要每 100 个训练步骤记录一次 `loss` 的摘要，请运行以下代码：" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.820622Z", "iopub.status.busy": "2022-12-14T22:22:14.820113Z", "iopub.status.idle": "2022-12-14T22:22:14.838888Z", "shell.execute_reply": "2022-12-14T22:22:14.838300Z" }, "id": "z6VInqhA6RH4" }, "outputs": [], "source": [ "logdir = \"./tb/\"\n", "writer = tf.summary.create_file_writer(logdir)\n", "\n", "steps = 1000\n", "with writer.as_default(): # or call writer.set_as_default() before the loop.\n", " for i in range(steps):\n", " step = i + 1\n", " # Calculate loss with your real train function.\n", " loss = 1 - 0.001 * step\n", " if step % 100 == 0:\n", " tf.summary.scalar('loss', loss, step=step)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:14.842203Z", "iopub.status.busy": "2022-12-14T22:22:14.841665Z", "iopub.status.idle": "2022-12-14T22:22:15.035754Z", "shell.execute_reply": "2022-12-14T22:22:15.034708Z" }, "id": "08QQD2j36TaI" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "events.out.tfevents.1671056534.kokoro-gcp-ubuntu-prod-129375217.162107.0.v2\r\n" ] } ], "source": [ "!ls tb/" ] }, { "cell_type": "markdown", "metadata": { "id": "xEL4yJe5qEhD" }, "source": [ "## 自动微分高级主题\n", "\n", "### 动态模型\n", "\n", "`tf.GradientTape` 也可以用于动态模型。下面这个[回溯线搜索](https://wikipedia.org/wiki/Backtracking_line_search)算法示例看起来就像普通的 NumPy 代码，但它的控制流比较复杂，存在梯度且可微分：" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.040247Z", "iopub.status.busy": "2022-12-14T22:22:15.039572Z", "iopub.status.idle": "2022-12-14T22:22:15.045177Z", "shell.execute_reply": "2022-12-14T22:22:15.044344Z" }, "id": "L518n5dkqEhE" }, "outputs": [], "source": [ "def line_search_step(fn, init_x, rate=1.0):\n", " with tf.GradientTape() as tape:\n", " # Variables are automatically tracked.\n", " # But to calculate a gradient from a tensor, you must `watch` it.\n", " tape.watch(init_x)\n", " value = fn(init_x)\n", " grad = tape.gradient(value, init_x)\n", " grad_norm = tf.reduce_sum(grad * grad)\n", " init_value = value\n", " while value > init_value - rate * grad_norm:\n", " x = init_x - rate * grad\n", " value = fn(x)\n", " rate /= 2.0\n", " return x, value" ] }, { "cell_type": "markdown", "metadata": { "id": "gieGOf_DqEhK" }, "source": [ "### 自定义梯度\n", "\n", "自定义梯度是重写梯度的一种简单方法。在前向函数中，定义相对于输入、输出或中间结果的梯度。例如，下面是在后向传递中裁剪梯度范数的一种简单方法：" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.048689Z", "iopub.status.busy": "2022-12-14T22:22:15.048236Z", "iopub.status.idle": "2022-12-14T22:22:15.051947Z", "shell.execute_reply": "2022-12-14T22:22:15.051221Z" }, "id": "-OwwsWUAqEhK" }, "outputs": [], "source": [ "@tf.custom_gradient\n", "def clip_gradient_by_norm(x, norm):\n", " y = tf.identity(x)\n", " def grad_fn(dresult):\n", " return [tf.clip_by_norm(dresult, norm), None]\n", " return y, grad_fn" ] }, { "cell_type": "markdown", "metadata": { "id": "JPLDHkF_qEhN" }, "source": [ "自定义梯度通常用来为运算序列提供数值稳定的梯度：" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.055429Z", "iopub.status.busy": "2022-12-14T22:22:15.054907Z", "iopub.status.idle": "2022-12-14T22:22:15.058746Z", "shell.execute_reply": "2022-12-14T22:22:15.058031Z" }, "id": "24WiLROnqEhO" }, "outputs": [], "source": [ "def log1pexp(x):\n", " return tf.math.log(1 + tf.exp(x))\n", "\n", "def grad_log1pexp(x):\n", " with tf.GradientTape() as tape:\n", " tape.watch(x)\n", " value = log1pexp(x)\n", " return tape.gradient(value, x)\n" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.062087Z", "iopub.status.busy": "2022-12-14T22:22:15.061624Z", "iopub.status.idle": "2022-12-14T22:22:15.073087Z", "shell.execute_reply": "2022-12-14T22:22:15.072364Z" }, "id": "n8fq69r9-B-c" }, "outputs": [ { "data": { "text/plain": [ "0.5" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# The gradient computation works fine at x = 0.\n", "grad_log1pexp(tf.constant(0.)).numpy()" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.076606Z", "iopub.status.busy": "2022-12-14T22:22:15.076085Z", "iopub.status.idle": "2022-12-14T22:22:15.082540Z", "shell.execute_reply": "2022-12-14T22:22:15.081745Z" }, "id": "_VFSU0mG-FSp" }, "outputs": [ { "data": { "text/plain": [ "nan" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# However, x = 100 fails because of numerical instability.\n", "grad_log1pexp(tf.constant(100.)).numpy()" ] }, { "cell_type": "markdown", "metadata": { "id": "-VcTR34rqEhQ" }, "source": [ "在此例中，`log1pexp` 函数可以通过自定义梯度进行分析简化。下面的实现重用了在前向传递期间计算的 `tf.exp(x)` 值，通过消除冗余计算使其变得更加高效：" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.086169Z", "iopub.status.busy": "2022-12-14T22:22:15.085653Z", "iopub.status.idle": "2022-12-14T22:22:15.090204Z", "shell.execute_reply": "2022-12-14T22:22:15.089466Z" }, "id": "Q7nvfx_-qEhS" }, "outputs": [], "source": [ "@tf.custom_gradient\n", "def log1pexp(x):\n", " e = tf.exp(x)\n", " def grad(dy):\n", " return dy * (1 - 1 / (1 + e))\n", " return tf.math.log(1 + e), grad\n", "\n", "def grad_log1pexp(x):\n", " with tf.GradientTape() as tape:\n", " tape.watch(x)\n", " value = log1pexp(x)\n", " return tape.gradient(value, x)\n" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.093593Z", "iopub.status.busy": "2022-12-14T22:22:15.093112Z", "iopub.status.idle": "2022-12-14T22:22:15.100054Z", "shell.execute_reply": "2022-12-14T22:22:15.099300Z" }, "id": "5gHPKMfl-Kge" }, "outputs": [ { "data": { "text/plain": [ "0.5" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# As before, the gradient computation works fine at x = 0.\n", "grad_log1pexp(tf.constant(0.)).numpy()" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.103447Z", "iopub.status.busy": "2022-12-14T22:22:15.102932Z", "iopub.status.idle": "2022-12-14T22:22:15.109452Z", "shell.execute_reply": "2022-12-14T22:22:15.108793Z" }, "id": "u38MOfz3-MDE" }, "outputs": [ { "data": { "text/plain": [ "1.0" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# And the gradient computation also works at x = 100.\n", "grad_log1pexp(tf.constant(100.)).numpy()" ] }, { "cell_type": "markdown", "metadata": { "id": "rnZXjfQzqEhV" }, "source": [ "## 性能\n", "\n", "在 Eager Execution 期间，计算会自动分流到 GPU。如果想控制计算运行的位置，可将其放在 `tf.device('/gpu:0')` 块（或 CPU 等效块）中：" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.112807Z", "iopub.status.busy": "2022-12-14T22:22:15.112361Z", "iopub.status.idle": "2022-12-14T22:22:15.687598Z", "shell.execute_reply": "2022-12-14T22:22:15.686740Z" }, "id": "Ac9Y64H-qEhX" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Time to multiply a (1000, 1000) matrix by itself 200 times:\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "CPU: 0.4679419994354248 secs\n", "GPU: 0.06870079040527344 secs\n" ] } ], "source": [ "import time\n", "\n", "def measure(x, steps):\n", " # TensorFlow initializes a GPU the first time it's used, exclude from timing.\n", " tf.matmul(x, x)\n", " start = time.time()\n", " for i in range(steps):\n", " x = tf.matmul(x, x)\n", " # tf.matmul can return before completing the matrix multiplication\n", " # (e.g., can return after enqueing the operation on a CUDA stream).\n", " # The x.numpy() call below will ensure that all enqueued operations\n", " # have completed (and will also copy the result to host memory,\n", " # so we're including a little more than just the matmul operation\n", " # time).\n", " _ = x.numpy()\n", " end = time.time()\n", " return end - start\n", "\n", "shape = (1000, 1000)\n", "steps = 200\n", "print(\"Time to multiply a {} matrix by itself {} times:\".format(shape, steps))\n", "\n", "# Run on CPU:\n", "with tf.device(\"/cpu:0\"):\n", " print(\"CPU: {} secs\".format(measure(tf.random.normal(shape), steps)))\n", "\n", "# Run on GPU, if available:\n", "if tf.config.experimental.list_physical_devices(\"GPU\"):\n", " with tf.device(\"/gpu:0\"):\n", " print(\"GPU: {} secs\".format(measure(tf.random.normal(shape), steps)))\n", "else:\n", " print(\"GPU: not found\")" ] }, { "cell_type": "markdown", "metadata": { "id": "RLw3IS7UqEhe" }, "source": [ "可以将 `tf.Tensor` 对象复制到不同设备来执行其运算：" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:22:15.691494Z", "iopub.status.busy": "2022-12-14T22:22:15.690806Z", "iopub.status.idle": "2022-12-14T22:22:15.698174Z", "shell.execute_reply": "2022-12-14T22:22:15.697454Z" }, "id": "ny6LX2BVqEhf" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_162107/1929933290.py:4: _EagerTensorBase.gpu (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.identity instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_162107/1929933290.py:5: _EagerTensorBase.cpu (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.identity instead.\n" ] } ], "source": [ "if tf.config.experimental.list_physical_devices(\"GPU\"):\n", " x = tf.random.normal([10, 10])\n", "\n", " x_gpu0 = x.gpu()\n", " x_cpu = x.cpu()\n", "\n", " _ = tf.matmul(x_cpu, x_cpu) # Runs on CPU\n", " _ = tf.matmul(x_gpu0, x_gpu0) # Runs on GPU:0" ] }, { "cell_type": "markdown", "metadata": { "id": "oA_qaII3-p6c" }, "source": [ "### 基准\n", "\n", "对于计算量繁重的模型（如在 GPU 上训练的 [ResNet50](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/resnet50)），Eager Execution 性能与 `tf.function` 执行相当。但是对于计算量较小的模型来说，这种性能差距会越来越大，并且在为有大量小运算的模型优化热代码路径方面，其性能还有待提升。\n", "\n", "## 使用函数\n", "\n", "虽然 Eager Execution 增强了开发和调试的交互性，但 TensorFlow 1.x 样式的计算图执行在分布式训练、性能优化和生产部署方面具有优势。为了弥补这一差距，TensorFlow 2.0 通过 `tf.function` API 引入了 `function`。有关详细信息，请参阅 [tf.function](https://render.githubusercontent.com/view/function.ipynb) 指南。" ] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "name": "eager.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" } }, "nbformat": 4, "nbformat_minor": 0 }