{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Tce3stUlHN0L" }, "source": [ "##### Copyright 2020 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2022-12-14T22:53:26.896212Z", "iopub.status.busy": "2022-12-14T22:53:26.895771Z", "iopub.status.idle": "2022-12-14T22:53:26.899798Z", "shell.execute_reply": "2022-12-14T22:53:26.899147Z" }, "id": "tuOe1ymfHZPu" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "qFdPvlXBOdUN" }, "source": [ "# 高级自动微分" ] }, { "cell_type": "markdown", "metadata": { "id": "MfBg1C5NB3X0" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
在 TensorFlow.org 上查看 在 Google Colab 中运行\n", " 在 GitHub 上查看源代码 下载笔记本
" ] }, { "cell_type": "markdown", "metadata": { "id": "8a859404ce7e" }, "source": [ "[梯度和自动微分简介](autodiff.ipynb)指南包括在 TensorFlow 中计算梯度所需的全部内容。本指南重点介绍 `tf.GradientTape` API 更深入、更不常见的功能。" ] }, { "cell_type": "markdown", "metadata": { "id": "MUXex9ctTuDB" }, "source": [ "## 设置" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:26.902953Z", "iopub.status.busy": "2022-12-14T22:53:26.902748Z", "iopub.status.idle": "2022-12-14T22:53:29.169891Z", "shell.execute_reply": "2022-12-14T22:53:29.169178Z" }, "id": "IqR2PQG4ZaZ0" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-12-14 22:53:27.845325: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n", "2022-12-14 22:53:27.845442: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n", "2022-12-14 22:53:27.845452: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n" ] } ], "source": [ "import tensorflow as tf\n", "\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "\n", "mpl.rcParams['figure.figsize'] = (8, 6)" ] }, { "cell_type": "markdown", "metadata": { "id": "uGRJJRi8TCkJ" }, "source": [ "## 控制梯度记录\n", "\n", "在[自动微分指南](autodiff.ipynb)中，您已了解构建梯度计算时如何控制条带监视变量和张量。\n", "\n", "条带还具有操作记录的方法。" ] }, { "cell_type": "markdown", "metadata": { "id": "gB_i0VnhQKt2" }, "source": [ "### 停止记录\n", "\n", "如果您希望停止记录梯度，可以使用 `tf.GradientTape.stop_recording` 暂时挂起记录。\n", "\n", "如果您不希望在模型中间对复杂运算微分，这可能有助于减少开销。其中可能包括计算指标或中间结果：" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:29.174186Z", "iopub.status.busy": "2022-12-14T22:53:29.173436Z", "iopub.status.idle": "2022-12-14T22:53:32.516698Z", "shell.execute_reply": "2022-12-14T22:53:32.515799Z" }, "id": "mhFSYf7uQWxR" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dz/dx: tf.Tensor(4.0, shape=(), dtype=float32)\n", "dz/dy: None\n" ] } ], "source": [ "x = tf.Variable(2.0)\n", "y = tf.Variable(3.0)\n", "\n", "with tf.GradientTape() as t:\n", " x_sq = x * x\n", " with t.stop_recording():\n", " y_sq = y * y\n", " z = x_sq + y_sq\n", "\n", "grad = t.gradient(z, {'x': x, 'y': y})\n", "\n", "print('dz/dx:', grad['x']) # 2*x => 4\n", "print('dz/dy:', grad['y'])" ] }, { "cell_type": "markdown", "metadata": { "id": "DEHbEZ1h4p8A" }, "source": [ "### 重置/从头开始纪录\n", "\n", "如果您希望完全重新开始，请使用 `tf.GradientTape.reset`。通常，直接退出梯度带块并重新开始比较易于读取，但在退出梯度带块有困难或不可行时，可以使用 `reset` 方法。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.520345Z", "iopub.status.busy": "2022-12-14T22:53:32.519757Z", "iopub.status.idle": "2022-12-14T22:53:32.526863Z", "shell.execute_reply": "2022-12-14T22:53:32.526265Z" }, "id": "lsMHsmrh4pqM" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dz/dx: tf.Tensor(4.0, shape=(), dtype=float32)\n", "dz/dy: None\n" ] } ], "source": [ "x = tf.Variable(2.0)\n", "y = tf.Variable(3.0)\n", "reset = True\n", "\n", "with tf.GradientTape() as t:\n", " y_sq = y * y\n", " if reset:\n", " # Throw out all the tape recorded so far.\n", " t.reset()\n", " z = x * x + y_sq\n", "\n", "grad = t.gradient(z, {'x': x, 'y': y})\n", "\n", "print('dz/dx:', grad['x']) # 2*x => 4\n", "print('dz/dy:', grad['y'])" ] }, { "cell_type": "markdown", "metadata": { "id": "6zS7cLmS6zMf" }, "source": [ "## 精确停止梯度流\n", "\n", "与上面的全局条带控制相比，`tf.stop_gradient` 函数更加精确。它可以用来阻止梯度沿着特定路径流动，而不需要访问条带本身：" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.530119Z", "iopub.status.busy": "2022-12-14T22:53:32.529526Z", "iopub.status.idle": "2022-12-14T22:53:32.539224Z", "shell.execute_reply": "2022-12-14T22:53:32.538586Z" }, "id": "30qnZMe48BkB" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dz/dx: tf.Tensor(4.0, shape=(), dtype=float32)\n", "dz/dy: None\n" ] } ], "source": [ "x = tf.Variable(2.0)\n", "y = tf.Variable(3.0)\n", "\n", "with tf.GradientTape() as t:\n", " y_sq = y**2\n", " z = x**2 + tf.stop_gradient(y_sq)\n", "\n", "grad = t.gradient(z, {'x': x, 'y': y})\n", "\n", "print('dz/dx:', grad['x']) # 2*x => 4\n", "print('dz/dy:', grad['y'])" ] }, { "cell_type": "markdown", "metadata": { "id": "mbb-9lnGVngH" }, "source": [ "## 自定义梯度\n", "\n", "在某些情况下，您可能需要精确控制梯度的计算方式，而不是使用默认值。这些情况包括：\n", "\n", "1. 正在编写的新运算没有定义的梯度。\n", "2. 默认计算在数值上不稳定。\n", "3. 您希望从前向传递缓存开销大的计算。\n", "4. 您想修改一个值（例如，使用 `tf.clip_by_value` 或 `tf.math.round`）而不修改梯度。\n", "\n", "对于第一种情况，要编写新运算，您可以使用 `tf.RegisterGradient` 自行设置。（请参阅 API 文档了解详细信息）。（注意，梯度注册为全局，需谨慎更改。）\n", "\n", "对于后三种情况，可以使用 `tf.custom_gradient`。" ] }, { "cell_type": "markdown", "metadata": { "id": "oHr31kc_irF_" }, "source": [ "以下示例将 `tf.clip_by_norm` 应用于中间梯度：" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.542484Z", "iopub.status.busy": "2022-12-14T22:53:32.541928Z", "iopub.status.idle": "2022-12-14T22:53:32.560962Z", "shell.execute_reply": "2022-12-14T22:53:32.560361Z" }, "id": "Mjj01w4NYtwd" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(2.0, shape=(), dtype=float32)\n" ] } ], "source": [ "# Establish an identity operation, but clip during the gradient pass.\n", "@tf.custom_gradient\n", "def clip_gradients(y):\n", " def backward(dy):\n", " return tf.clip_by_norm(dy, 0.5)\n", " return y, backward\n", "\n", "v = tf.Variable(2.0)\n", "with tf.GradientTape() as t:\n", " output = clip_gradients(v * v)\n", "print(t.gradient(output, v)) # calls \"backward\", which clips 4 to 2" ] }, { "cell_type": "markdown", "metadata": { "id": "n4t7S0scYrD3" }, "source": [ "请参阅 `tf.custom_gradient` 装饰器 API 文档，了解更多详细信息。" ] }, { "cell_type": "markdown", "metadata": { "id": "v0ODp4Oi--I0" }, "source": [ "### SavedModel 中的自定义梯度\n", "\n", "注：此功能将从 TensorFlow 2.6 开始提供。\n", "\n", "可以使用选项 `tf.saved_model.SaveOptions(experimental_custom_gradients=True)` 将自定义梯度保存到 SavedModel。\n", "\n", "要保存到 SavedModel，梯度函数必须可以跟踪（要了解详情，请参阅[使用 tf.function 提高性能](function.ipynb)指南）。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.564363Z", "iopub.status.busy": "2022-12-14T22:53:32.563821Z", "iopub.status.idle": "2022-12-14T22:53:32.567723Z", "shell.execute_reply": "2022-12-14T22:53:32.567122Z" }, "id": "Q5JBgIBYjN1I" }, "outputs": [], "source": [ "class MyModule(tf.Module):\n", "\n", " @tf.function(input_signature=[tf.TensorSpec(None)])\n", " def call_custom_grad(self, x):\n", " return clip_gradients(x)\n", "\n", "model = MyModule()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.570540Z", "iopub.status.busy": "2022-12-14T22:53:32.570316Z", "iopub.status.idle": "2022-12-14T22:53:32.767847Z", "shell.execute_reply": "2022-12-14T22:53:32.767122Z" }, "id": "xZTrgy2q-9pq" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Assets written to: saved_model/assets\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(2.0, shape=(), dtype=float32)\n" ] } ], "source": [ "tf.saved_model.save(\n", " model,\n", " 'saved_model',\n", " options=tf.saved_model.SaveOptions(experimental_custom_gradients=True))\n", "\n", "# The loaded gradients will be the same as the above example.\n", "v = tf.Variable(2.0)\n", "loaded = tf.saved_model.load('saved_model')\n", "with tf.GradientTape() as t:\n", " output = loaded.call_custom_grad(v * v)\n", "print(t.gradient(output, v))" ] }, { "cell_type": "markdown", "metadata": { "id": "d-LfRs5FbJCk" }, "source": [ "关于上述示例的注意事项：如果您尝试用 `tf.saved_model.SaveOptions(experimental_custom_gradients=False)` 替换上面的代码，梯度仍会在加载时产生相同的结果。原因在于，梯度注册表仍然包含函数 `call_custom_op` 中使用的自定义梯度。但是，如果在没有自定义梯度的情况下保存后重新启动运行时，则在 `tf.GradientTape` 下运行加载的模型会抛出错误：`LookupError: No gradient defined for operation 'IdentityN' (op type: IdentityN)`。" ] }, { "cell_type": "markdown", "metadata": { "id": "8aENEt6Veryb" }, "source": [ "## 多个条带\n", "\n", "多个条带无缝交互。\n", "\n", "例如，下面每个条带监视不同的张量集：" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.771447Z", "iopub.status.busy": "2022-12-14T22:53:32.770894Z", "iopub.status.idle": "2022-12-14T22:53:32.778938Z", "shell.execute_reply": "2022-12-14T22:53:32.778242Z" }, "id": "BJ0HdMvte0VZ" }, "outputs": [], "source": [ "x0 = tf.constant(0.0)\n", "x1 = tf.constant(0.0)\n", "\n", "with tf.GradientTape() as tape0, tf.GradientTape() as tape1:\n", " tape0.watch(x0)\n", " tape1.watch(x1)\n", "\n", " y0 = tf.math.sin(x0)\n", " y1 = tf.nn.sigmoid(x1)\n", "\n", " y = y0 + y1\n", "\n", " ys = tf.reduce_sum(y)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.782321Z", "iopub.status.busy": "2022-12-14T22:53:32.781826Z", "iopub.status.idle": "2022-12-14T22:53:32.791918Z", "shell.execute_reply": "2022-12-14T22:53:32.791320Z" }, "id": "6ApAoMNFfNz6" }, "outputs": [ { "data": { "text/plain": [ "1.0" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tape0.gradient(ys, x0).numpy() # cos(x) => 1.0" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.795148Z", "iopub.status.busy": "2022-12-14T22:53:32.794645Z", "iopub.status.idle": "2022-12-14T22:53:32.800473Z", "shell.execute_reply": "2022-12-14T22:53:32.799873Z" }, "id": "rF1jrAJsfYW_" }, "outputs": [ { "data": { "text/plain": [ "0.25" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tape1.gradient(ys, x1).numpy() # sigmoid(x1)*(1-sigmoid(x1)) => 0.25" ] }, { "cell_type": "markdown", "metadata": { "id": "DK05KXrAAld3" }, "source": [ "### 高阶梯度\n", "\n", "`tf.GradientTape` 上下文管理器内的运算会被记录下来，以供自动微分。如果在该上下文中计算梯度，梯度计算也会被记录。因此，完全相同的 API 也适用于高阶梯度。\n", "\n", "例如：" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.803497Z", "iopub.status.busy": "2022-12-14T22:53:32.803216Z", "iopub.status.idle": "2022-12-14T22:53:32.811810Z", "shell.execute_reply": "2022-12-14T22:53:32.811227Z" }, "id": "cPQgthZ7ugRJ" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dy_dx: 3.0\n", "d2y_dx2: 6.0\n" ] } ], "source": [ "x = tf.Variable(1.0) # Create a Tensorflow variable initialized to 1.0\n", "\n", "with tf.GradientTape() as t2:\n", " with tf.GradientTape() as t1:\n", " y = x * x * x\n", "\n", " # Compute the gradient inside the outer `t2` context manager\n", " # which means the gradient computation is differentiable as well.\n", " dy_dx = t1.gradient(y, x)\n", "d2y_dx2 = t2.gradient(dy_dx, x)\n", "\n", "print('dy_dx:', dy_dx.numpy()) # 3 * x**2 => 3.0\n", "print('d2y_dx2:', d2y_dx2.numpy()) # 6 * x => 6.0" ] }, { "cell_type": "markdown", "metadata": { "id": "k0HV-Ah4_76i" }, "source": [ "虽然这确实可以得到*标量*函数的二次导数，但这种模式并不能通用于生成黑塞矩阵，因为 `tf.GradientTape.gradient` 只计算标量的梯度。要构造[黑塞矩阵](https://en.wikipedia.org/wiki/Hessian_matrix)，请参见[“雅可比矩阵”部分](https://en.wikipedia.org/wiki/Hessian_matrix)下的[“黑塞矩阵”示例](#hessian)。\n", "\n", "当您从梯度计算标量，然后产生的标量作为第二个梯度计算的源时，“嵌套调用 `tf.GradientTape.gradient`”是一种不错的模式，如以下示例所示。\n" ] }, { "cell_type": "markdown", "metadata": { "id": "t7LRlcpVKHv1" }, "source": [ "#### 示例：输入梯度正则化\n", "\n", "许多模型都容易受到“对抗样本”影响。这种技术的集合会修改模型的输入，进而混淆模型输出。最简单的实现（例如，[使用 Fast Gradient Signed Method 攻击的对抗样本](https://tensorflow.google.cn/tutorials/generative/adversarial_fgsm)）沿着输出相对于输入的梯度（即“输入梯度”） 迈出一步。\n", "\n", "一种增强相对于对抗样本的稳健性的方法是[输入梯度正则化](https://arxiv.org/abs/1905.11468)（Finlay 和 Oberman，2019 年），这种方法会尝试将输入梯度的幅度最小化。如果输入梯度较小，那么输出的变化也应该较小。\n", "\n", "以下是输入梯度正则化的简单实现：\n", "\n", "1. 使用内条带计算输出相对于输入的梯度。\n", "2. 计算该输入梯度的幅度。\n", "3. 计算该幅度相对于模型的梯度。" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.815159Z", "iopub.status.busy": "2022-12-14T22:53:32.814659Z", "iopub.status.idle": "2022-12-14T22:53:32.823086Z", "shell.execute_reply": "2022-12-14T22:53:32.822530Z" }, "id": "tH3ZFuUfDLrR" }, "outputs": [], "source": [ "x = tf.random.normal([7, 5])\n", "\n", "layer = tf.keras.layers.Dense(10, activation=tf.nn.relu)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:32.826170Z", "iopub.status.busy": "2022-12-14T22:53:32.825717Z", "iopub.status.idle": "2022-12-14T22:53:33.181410Z", "shell.execute_reply": "2022-12-14T22:53:33.180473Z" }, "id": "E6yOFsjEDR9u" }, "outputs": [], "source": [ "with tf.GradientTape() as t2:\n", " # The inner tape only takes the gradient with respect to the input,\n", " # not the variables.\n", " with tf.GradientTape(watch_accessed_variables=False) as t1:\n", " t1.watch(x)\n", " y = layer(x)\n", " out = tf.reduce_sum(layer(x)**2)\n", " # 1. Calculate the input gradient.\n", " g1 = t1.gradient(out, x)\n", " # 2. Calculate the magnitude of the input gradient.\n", " g1_mag = tf.norm(g1)\n", "\n", "# 3. Calculate the gradient of the magnitude with respect to the model.\n", "dg1_mag = t2.gradient(g1_mag, layer.trainable_variables)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.185576Z", "iopub.status.busy": "2022-12-14T22:53:33.185054Z", "iopub.status.idle": "2022-12-14T22:53:33.189613Z", "shell.execute_reply": "2022-12-14T22:53:33.188983Z" }, "id": "123QMq6PqK_d" }, "outputs": [ { "data": { "text/plain": [ "[TensorShape([5, 10]), TensorShape([10])]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[var.shape for var in dg1_mag]" ] }, { "cell_type": "markdown", "metadata": { "id": "E4xiYigexMtQ" }, "source": [ "## 雅可比矩阵\n" ] }, { "cell_type": "markdown", "metadata": { "id": "4-hVHVIeExkI" }, "source": [ "以上所有示例都取标量目标相对于某些源张量的梯度。\n", "\n", "[雅可比矩阵](https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant)代表向量值函数的梯度。每行都包含其中一个向量元素的梯度。\n", "\n", "`tf.GradientTape.jacobian` 方法让您能够有效计算雅可比矩阵。" ] }, { "cell_type": "markdown", "metadata": { "id": "KzNyIM0QBYIH" }, "source": [ "注意：\n", "\n", "- 类似于 `gradient`：`sources` 参数可以是张量或张量的容器。\n", "- 不同于 `gradient`：`target` 张量必须是单个张量。" ] }, { "cell_type": "markdown", "metadata": { "id": "O74K3hlxBC8a" }, "source": [ "### 标量源" ] }, { "cell_type": "markdown", "metadata": { "id": "B08OKn1Orkuc" }, "source": [ "作为第一个示例，以下是矢量目标相对于标量源的雅可比矩阵。" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.193558Z", "iopub.status.busy": "2022-12-14T22:53:33.193031Z", "iopub.status.idle": "2022-12-14T22:53:33.361152Z", "shell.execute_reply": "2022-12-14T22:53:33.360438Z" }, "id": "bAFeIE8EuVIq" }, "outputs": [], "source": [ "x = tf.linspace(-10.0, 10.0, 200+1)\n", "delta = tf.Variable(0.0)\n", "\n", "with tf.GradientTape() as tape:\n", " y = tf.nn.sigmoid(x+delta)\n", "\n", "dy_dx = tape.jacobian(y, delta)" ] }, { "cell_type": "markdown", "metadata": { "id": "BgHbUk3zr-WU" }, "source": [ "当您相对于标量取雅可比矩阵时，结果为**目标**的形状，并给出每个元素相对于源的梯度：" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.365004Z", "iopub.status.busy": "2022-12-14T22:53:33.364752Z", "iopub.status.idle": "2022-12-14T22:53:33.368584Z", "shell.execute_reply": "2022-12-14T22:53:33.367924Z" }, "id": "iZ6awnDzr_BA" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(201,)\n", "(201,)\n" ] } ], "source": [ "print(y.shape)\n", "print(dy_dx.shape)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.372097Z", "iopub.status.busy": "2022-12-14T22:53:33.371423Z", "iopub.status.idle": "2022-12-14T22:53:33.548977Z", "shell.execute_reply": "2022-12-14T22:53:33.548264Z" }, "id": "siNZaklc0_-e" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAp8AAAINCAYAAAB4RhRAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABdXElEQVR4nO3dd3hUZf7+8XtmMpNegJAChN6lSonBgiWK6KKsZbH8BMEua1lWV7Gxul/FtsquDXUVdV0V3RUbCGIUFEFQigLSCS2QkFDSk0lmzu+PSQKBBBJI5kx5v65rrpk55zlnPjOOw53znPM8FsMwDAEAAABeYDW7AAAAAAQPwicAAAC8hvAJAAAAryF8AgAAwGsInwAAAPAawicAAAC8hvAJAAAAryF8AgAAwGtCzC6gIdxut3bv3q3o6GhZLBazywEAAMARDMNQYWGh2rRpI6u1/uObfhE+d+/erZSUFLPLAAAAwHHs3LlT7dq1q3e9X4TP6OhoSZ43ExMTY3I1AAAAOFJBQYFSUlJqclt9/CJ8Vne1x8TEED4BAAB82PFOkeSCIwAAAHgN4RMAAABeQ/gEAACA1/jFOZ8N4XK5VFFRYXYZAcdmsykkJIQhrgAAQJMIiPBZVFSkXbt2yTAMs0sJSBEREUpOTpbD4TC7FAAA4Of8Pny6XC7t2rVLERERat26NUfompBhGHI6ncrNzVVmZqa6det2zEFjAQAAjsfvw2dFRYUMw1Dr1q0VHh5udjkBJzw8XHa7Xdu3b5fT6VRYWJjZJQEAAD8WMIexOOLZfDjaCQAAmgqpAgAAAF5D+AQAAIDXED4BAADgNYRPAAAAeA3h0yTvvPOOWrVqpfLy8lrLR48ereuuu86kqgAAAJpXo8Pnd999p1GjRqlNmzayWCz65JNPjrvNggULdOqppyo0NFRdu3bVW2+9dQKlNoxhGCpxVppya8wg91deeaVcLpc+++yzmmV79+7V7NmzNWHChOb4aAAAAEzX6HE+i4uL1b9/f02YMEGXXXbZcdtnZmbq4osv1q233qr//Oc/ysjI0I033qjk5GSNGDHihIo+ltIKl3o/Mq/J99sQvz02QhGOhn2k4eHhuuaaazRjxgxdeeWVkqR3331X7du319lnn92MVQIAAJin0eFz5MiRGjlyZIPbT58+XZ06ddLf//53SVKvXr20aNEiPf/8880SPv3JTTfdpCFDhigrK0tt27bVW2+9peuvv54xSwEAQMBq9hmOlixZovT09FrLRowYobvvvrtZXi/cbtNvj5kTasPttka1HzhwoPr376933nlHF1xwgdauXavZs2c3U3UAAPgXwzDkchtyVd1Xug0Zhme5YUhuw5DbkAxVL/csMyS53Z5T4dxHtJWqtjFqr6tu667af11tPTUdVp+M6geH39XZxjiiTfX7q7XsWNvVWlf39rXP/vM8SWkZoVPaxMqXNHv4zM7OVmJiYq1liYmJKigoUGlpaZ1TYpaXl9e6EKegoKDBr2exWBrc9e0LbrzxRk2bNk1ZWVlKT09XSkqK2SUBAIKEYRgqr3SrrMKlEqdLpRUulR52X+J0qazC87y8wiWny60Kl2cbZ9WtwlX1+Mj7qnUVbkPuquDocrs9YbLmuVHrufvw5VWBEyfnutM66G+jgyx8noipU6fq0UcfNbsMr7jmmmt0zz336PXXX9c777xjdjkAAD9T6nQpr6hc+4udOlDiVEFZpQpKK1RYVqmCsgoVllWooLTSc3/YusKyCpVWuBQI+c5q8Rx8slokiyyyWCSLRbJaLLKo6v7wNlX3UvVzTxtr1WlvVqtnP9VtLRbJokNTeR9+clz1mXKWqqV1nTl35HaHtznW9pYjGtX9unW/RnWbDq0iji7IZM0ePpOSkpSTk1NrWU5OjmJiYuo86ilJkydP1qRJk2qeFxQUBOwRwdjYWF1++eWaPXu2Ro8ebXY5AAAfYBiGCsoqtSe/VLsPlmr3wTLtyS9VXqFT+4rLlVfkud9X5FSJ09Ukr2m3WRRutyncYVOEI0RhdpsiHDaF220Ks9sUarfKYau6hVhlr7p3hFjlsFmq7q1yhNhkP+x5iM2qEKtFtsNuRz+3HrW8rjZWa92BEv6l2cNnWlqa5syZU2vZ/PnzlZaWVu82oaGhCg0Nbe7SfEZWVpauvfbaoHrPABDsyipc2r6vRJl5RcrMK9GO/SXafbC0KnCWqai8ssH7coRYFR/pUFyEQzHhIYoJsys6zK6Y8BDPfZhn2eHrosJCFOmwKawqYNptDP0N72h0+CwqKtLmzZtrnmdmZmrVqlVq2bKl2rdvr8mTJysrK6umC/nWW2/Viy++qL/85S+aMGGCvvnmG3344YdcWCPpwIEDWrBggRYsWKCXX37Z7HIAAM2grMKlTTlFWpddoPV7CrUxp1CZecXanV+q4w0P3SLCruTYcLWJC1ebuDC1jgpVq6hQtYpyKD7KoVaRnsdRoSEcAYTfaHT4/Pnnn3XOOefUPK/uHh83bpzeeust7dmzRzt27KhZ36lTJ82ePVt/+tOf9I9//EPt2rXTv/71r6AfZknyXO1+4MABPfXUU+rRo4fZ5QAATlJ5pUtrdxdo5Y6DWrXzoH7bna/MvOJ6z6uMDgtR5/hIdYqPVPtWkWobF1YVNMOVHBvmVxfQAg1lMRozLY9JCgoKFBsbq/z8fMXExNRaV1ZWpszMTHXq1ElhYWEmVRjY+IwBoG4Hip1asnWflmXu18qqsFnhOvqf1RYRdvVKjlGPpGj1TIpWl9ZR6hQfqZaRDo5YImAcK68djj+pAABooFKnSz9t268ftuTph815Wru74Kiu81aRDg1sH6eB7VuoT9tY9UqKVuvoUEImUIXwCQDAMewvdurrdTmatyZb32/Ok7PSXWt998QoDesSr4Ht43Rq+xZq1yKcoAkcA+ETAIAj5BSUad7abM1dk62lmftrDXaeHBum07vG64yu8RrWpZUSYjgdCWgMwicAAJKclW59sz5HM3/aqYUbc2tdJNQ7OUYjTknSiD6J6pEYzZFN4CQQPgEAQW3z3kLN/GmnPl6RpX3Fzprlp7aP08g+yRpxSpLa++AsMYC/InwCAIKOYRj6flOepi/cosVb9tUsbx0dqisGtdOVg9qpc+soEysEAhfh08ecffbZGjBggKZNm9Zk+7RYLJo1axbTdwIIepUut75ck63pC7do7e4CSZLNatG5PRM0ZnCKzu7RWiHM9AM0K8JnAHj00Ue1adMmvfvuu2aXAgA+yVnp1oc/79Rr323Vjv0lkqRwu01XDU3RjWd2Vtu4cJMrBIIH4TMAfPrpp7r//vvNLgMAfI7bbejzX3fr719trAmdLSLsGjeso8aldVSLSIfJFQLBh74FExUXF2vs2LGKiopScnKy/v73v9ese+yxx9SnT5+jthkwYIAefvjhmuc7d+7U2rVrdeGFF0qSNm3apLPOOkthYWHq3bu35s+fX2v7d955R1FRUdq0aVPNsttvv109e/ZUSUlJU79FADDNT9v2a9SLi3TXB6u0Y3+J4qNC9cjveuuH+8/V3endCZ6ASQLvyKdhSBUmhSh7hNSI4TfuvfdeLVy4UJ9++qkSEhL0wAMPaMWKFRowYIAmTJigRx99VD/99JOGDBkiSVq5cqV+/fVXffzxxzX7+Oyzz3T22WcrJiZGbrdbl112mRITE7V06VLl5+fr7rvvrvWaY8eO1RdffKFrr71Wixcv1rx58/Svf/1LS5YsUUQEV3MC8H/Z+WWa+uU6fbpqtyQpOjREtwzvrAlndGKudMAHBN7/hRUl0hNtzHntB3ZLjsgGNS0qKtIbb7yhd999V+edd54k6e2331a7du0kSe3atdOIESM0Y8aMmvA5Y8YMDR8+XJ07d67Zz6effqpLL71UkvT1119r/fr1mjdvntq08XwGTzzxhEaOHFnrtV999VX169dPd955pz7++GP99a9/1aBBg07uvQOAydxuQ/9Zul1Pzd2govJKWSzSVUPa654LuqtVVKjZ5QGoQre7SbZs2SKn06nU1NSaZS1btlSPHj1qnt900016//33VVZWJqfTqffee08TJkyoWV9QUKCFCxfqkksukSStW7dOKSkpNcFTktLS0o567RYtWuiNN97QK6+8oi5dunC+KAC/tyW3SH94dYke/nStisorNbB9nD7/4xmaellfgifgYwLvyKc9wnME0qzXbkKjRo1SaGioZs2aJYfDoYqKCl1xxRU167/88kv17t1bKSkpjd73d999J5vNpj179qi4uFjR0dFNWToAeIVhGHp36Q49Pvs3lVW4FeGw6S8jeui6tI6yWZmFCPBFgRc+LZYGd32bqUuXLrLb7Vq6dKnat28vSTpw4IA2btyo4cOHS5JCQkI0btw4zZgxQw6HQ1dddZXCww8NB3J4l7sk9erVSzt37tSePXuUnJwsSfrxxx+Peu3Fixfrqaee0ueff6777rtPf/zjH/X2228359sFgCaXV1Suv/z3V32zfq8k6Yyu8Xry8r5q14Lz1wFfFnjh009ERUXphhtu0L333qtWrVopISFBDz74oKzW2mdC3HjjjerVq5ck6YcffqhZXllZqS+//FL33HNPzbL09HR1795d48aN0zPPPKOCggI9+OCDtfZXWFio6667TnfeeadGjhypdu3aaciQIRo1alSto6oA4Mt+3rZfE99boZyCcjlCrLrvwp4aP6yjrBztBHwe4dNEzzzzjIqKijRq1ChFR0frz3/+s/Lz82u16datm4YNG6b9+/fXOj904cKFioqK0qmnnlqzzGq1atasWbrhhhs0dOhQdezYUf/85z9rhmGSpLvuukuRkZF64oknJEl9+/bVE088oVtuuUVpaWlq27ZtM79rADhxhmHozR+2aeqcdap0G+rSOlIvXXuqeibFmF0agAayGIZhmF3E8RQUFCg2Nlb5+fmKian9A1NWVqbMzEx16tRJYWFhJlXYfAzDULdu3XT77bdr0qRJNcvvvPNOVVZW6uWXX272GgL9MwbgH8orXZr8v9X6eGWWJOl3/ZL15OX9FBXKcRTAFxwrrx2O/2N9WG5urj744ANlZ2dr/Pjxtdb16dOnzivZASAQ7Ssq1y3/Xq6ftx+QzWrRQxf30vXDOsrSiLGVAfgGwqcPS0hIUHx8vF577TW1aNGi1rqbb77ZpKoAwLu25hZp3Ixl2rm/VNFhIXr52lN1ZrfWZpcF4AQRPn2YH5wRAQDNak1Wvsa9uUz7ip1q3zJCb14/WF0TGBoO8GeETwCAT1qyZZ9ueudnFZVXqk/bGL01fqjiGTAe8HuETwCAz/luY65ueudnlVe6dVrnlnp97GBFh9nNLgtAEwiY8EkXdfPhswXgTd9vOhQ803sl6MVrTlWY3WZ2WQCaiN/P7W6zeX6QnE6nyZUErpKSEkmS3c5RBwDNa9GmPN34dnXwTNTL1w4ieAIBxu+PfIaEhCgiIkK5ubmy2+1HzRCEE2cYhkpKSrR3717FxcXVBH0AaA7Ltx847Ihnol6+9lQ5QvhNBwKN34dPi8Wi5ORkZWZmavv27WaXE5Di4uKUlJRkdhkAAtjGnEJNeOsnlVa4NLx7a4InEMD8PnxKksPhULdu3eh6bwZ2u50jngCa1a4DJRr7xjLll1bo1PZxeuX/ETyBQBYQ4VPyzGvO1I8A4F8Kyio0fsZPyi4oU/fEKL15/RBFOALmnyYAdeBPSwCAKSpdbv3xvZXatLdIiTGhenvCUMVFOMwuC0AzI3wCAEzxty9+03cbcxVut+mNcUOUHBtudkkAvIDwCQDwund/3K63l2yXxSJNu2qA+rSNNbskAF5C+AQAeNXKHQf06OdrJUn3juihEacwmgYQTAifAACv2VdUrtv/s0IVLkMj+yTptuFdzC4JgJcRPgEAXuFyG7rzg5Xak1+mzvGRevqKfrJYLGaXBcDLCJ8AAK948ZvN+mHzPoXbbZp+3SBFhzFlLxCMCJ8AgGa3fPt+/fObTZKkx3/fR90To02uCIBZCJ8AgGZVUFahuz5YJZfb0OgBbXTZqe3MLgmAiQifAIBm9cgna7TrQKnatQjXY6P7mF0OAJMRPgEAzWb2r3v0yardslkt+sdVAxXDeZ5A0CN8AgCaxb6icj3y6RpJ0u1nd9GgDi1MrgiALyB8AgCaxZTP1mpfsVM9k6J1x7ndzC4HgI8gfAIAmtyXq/foi1/3yGa16Jkr+ssRwj83ADz4NQAANKn8kgo9XNXdfuvwzurbjnnbARxC+AQANKmn561XXpFTXVpH6s7z6G4HUBvhEwDQZFbuOKD3lu2QJP3f6L4KDbGZXBEAX0P4BAA0iUqXWw/OWiPDkC4b2FZpXVqZXRIAH0T4BAA0iXeWbNdvewoUExaiBy7uZXY5AHwU4RMAcNL2FZXr+fkbJUn3jeyp+KhQkysC4KsInwCAk/bc/I0qLK/UKW1idNWQ9maXA8CHET4BACdlfXaB3q+6yOiR3/WWzWoxuSIAvozwCQA4YYZh6G9f/Ca3IV3UN0mpnbnICMCxET4BACcsY91e/bB5nxw2qyaP5CIjAMdH+AQAnJBKl1tPzl0vSZpwRieltIwwuSIA/oDwCQA4IR+vzNLmvUWKi7Dr9nO6mF0OAD9B+AQANFpZhUvTqoZWuv3sLooJs5tcEQB/QfgEADTauz9u1+78MiXHhmlsWkezywHgRwifAIBGKSir0EvfbpYk3Z3eTWF25m8H0HCETwBAo8xYtE0HSirUuXWkLj+1ndnlAPAzhE8AQIMVlFXojUVbJUl/Su+uEBv/jABoHH41AAAN9vYP21RQVqmuCVG6qG+y2eUA8EOETwBAgxSWVehfizIlSXec25VpNAGcEMInAKBB3lmyXfmlFerSOlK/69fG7HIA+CnCJwDguIrKK/X6955zPe84txtHPQGcMMInAOC43l+6QwdLKtQ5PlKj+nPUE8CJI3wCAI7JWenWG1Xnet4yvDNHPQGcFMInAOCYPl2VpeyCMiXGhGr0wLZmlwPAzxE+AQD1crsNvfqd51zPCad3UmgIsxkBODmETwBAvTLW79XmvUWKDg3RNantzS4HQAAgfAIA6jV94RZJ0rWndVB0mN3kagAEAsInAKBOK3Yc0PLtB+SwWTXh9I5mlwMgQBA+AQB1mvHDNknSJQPaKCEmzNxiAAQMwicA4Ch78ks1Z/UeSdJ4jnoCaEInFD5feukldezYUWFhYUpNTdWyZcuO2X7atGnq0aOHwsPDlZKSoj/96U8qKys7oYIBAM3v30u2y+U2lNqppU5pE2t2OQACSKPD58yZMzVp0iRNmTJFK1asUP/+/TVixAjt3bu3zvbvvfee7r//fk2ZMkXr1q3TG2+8oZkzZ+qBBx446eIBAE2v1OnSe8t2SJImnNHJ5GoABJpGh8/nnntON910k8aPH6/evXtr+vTpioiI0Jtvvlln+8WLF+v000/XNddco44dO+qCCy7Q1VdffdyjpQAAc8xamaWDJRVKaRmu9F6JZpcDIMA0Knw6nU4tX75c6enph3ZgtSo9PV1Lliypc5thw4Zp+fLlNWFz69atmjNnji666KKTKBsA0BwMw9Bbiz1TaY5L68hUmgCaXEhjGufl5cnlcikxsfZfwomJiVq/fn2d21xzzTXKy8vTGWecIcMwVFlZqVtvvfWY3e7l5eUqLy+veV5QUNCYMgEAJ2hZ5n5tzClSuN2mPwxJMbscAAGo2a92X7BggZ544gm9/PLLWrFihT7++GPNnj1bf/vb3+rdZurUqYqNja25paTwAwgA3vDuUs+5nqMHtlEMg8oDaAaNOvIZHx8vm82mnJycWstzcnKUlJRU5zYPP/ywrrvuOt14442SpL59+6q4uFg333yzHnzwQVmtR+ffyZMna9KkSTXPCwoKCKAA0MxyC8s1d41neKVrUzuYXA2AQNWoI58Oh0ODBg1SRkZGzTK3262MjAylpaXVuU1JSclRAdNms0nynFtUl9DQUMXExNS6AQCa14c/71SFy9CAlDj1acvwSgCaR6OOfErSpEmTNG7cOA0ePFhDhw7VtGnTVFxcrPHjx0uSxo4dq7Zt22rq1KmSpFGjRum5557TwIEDlZqaqs2bN+vhhx/WqFGjakIoAMBcLreh96q63P/faRz1BNB8Gh0+x4wZo9zcXD3yyCPKzs7WgAEDNHfu3JqLkHbs2FHrSOdDDz0ki8Wihx56SFlZWWrdurVGjRqlxx9/vOneBQDgpCzYsFdZB0sVG27X7/olm10OgABmMerr+/YhBQUFio2NVX5+Pl3wANAMbnjrJ2Ws36ubzuykBy/ubXY5APxQQ/Mac7sDQJDLzi/Ttxs8s9RdNbS9ydUACHSETwAIcv9bsUtuQxrSsYW6tI4yuxwAAY7wCQBBzO029OHPOyVJfxjMkHYAmh/hEwCC2NLM/dq+r0RRoSG6mAuNAHgB4RMAglj1Uc9R/ZMV4Wj0ACgA0GiETwAIUvmlFZqz2jOj0ZghXGgEwDsInwAQpD77ZbfKK93qkRit/u2Y0QiAdxA+ASBI/W/5LknSlYPbyWKxmFwNgGBB+ASAILQlt0irdh6UzWrRJQPamF0OgCBC+ASAIDRrRZYk6axu8UqIDjO5GgDBhPAJAEHG7TY0a6UnfF52ajuTqwEQbAifABBklmbuV9bBUkWHhej83olmlwMgyBA+ASDIfLzCc6HR7/olK8xuM7kaAMGG8AkAQaTU6aoZ25MudwBmIHwCQBD56rdsFTtdSmkZrsEdWphdDoAgRPgEgCDy+S+7JUmjB7RlbE8ApiB8AkCQOFji1MKNuZKkSxnbE4BJCJ8AECS+XJOtCpehXskx6poQbXY5AIIU4RMAgsSnqzxje17Sn6OeAMxD+ASAIJCdX6almfslSaP6J5tcDYBgRvgEgCDwxa+7ZRjS4A4t1K5FhNnlAAhihE8ACAKfVV3lzoVGAMxG+ASAALctr1i/7sqXzWrRRX3pcgdgLsInAAS4L9dkS5LSOrdSq6hQk6sBEOwInwAQ4L5c45lOc2TfJJMrAQDCJwAEtF0HSvTrrnxZLdIFvQmfAMxH+ASAADa3qst9aKeWah1NlzsA8xE+ASCAzVld1eXehwuNAPgGwicABKjs/DKt2HFQknRhH7rcAfgGwicABKi5VRcaDerQQokxYSZXAwAehE8ACFDVQyyN5KgnAB9C+ASAAJRbWK5l2zxzuY9kYHkAPoTwCQABaN7abBmG1L9drNrGhZtdDgDUIHwCQACqHmKJo54AfA3hEwACzP5ip5Zs3SeJ8z0B+B7CJwAEmPm/ZcvlNtQ7OUYdWkWaXQ4A1EL4BIAAU32V+0XM5Q7ABxE+ASCA5JdU6IfNeZKkC5nVCIAPInwCQAD5el2OKlyGuidGqWtClNnlAMBRCJ8AEEAODSzPUU8AvonwCQABotTp0qLNuZKkEadwvicA30T4BIAA8cPmPJVVuNU2Lly9kqPNLgcA6kT4BIAAkbE+R5KU3itBFovF5GoAoG6ETwAIAG63oa/X7ZUkndcr0eRqAKB+hE8ACACrs/KVW1iuqNAQpXZuaXY5AFAvwicABICv13m63M/qHq/QEJvJ1QBA/QifABAAqrvc0+lyB+DjCJ8A4Od2HSjRuj0Fslqkc3okmF0OABwT4RMA/Nw36z1HPQd3aKkWkQ6TqwGAYyN8AoCfm/+b53zP83px1BOA7yN8AoAfKyyr0I9b90mS0ntzvicA30f4BAA/9v2mPFW4DHWKj1SX1lFmlwMAx0X4BAA/Vj3EUjpd7gD8BOETAPyUy23o2/UMsQTAvxA+AcBPrdhxQAdKKhQbbtegDi3MLgcAGoTwCQB+6uuqq9zP7ZmgEBs/5wD8A79WAOCnqs/3ZIglAP6E8AkAfigzr1hbcotlt1l0VvfWZpcDAA1G+AQAP5RRddQztVMrxYTZTa4GABqO8AkAfmjBhlxJ0jk96XIH4F8InwDgZ4rLK7Usc78k6ewedLkD8C+ETwDwM0u27JPT5VZKy3B1jo80uxwAaBTCJwD4mQUbPQPLn909QRaLxeRqAKBxCJ8A4EcMw6g535MudwD+iPAJAH5kS26xdh0olcNmVVqXVmaXAwCNRvgEAD+yYIOnyz21c0tFOEJMrgYAGo/wCQB+pLrLfTgDywPwU4RPAPATtYdYYnxPAP6J8AkAfqJ6iKV2LcLVpTVDLAHwT4RPAPAT1UMsndODIZYA+C/CJwD4AYZYAhAoCJ8A4AcYYglAoCB8AoAfYIglAIHihMLnSy+9pI4dOyosLEypqalatmzZMdsfPHhQEydOVHJyskJDQ9W9e3fNmTPnhAoGgGDEEEsAAkWj/3yeOXOmJk2apOnTpys1NVXTpk3TiBEjtGHDBiUkHD30h9Pp1Pnnn6+EhAT997//Vdu2bbV9+3bFxcU1Rf0AEPAYYglAIGl0+Hzuued00003afz48ZKk6dOna/bs2XrzzTd1//33H9X+zTff1P79+7V48WLZ7XZJUseOHU+uagAIIgyxBCCQNKrb3el0avny5UpPTz+0A6tV6enpWrJkSZ3bfPbZZ0pLS9PEiROVmJioPn366IknnpDL5Tq5ygEgSFQPsXR2j9YMsQTA7zXqyGdeXp5cLpcSExNrLU9MTNT69evr3Gbr1q365ptvdO2112rOnDnavHmzbr/9dlVUVGjKlCl1blNeXq7y8vKa5wUFBY0pEwACRq0hlrrT5Q7A/zX71e5ut1sJCQl67bXXNGjQII0ZM0YPPvigpk+fXu82U6dOVWxsbM0tJSWlucsEAJ+UmccQSwACS6PCZ3x8vGw2m3Jycmotz8nJUVJSUp3bJCcnq3v37rLZbDXLevXqpezsbDmdzjq3mTx5svLz82tuO3fubEyZABAwvt+UJ0ka3LGFIkMZYgmA/2tU+HQ4HBo0aJAyMjJqlrndbmVkZCgtLa3ObU4//XRt3rxZbre7ZtnGjRuVnJwsh8NR5zahoaGKiYmpdQOAYPT9Jk+X+5ndGGIJQGBodLf7pEmT9Prrr+vtt9/WunXrdNttt6m4uLjm6vexY8dq8uTJNe1vu+027d+/X3fddZc2btyo2bNn64knntDEiROb7l0AQAByVrq1ZMs+SdKZ3eJNrgYAmkaj+3DGjBmj3NxcPfLII8rOztaAAQM0d+7cmouQduzYIav1UKZNSUnRvHnz9Kc//Un9+vVT27Ztddddd+m+++5runcBAAFo5Y4DKna61CrSod7J9AABCAwWwzAMs4s4noKCAsXGxio/P58ueABB49l5G/Tit5t1Sf82+ufVA80uBwCOqaF5jbndAcBHHTrfky53AIGD8AkAPuhAsVO/ZuVL4mIjAIGF8AkAPuiHLXkyDKl7YpSSYsPMLgcAmgzhEwB80PcbPeN7ctQTQKAhfAKAjzEMQ4s2V4dPzvcEEFgInwDgY7bmFSvroGdKzdROTKkJILAQPgHAx3y/0XOV+5BOLRTusB2nNQD4F8InAPiY6vncOd8TQCAifAKAD3FWurVkK1NqAghchE8A8CErdhxQidOl+CiHeiUxoxuAwEP4BAAfUj2r0Rld42W1WkyuBgCaHuETAHxI9fmeZ3C+J4AARfgEAB+xv9ip1TVTanK+J4DARPgEAB/xw2bPlJo9EqOVGMOUmgACE+ETAHxE9fmeHPUEEMgInwDgAwzD0KLq8T27c74ngMBF+AQAH7Alt1i788vkCLFqaMeWZpcDAM2G8AkAPqC6y31ox5ZMqQkgoBE+AcAHHJpSk/M9AQQ2wicAmKy80qUlW6qn1OR8TwCBjfAJACZbsf2gSitcio8KVc+kaLPLAYBmRfgEAJMdPsQSU2oCCHSETwAwWc2Uml053xNA4CN8AoCJ9hWVa81uptQEEDwInwBgoh+27JNhSD2TopXAlJoAggDhEwBMtKjqfE+63AEEC8InAJjEMIxD43sypSaAIEH4BACTbMkt1p78MjlsTKkJIHgQPgHAJNVd7oM7tmBKTQBBg/AJACZZtLlqiCWucgcQRAifAGCCCpdbP27dL0k6syvnewIIHoRPADDBqp0HVVReqRYRdp3SJsbscgDAawifAGCC6qvcT+/KlJoAggvhEwBMsOiw+dwBIJgQPgHAy/JLK/TLLs+Ummd043xPAMGF8AkAXrZkyz653IY6x0eqbVy42eUAgFcRPgHAyxZtrppSky53AEGI8AkAXrao6mIj5nMHEIwInwDgRTv3l2jbvhLZrBad1qWV2eUAgNcRPgHAi6pnNRqQEqeYMLvJ1QCA9xE+AcCL6HIHEOwInwDgJS63oR+2eMIn43sCCFaETwDwkrW783WwpELRoSHqnxJndjkAYArCJwB4SfWUmqd1aSW7jZ9fAMGJXz8A8JLvmVITAAifAOANJc5KLd9+QBIXGwEIboRPAPCCpZn7VeEy1DYuXJ3iI80uBwBMQ/gEAC84fIgli8VicjUAYB7CJwB4QU345HxPAEGO8AkAzWxvQZk25BTKYpFO53xPAEGO8AkAzax6Ss1T2sSoZaTD5GoAwFyETwBoZtVd7md2a21yJQBgPsInADQjwzD0fdWRzzPpcgcAwicANKcNOYXKLSxXmN2qQR1bmF0OAJiO8AkAzai6y31op1YKDbGZXA0AmI/wCQDNqHo+d7rcAcCD8AkAzaS80qWlmfskMb4nAFQjfAJAM1m+/YDKKtyKjwpVz6Ros8sBAJ9A+ASAZnJoSs1WTKkJAFUInwDQTKoHlz+D8T0BoAbhEwCawYFip1Zn5UuSzuR8TwCoQfgEgGbww5Y8GYbUPTFKiTFhZpcDAD6D8AkAzeDQ+Z50uQPA4QifANDEDMM4NL4nXe4AUAvhEwCa2LZ9Jco6WCq7zaLUzi3NLgcAfArhEwCa2KJNuZKkU9u3UIQjxORqAMC3ED4BoInR5Q4A9SN8AkATqnC5tXiLZ0rNMxnfEwCOQvgEgCa0csdBFZVXqmWkQ33bxppdDgD4HMInADShhRv3SpLO6Bovq5UpNQHgSIRPAGhCCzd6LjYa3p0udwCoC+ETAJpIXlG51mQVSJLO7M7FRgBQF8InADSR76uGWOqdHKOEaKbUBIC6ED4BoIl8t9EzxNJZdLkDQL1OKHy+9NJL6tixo8LCwpSamqply5Y1aLsPPvhAFotFo0ePPpGXBQCf5XYb+o7zPQHguBodPmfOnKlJkyZpypQpWrFihfr3768RI0Zo7969x9xu27Ztuueee3TmmWeecLEA4Kt+21OgfcVORTpsGtShhdnlAIDPanT4fO6553TTTTdp/Pjx6t27t6ZPn66IiAi9+eab9W7jcrl07bXX6tFHH1Xnzp1PqmAA8EXVV7mndYmXI4QzmgCgPo36hXQ6nVq+fLnS09MP7cBqVXp6upYsWVLvdo899pgSEhJ0ww03NOh1ysvLVVBQUOsGAL6sZoilHnS5A8CxNCp85uXlyeVyKTExsdbyxMREZWdn17nNokWL9MYbb+j1119v8OtMnTpVsbGxNbeUlJTGlAkAXlVYVqEV2w9IkoYzpSYAHFOz9g0VFhbquuuu0+uvv674+IaPeTd58mTl5+fX3Hbu3NmMVQLAyVm8ZZ8q3YY6xUeqfasIs8sBAJ8W0pjG8fHxstlsysnJqbU8JydHSUlJR7XfsmWLtm3bplGjRtUsc7vdnhcOCdGGDRvUpUuXo7YLDQ1VaGhoY0oDANNUd7mf1Y2B5QHgeBp15NPhcGjQoEHKyMioWeZ2u5WRkaG0tLSj2vfs2VOrV6/WqlWram6XXHKJzjnnHK1atYrudAB+zzAOG2KJ8z0B4LgadeRTkiZNmqRx48Zp8ODBGjp0qKZNm6bi4mKNHz9ekjR27Fi1bdtWU6dOVVhYmPr06VNr+7i4OEk6ajkA+KOtecXadaBUDptVp3VuZXY5AODzGh0+x4wZo9zcXD3yyCPKzs7WgAEDNHfu3JqLkHbs2CGrlWFGAASHhRs8Rz2HdGqhCEejf1IBIOhYDMMwzC7ieAoKChQbG6v8/HzFxMSYXQ4A1Lh+xjIt2JCrBy7qqZvPOvocdgAIFg3NaxyiBIATVFbh0o9b90liPncAaCjCJwCcoJ+27VdZhVuJMaHqkRhtdjkA4BcInwBwgr5dXz3EUmtZLBaTqwEA/0D4BIAT9O2GvZKkc3smmFwJAPgPwicAnIDMvGJl5hXLbrPoDAaXB4AGI3wCwAn4Zr3nqOeQji0VHWY3uRoA8B+ETwA4Ad+up8sdAE4E4RMAGqmovFJLMz1DLBE+AaBxCJ8A0EiLNuWqwmWoY6sIdW4dZXY5AOBXCJ8A0EjV53uew1FPAGg0wicANILbbejbqvnc6XIHgMYjfAJAI6zdXaDcwnJFOGwa2qml2eUAgN8hfAJAI1R3uZ/RNV6hITaTqwEA/0P4BIBG+IZZjQDgpBA+AaCB8orK9euug5K42AgAThThEwAaaMGGXBmG1KdtjBJjwswuBwD8EuETABqoZlajHhz1BIATRfgEgAaocLn13UbPEEt0uQPAiSN8AkAD/LztgArLK9Uq0qH+7eLMLgcA/BbhEwAa4Nuqq9yH92gtq9VicjUA4L8InwDQANXjezLEEgCcHMInABzHtrxibd5bpBCrRWd2a212OQDg1wifAHAcX/2WLUk6rXMrxYbbTa4GAPwb4RMAjuOrtTmSpAtOSTS5EgDwf4RPADiG3MJyLd9xQJJ0fm/CJwCcLMInABzD1+tyZBhS/3axSo4NN7scAPB7hE8AOIav1nrO97zglCSTKwGAwED4BIB6FJZV6IfN+yRJF9DlDgBNgvAJAPVYuDFXTpdbneMj1TUhyuxyACAgED4BoB7zqq5yP/+URFkszGoEAE2B8AkAdSivdOnbqlmNLujN+Z4A0FQInwBQhx+37ldReaVaR4dqYEqc2eUAQMAgfAJAHeZVXeV+fu9EWa10uQNAUyF8AsAR3G5D83+rmtWIq9wBoEkRPgHgCKt2HVRuYbmiQ0M0rEu82eUAQEAhfALAEaq73M/umSBHCD+TANCU+FUFgMMYhqGv1tLlDgDNhfAJAIfZklukzLxiOWxWnd2jtdnlAEDAIXwCwGGqB5Yf1rWVosPsJlcDAIGH8AkAh/lyzR5JDCwPAM2F8AkAVbbvK9aarALZrBaNOIXzPQGgORA+AaDK7NWeo56ndW6pVlGhJlcDAIGJ8AkAVeZUhc+L+7YxuRIACFyETwAQXe4A4C2ETwDQoS73tM6t6HIHgGZE+AQAHepyv6hvssmVAEBgI3wCCHp0uQOA9xA+AQS9L36lyx0AvIXwCSDoff7LbknSxf3ocgeA5kb4BBDUNmQXan12oew2iy7qQ/gEgOZG+AQQ1D77JUuSNLx7gmIjmMsdAJob4RNA0DIMQ59VdblfMoCB5QHAGwifAILWyp0HtXN/qSIcNqX3SjC7HAAICoRPAEHrs1Weo57n905UhCPE5GoAIDjwawsgKLncRs0QS5d6q8vdVSkV5XhuxXmSI1KKSpCiEqWwGO/UAAAmI3wCCEqLt+Qpr6hccRF2ndG1dfO90MGd0oY50uYMadv3UkVJ3e1a95K6nid1O1/qeJZkpWMKQGAifAIISrNWeK5yv7hvshwhzRD0dq+SFv9TWvuJZLgOLbeGSJEJUmS8J4gW7ZXKC6TcdZ7bkhelVt2kYX+U+l0l2cOavjYAMBHhE0DQKS6v1JdrsiVJlw9q17Q7P7hT+vI+acPsQ8vap0ndR0hdzpMS+xx9VLN4n7T1W2nLN9K6L6R9m6TP75K+nSqNeFzqc7lksTRtnQBgEsIngKAzd022Sitc6hQfqYEpcU2zU1eltOw16Zv/kyqKJYtN6nOZNOwOKbn/sbeNbCX1vcJzG/mUtOIdacnLUsEu6X83SL+8L138d6lFx6apFQBMxElFAILOxyt3SZIuG9hWlqY4oliUK/17tDRvsid4ppwm3faDdPm/jh88jxQaLaVNlO5cIZ39gGRzSJu/ll45Xfrt05OvFQBMRvgEEFR2HyzV4i37JEmjB7Y9+R3u/El69SzPxUSOKGnUP6TxX0oJvU5uvyGh0tn3SbctkdoPk5xF0odjpa8e9hxlBQA/RfgEEFQ+WZUlw5BSO7VUSsuIk9vZrx9JM0ZKhbul+O7STd9Kg65v2ivV47tK4z73dN9LnouY3rtSKi9qutcAAC8ifAIIGoZh6OOqq9wvP/UkLzRa+pr08Y2Su0LqNUq66RupdfcmqLIOthDpgv+Trnxbskd6Lkx651KpZH/zvB4ANCPCJ4Cg8cuufG3eW6TQEKtG9k06sZ0YhrTwaenLez3Ph94iXfmO51zN5nbKaGncZ1J4CynrZ2nGRVLBnuZ/XQBoQoRPAEFj5k87JUkX9U1WdJj9xHby3TPSt497Hp892XN1ujcHhG832HNOaXSyZ1zQt0d5LngCAD9B+AQQFEqclfr8F89c7n8YnHJiO1ny8qHgecHj0tn3mzP+ZkIvacI8KTbFMybov38vlR7wfh0AcAIInwCCwuxf96iovFIdWkXotM4tG7+D5W97hlKSpHMe9MxAZKYWHaSxn3pmS8pZLb17hVReaG5NANAAhE8AQeHDnz1d7n8YnNL4sT03zJW+uNvzeNid0ln3Nm1xJ6pVF08ArT4H9KPxDMMEwOcRPgEEvC25Rfpp2wFZLSdwlfvuVdJ/J0iGWxp4nXT+Y7411WVib+na/0kh4dLm+dKXf/FcFAUAPorwCSDgfVh1odE5PRKUFBvW8A3zd0nvjfHMWtT5bOl3z/tW8KzWbpB0+euSLNLPb0hLXjK7IgCoF+ETQEBzVrr1vxWe6TT/MKQRFxo5S6T3r5KKsqXWPaU/vCPZTvAKeW/oNcozFqgkffWQ51QBAPBBhE8AAW3e2mzlFTmVEB2qc3smNGwjw5A+v1PKXi1FxEvXfCiFxTZvoU0hbaI0eIIkQ/r4Jilvs9kVAcBRCJ8AAtq7P26XJF01JEV2WwN/8n58WVr9kWQN8RzxbNGhGStsQhaLdOFTUvs0qbxAmnktV8AD8DknFD5feukldezYUWFhYUpNTdWyZcvqbfv666/rzDPPVIsWLdSiRQulp6cfsz0ANJVNOYVamrlfVot01dD2Ddso8zvpq4c9j0c8IXU8vfkKbA4hDs80nNHJUu566ZPbuAAJgE9pdPicOXOmJk2apClTpmjFihXq37+/RowYob1799bZfsGCBbr66qv17bffasmSJUpJSdEFF1ygrKysky4eAI7lP0t3SJLO65WoNnHhx9+gMLvqynaX1P9qaejNzVxhM4lOlMa8K9kc0rrPpaXTza4IAGpYDKNxfxKnpqZqyJAhevHFFyVJbrdbKSkpuuOOO3T//fcfd3uXy6UWLVroxRdf1NixYxv0mgUFBYqNjVV+fr5iYmIaUy6AIFXirFTq4xkqLK/U2xOGanj31sfewFUpvXOptH2RlNhHuvFryd6AwOrLlr0uzblHstqlG76S2p5qdkUAAlhD81qjjnw6nU4tX75c6enph3ZgtSo9PV1Llixp0D5KSkpUUVGhli3rn2GkvLxcBQUFtW4A0BifrdqtwqoZjc7sGn/8DRZM9QRPR5Sn29rfg6ckDblR6nWJ5K6Q/jteKss3uyIAaFz4zMvLk8vlUmJiYq3liYmJys7ObtA+7rvvPrVp06ZWgD3S1KlTFRsbW3NLSTnBeZgBBCXDMPTvqguNrhnaXlbrccbm3Jwhff+s5/El/5TiuzZzhV5isUiXvCDFtZcObJM+v4vzPwGYzqtXuz/55JP64IMPNGvWLIWF1T/Q8+TJk5Wfn19z27lzpxerBODvlmXu19rdBQqzW/WHwcf547U4z3NRjuQZpqjP5c1foDeFx0lXzPBcub92lrT8LbMrAhDkGhU+4+PjZbPZlJOTU2t5Tk6OkpKSjrnts88+qyeffFJfffWV+vXrd8y2oaGhiomJqXUDgIaa8cM2SdLvB7ZTi0hH/Q0NQ/rsTqkoxzOQ/IgnvFOgt7UbLJ03xfN47v1Szlpz6wEQ1BoVPh0OhwYNGqSMjIyaZW63WxkZGUpLS6t3u6efflp/+9vfNHfuXA0ePPjEqwWA49i5v0Rf/eY5DWjC6R2P3XjF29KG2Z4Lci57PTDO86xP2h+lrudLlWXSR9dLzmKzKwIQpBrd7T5p0iS9/vrrevvtt7Vu3TrddtttKi4u1vjx4yVJY8eO1eTJk2vaP/XUU3r44Yf15ptvqmPHjsrOzlZ2draKioqa7l0AQJW3F2+T25DO7BavbonR9TfM2yzNrfqtOu8RKfnYPTJ+z2qVfj/dM/5n3kZpzl/MrghAkGp0+BwzZoyeffZZPfLIIxowYIBWrVqluXPn1lyEtGPHDu3Zs6em/SuvvCKn06krrrhCycnJNbdnn3226d4FAEgqKq/UzJ8954hPOL1T/Q1dFZ7pJytKpE5neY4KBoPIeOnyf0mySKveldZ+YnZFAIJQo8f5NAPjfAJoiLcXb9OUz9aqc3ykvp40vP6r3DP+5rm6PSxOum2xFNvWq3WaLuMx6fu/B+/7B9AsmmWcTwDwVZUut17/fqskafzpHesPntuXSIue8zweNS04g9fZk6U2A6Wyg9Int0put9kVAQgihE8AAWH26j3adaBULSMdumJQPcMrleVLH98sGW6p/zXSKb/3bpG+wmaXLvuXZI/wzGW/5EWzKwIQRAifAPyeYRiavtBz1PP6YR0V7rDV3XDOvVL+DimugzTyKS9W6IPiu0oXTvU8znhM2vOLufUACBqETwB+77tNeVq3p0DhdpvGpnWou9Hq/0q/zpQsVs+wSmGcP65Tx0k9f+eZfvN/N0nOErMrAhAECJ8A/N70BVskSVcNTVFcRB2Dyh/cKX0xyfP4rHul9qlerM6HWSzSqH9KUUlS3gZp/sNmVwQgCBA+Afi1X3Ye1JKt+xRitejGMzsf3cDtkmbdKpXnS20He8InDolsJf3+Fc/jn/4lbZhrbj0AAh7hE4Bfe+GbTZKkS/q3Udu4OmYoWvyCtH2RZI+ULnvNc7ENautyrnTaRM/jTydKRXvNrQdAQCN8AvBba7Ly9fW6vbJapInndj26we5V0jf/53k88kmpVRev1udXzntESjhFKsnzBFDfHwIagJ8ifALwW//I8Bz1HNW/jbq0jqq90lnimcXIXeG5qGbgdSZU6EfsYZ7Zj2yh0qavPF3wANAMCJ8A/NLa3fma/1uOLBbpjnO7Hd1g/sOeOcyjkqRLXvBcXINjS+wtnf+Y5/FXD0l715tbD4CARPgE4Jf+WX3Us18bdU044qjnxnmHjtz9/hUpoqWXq/NjqbdIXdOlyjLpfzdKleVmVwQgwBA+AfidtbvzNW+t56jnnecdca5nUa7nnEVJOu12z8U0aDiLRbr0ZSmilZSz2jMAPQA0IcInAL/zzLwNkqqPekYfWmEYnuBZnOu5eOa8KSZV6OeiE6VLqqbcXPKitOVbc+sBEFAInwD8yo9b92nBhlyFWC2adH732it/fkPaNM9z0czlr3suosGJ6XmRNGi85/Ent0kl+82tB0DAIHwC8BuGYejpuZ6LYK4amqKO8ZGHVub8Js170PM4/a9S4ineLzDQjHhcatVNKtwjfX4Xwy8BaBKETwB+4+t1e7Vix0GF2a268/Ar3CtKpf9O8Fwk0zVdSr3VvCIDiSPScwTZGiKt+0xa+a7ZFQEIAIRPAH6h0uXWM/M8Rz0nnN5JCTGHdal/9ZCUu06KTJBGvyJZ+WlrMm0GSuc+5Hn85X3Svi3m1gPA7/ELDcAvzPx5pzbmFCk23K5bhh82U9H62YcNqzRdikowp8BANuxOqeOZUkWxZ+B+V4XZFQHwY4RPAD6voKxCf/9qoyTpT+ndFBteNT97ftahYZXS/ih1Pc+kCgOc1eYJ9mGxUtZyaeHTZlcEwI8RPgH4vBcyNml/sVNdWkfq2tM6eBa6XdKsW6TSA1LyAIZVam6x7aTfTfM8/v5ZafsSU8sB4L8InwB8WmZesd5avE2S9NDvestuq/rZWvS8tO17yR4pXfGmFOIwr8hg0ecyqf/VkuGWZt0sleWbXREAP0T4BODTHp+9ThUuQ2f3aK1zelSdz7l9ifTtE57HFz0jtepS/w7QtEY+LcV1kA7ukD6/m+GXADQa4ROAz5r/W46+XpejEKtFD13cy7OwaK/00fWS4ZL6XCENuMbUGoNOWIx0+b88wy+t/Vha9rrZFQHwM4RPAD6pxFmpv362VpJ045mdPdNouio943kWZUvxPaRR//DMRQ7vShkqnV815/u8B6SdP5lbDwC/QvgE4JP+mbFZWQdL1TYuXHee19Wz8NvHD53nOebfUmiUuUUGs9Nul3pfKrkrPEeii/eZXREAP0H4BOBzNmQX6l/fb5Uk/fWSUxThCJE2fCktes7T4JJ/Sq17mFghZLFIl7woteoqFeySPr7RMwIBABwH4ROAT3G5DU3++FdVug2d3ztR5/dOlPZneoZVkqSht0h9rzC3SHiExUh/+Ldkj5C2fMP4nwAahPAJwKfM+CFTK3YcVFRoiP56ySlSRZn04VjPsD7thkgX/J/ZJeJwib0Pjf+58Clp09emlgPA9xE+AfiMrblFembeBknSgxf3UtvYMGnOPVL2r1JEK+nKtxjP0xf1HyMNniDJ8HS/H9hmdkUAfBjhE4BPcLkN/eW/v6q80q0zusbrqiEp0o8vSyv/LckiXfa6Z5Yd+KYLn5TaDPTMOPXeVVJZgdkVAfBRhE8APuHNRZn6efsBRTpsevLyvrJs+kqa96Bn5QX/x7ztvi4kVBrzHykqScpdJ/3vBi5AAlAnwicA063JytfT89ZLkh68uLfaObd5xvOUIZ06VkqbaGp9aKDYttLV70khYdKmr6SvHja7IgA+iPAJwFQlzkrd+cFKVbgMXdA7UVefEia9N0ZyFkkdzpAu+jsDyfuTtoOk30/3PP7xJWn5W6aWA8D3ED4BmOpvX6zT1txiJcaE6qlLu8sy8/9J+Tuklp09A8lzgZH/OeX30jlVp0zM/rOU+Z259QDwKYRPAKb54tfden/ZDlks0nNX9leLjHuknUul0Fjp6plSREuzS8SJOuteqc8VkrtSmnmdlLfZ7IoA+AjCJwBTbMop1F/++6sk6dbhXXR61hvSrzMli036w9tS6+4mV4iTYrFIl74otR0slR2U3rtSKs4zuyoAPoDwCcDrisordeu7y1XidCmtcyvd0/IHacFUz8qLnpG6nGNugWga9nDpqvek2PbS/q3Sf66QygvNrgqAyQifALzKMAz95b+/aEtusZJiwvTq4CzZvrzHs/Ksv0hDbjC3QDSt6ETpuo+l8JbS7pWeLvhKp9lVATAR4ROAV/0zY7PmrM6W3WbRO+eUKWb2bZLhlk4dJ53zgNnloTnEd5Ou/a9kj5S2fivNupkxQIEgRvgE4DWf/7Jbz3+9UZL08pkV6v7NjZKrXOr5O+ni5xhSKZC1GySNeUey2qW1s6TP7pDcbrOrAmACwicAr1i544Du+egXSdIjA0t1/oqJUkWJ1OVc6Yo3JVuIyRWi2XVNl654w3NR2ar/SHPukQzD7KoAeBnhE0Czy8wr1k3v/KzySrdu6HxQ4zP/LDkLPYPIj/mPZ2pGBIfel1YNQm+Rfn5D+vI+AigQZAifAJpVTkGZrntjqfKKnLosYY8e2ne/LGUHpXZDpWs+kBwRZpcIb+v3B+mSf3oeL3vVMxA9XfBA0CB8Amg2+SUVGvvGMu06UKqL43bo2bIpspQXSO3TpP/3Pyk02uwSYZZTx0qXvKiaI6Bf3EUABYIE4RNAsygsq9D1by3ThpxCXRL5m16ofFRWZ5HU8UzPlc9hMWaXCLOdep2nC95ilVa8I318I8MwAUGA8AmgyRWWVWjsm8u0csdBjQlbpn8YT8laWeq5uOiaD6XQKLNLhK/of5V02euSNURa8z/p/askZ7HZVQFoRoRPAE3q8OB5a9jXelL/kMVdIfW53DNfO+d44kh9r/B8N+wR0pYM6e1LmIoTCGCETwBNZl9Rua55fal+2bFfj4f9W/frTVlkSENvli77lxTiMLtE+Kpu6dLYz6TwFlLWz9K/zpNyN5pdFYBmQPgE0CR2HSjRldOXaEtWjt4Ke17X6kvPivS/SiOflqz83OA4UoZIE76S4jpIB7ZJb6RLWxeaXRWAJsa/BgBO2vrsAl3xyhK59m3RF+FTdJaWSyFh0pVvS2f8iZmL0HCtu0s3feMZiqssX3r3Mmnpq4wFCgQQwieAk/LN+hxd/vJi9Sz6UV+EPqzOxk4pKlEa94V0ymizy4M/ioyXxn0u9b1ScldKX/5F+uQ2qaLU7MoANAHCJ4ATYhiG3liUqVveXqqb3R/oTcczilax54jVzQs9XajAibKHea6CH/GEZzrOX96X3jhf2rfF7MoAnCTCJ4BGK3FW6u6Zq/TaF4v0rv1x3RUyS1YZ0qDx0vWzpZhks0tEILBYpLSJ0nWzpIhWUvZq6dWzpF8/MrsyACeB8AmgUTbvLdKlL/6g8l8/0Zeh9yvVul6GI0q6/A1p1DSuaEfT6zxcunWR1OEMyVnkGYx+1m2ec0IB+B3CJ4AGMQxDHyzboWtfnKfbDjyt6Y5pamkpkpL6yXLLd56xGoHmEtNGGveZNPx+z4xIv7wnvTyMq+EBPxRidgEAfN/+Yqfu/9+v0vov9In9bSVb98uwWGU540+eMMDRTniD1SadM1nqco4061bpQKb0ziXS4AmeIb3CYs2uEEADWAzD98evKCgoUGxsrPLz8xUTw3zQgLcYhqHPf92jlz/9TndX/EsX2n7yLG/ZWZbfvyqlDDW5QgSt8iJp/sPSz296nkclSRc9LfW6hKG9AJM0NK8RPgHUKetgqR6btUKdNr+jO0JmKdJSLsMSIsvpd0rD/yLZw80uEZAyv5e+uFvat9nzvMt50oVPesYLBeBVhE8AJ6SswqXXF27RhoXva5LlPXW2ZkuS3G2HyDpqmpTUx9wCgSNVlEnfPyv98A/J5ZSsIZ4pXc+8R4psZXZ1QNAgfAJoFLfb0JzVu/XtnPc1rvRd9bNmSpIqIxIUcsFjUr8xTJEJ37ZvizTvQWlj1dSujijptNuktD9K4XGmlgYEA8IngAYxDEOLNudpzuf/1e8PztBQ6wZJUqUtXLZhE2U5/S4pjP/v4Ec2fy1lPCbt+cXzPCxWGnanlHqrFBplbm1AACN8AjgmwzC0cMNefTf3Q527732dYVsrSaq0hsoYfKPswyd5pjkE/JFhSOs+l759XMpd71kWEe8ZtH7Q9VJES1PLAwIR4RNAncorXZq7Yqt2LXxLFxTOUjdrliTJZQlRRb//p7Dz7vOMqQgEArdLWvM/6dsnPEMzSVJIuNT/Ks+R0ISe5tYHBBDCJ4Ba9uSX6rOFy+RYOUOj3fPVwlIkSSq3RsjV/xpFnHWH1KKjuUUCzcVVIa3+SPrxZc80ndW6nCul3iZ1Pc8zjiiAE0b4BCCX29DS9du0ZeH76rrnc6Va1slq8fwvnx/WViFptyoydRyDcyN4GIa0fbEnhK6fLanqn8DoZKnvlVL/q6XE3qaWCPgrwicQpAzD0KrtuVr7/edqtXWWznYvVbjFWbM+r/VpanHOHbL1HMmRHgS3A9ukpa9Jq/4jlR08tDypryeE9rlcik4yqzrA7xA+gSDidhtam7lT25d+ptAt8zS08mfFWkpq1ueGtpfR7yolnH6dFNfexEoBH1RZLm36SvrlA2njPMldcWhd28FSz4ukHhdJrXsyexJwDIRPIMDlFzu1csViHVjztZJzFmqQsVZ2i6tmfaEtTgc7/05JZ46XPWUQ/2gCDVGy33OB0q8zpV0/1V7XoqMnhHY5T2p/GsM2AUcgfAIBJr/EqXVrVujgb98oZs9i9Sj7Ra0sBbXa7LG3V1HH85Vy2uUK63Qa3erAySjYI22cK22YI21dKLnKD62zhkhtTpU6nSl1PFNKSZUcEebVCvgAwifgx1xuQ9t37lD2usUq3/6TInN/UZeKDWplKazVrkwO7Y4ZIGvXc9Qm9Qo5EpnPGmgW5UXS1m89YTTzO+ngjtrrrSFS4ilS20GHbvHd+QMQQYXwCfiJwpJS7dy8Rge3rVLF7jUKO7BBbcq2KMWy96i2Ttm1I7KPytqertb90pXY83QpxGFC1UCQO7Bd2va9lPm9574g6+g2jijPxUsJvaWEXp5wmtBLCm/h/XoBLyB8Aj6k3FmunB2bdGDXBpXlbJFr/1Y5CrYrrixLKe4shVoq69wuy9ZOebF9ZEsZrDa9T1fLLoOkkFAvVw/gmAxDyt8pZa2QspZ77nevlCqK624f3UZq1cVza3nYfYuOkj3Mq6UDTYnwCXhJRUWF8vZmKT9nu4pzd6n8QJaMgt2yFecovDRbrZy7lWTsVYjFXe8+ShSmLHtHFcZ2V0jSKWrVeYCSeqbKFsEREsAvuV1S7gYpZ620d62U85u09zdPSD2WyNZSTFsptl3Vfdvaz6MS+AMUPqtZw+dLL72kZ555RtnZ2erfv79eeOEFDR06tN72H330kR5++GFt27ZN3bp101NPPaWLLrqowa9H+IS3lJeXqbjggEoLD6q06IBK83NVlp+ryqI8GcX7ZC3bL1vZAYU6Dyq88qBiXAfUyjh4zGBZs2/Drmxbkg6EtVN5dHvZ47souk03JXbuq5jELpLV6oV3CMBUZfmeULpvi7R/y2H3WyVn4fG3l6TQGCky3hNUI+KrHlc9D2/pmTQiLMZzH1p174jiNwbNrqF5LaSxO545c6YmTZqk6dOnKzU1VdOmTdOIESO0YcMGJSQkHNV+8eLFuvrqqzV16lT97ne/03vvvafRo0drxYoV6tOnT2NfHkHKcLtVWVmhygqnKiqcqigvlbOsWM6yUlWUFauyvMRzc5bKVV4qt7NE7opSGVU3S0WZVFkmVZbI5ixWSGWhHJXFCnUVK8xdonCjRJFGicIsFWr0MQWL5DIsOmCJ08GQeBWHtpYzIlFGVLJCYpMVmdxN8Sk91TKpvTpYberQHB8QAP8QFiulDPXcDmcYUukBKX+X5/zRmvusw57v9oxBWl7gue3f2ogXtngCaWjsoXAaGuO5Qt8eLtkjDt3qWmYPl0LCJJvdc+TV5vDcjnzMBVZogEYf+UxNTdWQIUP04osvSpLcbrdSUlJ0xx136P777z+q/ZgxY1RcXKwvvviiZtlpp52mAQMGaPr06Q16TW8e+SwuPKiNiz9VzcdiGDr0CblV/cSQUfO49n3VeuOw9TIO7a9qvcU4Yplx2HY6Yr8y6nzcmLa12tW8B0mGu2a9xZAMuSW327PccHnu3W5Zqh8bdT+21HpuyGK4ZNHhy901zy2GS1ajUjZ3pWyqlNWoVIhRKVv1TS7Z5VkXYrgUoko5Dhu/0htKDYdKLBEqskarJCRO5fY4VYTGyR3eSpaIlrJFxcsR01oRLZLUIqmD4uLbyBpi92qNAIKMYXhmYireJxXnHrqVHPa8ZL8nmJYVeI6yluXXHjS/uVlsVYHULtlCDwVSa4jnZrHVfl7z+HhtbIfWWayecYstVs9NlqrnlsOeW+t4bj3U7rht6ttv9XjJh42bfOSyRrVp6HZ1Pa9nuyPbtOgoJfeTNzTLkU+n06nly5dr8uTJNcusVqvS09O1ZMmSOrdZsmSJJk2aVGvZiBEj9Mknn9T7OuXl5SovPzSeWkFBQb1tm9qBnF0auOROr70ejuMY46KXG3aVWxwql0NOi0MVllBVWD03V/XNFia3LUzukDAZIWFSSLgUFiNrWLRCwmMUEhErR2SswiLjFBYVp4joFoqMjlO43aFwSa289kYB4DgsFs+V8uEtpPiuDdvGMDy9PtVhtLzAE2DLqo6eVpRKFSWSs6TqcXEdy0o8t8pyyeX03CqdnnFPXc4jXs9V1b7J3z1O1JAbpYv/bnYVtTQqfObl5cnlcikxMbHW8sTERK1fv77ObbKzs+tsn52dXe/rTJ06VY8++mhjSmsy9rBwrbOfIkkyDvsr4tDh4aq/mCQZh/11UdNWlsOWH9a2aj+HlldtY7EcsX3V+qOWWw4LYpZatdXaX63Xqaue+vavmr/uDIvV89elLDKs1X9l2jz7qf6rs2Z59WObLFZrTVtZrLJYa9/LapOl6rk1xCFLiEPWkBBZbQ5ZQxyyhdhltYfKarMrxO55brOHymZ3KMTukD0kVDa7XSEOh+z2MIXabI3vIgeAYGKxVHWhh0vRicdv31iGIbkqqoJoRVVAPeKx21V1q6y6uTwhteZ5pafH7fDnRh3bVD+XUdX7ZtT0wtVeZhzdRoe1rbXMaECbw/cr6bBEUGfvYoPbnMx2jdh3i05Hb2eyRp/z6Q2TJ0+udbS0oKBAKSkpXnntxHZdlPjgYq+8FgAAfs1i8Yw1zHjDaIRGhc/4+HjZbDbl5OTUWp6Tk6OkpKQ6t0lKSmpUe0kKDQ1VaCjHtAAAAAJNo8ZdcDgcGjRokDIyMmqWud1uZWRkKC0trc5t0tLSarWXpPnz59fbHgAAAIGr0d3ukyZN0rhx4zR48GANHTpU06ZNU3FxscaPHy9JGjt2rNq2baupU6dKku666y4NHz5cf//733XxxRfrgw8+0M8//6zXXnutad8JAAAAfF6jw+eYMWOUm5urRx55RNnZ2RowYIDmzp1bc1HRjh07ZD1sINthw4bpvffe00MPPaQHHnhA3bp10yeffMIYnwAAAEGI6TUBAABw0hqa15hrCwAAAF5D+AQAAIDXED4BAADgNYRPAAAAeA3hEwAAAF5D+AQAAIDXED4BAADgNYRPAAAAeA3hEwAAAF5D+AQAAIDXED4BAADgNYRPAAAAeA3hEwAAAF4TYnYBDWEYhiSpoKDA5EoAAABQl+qcVp3b6uMX4bOwsFCSlJKSYnIlAAAAOJbCwkLFxsbWu95iHC+e+gC3263du3crOjpaFoul2V+voKBAKSkp2rlzp2JiYpr99fwFn0v9+GzqxudSPz6buvG51I/Ppm58LvXz9mdjGIYKCwvVpk0bWa31n9npF0c+rVar2rVr5/XXjYmJ4YtcBz6X+vHZ1I3PpX58NnXjc6kfn03d+Fzq583P5lhHPKtxwREAAAC8hvAJAAAAryF81iE0NFRTpkxRaGio2aX4FD6X+vHZ1I3PpX58NnXjc6kfn03d+Fzq56ufjV9ccAQAAIDAwJFPAAAAeA3hEwAAAF5D+AQAAIDXED4BAADgNUEZPh9//HENGzZMERERiouLq7PNjh07dPHFFysiIkIJCQm69957VVlZecz97t+/X9dee61iYmIUFxenG264QUVFRc3wDrxjwYIFslgsdd5++umnerc7++yzj2p/6623erFy7+jYseNR7/PJJ5885jZlZWWaOHGiWrVqpaioKF1++eXKycnxUsXNb9u2bbrhhhvUqVMnhYeHq0uXLpoyZYqcTucxtwvU78xLL72kjh07KiwsTKmpqVq2bNkx23/00Ufq2bOnwsLC1LdvX82ZM8dLlXrH1KlTNWTIEEVHRyshIUGjR4/Whg0bjrnNW2+9ddR3IywszEsVe89f//rXo95nz549j7lNoH9fpLp/Zy0WiyZOnFhn+0D+vnz33XcaNWqU2rRpI4vFok8++aTWesMw9Mgjjyg5OVnh4eFKT0/Xpk2bjrvfxv5ONYWgDJ9Op1NXXnmlbrvttjrXu1wuXXzxxXI6nVq8eLHefvttvfXWW3rkkUeOud9rr71Wa9eu1fz58/XFF1/ou+++080339wcb8Erhg0bpj179tS63XjjjerUqZMGDx58zG1vuummWts9/fTTXqraux577LFa7/OOO+44Zvs//elP+vzzz/XRRx9p4cKF2r17ty677DIvVdv81q9fL7fbrVdffVVr167V888/r+nTp+uBBx447raB9p2ZOXOmJk2apClTpmjFihXq37+/RowYob1799bZfvHixbr66qt1ww03aOXKlRo9erRGjx6tNWvWeLny5rNw4UJNnDhRP/74o+bPn6+KigpdcMEFKi4uPuZ2MTExtb4b27dv91LF3nXKKafUep+LFi2qt20wfF8k6aeffqr1mcyfP1+SdOWVV9a7TaB+X4qLi9W/f3+99NJLda5/+umn9c9//lPTp0/X0qVLFRkZqREjRqisrKzefTb2d6rJGEFsxowZRmxs7FHL58yZY1itViM7O7tm2SuvvGLExMQY5eXlde7rt99+MyQZP/30U82yL7/80rBYLEZWVlaT124Gp9NptG7d2njssceO2W748OHGXXfd5Z2iTNShQwfj+eefb3D7gwcPGna73fjoo49qlq1bt86QZCxZsqQZKvQNTz/9tNGpU6djtgnE78zQoUONiRMn1jx3uVxGmzZtjKlTp9bZ/g9/+INx8cUX11qWmppq3HLLLc1ap5n27t1rSDIWLlxYb5v6fqcDzZQpU4z+/fs3uH0wfl8MwzDuuusuo0uXLobb7a5zfbB8XyQZs2bNqnnudruNpKQk45lnnqlZdvDgQSM0NNR4//33691PY3+nmkpQHvk8niVLlqhv375KTEysWTZixAgVFBRo7dq19W4TFxdX64hgenq6rFarli5d2uw1e8Nnn32mffv2afz48cdt+5///Efx8fHq06ePJk+erJKSEi9U6H1PPvmkWrVqpYEDB+qZZ5455qkZy5cvV0VFhdLT02uW9ezZU+3bt9eSJUu8Ua4p8vPz1bJly+O2C6TvjNPp1PLly2v9t7ZarUpPT6/3v/WSJUtqtZc8vzuB/t2QdNzvR1FRkTp06KCUlBRdeuml9f4O+7tNmzapTZs26ty5s6699lrt2LGj3rbB+H1xOp169913NWHCBFkslnrbBcv35XCZmZnKzs6u9Z2IjY1Vampqvd+JE/mdaiohzbp3P5WdnV0reEqqeZ6dnV3vNgkJCbWWhYSEqGXLlvVu42/eeOMNjRgxQu3atTtmu2uuuUYdOnRQmzZt9Ouvv+q+++7Thg0b9PHHH3upUu+48847deqpp6ply5ZavHixJk+erD179ui5556rs312drYcDsdR5xknJiYGzHfkSJs3b9YLL7ygZ5999pjtAu07k5eXJ5fLVefvyPr16+vcpr7fnUD9brjdbt199906/fTT1adPn3rb9ejRQ2+++ab69eun/Px8Pfvssxo2bJjWrl173N8if5Kamqq33npLPXr00J49e/Too4/qzDPP1Jo1axQdHX1U+2D7vkjSJ598ooMHD+r666+vt02wfF+OVP3fvTHfiRP5nWoqARM+77//fj311FPHbLNu3brjnsAdDE7ks9q1a5fmzZunDz/88Lj7P/w81759+yo5OVnnnXeetmzZoi5dupx44V7QmM9m0qRJNcv69esnh8OhW265RVOnTvW5qcxO1ol8Z7KysnThhRfqyiuv1E033XTMbf35O4MTM3HiRK1Zs+aY5zVKUlpamtLS0mqeDxs2TL169dKrr76qv/3tb81dpteMHDmy5nG/fv2UmpqqDh066MMPP9QNN9xgYmW+44033tDIkSPVpk2betsEy/fF3wVM+Pzzn/98zL+GJKlz584N2ldSUtJRV3tVX5GclJRU7zZHnqBbWVmp/fv317uNWU7ks5oxY4ZatWqlSy65pNGvl5qaKslzFMzXg8TJfI9SU1NVWVmpbdu2qUePHketT0pKktPp1MGDB2sd/czJyfG578iRGvu57N69W+ecc46GDRum1157rdGv50/fmbrEx8fLZrMdNZLBsf5bJyUlNaq9P/vjH/9Yc1FmY49G2e12DRw4UJs3b26m6nxDXFycunfvXu/7DKbviyRt375dX3/9daN7Q4Ll+1L93z0nJ0fJyck1y3NycjRgwIA6tzmR36mmEjDhs3Xr1mrdunWT7CstLU2PP/649u7dW9OVPn/+fMXExKh37971bnPw4EEtX75cgwYNkiR98803crvdNf+Q+orGflaGYWjGjBkaO3as7HZ7o19v1apVklTrfwhfdTLfo1WrVslqtR51+kW1QYMGyW63KyMjQ5dffrkkacOGDdqxY0etv9R9UWM+l6ysLJ1zzjkaNGiQZsyYIau18aeW+9N3pi4Oh0ODBg1SRkaGRo8eLcnTzZyRkaE//vGPdW6TlpamjIwM3X333TXL5s+f7/PfjcYwDEN33HGHZs2apQULFqhTp06N3ofL5dLq1at10UUXNUOFvqOoqEhbtmzRddddV+f6YPi+HG7GjBlKSEjQxRdf3KjtguX70qlTJyUlJSkjI6MmbBYUFGjp0qX1juxzIr9TTaZZL2fyUdu3bzdWrlxpPProo0ZUVJSxcuVKY+XKlUZhYaFhGIZRWVlp9OnTx7jggguMVatWGXPnzjVat25tTJ48uWYfS5cuNXr06GHs2rWrZtmFF15oDBw40Fi6dKmxaNEio1u3bsbVV1/t9ffX1L7++mtDkrFu3bqj1u3atcvo0aOHsXTpUsMwDGPz5s3GY489Zvz8889GZmam8emnnxqdO3c2zjrrLG+X3awWL15sPP/888aqVauMLVu2GO+++67RunVrY+zYsTVtjvxsDMMwbr31VqN9+/bGN998Y/z8889GWlqakZaWZsZbaBa7du0yunbtapx33nnGrl27jD179tTcDm8TDN+ZDz74wAgNDTXeeust47fffjNuvvlmIy4urmYUjeuuu864//77a9r/8MMPRkhIiPHss88a69atM6ZMmWLY7XZj9erVZr2FJnfbbbcZsbGxxoIFC2p9N0pKSmraHPm5PProo8a8efOMLVu2GMuXLzeuuuoqIywszFi7dq0Zb6HZ/PnPfzYWLFhgZGZmGj/88IORnp5uxMfHG3v37jUMIzi/L9VcLpfRvn1747777jtqXTB9XwoLC2vyiiTjueeeM1auXGls377dMAzDePLJJ424uDjj008/NX799Vfj0ksvNTp16mSUlpbW7OPcc881XnjhhZrnx/udai5BGT7HjRtnSDrq9u2339a02bZtmzFy5EgjPDzciI+PN/785z8bFRUVNeu//fZbQ5KRmZlZs2zfvn3G1VdfbURFRRkxMTHG+PHjawKtP7v66quNYcOG1bkuMzOz1me3Y8cO46yzzjJatmxphIaGGl27djXuvfdeIz8/34sVN7/ly5cbqampRmxsrBEWFmb06tXLeOKJJ4yysrKaNkd+NoZhGKWlpcbtt99utGjRwoiIiDB+//vf1wpm/m7GjBl1/r91+N+5wfSdeeGFF4z27dsbDofDGDp0qPHjjz/WrBs+fLgxbty4Wu0//PBDo3v37obD4TBOOeUUY/bs2V6uuHnV992YMWNGTZsjP5e777675jNMTEw0LrroImPFihXeL76ZjRkzxkhOTjYcDofRtm1bY8yYMcbmzZtr1gfj96XavHnzDEnGhg0bjloXTN+X6txx5K36/bvdbuPhhx82EhMTjdDQUOO888476jPr0KGDMWXKlFrLjvU71VwshmEYzXtsFQAAAPBgnE8AAAB4DeETAAAAXkP4BAAAgNcQPgEAAOA1hE8AAAB4DeETAAAAXkP4BAAAgNcQPgEAAOA1hE8AAAB4DeETAAAAXkP4BAAvyc3NVVJSkp544omaZYsXL5bD4VBGRoaJlQGA9zC3OwB40Zw5czR69GgtXrxYPXr00IABA3TppZfqueeeM7s0APAKwicAeNnEiRP19ddfa/DgwVq9erV++uknhYaGml0WAHgF4RMAvKy0tFR9+vTRzp07tXz5cvXt29fskgDAazjnEwC8bMuWLdq9e7fcbre2bdtmdjkA4FUc+QQAL3I6nRo6dKgGDBigHj16aNq0aVq9erUSEhLMLg0AvILwCQBedO+99+q///2vfvnlF0VFRWn48OGKjY3VF198YXZpAOAVdLsDgJcsWLBA06ZN07///W/FxMTIarXq3//+t77//nu98sorZpcHAF7BkU8AAAB4DUc+AQAA4DWETwAAAHgN4RMAAABeQ/gEAACA1xA+AQAA4DWETwAAAHgN4RMAAABeQ/gEAACA1xA+AQAA4DWETwAAAHgN4RMAAABeQ/gEAACA1/x/SWhogTu+XXQAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.plot(x.numpy(), y, label='y')\n", "plt.plot(x.numpy(), dy_dx, label='dy/dx')\n", "plt.legend()\n", "_ = plt.xlabel('x')" ] }, { "cell_type": "markdown", "metadata": { "id": "DsOMSD_1BGkD" }, "source": [ "### 张量源" ] }, { "cell_type": "markdown", "metadata": { "id": "g3iXKN7KF-st" }, "source": [ "无论输入是标量还是张量，`tf.GradientTape.jacobian` 都能有效计算源的每个元素相对于目标的每个元素的梯度。\n", "\n", "例如，此层的输出的形状为 `(10，7)`。" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.552940Z", "iopub.status.busy": "2022-12-14T22:53:33.552263Z", "iopub.status.idle": "2022-12-14T22:53:33.562815Z", "shell.execute_reply": "2022-12-14T22:53:33.562242Z" }, "id": "39YXItgLxMBk" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([7, 10])" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = tf.random.normal([7, 5])\n", "layer = tf.keras.layers.Dense(10, activation=tf.nn.relu)\n", "\n", "with tf.GradientTape(persistent=True) as tape:\n", " y = layer(x)\n", "\n", "y.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "tshNRtfKuVP_" }, "source": [ "层内核的形状是 `(5，10)`。" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.566325Z", "iopub.status.busy": "2022-12-14T22:53:33.565753Z", "iopub.status.idle": "2022-12-14T22:53:33.570051Z", "shell.execute_reply": "2022-12-14T22:53:33.569411Z" }, "id": "CigTWyfPvPuv" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([5, 10])" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "layer.kernel.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "mN96JRpnAjpx" }, "source": [ "将这两个形状连在一起就是输出相对于内核的雅可比矩阵的形状：" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.573259Z", "iopub.status.busy": "2022-12-14T22:53:33.572715Z", "iopub.status.idle": "2022-12-14T22:53:33.688633Z", "shell.execute_reply": "2022-12-14T22:53:33.687891Z" }, "id": "pRLzTTbvEimH" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([7, 10, 5, 10])" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "j = tape.jacobian(y, layer.kernel)\n", "j.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "2Lrv7miMvTll" }, "source": [ "如果您在目标的维度上求和，会得到由 `tf.GradientTape.gradient` 计算的总和的梯度：" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.692551Z", "iopub.status.busy": "2022-12-14T22:53:33.691997Z", "iopub.status.idle": "2022-12-14T22:53:33.701294Z", "shell.execute_reply": "2022-12-14T22:53:33.700672Z" }, "id": "FJjZpYRnDjVa" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "g.shape: (5, 10)\n", "delta: 4.7683716e-07\n" ] } ], "source": [ "g = tape.gradient(y, layer.kernel)\n", "print('g.shape:', g.shape)\n", "\n", "j_sum = tf.reduce_sum(j, axis=[0, 1])\n", "delta = tf.reduce_max(abs(g - j_sum)).numpy()\n", "assert delta < 1e-3\n", "print('delta:', delta)" ] }, { "cell_type": "markdown", "metadata": { "id": "ZKajuGlk_krs" }, "source": [ " \n", "\n", "#### 示例：黑塞矩阵" ] }, { "cell_type": "markdown", "metadata": { "id": "NYcsXeo8TDLi" }, "source": [ "虽然 `tf.GradientTape` 并没有给出构造[黑塞矩阵](https://en.wikipedia.org/wiki/Hessian_matrix)的显式方法，但可以使用 tf.GradientTape.jacobian 方法进行构建。\n", "\n", "注：黑塞矩阵包含 `N**2` 个参数。由于这个原因和其他原因，它对于大多数模型都不实际。此示例主要是为了演示如何使用 `tf.GradientTape.jacobian` 方法，并不是对直接黑塞矩阵优化的认可。黑塞矩阵向量积可以[通过嵌套条带有效计算](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/eager/benchmarks/resnet50/hvp_test.py)，这也是一种更有效的二阶优化方法。" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.704964Z", "iopub.status.busy": "2022-12-14T22:53:33.704370Z", "iopub.status.idle": "2022-12-14T22:53:33.982510Z", "shell.execute_reply": "2022-12-14T22:53:33.981813Z" }, "id": "ELGTaell_j81" }, "outputs": [], "source": [ "x = tf.random.normal([7, 5])\n", "layer1 = tf.keras.layers.Dense(8, activation=tf.nn.relu)\n", "layer2 = tf.keras.layers.Dense(6, activation=tf.nn.relu)\n", "\n", "with tf.GradientTape() as t2:\n", " with tf.GradientTape() as t1:\n", " x = layer1(x)\n", " x = layer2(x)\n", " loss = tf.reduce_mean(x**2)\n", "\n", " g = t1.gradient(loss, layer1.kernel)\n", "\n", "h = t2.jacobian(g, layer1.kernel)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.986560Z", "iopub.status.busy": "2022-12-14T22:53:33.986051Z", "iopub.status.idle": "2022-12-14T22:53:33.989710Z", "shell.execute_reply": "2022-12-14T22:53:33.989070Z" }, "id": "FVqQuZj4XGjm" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "layer.kernel.shape: (5, 8)\n", "h.shape: (5, 8, 5, 8)\n" ] } ], "source": [ "print(f'layer.kernel.shape: {layer1.kernel.shape}')\n", "print(f'h.shape: {h.shape}')" ] }, { "cell_type": "markdown", "metadata": { "id": "_M7XElgaiMeP" }, "source": [ "要将此黑塞矩阵用于[牛顿方法](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization)步骤，首先需要将其轴展平为矩阵，然后将梯度展平为向量：" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:33.993118Z", "iopub.status.busy": "2022-12-14T22:53:33.992570Z", "iopub.status.idle": "2022-12-14T22:53:33.999939Z", "shell.execute_reply": "2022-12-14T22:53:33.999275Z" }, "id": "6te7N6wVXwXX" }, "outputs": [], "source": [ "n_params = tf.reduce_prod(layer1.kernel.shape)\n", "\n", "g_vec = tf.reshape(g, [n_params, 1])\n", "h_mat = tf.reshape(h, [n_params, n_params])" ] }, { "cell_type": "markdown", "metadata": { "id": "L9rO8b-0mgOH" }, "source": [ "黑塞矩阵应当对称：" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.003205Z", "iopub.status.busy": "2022-12-14T22:53:34.002673Z", "iopub.status.idle": "2022-12-14T22:53:34.006052Z", "shell.execute_reply": "2022-12-14T22:53:34.005510Z" }, "id": "8TCHc7Vrf52S" }, "outputs": [], "source": [ "def imshow_zero_center(image, **kwargs):\n", " lim = tf.reduce_max(abs(image))\n", " plt.imshow(image, vmin=-lim, vmax=lim, cmap='seismic', **kwargs)\n", " plt.colorbar()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.009264Z", "iopub.status.busy": "2022-12-14T22:53:34.008630Z", "iopub.status.idle": "2022-12-14T22:53:34.222801Z", "shell.execute_reply": "2022-12-14T22:53:34.222222Z" }, "id": "DExOxd7Ok2H0" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAH5CAYAAADENpm/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABSiklEQVR4nO3df3RU5bXH/08GTGIMMzEGEmKCgKhIEVAQiLWUSirg1cqV6wVrv4K1sLBgq2hb6ddfte2Nv2ppLYXl9feqqNVV6JV7iwo2uFoDCpgvRQUNQiGGBBCTIUmTkMz5/lFJjfLj2cwZ5yR5v9aapUx29jzPnDMzT/aZc3aK53meAAAAEAihZA8AAAAA/8LiDAAAIEBYnAEAAAQIizMAAIAAYXEGAAAQICzOAAAAAoTFGQAAQID0TPYAPisWi6mqqkq9evVSSkpKsocDAECgeZ6nAwcOKD8/X6HQF19zaWpqUktLS0Jyp6amKj09PSG5gyxwi7OqqioVFhYmexgAAHQqu3btUkFBwRf6mE1NTep94omqT1D+vLw8bd++vdst0AK3OOvVq5ckaddbbyn8yf8fS0ukt3P+1Pr9tgFlZTmHRuttf7GEM2POsXv22XL3yXHP/dcyW+4vn+/+F9Luj1JNufvmuo9bkvTBB+6x/frZcpeVucd+5Sum1Nv/7v6cDzjN9pxY9kPLPihJqq01hceysp1jQ3/fbhvLqae6x65aZUrdOP4S59iMqgpT7tjAQc6x77xjSq0hQ9xjV6605Z40yRb/4YfusYW5tqrL/nr39xXDW7gkKVTr/hlh2b8lKfThLvfcp7oXKaLRqE47rbD98/OL1NLSonpJN0lK8zl3s6RfVlerpaWFxZlfFi1apPvvv1/V1dUaPny4HnroIY0ePfqYv3foUGa4Vy/3xVk47Dyu1FCrc+w/B+KeW8ZysuWD8R/Nxtxh99wnnWTN7f5GWt9iW5xZxi1Jysy0JLflPumkhOXu1cuwgLI+J4b90Lw4i9niY4bnJWT9YLE85xkZptQ9DbkzooZ9ULbnxLJ7Swl9Sswvn2jUktu2OGsNub+vWMcdirl/Rli2pSSFou77uDW3pKR+FehESX4vn7rzl+ITMvfnnntO8+fP15133qmNGzdq+PDhmjhxovbs2ZOIhwMAAOgyErI4e/DBBzVr1ixde+21GjJkiJYsWaKMjAw99thjiXg4AACQRKEE3bor3+fe0tKiDRs2qLi4+F8PEgqpuLhYZYf5Dk9zc7Oi0WiHGwAAQHfl++Js3759amtrU25ubof7c3NzVV1d/bn4kpISRSKR9htnagIA0LlQOfNX0ue+YMEC1dXVtd927XI/mwUAACQfizN/+X62Zk5Ojnr06KGampoO99fU1CgvL+9z8WlpaUpL8/sEXAAAgM7J94VpamqqRo4cqdWrV7ffF4vFtHr1ahUVFfn9cAAAIMmonPkrIdc5mz9/vmbMmKFRo0Zp9OjRWrhwoRoaGnTttdcm4uEAAAC6jIQszqZNm6a9e/fqjjvuUHV1tUaMGKGVK1d+7iQBAADQ+SWi0tWdK2cpnud5yR7Ep0WjUUUiEe3dW6ew4xWSU6t3Oud/r8nWwmeQe6cVhWS7evr+WvddLzvTdgXtqn3uV9DOzzE2rD3MWbdH8toO2/M97kLjFegNL9/Nm02pZWlR19P4Z47pyvz79plyf1Dfxzm2f39TarNQdZVzbEtOvin3+vXusSNGmFKbZDQlriWcWo0dTRKpqckWb2i3U/OxrZNIbrP7e77phSzbe4ql1ZNkbGdmyB2NRhUZMEB1de6fm3459Jn9M/nfIaBJ0m1SUuaVbIHrrQkAADoXKmf+6s5zBwAACBwqZwAAIC5UzvzVnecOAAAQOFTOAABAXFLkf7Unxed8nQmLMwAAEJcU+b+Y6s6LMw5rAgAABAiVMwAAEJcen9z8ztldUTkDAAAIECpnAAAgLlxKw1+BXZyl1u9XasitdYmlJdOZa5+yDWTQt5xDG5tsu1J2k3trmw9qba1tBra+5xxbpTNNuSsMLZmGDjWlNgtVurdxGVa7w5a83BB/4YWm1FENdI7dZ2jHJNlaMoVaja27jC18Pmhy328Hbtlkyn1B5Rb34JwRptzKyXGPXb7clDo289vOsTsqbW2N8vLcYx95xJRa3/qWbSyqdw81tWOS9Ngq9/egb8+0tYQL1UedY3fWu7djkqR+le77eFXOMOfYA82B/SjHcWKLAgCAuFA581d3njsAAEDgUDkDAABxoXLmLxZnAAAgLizO/NWd5w4AABA4VM4AAEBcqJz5qzvPHQAAIHConAEAgLhQOfNXd547AABA4FA5AwAAcUn55OZ3zu6KyhkAAECApHie5yV7EJ8WjUYViURU9/HHCofDTr8TM6wxQ7L1WVOrW39PSVJPCpHxeq/C9veCpY+kdfPUG3oDhuXej0+SlJnpHGrZvyVbv8wW2folpvY0vn4MzPNsanTPnZ5hy214n7COG59nfl9OoM64PaPRqE4+OaK6ujrnz00/HzsSiegRSbZX2bE1SvqOlJR5JRurCQAAEBdOCPBXd547AABA4FA5AwAAcaFy5q/uPHcAAIDAoXIGAADiQuXMX9157gAAAIFD5QwAAMSFypm/uvPcAQAAAofKGQAAiAuVM3+xOAMAAHFhceavwC7OovUhKeS2acKZ7q0/Gptsmzsj3fAUWfr9SKYWPqqtteXOynKPragwpY4NOtM5NlS505S7f/9+pvjNm91jR4wwpTY9Lf3721qLZNfudw/OyjblVlOTc2hqpu0tYH+t7fWTneX+2ty715RavXu7N4ux7CeSNGywe9u2HZW2FliWlmOhHR+Ycsf6D3TPvf4NW+5Ro03xIcM+HjPu46F693ZpsUzba9PyNm55C5dsb+OW106Q2l/BH4FdnAEAgM4h5ZOb3zm7q+5cNQQAAAgcKmcAACAuPT65+Z2zu6JyBgAAECBUzgAAQFxS5H+1h++cAQAAIBConAEAgLhwnTN/sTgDAABxYXHmr+48dwAAgMChcgYAAOJC5cxf3XnuAAAAgRPYylk4M+bcM9PS7y+7qco2kLw891hro7XKSufQWIGt56Sp15ql2Z9sfQqH1e4w5e5pnKelX2boT/9ryn3e5MmmeIuY3HsJhqpt++wHTfnOsQMNfWklW78/SXqvwv21OWiQKbVpHx861Ph3qHtrTdNbhJWlV6ZkaquqjFGjTLmt/Rst/TItfTglaWe9e+4C49uy5W3c2JZYZw7quj0wqZz5qzvPHQAAIHB8X5zdddddSklJ6XAbPHiw3w8DAAACIpSgW3eVkMOaX/rSl7Rq1ap/PUjPwB49BQAACJSErJp69uypvER+EQMAAARGivxvt0T7Jp+9//77ys/P18CBA3X11Vdr586dR4xtbm5WNBrtcAMAAJ1HjwTduivfF2djxozRE088oZUrV2rx4sXavn27vvKVr+jAgQOHjS8pKVEkEmm/FRYW+j0kAACATsP3xdnkyZN15ZVXatiwYZo4caL+7//+T7W1tfr9739/2PgFCxaorq6u/bZr1y6/hwQAABKIEwL8lfBv6mdlZenMM89UxREuCJOWlqa0tLREDwMAAKBTSPjCtL6+Xtu2bVPfvn0T/VAAACAJUuR/1YwTAnx0yy23aM2aNdqxY4def/11/fu//7t69Oihq666yu+HAgAA6HJ8P6xZWVmpq666Sh999JF69+6tCy+8UGvXrlXv3r1NefbsC+kfzW5rx9yTW5zzflDr3tpGkgZaWpbU1ppyW1oyhTZvsuUeOsw991/+YspdMOIi9+DyHabc9SPGmeIt7VOs7Ziun+v+t8vie21nGYf27XOObSmwtfAZuMV9X4nJfT+RpL17TeGmlkzG3VDjLnSPXbHClvsbk9xjH3nElnvePPfY0Po3TLlNLZnuusuU2xofqjzyWfqfZW1P16/SsI8X2PZxy9u4tR3TxnL39xRLa7pYAL6dRfsmf/m+OHv22Wf9TgkAANBtcOl+AAAQFypn/mJxBgAA4sLizF/dee4AAKCLWbRokfr376/09HSNGTNGb7xx5O9uvv3225o6dar69++vlJQULVy4MO6cfmBxBgAA4hKUi9A+99xzmj9/vu68805t3LhRw4cP18SJE7Vnz57Dxjc2NmrgwIG65557jtgT3JrTDyzOAABAl/Dggw9q1qxZuvbaazVkyBAtWbJEGRkZeuyxxw4bf/755+v+++/X9OnTj3hBfGtOP7A4AwAAcUlJ0E2SotFoh1tzc/Nhx9DS0qINGzaouLi4/b5QKKTi4mKVlZUd17wSkdMFizMAABBYhYWFikQi7beSkpLDxu3bt09tbW3Kzc3tcH9ubq6qq6uP67ETkdMFZ2sCAIC49Pjk5ndOSdq1a5fC4XD7/d2hHzeLMwAAEFjhcLjD4uxIcnJy1KNHD9XU1HS4v6am5ohf9k9GThcc1gQAAHEJwtmaqampGjlypFavXt1+XywW0+rVq1VUVHRc80pETheBrZz1yYkpHHbrW1ZVneqcd2Dre8aRGJoDZmWZMocMfTstvTIlYy/O8eNNuXvWG4IvNDRAlBSWrUdl//7H/mvqeJn6ZWZm2pKnpzuHrl9vS31B5Rbn2JCl+aWk3r0zTPGWfdzSK1OStHy5c+ilU66w5a5vcg791rfc33+sYqNGm+Itz7e1V2bNXttHZW9L7+Da/abcVTnu74fW2kZ2lq1fpoWlX6alj+2BA+ah+C4oF6GdP3++ZsyYoVGjRmn06NFauHChGhoadO2110qSrrnmGp166qnt31traWnRO++80/7/H374ocrLy5WZmalBn7w/HitnIgR2cQYAAGAxbdo07d27V3fccYeqq6s1YsQIrVy5sv0L/Tt37lQo9K9lX1VVlc4999z2fz/wwAN64IEH9NWvflWlpaVOOROBxRkAAIhLivyvnKUcO+Sw5s2bp3nz5h32Z4cWXIf0799fnufFlTMR+M4ZAABAgFA5AwAAcQnKd866iu48dwAAgMChcgYAAOJC5cxf3XnuAAAAgUPlDAAAxIXKmb+689wBAAACh8oZAACIS4qO/7pkR8vZXQV2cfbXspBOOsmtsDdubItz3iqdaRpHvqUdSkWFKbf693cODf3lL7bclpZMkyaZUlcufNk5dsiqFabcMl7k73e/cY/93kxL3ynpoinuraFeXWVs+fLss86hQ6dcY8udOdg91tBGSpLefNM2lHPPdS/O/+53ttzf/talzrHGLlX6YGW1c+x/L7O1EPvBD9xjQ3fdYcodu+tu99yn9jXl7v3hblN86IXfO8c2Xvqfptz5+9zb8MWM7/kby9332aFDTam1bJl77JVXusempdnGkQg9Prn5nbO74rAmAABAgAS2cgYAADoHTgjwV3eeOwAAQOBQOQMAAHGhcuav7jx3AACAwKFyBgAA4kLlzF/dee4AAACBQ+UMAADEJUX+V3u4CC0AAMBx4rCmv7rz3AEAAAKHyhkAAIgLlTN/pXie5yV7EJ8WjUYViURUt3evwmHHnnXV7n3wXtvRzzSecRe690yMGXelzZvdYwsKTKnV07Dsrqy05R7ygnv/vl9n2XoD/sd/2MaSn9XoHPvbJzJMub972v86x758wr+Zco8d6x4b7uk+R0mmfpmNTbZ9NqOnex9bSdL69e6xxgaYr23p4xw7Ls+9F6Mk7Ux378fYr2eVKbfy8pxD99fatk92z6hzbM0/bD1Bc0/Yb4pvTM92jrW+B52573X3YMuLTZJaW91jt2yx5bY049y3zzk0euCAIoMGqa6uzv1z0yeHPrMrJfn9yFFJBVJS5pVsVM4AAEBcqJz5qzvPHQAAIHConAEAgLhQOfNXd547AABA4FA5AwAAcaFy5i8WZwAAIC4szvzVnecOAAAQOFTOAABAXKic+as7zx0AACBwqJwBAIC4UDnzV2AXZ7s/SlV9S6pTbIWhJZOle4ZVqHKnKX5Y7Q734HJDrCRdeKFz6JBVK0ypLS2Zen4/xZQ7f16bKf65591bMn23fLYpd8rcB5xjvW0fmHJrR717rHWnveUW59CMn/3MlPqdClsLrPIdFzjHbllpSq27x7i317q3zNZe60f/j3tLppRTl5tyt7V91zk2+4WHTblrLnffxxvybK/NnX+3dfrrt8+9ZdaZhlZFkqR699ePtUXZjh1unzuSVFs/zJR7+a3usTfe6N6e7ECre8s2dA6BXZwBAIDOIUVSSoptwX/MnMFq/f2F6s5VQwAAgMAxL85ee+01XXbZZcrPz1dKSoqWL1/e4eee5+mOO+5Q3759deKJJ6q4uFjvv/++X+MFAABB07NnYm7dlHlx1tDQoOHDh2vRokWH/fl9992nX//611qyZInWrVunk046SRMnTlRTU1PcgwUAAAHE4sxX5plPnjxZkydPPuzPPM/TwoULddttt+nyyy+XJD311FPKzc3V8uXLNX369PhGCwAA0MX5uizdvn27qqurVVxc3H5fJBLRmDFjVFZWdtjFWXNzs5qbm9v/HY1G/RwSAABItJ49JZ9PCBAnBPijurpakpSbm9vh/tzc3PaffVZJSYkikUj7rbCw0M8hAQAAdCpJP1tzwYIFqqura7/t2rUr2UMCAAAWfOfMV74uzvLy8iRJNTU1He6vqalp/9lnpaWlKRwOd7gBAAB0V74uzgYMGKC8vDytXr26/b5oNKp169apqKjIz4cCAABB0aOH/1WzHj2SPaukMdcM6+vrVVFR0f7v7du3q7y8XNnZ2erXr59uvPFG/exnP9MZZ5yhAQMG6Pbbb1d+fr6mTJni57gBAAC6pBTPs50OUVpaqq997Wufu3/GjBl64okn5Hme7rzzTj388MOqra3VhRdeqN/+9rc688wznfJHo1FFIhHVffxxIA5xvlfhXlzs39+W23I43dBKTpIUluGs18xMU+6qavfnJD8vZsot6yL+kUfcY3NybLlLS51DX0+/yJT6grHuz4u1N2CGGt2DrT0NCwpM4TFDcT7U2mLK3SL3HojWr66E5L59LHO05g4S6zwtrM+J5TVh3fapsu2HCWMYeDQaVeTkk1VXV/eFf262f2bn5ioc8ncficZiitTUJGVeyWaunI0fP15HW8+lpKTo7rvv1t133x3XwAAAQCfRs6fk8+JMsc75h4wfkn62JgAAAP6l+56nCgAA/EHlzFdUzgAAAAKEyhkAAIhPjx7+X/qirc3ffJ0IlTMAAIAAoXIGAADik4iLxvrdSL0ToXIGAAAQIFTOAABAfKic+YrFGQAAiA+LM19xWBMAACBAgls5++AD556PsUFufTslKVS50zSM/v37Ocdu3mxKrREj3GM/1WveSf/+7n3IfvcbW+7vfce9d+Nzz2eYck+z9MqUpJkz3WNXrLDlfuEF59ALxtt6VK578z+dY8eca+v1906F+3M+ZLCtV6bWrjWFt466wDk21dDLVJJSR41yD1640JR7/43u7eeyX3jYlFvf+Y5z6MOP2P5+nv0d94t2zp5jy/3wEtsFQTeWu+c/b2irKfeOHe59VYcMMvbK3LHDPdbYa1bl5c6hjSPcXzvW/rsJQeXMVwHYogAAADgkuJUzAADQOfTo8c/qGXxB5QwAACBAWOYCAID49OxJ5cxHVM4AAAAChGUuAACID5UzX/FMAgCA+LA48xWHNQEAAAKEZS4AAIhPIi6l4Xn+5utEqJwBAAAESIrnBWtpGo1GFYlEVLd3r8JhtxZEm7a4t/IYVvuaaTyxC8eZ4i1Cf/pf9+DJkxM2DtXXm8J/+zv31lDfLZ9tG8uSJbZ4i0mTTOGxlS87x4Zka23T0ur+d1Hq+tdNuZfucG/7Mn26KbVabV12TJvze3OMbXYMf6XHjH+HhuqjzrE1/3B/PUhSbm/bvmJRVe0+z7w8W+6E7uNbNplyv14/zDn2glHG/cqwk6/7m6093dlnu8eGM92f72g0qsjJJ6uurs75c9Mv7Z/ZkyYpfMIJ/uY+eFCRlSvN81q0aJHuv/9+VVdXa/jw4XrooYc0evToI8Y///zzuv3227Vjxw6dccYZuvfee3XJJZe0/3zmzJl68sknO/zOxIkTtXLlSvukHFE5AwAAXcJzzz2n+fPn684779TGjRs1fPhwTZw4UXv27Dls/Ouvv66rrrpK1113nd566y1NmTJFU6ZM0ebPNMueNGmSdu/e3X575plnEjoPFmcAACA+h87W9Ptm9OCDD2rWrFm69tprNWTIEC1ZskQZGRl67LHHDhv/q1/9SpMmTdIPfvADnX322frpT3+q8847T7/5zW86xKWlpSkvL6/9dvLJJx/X0+SKxRkAAAisaDTa4dbc3HzYuJaWFm3YsEHFxcXt94VCIRUXF6usrOywv1NWVtYhXvrnIcvPxpeWlqpPnz4666yzdP311+ujjz6Kc1ZHx+IMAADEJ4GVs8LCQkUikfZbSUnJYYewb98+tbW1KTc3t8P9ubm5qq6uPuzvVFdXHzN+0qRJeuqpp7R69Wrde++9WrNmjSZPnqy2trZ4nrGj4lIaAAAgPom4CO0n5yvu2rWrwwkBaWlp/j7OMUz/1JlT55xzjoYNG6bTTz9dpaWlmjBhQkIek8oZAAAIrHA43OF2pMVZTk6OevTooZqamg7319TUKO8Ipyfn5eWZ4iVp4MCBysnJUUVFhXEm7licAQCA+By6CK2ftx49TENITU3VyJEjtXr16vb7YrGYVq9eraKiosP+TlFRUYd4SXrllVeOGC9JlZWV+uijj9S3b1/T+CxYnAEAgC5h/vz5+u///m89+eSTevfdd3X99deroaFB1157rSTpmmuu0YIFC9rjv//972vlypX6xS9+oS1btuiuu+7S+vXrNW/ePElSfX29fvCDH2jt2rXasWOHVq9ercsvv1yDBg3SxIkTEzYPvnMGAADik4jvnMXsF2yeNm2a9u7dqzvuuEPV1dUaMWKEVq5c2f6l/507dyoU+ldd6oILLtDSpUt122236cc//rHOOOMMLV++XEOHDpUk9ejRQ5s2bdKTTz6p2tpa5efn6+KLL9ZPf/rThH73jcUZAADoMubNm9de+fqs0tLSz9135ZVX6sorrzxs/IknnqiXXnrJz+E5YXEGAADiE5DKWVcR3MVZWZl00klOoQUjLnLPW77DNIz6Ee69Na0nbpxn6Jd5/Vzb1wMX3+veG/CiKbZebK/e7N4TNGXuA6bc3vRSU7xeeME51NIrU5Lu7pHiHHveH20tavftc4/99qWDTLm3GNq9hVptfQdTD/NX59F8b8549+BLLzXljr7gvj3DZ9m+uFtTvts5tiHPfT+RJBmujTR7ju11b+ll2qOH+2tHktra/sMUv2yZe+y0K4eaci+/1T32glGm1FJ5uXPo2UPd+9hKUvi277kH33OPe2xTk2kcCL7gLs4AAEDnQOXMV5ytCQAAECBUzgAAQHyonPmKxRkAAIjPoYvQ+imBvSuDjsOaAAAAAULlDAAAxCcRhzWpnAEAACAIqJwBAID4UDnzFZUzAACAAKFyBgAA4kPlzFcpnufZ+s4kWDQaVSQSUd3HHyscdmsrFK13LwCG931gG1BOjnPo/lZbG6TsLMM1XOrrTbmVmWmLN3h5lfvzffEg2/P9evVAU/wFlb93D/4PW/uZ/1nhPs9vfH+AKbe2bnUOfW1tqin1uAPu7bVavv5vptyp9ftN8crKcg61vI4lKbzkPufY2C0/NOUO1brPc2d9til3QYFhHErcdZ5ixgMnlZW2/KZ57ttjyl3V2sc5Nj/P9hw2Nrk/Lxnpxu1jaLO0cUuGc2x9fVRf/WpEdXV1zp+bfmn/zP7hDxVOS/M3d3OzIvfdl5R5JRuVMwAAEJ9EXOestdXffJ0IizMAABCfRBzW9DtfJ2I+IeC1117TZZddpvz8fKWkpGj58uUdfj5z5kylpKR0uE2aNMmv8QIAAHRp5mVpQ0ODhg8frm9/+9u64oorDhszadIkPf744+3/TvP5ODQAAAgQKme+Ms988uTJmjx58lFj0tLSlJeX55SvublZzc3N7f+ORqPWIQEAAHQZCbnOWWlpqfr06aOzzjpL119/vT766KMjxpaUlCgSibTfCgsLEzEkAACQKIcqZ37fuinfF2eTJk3SU089pdWrV+vee+/VmjVrNHnyZLUd4XolCxYsUF1dXftt165dfg8JAACg0/B9WTp9+vT2/z/nnHM0bNgwnX766SotLdWECRM+F5+WlsZ30gAA6Mz4zpmvEt6+aeDAgcrJyVFFRUWiHwoAAKDTS/iytLKyUh999JH69u2b6IcCAADJkIiL0Pbo4W++TsT8TNbX13eogm3fvl3l5eXKzs5Wdna2fvKTn2jq1KnKy8vTtm3b9MMf/lCDBg3SxIkTfR04AAAICA5r+so88/Xr1+trX/ta+7/nz58vSZoxY4YWL16sTZs26cknn1Rtba3y8/N18cUX66c//an5e2Xb/x5Sr15uR10H9nfvbxaVrXdjONM9d7ahH58kxeTeky+0b58pt9LT3WOffdaUeuyUa9yDd9h6gl4w1tarbt2b/+kce66xE4jpKTf0ypQkbd7sHHrWWefZcte4n/Gc2tP2fFc12fpI5re2OMdWVNh6iJ73ne84xy5caEqt+TPdYxPZc1Klpbbk48c7h4YWPmhKXXDjfFN8aMs7zrEtg4aYcufXuvfijMm9D6ck1da6x2a4XTGq3aYK936ZQ4e65+UKVF2PeXE2fvx4Ha1X+ksvvRTXgAAAQCdD5cxXCT8hAAAAAO6677IUAAD4g8qZr6icAQAABEj3XZYCAAB/cCkNX1E5AwAACBAqZwAAID5858xX3XfmAADAHyzOfMVhTQAAgADpvstSAADgDypnvkrxjna5/ySIRqOKRCKq+/hjhcNht18y9Nn5oN7WyqN/f1O4Sai6yjm2JSfflHv9evdYS5sQSQr3bHQPtrSRktTYZCvmZvR0bw9kelIkadAg59DXttj2q7POco/NfdrWZmf2Fvc2O9a2RhnptnZPH+xw354DCwzbUtL8W93bPT14m621WmO6e5sq0z4omT5wrK+HRMpotfUIakl3fP8+DqlNhrFkZiZsHFXVtu2Tn2d4/Xyqh/WxROvrFRk5UnV1de6fmz5p/8x+5hmFM9zbUznlbmxU5KqrkjKvZOu+y1IAAOAPKme+Cs6fZQAAAKByBgAA4sRFaH1F5QwAACBAqJwBAID48J0zX3XfmQMAAH+wOPMVhzUBAAACpPsuSwEAgD+onPmKyhkAAECAdN9lKQAA8AeX0vAVlTMAAIAACWzlLFofkkJua8d9hn6Z1l6ZoVZD37ymJlPuD5rc+2UO3LLJlPuCyi3uwZmDTblNzThvucWUOuNnPzPFv1Ph3sutfMcFptxbVrrH3j3mf025VVPoHGrplSlJd/x3inNsxsIGU+6qalvvvIH73nCO/UP5aFPuB8uKnGNf31Jmyn3BKPfX/bhi9x6fklRa6h6bsWWjKbcGG17Lc+aYUtfc/5QpPrdpj3uwsQevamvdY43VnGir+z6e39MwR0nvVbh/VtXXn2mItfU9TQi+c+YrKmcAAAAB0n2XpQAAwB9UznzVfWcOAAD8weLMVxzWBAAACJDuuywFAAD+oHLmKypnAAAAAdJ9l6UAAMAfXITWV1TOAAAAAoTKGQAAiA/fOfMVlTMAAIAASfE8z0v2ID4tGo0qEomo7uOPFQ6HnX4nZlhjmtoxSWqRe2uW1J4xU24LyxwlKdTU6B5sbJ3S2OQ+lgwZxiFJ+/bZ4gsKnEPNz6FhX7HsJ5JtX7E835LxOTe2HFNmpi3e8pevdSytrc6hsUy395JDQnLfPtb9Cp9neb4lmfaVqlpbyzHLLm7tOvXxx+6xvXu7x0ajUZ18ckR1dXXOn5t+af/M3rJF4V69/M194IAigwcnZV7J1n1rhgAAwB8c1vQVf/IBAAAESPddlgIAAH9wKQ1fUTkDAAAIECpnAAAgPnznzFdUzgAAAAKk+y5LAQCAP6ic+YrKGQAA6DIWLVqk/v37Kz09XWPGjNEbb7xx1Pjnn39egwcPVnp6us455xz93//9X4efe56nO+64Q3379tWJJ56o4uJivf/++4mcAoszAAAQp0OVM79vRs8995zmz5+vO++8Uxs3btTw4cM1ceJE7dmz57Dxr7/+uq666ipdd911euuttzRlyhRNmTJFmzdvbo+577779Otf/1pLlizRunXrdNJJJ2nixIlqsl4424AOAcdAh4DPo0PA59Eh4AjoEAAHdAj4vE7XIcDwmW3KffLJpnmNGTNG559/vn7zm99IkmKxmAoLC3XDDTfo1ltv/Vz8tGnT1NDQoBUrVrTfN3bsWI0YMUJLliyR53nKz8/XzTffrFtuuUWSVFdXp9zcXD3xxBOaPn26DzP9PN5VAABAYEWj0Q635ubmw8a1tLRow4YNKi4ubr8vFAqpuLhYZWVlh/2dsrKyDvGSNHHixPb47du3q7q6ukNMJBLRmDFjjpjTD8H9tl1trRRz/GsqK9s9r/Gv89RM96dof61trZud5f7X4t69ptTq3dv9r8U337TlHnOue0XpnQrbX61DBrtXwiRJa9c6h7aOusCUOrW01D121ChT7qom9302P89WVaiqdn/O83OMbwGVlbZ4Q2VT1dW23P37O4eGSl81pW4ce5FzbMaWjabcsRHnOcdaX5vnn+8e+6lCgZNvXGrdD93fD/PzbGOJtrrv49ZibzjTfZ7W9/zc5p3OsTH1M+VOtphCvleRD+UrLCzscP+dd96pu+6663Px+/btU1tbm3Jzczvcn5ubqy1bthz2Maqrqw8bX/3J+9Gh/x4tJhGCuzgDAADd3q5duzoc1kxLS0viaL4YpmVuSUmJzj//fPXq1Ut9+vTRlClTtHXr1g4xTU1Nmjt3rk455RRlZmZq6tSpqqmp8XXQAAAgOFpbE3OTpHA43OF2pMVZTk6OevTo8bk1R01NjfLyDl+ezcvLO2r8of9acvrBtDhbs2aN5s6dq7Vr1+qVV17RwYMHdfHFF6uhoaE95qabbtKLL76o559/XmvWrFFVVZWuuOIK3wcOAABwSGpqqkaOHKnVq1e33xeLxbR69WoVFRUd9neKioo6xEvSK6+80h4/YMAA5eXldYiJRqNat27dEXP6wXRYc+XKlR3+/cQTT6hPnz7asGGDxo0bp7q6Oj366KNaunSpLrron9/ZePzxx3X22Wdr7dq1Gjt2rH8jBwAAgfDpSpefOa3mz5+vGTNmaNSoURo9erQWLlyohoYGXXvttZKka665RqeeeqpKSkokSd///vf11a9+Vb/4xS/0b//2b3r22We1fv16Pfzww5KklJQU3XjjjfrZz36mM844QwMGDNDtt9+u/Px8TZkyxa+pfk5c3zmrq6uTJGVn//PLzRs2bNDBgwc7nNUwePBg9evXT2VlZYddnDU3N3c48yIajcYzJAAA0E1NmzZNe/fu1R133KHq6mqNGDFCK1eubP9C/86dOxUK/eug4QUXXKClS5fqtttu049//GOdccYZWr58uYYOHdoe88Mf/lANDQ2aPXu2amtrdeGFF2rlypVKt15LxeC4F2exWEw33nijvvzlL7dPorq6WqmpqcrKyuoQe7SzGkpKSvSTn/zkeIcBAACSLCiVM0maN2+e5s2bd9iflR7mLPwrr7xSV1555RHzpaSk6O6779bdd999fAM6Dsd93uvcuXO1efNmPfvss3ENYMGCBaqrq2u/7dq1K658AADgi5XIEwK6o+OqnM2bN08rVqzQa6+9poJPXccoLy9PLS0tqq2t7VA9O9pZDWlpad3itFgAAAAXpsqZ53maN2+eli1bpldffVUDBgzo8PORI0fqhBNO6HBWw9atW7Vz586EntUAAACSp63N/6pZW1uyZ5U8psrZ3LlztXTpUv3xj39Ur1692r9HFolEdOKJJyoSiei6667T/PnzlZ2drXA4rBtuuEFFRUWcqQkAAODAtDhbvHixJGn8+PEd7n/88cc1c+ZMSdIvf/lLhUIhTZ06Vc3NzZo4caJ++9vf+jJYAAAQPEE6IaArSPE8z0v2ID7tUIf7jz9270Ifqq5yzv9BU75pPAP72/rJWbxX4X5UedAgW+6Q3Mfd0mo7LyR1/evOsUt32PpZTp9uCje9eJcsseX+3hz3HqLqafz6pmHgH1SmmlIP3PeGe7CxJ6j53fKVV5xDG7/2b6bUGemJe202Nrm/JjLUaModS7f1m7UI1RsuRWRtOplAlj6ckpTfc49zbEtWH1Pu+nr3WEt/ZEnSvn3usTk5zqHRaFSRk09WXZ3756ZfDn1mb9lSp169/H3sAweiGjw4kpR5JRu9NQEAQFyonPnL3xbyAAAAiAuVMwAAEBcqZ/6icgYAABAgVM4AAEBcDl3nzO+c3RWLMwAAEBcOa/qLw5oAAAABQuUMAADEhcqZv6icAQAABAiVMwAAEBcqZ/4K7OIs9PftCvXq5RTbUjDQOe/ALZtM44hpmHPs3r2m1KaWTH/5iy33uAvdY3/3O1vub1/qPvAtK225Q62GlkmSUktLnWO/N2e8bTCXXuocGn3hZVPqigr3lkznDbU9J38oH+0ce8VQW+shVVebwi0tmTIeX2TKHbt+rnNsaOGDptyaM98QO8eW+4mnnENXrLCl/sal7i2ZLiq2HThZtco2ls2b3WOHDbW1QXqvwr0lU8TYoDC3ead7cGu6LXlTky0e3VZgF2cAAKBzoHLmL75zBgAAECBUzgAAQFy4CK2/WJwBAIC4cFjTXxzWBAAACBAqZwAAIC5UzvxF5QwAACBAqJwBAIC4UDnzF5UzAACAAKFyBgAA4sKlNPxF5QwAACBAUjzPM3YeS6xoNKpIJKK6vXsVDoedfuf19e59Ci+o/L1tQIb+irH0DFPqkGz95EyWL3ePNcxRkl5b6/58jzvwv6bcLV9378UoSan1+92Ds7JMuaP17n+7hJfcZ8qt73zHOXT+z7JNqR8sK3IPfuklU25luvdutIoZ/1YMnT7APfe27bbc9VHn2Jp/uL1PHdK7t2EcCXyPsD7fxraqyslxj03d8Z4p98b6M51jR4wwpTZJ5PZ5fa379mloiOriiyOqq6tz/tz0y6HP7GeeqVNGhr+P3dgY1VVXJWdeycZhTQAAEBdOCPAXhzUBAAAChMoZAACIC5Uzf1E5AwAACBAqZwAAIC5UzvxF5QwAACBAqJwBAIC4cBFaf1E5AwAACBAqZwAAIC5858xfLM4AAEBcWJz5K7iLs1WrpAy3dkgjir/hnjdnhGkYlpZMmzebUmvoUPejyitW2HJfOuUK59hBg2y5P1jp3mrl3jJbO6YfTLaNRQsXOofG7rrblDp8Vl/33B/uNuU2DFsP3mZoUSXp9S1lzrFjjd2YQqWv2n5h/Hj33AsfNKW2tGRq7ZFiyr3l/3Pvajfsifmm3HrgAefQ2XNs3zx5eIl7O6EePdxbVEmS15xuG8sT7m3eZk/PM+VWhXuoucXS2rXusWPH2nIvXuwcesH11zvHRqMJbAWIpAju4gwAAHQKVM78xQkBAAAAAULlDAAAxIVLafiLyhkAAECAUDkDAABx4Ttn/qJyBgAAECBUzgAAQFyonPmLxRkAAIgLizN/cVgTAAAgQKicAQCAuFA58xeVMwAAgABJ8TzPvYncFyAajSoSiWj37jqFw2Hf82c02foUKivLPTZIy/ymJvfY6mpT6p3pZzrH9utZZcqtPFuPvf217n9fZPe09RKs+Yf7/pd7gnG/MmhMzzbFZ/RscQ/uaSueNzYF5++5jFb37blph+29ZFi6e//Y9+T+epBsvWxD+/aYcsdy+rjnrra9Nt+pzTfFDymwvd4squrdt6fxLcX01ml8+Zg+Ig4csMRGdcYZEdXVJeZz82gOfWb/8Id1Skvz97Gbm6O6777kzCvZgvNOCwAAANvirKSkROeff7569eqlPn36aMqUKdq6dWuHmPHjxyslJaXDbc6cOb4OGgAABMeh75z5feuuTIuzNWvWaO7cuVq7dq1eeeUVHTx4UBdffLEaGho6xM2aNUu7d+9uv913332+DhoAAKCrMh0xX7lyZYd/P/HEE+rTp482bNigcePGtd+fkZGhPOuBfgAA0Cm1tko9evifs7uK6ztndXV1kqTs7I5fWH766aeVk5OjoUOHasGCBWpsbDxijubmZkWj0Q43AADQeXBY01/HfZ2zWCymG2+8UV/+8pc1dOjQ9vu/+c1v6rTTTlN+fr42bdqkH/3oR9q6dav+8Ic/HDZPSUmJfvKTnxzvMAAAALqU416czZ07V5s3b9Zf/vKXDvfPnj27/f/POecc9e3bVxMmTNC2bdt0+umnfy7PggULNH/+/PZ/R6NRFRYWHu+wAADAF6ytzf9KV1ubv/k6k+NanM2bN08rVqzQa6+9poKCgqPGjhkzRpJUUVFx2MVZWlqa0tLSjmcYAAAAXY5pceZ5nm644QYtW7ZMpaWlGjBgwDF/p7y8XJLUt2/f4xogAAAIttZWKeTzlVP5zpmjuXPnaunSpfrjH/+oXr16qfqTK8tHIhGdeOKJ2rZtm5YuXapLLrlEp5xyijZt2qSbbrpJ48aN07BhwxIyAQAAgK7EtDhbvHixpH9eaPbTHn/8cc2cOVOpqalatWqVFi5cqIaGBhUWFmrq1Km67bbbzAPLqKpQRjTTLTgnxz3x8uWmccRmfts5dkdlqim35WojjzxiSq1vfct9LP+9zNYW40f/j3vbl5RTl5tyt7V91xSf/cLDzrE1l88+dtCnNOSlOMfu/LutC1plpXvsBaMM7ZgkjSt23/alpabUytiy0fYLgwe7xxovVl1z/1POscOemH/soE95b86DzrHLz3LfTyTph83NzrH/9Yh7OyZJ+vGtMefYlFPXmHJ7a041xW9sGnfsoE9Yr7xkeT+84zb350SSMio2uwdbenFJSjV8FmbceKNz7InNhl5PCULlzF/mw5pHU1hYqDVrbC94AAAA/Mtxn60JAAAgUTnzG4szAAAQFxZn/vL5qQQAAEA8WJwBAIC4HLoIrZ+3RF+Edv/+/br66qsVDoeVlZWl6667TvX19Uf9naamJs2dO1ennHKKMjMzNXXqVNXU1HSISUlJ+dzt2WefNY2NxRkAAOh2rr76ar399tt65ZVX2i+s/+kuR4dz00036cUXX9Tzzz+vNWvWqKqqSldcccXn4h5//HHt3r27/TZlyhTT2PjOGQAAiEtrq5Riu6qMU07pn20dP82PzkLvvvuuVq5cqTfffFOjRo2SJD300EO65JJL9MADDyg/P/9zv1NXV6dHH31US5cu1UUXXSTpn4uws88+W2vXrtXYsWPbY7OyspRnvUbMp1A5AwAAgVVYWKhIJNJ+KykpiTtnWVmZsrKy2hdmklRcXKxQKKR169Yd9nc2bNiggwcPqri4uP2+wYMHq1+/fiorK+sQO3fuXOXk5Gj06NF67LHHjnkpss+icgYAAOKSyMrZrl27FA7/62LpfvTjrq6uVp8+HS/y3LNnT2VnZ7d3Pzrc76SmpiorK6vD/bm5uR1+5+6779ZFF12kjIwMvfzyy/rud7+r+vp6fe9733MeH4szAAAQWOFwuMPi7GhuvfVW3XvvvUeNeffdd/0Y1hHdfvvt7f9/7rnnqqGhQffffz+LMwAA8MVJZOXM4uabb9bMmTOPGjNw4EDl5eVpz549n3m8Vu3fv/+I3xXLy8tTS0uLamtrO1TPampqjvr9sjFjxuinP/2pmpubnat+KZ71QGiCRaNRRSIRffxxnfNKOST33mkxvmYXt0Q+35bcQZLI/cr6nLCPf14in8NQq633qXp20r+JrZ+Uhnl2l/eJRIlGo4qcfLLq6tw/N3197EhExcV1OuEEfx/74MGoVq2KJGRe7777roYMGaL169dr5MiRkqSXX35ZkyZNUmVl5RFPCOjdu7eeeeYZTZ06VZK0detWDR48WGVlZR1OCPi0n//85/rFL36h/fv3O4+vk75LAAAAHJ+zzz5bkyZN0qxZs7RkyRIdPHhQ8+bN0/Tp09sXZh9++KEmTJigp556SqNHj1YkEtF1112n+fPnKzs7W+FwWDfccIOKioraF2YvvviiampqNHbsWKWnp+uVV17Rf/3Xf+mWW24xjY/FGQAAiEtbm/+HNRN9Edqnn35a8+bN04QJExQKhTR16lT9+te/bv/5wYMHtXXrVjU2Nrbf98tf/rI9trm5WRMnTtRvf/vb9p+fcMIJWrRokW666SZ5nqdBgwbpwQcf1KxZs0xj47AmzDis+Xkc1gw2Dmv6gMOagRWEw5pf+1qdevb097FbW6P6858Tc1gz6DrpuwQAAAiKRDQpp/E5AAAAAoHKGQAAiAuVM39ROQMAAAgQKmcAACAuVM78ReUMAAAgQKicAQCAuCTimmSJvs5ZkAV2cfbOO1JmplvssMHutc8dlammcfTv7x4b2vGBKXes/0D33OvfsOUeNdo99113mHLvv/Fu59jsFx425dZ3vmMKf/gR9+Lv7O/Yro00e4577oeXGK+7VFrqHNo49iJT6owtG51jYyPOM+V+801TuM4/3z12xQpb7m9c6v6cW7alJD38sz3HDvrEfz3Sx5T7x7ca9pWSElNuLVjgHnvppbbc1g20dq1zaNOIC0ypMyo2uwcPHWrKvWmz+74ybKjtdf/yKvfcxcXueYNwbcPWVsnvq6Z258VZ8rcoAAAA2gW2cgYAADoHKmf+onIGAAAQIFTOAABAXKic+YvKGQAAQIBQOQMAAHGhcuYvKmcAAAABQuUMAADEpa3N/8pZzHj5yK6ExRkAAIhLa6sU8vlYXHdenHFYEwAAIEConAEAgLhQOfNXiuf5fZQ4PtFoVJFIRB9/XKdwOOz0O6HWFuf8ja223prp6aZwk6Ym99iM9MTtpda+bKH6qHNszT/ctuEhub0TN8+qats88/LcY0NK3Lgbm2zjzlCjc2wsPcM6HBPLvuLcTPcLYHlNJHLbm7W69xmO9bS9FyZyni2ttn08tdV9H0/om3h9vSk8lun+fmh5vqPRqCInn6y6OvfPTb8c+szOza1TKOTvY8diUdXURJIyr2SjcgYAAOJC5cxffOcMAAAgQKicAQCAuLS1+V/pCtaXrr5YVM4AAAAChMoZAACIS2urlJLib87uXDljcQYAAOLC4sxfHNYEAAAIECpnAAAgLlTO/EXlDAAAIEConAEAgLhQOfNXYBdnK1dKGY6dZb4xyT3vI4/YxjFvnntsaP0bptwZo0a5B991lym3JT50al9T6pry3c6xDXnGV2tbmyl89hz34u+SJbah9OjxgnNsW9t/mHKHFj7oHjxnvim35sxxj33iKVPqFStsQ/nGpe4tmS4qthXyV61yj+3Rw9BGSpL3oXtbnpRT19hyt01zD770UlPu2Ir/c459qofttXlNm+2TMrR4kXNs67VzTblTb7vNPfiBB0y5X17lvh8WF9taCn3rW+6xS39nSo0uJrCLMwAA0FnE5Hl+91vqvv2b+M4ZAABAgJgWZ4sXL9awYcMUDocVDodVVFSkP/3pT+0/b2pq0ty5c3XKKacoMzNTU6dOVU1Nje+DBgAAQdKWoFv3ZFqcFRQU6J577tGGDRu0fv16XXTRRbr88sv19ttvS5Juuukmvfjii3r++ee1Zs0aVVVV6YorrkjIwAEAQFCwOPOT6Ttnl112WYd///znP9fixYu1du1aFRQU6NFHH9XSpUt10UUXSZIef/xxnX322Vq7dq3Gjh3r36gBAAC6qOM+IaCtrU3PP/+8GhoaVFRUpA0bNujgwYMqLi5ujxk8eLD69eunsrKyIy7Ompub1dzc3P7vaNR2VhUAAEi2RFS6um/lzHxCwN/+9jdlZmYqLS1Nc+bM0bJlyzRkyBBVV1crNTVVWVlZHeJzc3NVXV19xHwlJSWKRCLtt8LCQvMkAAAAugrz4uyss85SeXm51q1bp+uvv14zZszQO++8c9wDWLBggerq6tpvu3btOu5cAAAgGWIJunVP5sOaqampGjRokCRp5MiRevPNN/WrX/1K06ZNU0tLi2praztUz2pqapSXl3fEfGlpaUpLS7OPHAAAoAuK+zpnsVhMzc3NGjlypE444QStXr26/Wdbt27Vzp07VVRUFO/DAACAwOJsTT+ZKmcLFizQ5MmT1a9fPx04cEBLly5VaWmpXnrpJUUiEV133XWaP3++srOzFQ6HdcMNN6ioqIgzNQEAAByleJ57a9HrrrtOq1ev1u7duxWJRDRs2DD96Ec/0te//nVJ/7wI7c0336xnnnlGzc3Nmjhxon77298e9bDmZ0WjUUUiEX38cZ3CYbe+ZaF69zM897faeqF95vwGX4USeDy9Zq97UbR3b1vuUO1+59id9dmm3AUFxrEk8DmMGQrLlZW23JZ5WvZvSar5h/s+bt72AXm+Jeko5xl9Tn5Oiyn3OxWpzrFD9r1myi3LH6s9E9dhz/p8h1a9bHuAT525fyyW9ytJym3e6R5sfFOxPC+JfD1Y3lSiBw4oMnSo6urcPzf9cugzW9ouqZfP2Q9IGpCUeSWb6ZX/6KOPHvXn6enpWrRokRYtcm94CwAAOjsupeEnemsCAAAESOJq5gAAoJugcuYnKmcAAAABQuUMAADEKREXje2+F6GlcgYAABAgVM4AAECc+M6Zn6icAQAABAiVMwAAECcqZ35icQYAAOLE4sxPHNYEAAAIkMBWzj78UIo6thTsl5funrj++MbjwtJzUpJiWe59J0OVhl5yknoX9HPP/cLvTbkbL/1P59h++94z5Y7pTFP8xnL3vy+GDjWl1rJl7rFXXmnLHdryjnNsy6Ahpty5TXucY2PqY8pdVW37ey4/z/1U+M2bTak1eLB77MNPuPfKlKTZ0937mW5sGmfKfV5Pw+UB1q415bb07QwtNrbZu/56W3xFhXNorwLb614phvd8o7173WNzjb1p36twf/2cOcjQE9T1wzKhYvK/0sWlNAAAABAAga2cAQCAzoKL0PqJyhkAAECAUDkDAABx4mxNP1E5AwAACBAqZwAAIE5UzvzE4gwAAMSJxZmfOKwJAAAQIFTOAABAnKic+YnKGQAAQICkeJ7nJXsQnxaNRhWJRFS3d6/C4bDT79R87N6aJbfZ1gZJBe4tNGLGta6l3ZOl1ZM1d2O6LXdlpXvsmfteN+W2tJ+RJLW2usdu2WLLben3tG+fKXVLlq1tkkVqk6GVS2ZmwsYh2do9WVo9STK1B1Jeni23QVW92/vUIZahNDXZxtLTcCzE8tKRpAMHbPG9ernHZtS7txyTpJ1N7q8fw1u4JClkuPDp/lrbe352Zot77nr3z7VoNKoBAyKqq6tz/tz0y6HPbGm1pJN8zt4gaUJS5pVsVM4AAEC3s3//fl199dUKh8PKysrSddddp/r6ozfgfvjhhzV+/HiFw2GlpKSotrbWl7yfxeIMAADEqS1Bt8S5+uqr9fbbb+uVV17RihUr9Nprr2n27NlH/Z3GxkZNmjRJP/7xj33N+1mcEAAAAAIrGu34VY20tDSlpaXFlfPdd9/VypUr9eabb2rUqFGSpIceekiXXHKJHnjgAeXn5x/292688UZJUmlpqa95P4vKGQAAiFPiKmeFhYWKRCLtt5KSkrhHW1ZWpqysrPYFlCQVFxcrFApp3bp1Sc9L5QwAAMQpcZfS2LVrV4cTAuKtmklSdXW1+vTpeGJJz549lZ2drerq6qTnpXIGAAACKxwOd7gdbXF26623KiUl5ai3LdYz95OAyhkAAIhTMC5Ce/PNN2vmzJlHjRk4cKDy8vK0Z0/HS7i0trZq//79yovj0jt+5WVxBgAAuoTevXurd+/ex4wrKipSbW2tNmzYoJEjR0qSXn31VcViMY0ZM+a4H9+vvBzWBAAAcYol6JYYZ599tiZNmqRZs2bpjTfe0F//+lfNmzdP06dPbz+j8sMPP9TgwYP1xhtvtP9edXW1ysvLVfHJRbD/9re/qby8XPv373fO64LFGQAA6HaefvppDR48WBMmTNAll1yiCy+8UA8//HD7zw8ePKitW7eqsbGx/b4lS5bo3HPP1axZsyRJ48aN07nnnqv/+Z//cc7rgsOaAAAgTsH4zplFdna2li5desSf9+/fX5/tcHnXXXfprrvuiiuvi8AuzvbXp6o15NZbzNIv87FV/Uzj+PZM97JqqN7Q01DSznr3npb9KjeZclflDHOOzd/3nin3mZY+ksaWFY1NtmLujh3u/edq692fE0lafqt77I032npl5tcaegmmp5ty6zDtRI7I0oxRUrQ1wxSf39N9nu9V2J7D+voz3YMNbTglW//LRx6x5b7jNvf3lIyKzbbkgwY5h6bedpspdcYnF990luK+31p6ZUpSv7u+7Z77rsdMuS0HlPpV2noHP7j2AufY+d9yf+30bDM2PkXgBXZxBgAAOouY/K90Je47Z0HHd84AAAAChMoZAACIU+f7zlmQsTgDAABxSsSlLzisCQAAgACgcgYAAOLEYU0/UTkDAAAIECpnAAAgTlTO/ETlDAAAIEConAEAgDhROfNTivfZxlFJFo1GFYlE9PHHdQqHw06/EwrI6baxblKItDzf1nZMxm5CSlWL7RcSxThwy75i3r+bmpxDq2pt7ZgyM21DsXSe+vhjW+7evd1jrc9hQrcPPsf63llZ6R7br/oN22AsvbusLwjLwIcOdQ6NRqOKnHyy6urcPzf9cugzW3pEku395NgaJX0nKfNKNipnAAAgTrRv8hOLMwAAECcuQusnUy158eLFGjZsmMLhsMLhsIqKivSnP/2p/efjx49XSkpKh9ucOXN8HzQAAEBXZaqcFRQU6J577tEZZ5whz/P05JNP6vLLL9dbb72lL33pS5KkWbNm6e67727/nYwMv49BAwCAYOGEAD+ZFmeXXXZZh3///Oc/1+LFi7V27dr2xVlGRobyDF+obG5uVnNzc/u/o9GoZUgAAABdynGfXtjW1qZnn31WDQ0NKioqar//6aefVk5OjoYOHaoFCxaosbHxqHlKSkoUiUTab4WFhcc7JAAAkBRtCbp1T+YTAv72t7+pqKhITU1NyszM1LJlyzRkyBBJ0je/+U2ddtppys/P16ZNm/SjH/1IW7du1R/+8Icj5luwYIHmz5/f/u9oNMoCDQAAdFvmxdlZZ52l8vJy1dXV6YUXXtCMGTO0Zs0aDRkyRLNnz26PO+ecc9S3b19NmDBB27Zt0+mnn37YfGlpaUpLSzv+GQAAgCTjO2d+Mh/WTE1N1aBBgzRy5EiVlJRo+PDh+tWvfnXY2DFjxkiSKioq4hslAABANxH3dc5isViHL/R/Wnl5uSSpb9++8T4MAAAILCpnfjItzhYsWKDJkyerX79+OnDggJYuXarS0lK99NJL2rZtm5YuXapLLrlEp5xyijZt2qSbbrpJ48aN07BhwxI1fgAAkHRchNZPpsXZnj17dM0112j37t2KRCIaNmyYXnrpJX3961/Xrl27tGrVKi1cuFANDQ0qLCzU1KlTddtttx3XwEK1+xWKtTrFxrKy3fPW2y7VEct07+dVX29KbWrLVltry52d5b5Tbyy3Hd0+b6jbdpGkHTtSTbmHDDL2ytyxwz22oMCW+5PKr4vGEReYUlu2Z76h1Z8kRVvdry1obQ0YzrS9We6vdd+3cpt3mnLH1M89eO1aU+4mw/bMqNhsym3pmbhps+21OWyo+/Z5eZUtd3GxKVx797rH5va2fggbxm7plSlJ1dXusVlZttzWeHRbpsXZo48+esSfFRYWas2aNXEPCAAAdDb01vTTcV/nDAAAAP6j8TkAAIgTJwT4icoZAABAgFA5AwAAcaJy5icqZwAAAAFC5QwAAMSJypmfWJwBAIA4sTjzE4c1AQAAAoTKGQAAiBPtm/wU2MVZLCtbsbBb66RQ7X7nvDvr3Vs9SVKBob2NtRVORYV77JmDEreTGrrJ/NOWLc6htfUJ7qtqaMm07m/ubY0k6eyh7i18wum27ZNh6ChTVW0rcOf33OMc25LVx5Tb0o5JsrURU2u6KXfI8sY9dqwpd0/3DmXSoEGm3BaWdkySTD3kiovdW9NJxudbUm5v91jrftWv8nX34MGDTblNLZaef96UeuPk/9c59ryC7rswQYAXZwAAoLOgfZOf+M4ZAABAgFA5AwAAceJsTT9ROQMAAAgQKmcAACBOVM78xOIMAADEiUtp+InDmgAAAAFC5QwAAMSJw5p+onIGAAAQIFTOAABAnKic+YnKGQAAQIAEtnIW+nCXQtFeTrGxgn7OeftVbjKNI1bg3huyttaU2tQvc2O5bR09YoR77LJlptSadqV7M87lt9pyXzDKFq/ycudQS69MSQrf9j334HvuMeXeVOHe59PaX/G9Cvd+mRHPlFq5zTttv2Dpl9nUZMttsXixKbz12rnOsam33WYbywMPOIe+vMr2urf0y/zWt0yptfR3tvj3KtzHfmb/FlPuB9e6v5bnZ9re8y29NS29MiXpvCWznWNjSx52jw1EnYXKmZ+CsEUBAADwicBWzgAAQGdB43M/sTgDAABx4iK0fuKwJgAAQIBQOQMAAHHihAA/UTkDAAAIECpnAAAgTlTO/ETlDAAAIEConAEAgDhROfMTlTMAAIAASfE8z9jAJbGi0agikYg+/rhO4bBbK5JQ7X7n/FVN2abx5OW5x4YSeE0Wa3uOvXvdY3v3to0ltG+Pc2xVq3srIUnKz7M9h41N7s9LRrpx+xjaCW3c4t6OSZKGunfAUuqO90y5N9af6RxrafN1PBL5mnh9rfu2v2CsbRw1e91zm1taFRQ4h1pf94l8vlVZaYs3zHN/rW2e2a3u70HKyTHlTiTL9gxVVznHRg8cUGTwYNXVuX9u+uXQZ7Y0TVKqz9lbJD2XlHklG4c1AQBAnOgQ4CcOawIAAAQIlTMAABAn2jf5icoZAABAgFA5AwAAcWqT//UeLqUBAACAAKByBgAA4kTlzE9UzgAAAAKEyhkAAIgTlTM/sTgDAABx4lIafgrc4uxQN6loNOr8OyFD7IFm25QzDF15gtS+6cAB99i0NNtYQobkB1rTTbmjGYlr39Takrj2TfX1rabUhl1WqfX1ptz19e7JLeM4Hol8TTQ0uG/7aNQ2jgMH3HOf2Gx4sf1zMM6hgWrfZHlTkUzzjEZt8+zZZhhLqt8thY6fqX2T4fmOfvIekdxujLb3wOTl7BwCtzg78MkOedpphUkeCQAAnceBAwc+6XP5xUlNTVVeXp6qq1clJH9eXp5SA7TA/qIErvF5LBZTVVWVevXqpZSUlPb7o9GoCgsLtWvXri7dAJV5dh3dYY4S8+xqusM8u9ocPc/TgQMHlJ+fr1Doiz/Pr6mpSS0tLQnJnZqaqvR02xGYriBwlbNQKKSCgoIj/jwcDneJF9OxMM+uozvMUWKeXU13mGdXmuMXXTH7tPT09G65gEokLqUBAAAQICzOAAAAAqTTLM7S0tJ05513Ks16amEnwzy7ju4wR4l5djXdYZ7dYY7o3AJ3QgAAAEB31mkqZwAAAN0BizMAAIAAYXEGAAAQICzOAAAAAoTFGQAAQIB0msXZokWL1L9/f6Wnp2vMmDF64403kj0kX911111KSUnpcBs8eHCyhxWX1157TZdddpny8/OVkpKi5cuXd/i553m644471LdvX5144okqLi7W+++/n5zBxuFY85w5c+bntu2kSZOSM9jjVFJSovPPP1+9evVSnz59NGXKFG3durVDTFNTk+bOnatTTjlFmZmZmjp1qmpqapI04uPjMs/x48d/bnvOmTMnSSM+PosXL9awYcPar5BfVFSkP/3pT+0/7wrbUjr2PLvCtkTX1CkWZ88995zmz5+vO++8Uxs3btTw4cM1ceJE7dmzJ9lD89WXvvQl7d69u/32l7/8JdlDiktDQ4OGDx+uRYsWHfbn9913n379619ryZIlWrdunU466SRNnDhRTU1NX/BI43OseUrSpEmTOmzbZ5555gscYfzWrFmjuXPnau3atXrllVd08OBBXXzxxWpoaGiPuemmm/Tiiy/q+eef15o1a1RVVaUrrrgiiaO2c5mnJM2aNavD9rzvvvuSNOLjU1BQoHvuuUcbNmzQ+vXrddFFF+nyyy/X22+/LalrbEvp2POUOv+2RBfldQKjR4/25s6d2/7vtrY2Lz8/3yspKUniqPx15513esOHD0/2MBJGkrds2bL2f8diMS8vL8+7//772++rra310tLSvGeeeSYJI/THZ+fpeZ43Y8YM7/LLL0/KeBJlz549niRvzZo1nuf9c9udcMIJ3vPPP98e8+6773qSvLKysmQNM26fnafned5Xv/pV7/vf/37yBpUgJ598svfII4902W15yKF5el7X3Zbo/AJfOWtpadGGDRtUXFzcfl8oFFJxcbHKysqSODL/vf/++8rPz9fAgQN19dVXa+fOnckeUsJs375d1dXVHbZrJBLRmDFjutx2laTS0lL16dNHZ511lq6//np99NFHyR5SXOrq6iRJ2dnZkqQNGzbo4MGDHbbn4MGD1a9fv069PT87z0Oefvpp5eTkaOjQoVqwYIEaGxuTMTxftLW16dlnn1VDQ4OKioq67Lb87DwP6UrbEl1Hz2QP4Fj27duntrY25ebmdrg/NzdXW7ZsSdKo/DdmzBg98cQTOuuss7R792795Cc/0Ve+8hVt3rxZvXr1SvbwfFddXS1Jh92uh37WVUyaNElXXHGFBgwYoG3btunHP/6xJk+erLKyMvXo0SPZwzOLxWK68cYb9eUvf1lDhw6V9M/tmZqaqqysrA6xnXl7Hm6ekvTNb35Tp512mvLz87Vp0yb96Ec/0tatW/WHP/whiaO1+9vf/qaioiI1NTUpMzNTy5Yt05AhQ1ReXt6ltuWR5il1nW2Jrifwi7PuYvLkye3/P2zYMI0ZM0annXaafv/73+u6665L4sgQr+nTp7f//znnnKNhw4bp9NNPV2lpqSZMmJDEkR2fuXPnavPmzZ3+O5HHcqR5zp49u/3/zznnHPXt21cTJkzQtm3bdPrpp3/RwzxuZ511lsrLy1VXV6cXXnhBM2bM0Jo1a5I9LN8daZ5DhgzpMtsSXU/gD2vm5OSoR48enztTqKamRnl5eUkaVeJlZWXpzDPPVEVFRbKHkhCHtl13266SNHDgQOXk5HTKbTtv3jytWLFCf/7zn1VQUNB+f15enlpaWlRbW9shvrNuzyPN83DGjBkjSZ1ue6ampmrQoEEaOXKkSkpKNHz4cP3qV7/qctvySPM8nM66LdH1BH5xlpqaqpEjR2r16tXt98ViMa1evbrD9wa6mvr6em3btk19+/ZN9lASYsCAAcrLy+uwXaPRqNatW9elt6skVVZW6qOPPupU29bzPM2bN0/Lli3Tq6++qgEDBnT4+ciRI3XCCSd02J5bt27Vzp07O9X2PNY8D6e8vFySOtX2PJxYLKbm5uYusy2P5NA8D6erbEt0Ack+I8HFs88+66WlpXlPPPGE984773izZ8/2srKyvOrq6mQPzTc333yzV1pa6m3fvt3761//6hUXF3s5OTnenj17kj2043bgwAHvrbfe8t566y1Pkvfggw96b731lvf3v//d8zzPu+eee7ysrCzvj3/8o7dp0ybv8ssv9wYMGOD94x//SPLIbY42zwMHDni33HKLV1ZW5m3fvt1btWqVd95553lnnHGG19TUlOyhO7v++uu9SCTilZaWert3726/NTY2tsfMmTPH69evn/fqq69669ev94qKiryioqIkjtruWPOsqKjw7r77bm/9+vXe9u3bvT/+8Y/ewIEDvXHjxiV55Da33nqrt2bNGm/79u3epk2bvFtvvdVLSUnxXn75Zc/zusa29Lyjz7OrbEt0TZ1iceZ5nvfQQw95/fr181JTU73Ro0d7a9euTfaQfDVt2jSvb9++Xmpqqnfqqad606ZN8yoqKpI9rLj8+c9/9iR97jZjxgzP8/55OY3bb7/dy83N9dLS0rwJEyZ4W7duTe6gj8PR5tnY2OhdfPHFXu/evb0TTjjBO+2007xZs2Z1uj8sDjc/Sd7jjz/eHvOPf/zD++53v+udfPLJXkZGhvfv//7v3u7du5M36ONwrHnu3LnTGzdunJedne2lpaV5gwYN8n7wgx94dXV1yR240be//W3vtNNO81JTU73evXt7EyZMaF+YeV7X2Jaed/R5dpVtia4pxfM874ur0wEAAOBoAv+dMwAAgO6ExRkAAECAsDgDAAAIEBZnAAAAAcLiDAAAIEBYnAEAAAQIizMAAIAAYXEGAAAQICzOAAAAAoTFGQAAQICwOAMAAAiQ/x9O/Sk1fK1Z9wAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "imshow_zero_center(h_mat)" ] }, { "cell_type": "markdown", "metadata": { "id": "13fBswmtQes4" }, "source": [ "牛顿方法更新步骤如下所示。" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.226346Z", "iopub.status.busy": "2022-12-14T22:53:34.225861Z", "iopub.status.idle": "2022-12-14T22:53:34.232357Z", "shell.execute_reply": "2022-12-14T22:53:34.231774Z" }, "id": "3DdnbynBdSor" }, "outputs": [], "source": [ "eps = 1e-3\n", "eye_eps = tf.eye(h_mat.shape[0])*eps" ] }, { "cell_type": "markdown", "metadata": { "id": "-zPdtyoWeUeV" }, "source": [ "注：[实际上不反转矩阵](https://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/)。" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.235864Z", "iopub.status.busy": "2022-12-14T22:53:34.235207Z", "iopub.status.idle": "2022-12-14T22:53:34.336240Z", "shell.execute_reply": "2022-12-14T22:53:34.335482Z" }, "id": "k1LYftgmswOO" }, "outputs": [], "source": [ "# X(k+1) = X(k) - (∇²f(X(k)))^-1 @ ∇f(X(k))\n", "# h_mat = ∇²f(X(k))\n", "# g_vec = ∇f(X(k))\n", "update = tf.linalg.solve(h_mat + eye_eps, g_vec)\n", "\n", "# Reshape the update and apply it to the variable.\n", "_ = layer1.kernel.assign_sub(tf.reshape(update, layer1.kernel.shape))" ] }, { "cell_type": "markdown", "metadata": { "id": "pF6qjlHKWxF4" }, "source": [ "虽然这对于单个 `tf.Variable` 来说相对简单，但将其应用于非平凡模型则需要仔细的级联和切片，以产生跨多个变量的完整黑塞矩阵。" ] }, { "cell_type": "markdown", "metadata": { "id": "PQWM0uN-GO5t" }, "source": [ "### 批量雅可比矩阵" ] }, { "cell_type": "markdown", "metadata": { "id": "hKtB3rY6EySJ" }, "source": [ "在某些情况下，您需要取各个目标堆栈相对于源堆栈的雅可比矩阵，其中每个目标-源对的雅可比矩阵都是独立的。\n", "\n", "例如，此处的输入 `x` 形状为 `(batch, ins)` ，输出 `y` 形状为 `(batch, outs)`：\n" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.340794Z", "iopub.status.busy": "2022-12-14T22:53:34.340282Z", "iopub.status.idle": "2022-12-14T22:53:34.356398Z", "shell.execute_reply": "2022-12-14T22:53:34.355804Z" }, "id": "tQMndhIUHMes" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([7, 6])" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = tf.random.normal([7, 5])\n", "\n", "layer1 = tf.keras.layers.Dense(8, activation=tf.nn.elu)\n", "layer2 = tf.keras.layers.Dense(6, activation=tf.nn.elu)\n", "\n", "with tf.GradientTape(persistent=True, watch_accessed_variables=False) as tape:\n", " tape.watch(x)\n", " y = layer1(x)\n", " y = layer2(y)\n", "\n", "y.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "Ff2spRHEJXBU" }, "source": [ "`y` 相对 `x` 的完整雅可比矩阵的形状为 `(batch, ins, batch, outs)`，即使您只想要 `(batch, ins, outs)`。" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.359583Z", "iopub.status.busy": "2022-12-14T22:53:34.359107Z", "iopub.status.idle": "2022-12-14T22:53:34.472362Z", "shell.execute_reply": "2022-12-14T22:53:34.471792Z" }, "id": "1zSl2A5-HhMH" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([7, 6, 7, 5])" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "j = tape.jacobian(y, x)\n", "j.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "UibJijPLJrpQ" }, "source": [ "如果堆栈中各项的梯度相互独立，那么此张量的每一个 `(batch, batch)` 切片都是对角矩阵：" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.475794Z", "iopub.status.busy": "2022-12-14T22:53:34.475245Z", "iopub.status.idle": "2022-12-14T22:53:34.713223Z", "shell.execute_reply": "2022-12-14T22:53:34.712612Z" }, "id": "ZFl9uj3ueVSH" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAIQCAYAAABDpCBuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA9CUlEQVR4nO3de3gU5d3/8c8mkE2EJAQICYdAOCgYKQcTiKgclBQQxENRDsWHEHmwB4Jg7AH0KWBpDRYPeEAQfw/YVhCrFUQfgdJwagUJhFIBUUGxBDAJEchCKAGy8/uDshqSTGbJ7G6yvF/XNddlZmdnvvfgRb585t57HYZhGAIAAECthQS6AAAAgGBBYwUAAGATGisAAACb0FgBAADYhMYKAADAJjRWAAAANqGxAgAAsAmNFQAAgE1orAAAAGxCY4WrktvtVteuXfXb3/7Ws2/WrFlyOBwqLi4OYGXf2rhxoxwOh95++23bzvnVV1/J4XDo6aeftu2ctZWYmKg777yzxuPWrFmjxo0b69ixY7Zde8CAARowYIDn50v357XXXrPtGgCuLjRWqLdefvllORwOpaamev3eN954Q/n5+crMzPRBZd96+eWXg/6X9JYtWzRr1iydPHnSp9cZMmSIOnXqpOzsbJ9eBwBqg8YK9dbSpUuVmJio3NxcHThwwKv3zp07V6NHj1Z0dLSPqrvoammsnnjiCZ83VpL0ox/9SK+88opOnTrlk/O3a9dO//73v/Vf//VfPjk/gOBHY4V66eDBg9qyZYueffZZxcbGaunSpZbf+49//EP//Oc/NXLkSB9WCF8YMWKEysrK9NZbb/nk/A6HQ+Hh4QoNDfXJ+QEEPxor1EtLly5VTEyMhg0bpvvuu8+rxmrlypUKCwtTv379qny9uLhYI0eOVFRUlJo1a6YpU6bo7NmzFY5ZsmSJbr/9drVo0UJOp1NJSUlasGBBhWMSExO1d+9ebdq0SQ6HQw6Ho8J8npMnT+qRRx5RYmKinE6n2rRpo3HjxlWa4+V2u/Xb3/5Wbdq0UXh4uAYOHOh1QleV5557Tu3atVNERIT69++vPXv2VHj9448/1vjx49WhQweFh4crPj5eDz74oL755hvPMbNmzdLPf/5zSVL79u094/zqq688x7z++uvq3bu3rrnmGsXExKhfv376y1/+Uqmev//97+rdu7fCw8PVoUMH/eEPf6h0TIsWLdStWze9++67NY6voKBAGRkZatOmjZxOp1q2bKm77767Qm2Xq26O1aeffqqRI0cqNjZWERER6ty5sx5//PEKxxw5ckQPPvig4uLi5HQ6dcMNN2jx4sU11gkguDQIdAHAlVi6dKl+8IMfKCwsTGPGjNGCBQu0fft29erVq8b3btmyRV27dlXDhg2rfH3kyJFKTExUdna2PvroI73wwgs6ceJEhV/0CxYs0A033KC77rpLDRo00Hvvvaef/vSncrvdmjRpkiRp3rx5mjx5sho3buz5JRwXFydJOn36tPr27at9+/bpwQcf1I033qji4mKtWrVKhw8fVvPmzT3XmjNnjkJCQvSzn/1MJSUl+t3vfqexY8dq27ZtV3z//vCHP+jUqVOaNGmSzp49q+eff1633367du/e7alx3bp1+vLLL5WRkaH4+Hjt3btXixYt0t69e/XRRx/J4XDoBz/4gT7//HO98cYbeu655zx1x8bGSpKeeOIJzZo1SzfffLN+/etfKywsTNu2bdP69es1aNAgTz0HDhzQfffdpwkTJig9PV2LFy/W+PHjlZycrBtuuKFC7cnJyVq5cmWNYxwxYoT27t2ryZMnKzExUUVFRVq3bp0OHTqkxMREy/fq448/Vt++fdWwYUM99NBDSkxM1BdffKH33nvP8+GHwsJC3XTTTXI4HMrMzFRsbKxWr16tCRMmyOVyaerUqZavB6CeM4B6ZseOHYYkY926dYZhGIbb7TbatGljTJkyxdL727RpY4wYMaLS/pkzZxqSjLvuuqvC/p/+9KeGJOOf//ynZ9+ZM2cqvX/w4MFGhw4dKuy74YYbjP79+1c6dsaMGYYk45133qn0mtvtNgzDMDZs2GBIMq6//nqjrKzM8/rzzz9vSDJ2795tPtAqHDx40JBkREREGIcPH/bs37ZtmyHJeOSRR0zH+MYbbxiSjM2bN3v2zZ0715BkHDx4sMKx+/fvN0JCQox7773XKC8vr3KMhmEY7dq1q3TOoqIiw+l0Go8++milGp588klDklFYWFjtOE+cOGFIMubOnVvtMYZhGP3796/w53Pp/ixZssSzr1+/fkZkZKTxr3/9q9oxTJgwwWjZsqVRXFxc4ZjRo0cb0dHRVd5LAMGJR4God5YuXaq4uDjddtttki7Oixk1apSWL1+u8vLyGt//zTffKCYmptrXLyVOl0yePFmS9MEHH3j2RUREeP67pKRExcXF6t+/v7788kuVlJTUWMOf//xnde/eXffee2+l1xwOR4WfMzIyFBYW5vm5b9++kqQvv/yyxutU55577lHr1q09P/fu3VupqanVjvHs2bMqLi7WTTfdJEnauXNnjddYuXKl3G63ZsyYoZCQin/VXD7GpKQkz7iki4lX586dqxzjpT87s2UxIiIiFBYWpo0bN+rEiRM11lqdY8eOafPmzXrwwQfVtm3bKsdgGIb+/Oc/a/jw4TIMQ8XFxZ5t8ODBKikpsXS/AAQHGivUK+Xl5Vq+fLluu+02HTx4UAcOHNCBAweUmpqqwsJC5eTkWDqPYRjVvnbttddW+Lljx44KCQmpMDfnww8/VFpamho1aqQmTZooNjZWjz32mCRZaqy++OILde3a1VKtl/9Cv9RY1KZhuHyMknTddddVGOPx48c1ZcoUxcXFKSIiQrGxsWrfvr0k62MMCQlRUlJSjcdePkbp4jirGuOlP7vLm7Pvcjqdeuqpp7R69WrFxcWpX79++t3vfqeCgoIaa/muS42d2Z/VsWPHdPLkSS1atEixsbEVtoyMDElSUVGRV9cFUH8xxwr1yvr16/X1119r+fLlWr58eaXXly5dWmHuTlWaNWvmVVNy+S/wL774QgMHDlSXLl307LPPKiEhQWFhYfrggw/03HPPye12Wz63FdV9Qs2sObTDyJEjtWXLFv385z9Xjx491LhxY7ndbg0ZMiSgY7z0Z/fdeWhVmTp1qoYPH66VK1dq7dq1+tWvfqXs7GytX79ePXv2rH3R/3HpXjzwwANKT0+v8phu3brZdj0AdRuNFeqVpUuXqkWLFpo/f36l19555x2tWLFCCxcurPAY63JdunTRwYMHq319//79nmRGujix2u12eyY8v/feeyorK9OqVasqJC0bNmyodK7qUpWOHTtW+hSeP+3fv7/Svs8//9wzxhMnTignJ0dPPPGEZsyYYfo+szG63W598skn6tGjhy11SxeX2mjevLlngryZjh076tFHH9Wjjz6q/fv3q0ePHnrmmWf0+uuvW7pWhw4dJMn0zyo2NlaRkZEqLy9XWlqatUEACFo8CkS98e9//1vvvPOO7rzzTt13332VtszMTJ06dUqrVq0yPU+fPn20Z88elZWVVfn65U3biy++KEm64447JH2brnw3TSkpKdGSJUsqnatRo0ZVLpw5YsQI/fOf/9SKFSsqvebrJEq6OP/pyJEjnp9zc3O1bds20zFKFz/peLlGjRpJUqVx3nPPPQoJCdGvf/3rSglXbcaYl5enPn36mB5z5syZSktkdOzYUZGRkdX+uVclNjZW/fr10+LFi3Xo0KEKr10aQ2hoqEaMGKE///nPVTZgdn4FD4C6j8QK9caqVat06tQp3XXXXVW+ftNNN3kWCx01alS157n77rs1e/Zsbdq0qcrHhgcPHtRdd92lIUOGaOvWrXr99df1wx/+UN27d5ckDRo0SGFhYRo+fLh+9KMf6fTp03r11VfVokULff311xXOlZycrAULFug3v/mNOnXqpBYtWuj222/Xz3/+c7399tu6//779eCDDyo5OVnHjx/XqlWrtHDhQs+1rNq4caNuu+02zZw5U7Nmzarx+E6dOunWW2/VT37yE5WVlWnevHlq1qyZfvGLX0iSoqKiPPOSzp8/r9atW+svf/lLlUlfcnKyJOnxxx/X6NGj1bBhQw0fPlydOnXS448/rtmzZ6tv3776wQ9+IKfTqe3bt6tVq1ZX9NU0RUVF+vjjjyt9wOByn3/+uQYOHKiRI0cqKSlJDRo00IoVK1RYWKjRo0d7dc0XXnhBt956q2688UY99NBDat++vb766iv93//9n3bt2iXp4pIYGzZsUGpqqiZOnKikpCQdP35cO3fu1F//+lcdP37c67ECqKcC9GlEwGvDhw83wsPDjdLS0mqPGT9+vNGwYcNKH3u/XLdu3YwJEyZU2HdpuYVPPvnEuO+++4zIyEgjJibGyMzMNP79739XOHbVqlVGt27djPDwcCMxMdF46qmnjMWLF1dadqCgoMAYNmyYERkZaUiq8NH+b775xsjMzDRat25thIWFGW3atDHS09M9tV9abuGtt96qcO2qlgR47733DEnGwoULTcd96b1z5841nnnmGSMhIcFwOp1G3759KywnYRiGcfjwYePee+81mjRpYkRHRxv333+/cfToUUOSMXPmzArHzp4922jdurUREhJS6R4sXrzY6Nmzp+F0Oo2YmBijf//+nqUyDOPicgvDhg2rVOvlSyEYhmEsWLDAuOaaawyXy2U6zuLiYmPSpElGly5djEaNGhnR0dFGamqq8ac//cn0GlXdW8MwjD179njuRXh4uNG5c2fjV7/6VYVjCgsLjUmTJhkJCQlGw4YNjfj4eGPgwIHGokWLTGsFEFwchuGH5w5AHfPHP/5RkyZN0qFDh9SkSZNAl1Nrv/jFL/TGG2/owIEDcjqdgS7HZ3r27KkBAwboueeeC3QpAFAl5ljhqjR27Fi1bdu2yknw9dGGDRv0q1/9KqibqjVr1mj//v2aPn16oEsBgGqRWAEAANiExAoAAMAmNFYAAAA2obECAACwCY0VAACATfy+QKjb7dbRo0cVGRlp+iWqAADg4ir/p06dUqtWrRQS4v885OzZszp37pxPzh0WFqbw8HCfnDtQ/N5YHT16VAkJCf6+LAAA9Vp+fr7atGnj12uePXtWsREROu2j88fHx+vgwYNB1Vz5vbGKjIz8z399JSnK35evU0pOkNgBAMy5XC4ltGv3nd+f/nPu3DmdlvSIJLtXySuT9FxBgc6dO0djVRvfPv6L0tXeWEVF0VgBAKwJ5PSZCEl2tz7BOsk7WMcFAADgdzRWAADAVIiPNm/Nnz9fiYmJCg8PV2pqqnJzc6s99p133lFKSoqaNGmiRo0aqUePHvrjH/94BVf1Do0VAACo8958801lZWVp5syZ2rlzp7p3767BgwerqKioyuObNm2qxx9/XFu3btXHH3+sjIwMZWRkaO3atT6t0+/fFehyuRQdHS3puK72OVZGOXOsAADmXC6XomNiVFJSoqgo//7evPQ7e4bsn2N1VtKvJcvjSk1NVa9evfTSSy9Jurh8U0JCgiZPnqxp06ZZuuaNN96oYcOGafbs2bWo3ByJFQAAMOXLR4Eul6vCVlZWVun6586dU15entLS0r6tKSREaWlp2rp1a431G4ahnJwcffbZZ+rXr98V3AHraKwAAEDAJCQkKDo62rNlZ2dXOqa4uFjl5eWKi4ursD8uLk4FBQXVnrukpESNGzdWWFiYhg0bphdffFHf//73bR/Dd/l9uQUAAFC/XOlk85rOKV1c+PS7jwKdTvtWzIqMjNSuXbt0+vRp5eTkKCsrSx06dNCAAQNsu8blaKwAAEDAREVF1TjHqnnz5goNDVVhYWGF/YWFhYqPj6/2fSEhIerUqZMkqUePHtq3b5+ys7N92ljxKBAAAJgK9HILYWFhSk5OVk5Ojmef2+1WTk6O+vTpY/k8bre7yjlcdiKxAgAAdV5WVpbS09OVkpKi3r17a968eSotLVVGRoYkady4cWrdurVnjlZ2drZSUlLUsWNHlZWV6YMPPtAf//hHLViwwKd10lgBAABTvpxjZdWoUaN07NgxzZgxQwUFBerRo4fWrFnjmdB+6NAhhYR8e9bS0lL99Kc/1eHDhxUREaEuXbro9ddf16hRo2wcRWWsYxVArGMFAKhJXVjH6kn5Zh2rx2R9Hav6gsQKAACYqguJVX0RrOMCAADwOxIrAABgyiH7k5hgnQxDYwUAAEw5ZH8jFKyNFY8CAQAAbEJiBQAATIX+Z7P7nMGIxAoAAMAmJFYAAMAUyy1YF6zjAgAA8DsSKwAAYIrEyrpgHRcAAIDfkVgBAABTJFbW0VgBAABTNFbWBeu4AAAA/O6KGqv58+crMTFR4eHhSk1NVW5urt11AQCAOiLER1sw8npcb775prKysjRz5kzt3LlT3bt31+DBg1VUVOSL+gAAAOoNrxurZ599VhMnTlRGRoaSkpK0cOFCXXPNNVq8eLEv6gMAAAFGYmWdV+M6d+6c8vLylJaW9u0JQkKUlpamrVu3VvmesrIyuVyuChsAAEAw8qqxKi4uVnl5ueLi4irsj4uLU0FBQZXvyc7OVnR0tGdLSEi48moBAIDfOXy0BSOfJ3HTp09XSUmJZ8vPz/f1JQEAAALCq3WsmjdvrtDQUBUWFlbYX1hYqPj4+Crf43Q65XQ6r7xCAAAQUCGSQn1wzmDk1bjCwsKUnJysnJwczz63262cnBz16dPH9uIAAEDgMXndOq9XXs/KylJ6erpSUlLUu3dvzZs3T6WlpcrIyPBFfQAAAPWG143VqFGjdOzYMc2YMUMFBQXq0aOH1qxZU2lCOwAACA58pY11V/RdgZmZmcrMzLS7FgAAgHqNL2EGAACmSKysC9ZxAQAA+B2JFQAAMEViZV2wjgsAAMDvSKwAAIApEivraKwAAIApGivrgnVcAAAAfkdiBQAATDn+s9l9zmBEYgUAAGATEisAAGAq9D+b3ecMRiRWAAAANiGxAgAAphyyP4lhjhUAAABMkVgBAABTrGNlHY0VAAAwRWNlXbCOCwAAwO9IrAAAgCkSK+uCdVwAAAB+R2IFAABMkVhZF6zjAgAA8DsSKwAAYIrEyrpgHRcAAIDfkVgBAABTDtn/FTTB+pU2NFYAAMBU6H82u88ZjHgUCAAAYBMSKwAAYIrJ69YFrLEqOeFQVFSwPmG1xhG6P9Al1AlG+bWBLgEAAFuQWAEAAFMO2Z8wBWu0EqxJHAAAgN+RWAEAAFPMsbIuWMcFAADgdyRWAADAFImVdTRWAADAFI2VdcE6LgAAAL8jsQIAAKZIrKwL1nEBAAD4HYkVAAAw5ZD9C3qyQCgAAABM0VgBAABToT7avDV//nwlJiYqPDxcqampys3NrfbYV199VX379lVMTIxiYmKUlpZmerxdaKwAAECd9+abbyorK0szZ87Uzp071b17dw0ePFhFRUVVHr9x40aNGTNGGzZs0NatW5WQkKBBgwbpyJEjPq3TYRiG4dMrXMblcik6OlolJ04oKirKn5eucxyh+wNdQp1glF8b6BIAoM5yuVyKjolRSUmJ339vXvqdvUNSY5vPfVpSimR5XKmpqerVq5deeuklSZLb7VZCQoImT56sadOm1fj+8vJyxcTE6KWXXtK4ceNqWX31SKwAAICpEB9t0sXm7btbWVlZpeufO3dOeXl5SktL+7amkBClpaVp69atlsZw5swZnT9/Xk2bNvVy9N6hsQIAAAGTkJCg6Ohoz5adnV3pmOLiYpWXlysuLq7C/ri4OBUUFFi6zi9/+Uu1atWqQnPmCyy3AAAATDlkfxJzabmF/Pz8Co8CnU6nzVeS5syZo+XLl2vjxo0KDw+3/fzfRWMFAAACJioqqsY5Vs2bN1doaKgKCwsr7C8sLFR8fLzpe59++mnNmTNHf/3rX9WtW7da11sTHgUCAABTvpxjZUVYWJiSk5OVk5Pj2ed2u5WTk6M+ffpU+77f/e53mj17ttasWaOUlBQvrnjlSKwAAECdl5WVpfT0dKWkpKh3796aN2+eSktLlZGRIUkaN26cWrdu7Zmj9dRTT2nGjBlatmyZEhMTPXOxGjdurMaN7f6M47dorAAAgKm68CXMo0aN0rFjxzRjxgwVFBSoR48eWrNmjWdC+6FDhxQS8u1ZFyxYoHPnzum+++6rcJ6ZM2dq1qxZtay+eqxjFUCsY3UR61gBQPXqwjpWuyVF2nzuU5K+J+vrWNUXJFYAAMBUXUis6otgHRcAAIDfkVgBAABTDn277pSd5wxGNFYAAMBU6H82u88ZjHgUCAAAYBMSKwAAYIrJ69Z5Pa7Nmzdr+PDhatWqlRwOh1auXOmDsgAAAOofrxur0tJSde/eXfPnz/dFPQAAoI4J9Ffa1CdePwq84447dMcdd/iiFgAAgHrN53OsysrKVFZW5vnZ5XL5+pIAAMBGzLGyzufjys7OVnR0tGdLSEjw9SUBAAACwueN1fTp01VSUuLZ8vPzfX1JAABgI4fsn1/FAqFXyOl0yul0+voyAADAR3gUaF2wjgsAAMDvvE6sTp8+rQMHDnh+PnjwoHbt2qWmTZuqbdu2thYHAAACj8TKOq8bqx07dui2227z/JyVlSVJSk9P12uvvWZbYQAAAPWN143VgAEDZBiGL2oBAAB1EImVdcE6LgAAAL/jS5gBAIApEivrgnVcAAAAfkdiBQAATJFYWUdjBQAATNFYWRes4wIAAPA7EisAAGCKxMq6YB0XAACA35FYAQAAUyRW1gXruAAAAPyOxAoAAJhySHI4HPaeM0i/Ho/ECgAAwCYkVgAAwFyDBpLNiZUMQ7pwwd5z1gE0VgAAwByNlWU8CgQAALAJiRUAADDnq8QqCJFYAQAA2ITECgAAmCOxsozECgAAwCYkVgAAwFxoqBRicxbjdtt7vjqCxAoAAMAmJFYAAMBcgwYkVhbRWAEAAHM0VpbxKBAAAMAmJFYAAMAciZVlJFYAAAA2IbEKIKP82kCXUCc4Qu8OdAl1glH+bqBLAICqhYZe3OxUXm7v+eoIEisAAACbkFgBAABzDRrYn1jZ/RU5dQSJFQAAgE1IrAAAgDkSK8torAAAgDkaK8t4FAgAAGATEisAAGCOxMoyEisAAACbkFgBAABzoaEXUyvUiMQKAADAJrSfAADAXIMGJFYWkVgBAADYhPYTAACYI7GyjLsEAADM0VhZxqNAAAAAm9B+AgAAc75YbsEw7D1fHUFiBQAAYBMSKwAAYM4Xc6xIrAAAAGCGxgoAAJi7lFjZvXlp/vz5SkxMVHh4uFJTU5Wbm1vtsXv37tWIESOUmJgoh8OhefPm1eIGWEdjBQAA6rw333xTWVlZmjlzpnbu3Knu3btr8ODBKioqqvL4M2fOqEOHDpozZ47i4+P9VieNFQAAMFcHEqtnn31WEydOVEZGhpKSkrRw4UJdc801Wrx4cZXH9+rVS3PnztXo0aPldDrtuAuW0FgBAABzAW6szp07p7y8PKWlpXn2hYSEKC0tTVu3bvXFiK8YnwoEAAAB43K5KvzsdDorJUzFxcUqLy9XXFxchf1xcXH69NNPfV6jN0isAACAuUsLhNq5hYZKkhISEhQdHe3ZsrOzAzzY2iGxAgAAAZOfn6+oqCjPz1XNh2revLlCQ0NVWFhYYX9hYaFfJ6ZbQWIFAADM+XCOVVRUVIWtqsYqLCxMycnJysnJ8exzu93KyclRnz59/HYbrCCxAgAAdV5WVpbS09OVkpKi3r17a968eSotLVVGRoYkady4cWrdurXnUeK5c+f0ySefeP77yJEj2rVrlxo3bqxOnTr5rE6vGqvs7Gy98847+vTTTxUREaGbb75ZTz31lDp37uyr+gAAQKD54itt3G6vDh81apSOHTumGTNmqKCgQD169NCaNWs8E9oPHTqkkJBvH8QdPXpUPXv29Pz89NNP6+mnn1b//v21ceNGW4ZQFa/u0qZNmzRp0iT16tVLFy5c0GOPPaZBgwbpk08+UaNGjXxVIwAAgDIzM5WZmVnla5c3S4mJiTIC8H2EXjVWa9asqfDza6+9phYtWigvL0/9+vWztTAAAFBH1IHEqr6o1V0qKSmRJDVt2rTaY8rKylRWVub5+fL1KgAAAILFFX8q0O12a+rUqbrlllvUtWvXao/Lzs6usD5FQkLClV4SAAAEQh34Spv64oobq0mTJmnPnj1avny56XHTp09XSUmJZ8vPz7/SSwIAgEDw4QKhweaK2sXMzEy9//772rx5s9q0aWN6bFVL0wMAAAQjrxorwzA0efJkrVixQhs3blT79u19VRcAAKgrfPHorrzc3vPVEV7dpUmTJmnZsmV69913FRkZqYKCAklSdHS0IiIifFIgAABAfeFVY7VgwQJJ0oABAyrsX7JkicaPH29XTQAAoC4hsbLM60eBAAAAqFpwftYRAADYh8TKsitebgEAAAAVkVgBAABzl9axstOFC/aer46gsQIAAOZ88SiQldcBAABgJjjbRQAAYB8SK8tIrAAAAGwSnO0iAACwD4mVZSRWAAAANgnOdhEAANiHxMoyEisAAACbBGe7CAAA7OOLBUJDQ+09Xx1BYwUAAMzxKNAyHgUCAADYJDjbRQAAYB8SK8tIrAAAAGwSnO0iAACwD4mVZSRWAAAANgnOdhEAANiH5RYsI7ECAACwCYkVAAAwxxwry4JzVAAAwD40VpbxKBAAAMAmwdkuAgAA+5BYWUZiBQAAYJPgbBcBAIB9SKwsC85RoV4xyt8NdAl1wrPzCJAlKWuqO9AlAMAVo7ECAADmWCDUMv6JDAAAYBMSKwAAYI45VpYF56gAAIB9aKws41EgAACATYKzXQQAAPYhsbKMxAoAAMAmwdkuAgAA+7DcgmUkVgAAADYhsQIAAOaYY2UZiRUAAIBNgrNdBAAA9iGxsiw4RwUAAOxDY2UZjwIBAABsEpztIgAAsA+JlWUkVgAAADYJznYRAADYhwVCLSOxAgAAsAmJFQAAMMccK8tIrAAAAGwSnO0iAACwD4mVZcE5KgAAYB8aK8t4FAgAAGCT4GwXAQCAfVhuwTISKwAAAJuQWAEAAHPMsbKMxAoAAMAmwdkuAgAA+5BYWeZVYrVgwQJ169ZNUVFRioqKUp8+fbR69Wpf1QYAAOAxf/58JSYmKjw8XKmpqcrNzTU9/q233lKXLl0UHh6u733ve/rggw98XqNXjVWbNm00Z84c5eXlaceOHbr99tt19913a+/evb6qDwAABNqlxMruzQtvvvmmsrKyNHPmTO3cuVPdu3fX4MGDVVRUVOXxW7Zs0ZgxYzRhwgT94x//0D333KN77rlHe/bsseOOVMthGIZRmxM0bdpUc+fO1YQJEywd73K5FB0drZITJxQVFVWbSwNB5dl5THmUpKyp7kCXANQpLpdL0TExKikp8fvvTV/+zvZ2XKmpqerVq5deeuklSZLb7VZCQoImT56sadOmVTp+1KhRKi0t1fvvv+/Zd9NNN6lHjx5auHChfQO5zBX/TV5eXq7ly5ertLRUffr0qfa4srIyuVyuChsAAICkSj1CWVlZpWPOnTunvLw8paWlefaFhIQoLS1NW7durfK8W7durXC8JA0ePLja4+3idWO1e/duNW7cWE6nUz/+8Y+1YsUKJSUlVXt8dna2oqOjPVtCQkKtCgYAAP7lVohPNklKSEio0CdkZ2dXun5xcbHKy8sVFxdXYX9cXJwKCgqqrLmgoMCr4+3i9ZT8zp07a9euXSopKdHbb7+t9PR0bdq0qdrmavr06crKyvL87HK5aK4AAIAkKT8/v8KjQKfTGcBqas/rxiosLEydOnWSJCUnJ2v79u16/vnn9corr1R5vNPprPc3CQCAq9mFCxc3u88pybPSgJnmzZsrNDRUhYWFFfYXFhYqPj6+yvfEx8d7dbxdaj1b1u12V/k8FAAAwA5hYWFKTk5WTk6OZ5/b7VZOTk6187z79OlT4XhJWrdunem8cDt4lVhNnz5dd9xxh9q2batTp05p2bJl2rhxo9auXeur+gAAQID5MrGyKisrS+np6UpJSVHv3r01b948lZaWKiMjQ5I0btw4tW7d2jNHa8qUKerfv7+eeeYZDRs2TMuXL9eOHTu0aNEiewdyGa8aq6KiIo0bN05ff/21oqOj1a1bN61du1bf//73fVUfAACARo0apWPHjmnGjBkqKChQjx49tGbNGs8E9UOHDikk5NsHcTfffLOWLVum//mf/9Fjjz2ma6+9VitXrlTXrl19Wmet17HyFutYAVVjHauLWMcKqKgurGOVn2//tS9+mC06IOPypeD8oh4AAGCbuvAosL7gn8gAAAA2IbECAACmysvtT5jKy+09X11BYgUAAGATEisAAGCKOVbWkVgBAADYhMQKAACYIrGyjsQKAADAJiRWAADAFImVdSRWAAAANiGxAgAApljHyjoaKwAAYIpHgdbxKBAAAMAmJFYAAMAUiZV1JFYAAAA2IbECAACmSKysI7ECAACwCYkVAAAwRWJlHYkVAACATUisAACAKRYItY7GCgAAmOJRoHU8CgQAALAJiRUAADBFYmUdiRUAAIBNSKwAAIApEivrSKwAAABsQmIF1BFZU92BLqFOWPT/+PeeJD303/z/gLqD5Ras428wAAAAm5BYAQAAU8yxso7GCgAAmKKxso5HgQAAADYhsQIAAKZIrKwjsQIAALAJiRUAADBFYmUdiRUAAIBNSKwAAIApFgi1jsQKAADAJiRWAADAFHOsrKOxAgAApmisrONRIAAAgE1IrAAAgCkSK+tIrAAAAGxCYgUAAEyx3IJ1JFYAAAA2IbECAACmmGNlHYkVAACATUisAACAKRIr62isAACAKRor63gUCAAAYBMSKwAAYIrEyjoSKwAAAJuQWAEAAFMsEGodiRUAAIBNSKwAAICpCxek0FD7zxmMSKwAAABsUqvGas6cOXI4HJo6dapN5QAAgLrm0qcC7d6C0RU/Cty+fbteeeUVdevWzc56AABAHcOjQOuuKLE6ffq0xo4dq1dffVUxMTF21wQAAFAvXVFjNWnSJA0bNkxpaWk1HltWViaXy1VhAwAA9cel5Rbs3IJ1uQWvHwUuX75cO3fu1Pbt2y0dn52drSeeeMLrwgAAAOobrxKr/Px8TZkyRUuXLlV4eLil90yfPl0lJSWeLT8//4oKBQAAgcHkdeu8Sqzy8vJUVFSkG2+80bOvvLxcmzdv1ksvvaSysjKFXja7zel0yul02lMtAABAHeZVYzVw4EDt3r27wr6MjAx16dJFv/zlLys1VQAAoP67cEEKsXnly2BNrLy6TZGRkeratWuFrVGjRmrWrJm6du3qqxoBAAAsO378uMaOHauoqCg1adJEEyZM0OnTp03fs2jRIg0YMEBRUVFyOBw6efLkFV2bldcBAICp+jbHauzYsdq7d6/WrVun999/X5s3b9ZDDz1k+p4zZ85oyJAheuyxx2p17Vp/V+DGjRtrewoAAFCH1adHgfv27dOaNWu0fft2paSkSJJefPFFDR06VE8//bRatWpV5fsufYtMbfsaEisAABA0tm7dqiZNmniaKklKS0tTSEiItm3b5vPr1zqxAgAAwe3SAqF2n1NSpYXDa7uaQEFBgVq0aFFhX4MGDdS0aVMVFBRc8XmtIrECAAABk5CQoOjoaM+WnZ1d5XHTpk2Tw+Ew3T799FM/V18ZiRUAADB14YLkcNh/Tuni4uNRUVGe/dWlVY8++qjGjx9ves4OHTooPj5eRUVFl13rgo4fP674+Pha1WwFjRUAAAiYqKioCo1VdWJjYxUbG1vjcX369NHJkyeVl5en5ORkSdL69evldruVmppa63prwqNAAABgqj4tt3D99ddryJAhmjhxonJzc/Xhhx8qMzNTo0eP9nwi8MiRI+rSpYtyc3M97ysoKNCuXbt04MABSdLu3bu1a9cuHT9+3Kvr01gBAICgsnTpUnXp0kUDBw7U0KFDdeutt2rRokWe18+fP6/PPvtMZ86c8exbuHChevbsqYkTJ0qS+vXrp549e2rVqlVeXZtHgQAAwJQv51j5QtOmTbVs2bJqX09MTJRhGBX2zZo1S7Nmzar1tWmsAACAqfrWWAUSjwIBAABsQmIFAABMlZfbn1hdWiA02JBYAQAA2ITECgAAmPLFfCjmWAEAAMAUiRUAADBFYmUdiRUAAIBNSKwAAIApEivrSKwAAABsQmIFAABM+WLNqWBdx4rGCgAAmLpwQbrsq/VqLVgbKx4FAgAA2ITECgAAmCKxso7ECgAAwCYkVgAAwBSJlXUkVgAAADYhsQIAAKZIrKyjsQJQpzz03+5Al1AnOEKXBbqEOsEo/2GgSwC8QmMFAABMlZfbn1i5g/TfUDRWAADA1IULUojNs7KDtbFi8joAAIBNSKwAAIApEivrSKwAAABsQmIFAABMkVhZR2IFAABgExIrAABgqrzc/oTJ7uUb6goSKwAAAJuQWAEAAFMXLkgOh73nDNbEisYKAACYorGyjkeBAAAANiGxAgAApkisrCOxAgAAsAmJFQAAMEViZR2JFQAAgE1IrAAAQA3cMgy7v4MmOL/ThsQKAADAJiRWAACgBuX/2ew+Z/ChsQIAADWgsbKKR4EAAAA2IbECAAA1ILGyisQKAADAJiRWAACgBm7ZvzwCyy0AAADABIkVAACoAXOsrCKxAgAAsAmJFQAAqIFb9idMzLHSrFmz5HA4KmxdunTxVW0AAKBOKPfRFny8TqxuuOEG/fWvf/32BA0IvQAAAKQraKwaNGig+Ph4X9QCAADqJCavW+X15PX9+/erVatW6tChg8aOHatDhw6ZHl9WViaXy1VhAwAACEZeNVapqal67bXXtGbNGi1YsEAHDx5U3759derUqWrfk52drejoaM+WkJBQ66IBAIA/uX20BR+HYRjGlb755MmTateunZ599llNmDChymPKyspUVlbm+dnlcikhIUElJ04oKirqSi8NAEHNEbos0CXUCUb5DwNdQsC5XC5Fx8SopKTE7783XS6XoqOjJe2TFGnz2U9Juj4g4/KlWs08b9Kkia677jodOHCg2mOcTqecTmdtLgMAAAKKOVZW1WqB0NOnT+uLL75Qy5Yt7aoHAACg3vKqsfrZz36mTZs26auvvtKWLVt07733KjQ0VGPGjPFVfQAAIOBYx8oqrx4FHj58WGPGjNE333yj2NhY3Xrrrfroo48UGxvrq/oAAEDA8SjQKq8aq+XLl/uqDgAAgHqPZdMBAEAN+K5Aq2o1eR0AAADfIrECAAA18MWCniRWAAAAMEFiBQAAasCnAq0isQIAALAJjRUAAKhB/Vog9Pjx4xo7dqyioqLUpEkTTZgwQadPnzY9fvLkyercubMiIiLUtm1bPfzwwyopKfH62jwKBAAANahfjwLHjh2rr7/+WuvWrdP58+eVkZGhhx56SMuWVf3l5kePHtXRo0f19NNPKykpSf/617/04x//WEePHtXbb7/t1bUdhmEYdgzCqkvflF1y4kRQfZs1ANjJEVr1L4CrjVH+w0CXEHAul0vRMTEqKSnx++/NS7+zpU2SGtt89tOS+ts+rn379ikpKUnbt29XSkqKJGnNmjUaOnSoDh8+rFatWlk6z1tvvaUHHnhApaWlatDAeg7Fo0AAAFAD3z0KdLlcFbaysrJaVbp161Y1adLE01RJUlpamkJCQrRt2zbL57nU8HnTVEk0VgAAIIASEhIUHR3t2bKzs2t1voKCArVo0aLCvgYNGqhp06YqKCiwdI7i4mLNnj1bDz30kNfXZ44VAACogSH7F/S8OBMpPz+/wqNAp9NZ5dHTpk3TU089ZXrGffv21boql8ulYcOGKSkpSbNmzfL6/TRWAAAgYKKioizNsXr00Uc1fvx402M6dOig+Ph4FRUVVdh/4cIFHT9+XPHx8abvP3XqlIYMGaLIyEitWLFCDRs2rLGuy9FYAQCAGgT+U4GxsbGKjY2t8bg+ffro5MmTysvLU3JysiRp/fr1crvdSk1NrfZ9LpdLgwcPltPp1KpVqxQeHu5VfZcwxwoAAASN66+/XkOGDNHEiROVm5urDz/8UJmZmRo9erTnE4FHjhxRly5dlJubK+liUzVo0CCVlpbqf//3f+VyuVRQUKCCggKVl3vXAJJYAQCAGgQ+sfLG0qVLlZmZqYEDByokJEQjRozQCy+84Hn9/Pnz+uyzz3TmzBlJ0s6dOz2fGOzUqVOFcx08eFCJiYmWr01jBQAAalC/GqumTZtWuxioJCUmJuq7y3gOGDBAdi3ryaNAAAAAm5BYAQCAGtSvxCqQSKwAAABsQmIFAABq4Jb9C4Tafb66gcQKAADAJiRWAACgBsyxsorECgAAwCYkVgAAoAZu2Z8wMccKAAAAJkisAKAOMsp/GOgS6gRH6DuBLqEOOBPoAsQcK+torAAAQA1YbsEqHgUCAADYhMQKAADUgEeBVpFYAQAA2ITECgAA1IDEyioSKwAAAJuQWAEAgBqQWFlFYgUAAGATEisAAFADvtLGKhorAABQAxYItYpHgQAAADYhsQIAADVg8rpVJFYAAAA2IbECAAA1ILGyisQKAADAJiRWAACgBiRWVpFYAQAA2ITECgAA1IDEyioaKwAAUAMWCLWKR4EAAAA2IbECAAA14LsCrSKxAgAAsAmJFQAAqAGT160isQIAALAJiRUAAKgBiZVVJFYAAAA2IbECAAA1ILGyisYKAADUgMbKKq8fBR45ckQPPPCAmjVrpoiICH3ve9/Tjh07fFEbAABAveJVYnXixAndcsstuu2227R69WrFxsZq//79iomJ8VV9AAAg4PhKG6u8aqyeeuopJSQkaMmSJZ597du3t70oAACA+sirR4GrVq1SSkqK7r//frVo0UI9e/bUq6++avqesrIyuVyuChsAAKhPLn2ljZ1bcCZWXjVWX375pRYsWKBrr71Wa9eu1U9+8hM9/PDD+v3vf1/te7KzsxUdHe3ZEhISal00AABAXeQwDMOwenBYWJhSUlK0ZcsWz76HH35Y27dv19atW6t8T1lZmcrKyjw/u1wuJSQkqOTECUVFRdWidABAsHOEvhPoEuqAM5LSVVJS4vffmy6XS9HR0ZJ+Lslp89nLJM0NyLh8yavEqmXLlkpKSqqw7/rrr9ehQ4eqfY/T6VRUVFSFDQAAIBh5NXn9lltu0WeffVZh3+eff6527drZWhQAAKhLWMfKKq8aq0ceeUQ333yznnzySY0cOVK5ublatGiRFi1a5Kv6AABAwLHcglVePQrs1auXVqxYoTfeeENdu3bV7NmzNW/ePI0dO9ZX9QEAANQbXn+lzZ133qk777zTF7UAAIA6iUeBVnn9lTYAAACoGl/CDAAAakBiZRWJFQAAgE1IrAAAQA1IrKwisQIAALAJiRUAAKjBpS9htvucwYfGCgAA1IAFQq3iUSAAAIBNSKwAAEANmLxuFYkVAACATUisAABADUisrCKxAgAAsAmJFQAAqAGJlVUkVgAAADahsQIAADUo99HmG8ePH9fYsWMVFRWlJk2aaMKECTp9+rTpe370ox+pY8eOioiIUGxsrO6++259+umnXl+bxgoAANTg0srrdm6+WyB07Nix2rt3r9atW6f3339fmzdv1kMPPWT6nuTkZC1ZskT79u3T2rVrZRiGBg0apPJy7xpAh2EYRm2K95bL5VJ0dLRKTpxQVFSUPy8NAKhnHKHvBLqEOuCMpHSVlJT4/ffmpd/Z0v2SGtp89vOS3rJ9XPv27VNSUpK2b9+ulJQUSdKaNWs0dOhQHT58WK1atbJ0no8//ljdu3fXgQMH1LFjR8vXJ7ECAAA1cPtou9i8fXcrKyurVaVbt25VkyZNPE2VJKWlpSkkJETbtm2zdI7S0lItWbJE7du3V0JCglfXp7ECAAABk5CQoOjoaM+WnZ1dq/MVFBSoRYsWFfY1aNBATZs2VUFBgel7X375ZTVu3FiNGzfW6tWrtW7dOoWFhXl1fRorAABQA99NXs/Pz1dJSYlnmz59epUVTJs2TQ6Hw3S7ksnm3zV27Fj94x//0KZNm3Tddddp5MiROnv2rFfnYB0rAAAQMFFRUZbmWD366KMaP3686TEdOnRQfHy8ioqKKuy/cOGCjh8/rvj4eNP3X0rNrr32Wt10002KiYnRihUrNGbMmBrru4TGCgAA1KBc9j/k8u7TdrGxsYqNja3xuD59+ujkyZPKy8tTcnKyJGn9+vVyu91KTU21fD3DMGQYhtdzvngUCAAAgsb111+vIUOGaOLEicrNzdWHH36ozMxMjR492vOJwCNHjqhLly7Kzc2VJH355ZfKzs5WXl6eDh06pC1btuj+++9XRESEhg4d6tX1aawAAEAN6tcCoUuXLlWXLl00cOBADR06VLfeeqsWLVrkef38+fP67LPPdObMGUlSeHi4/va3v2no0KHq1KmTRo0apcjISG3ZsqXSRPia8CgQAADU4NvlEew9p280bdpUy5Ytq/b1xMREfXcZz1atWumDDz6w5dp+b6wuDcTlcvn70gCAeudMoAuoA/4tSfLzet6XuVBPzhl4fm+sTp06JUlKaNfO35cGAKDeOnXq1H9WQfefsLAwxcfHq6Dgrz45f3x8vNfrRNV1fv9KG7fbraNHjyoyMlIOh8Ofl/ZwuVxKSEhQfn7+Vf21OtyHi7gPF3EfLuI+XMR9uKgu3AfDMHTq1Cm1atVKISH+nxp99uxZnTt3zifnDgsLU3h4uE/OHSh+T6xCQkLUpk0bf1+2SlbXzgh23IeLuA8XcR8u4j5cxH24KND3wd9J1XeFh4cHXfPjS3wqEAAAwCY0VgAAADa5Khsrp9OpmTNnyul0BrqUgOI+XMR9uIj7cBH34SLuw0XcB3jL75PXAQAAgtVVmVgBAAD4Ao0VAACATWisAAAAbEJjBQAAYJOrrrGaP3++EhMTFR4ertTUVOXm5ga6JL/bvHmzhg8frlatWsnhcGjlypWBLsnvsrOz1atXL0VGRqpFixa655579NlnnwW6LL9bsGCBunXr5ln8sE+fPlq9enWgywq4OXPmyOFwaOrUqYEuxa9mzZolh8NRYevSpUugywqII0eO6IEHHlCzZs0UERGh733ve9qxY0egy0I9cFU1Vm+++aaysrI0c+ZM7dy5U927d9fgwYNVVFQU6NL8qrS0VN27d9f8+fMDXUrAbNq0SZMmTdJHH32kdevW6fz58xo0aJBKS0sDXZpftWnTRnPmzFFeXp527Nih22+/XXfffbf27t0b6NICZvv27XrllVfUrVu3QJcSEDfccIO+/vprz/b3v/890CX53YkTJ3TLLbeoYcOGWr16tT755BM988wziomJCXRpqAeuquUWUlNT1atXL7300kuSLn5vYUJCgiZPnqxp06YFuLrAcDgcWrFihe65555AlxJQx44dU4sWLbRp0yb169cv0OUEVNOmTTV37lxNmDAh0KX43enTp3XjjTfq5Zdf1m9+8xv16NFD8+bNC3RZfjNr1iytXLlSu3btCnQpATVt2jR9+OGH+tvf/hboUlAPXTWJ1blz55SXl6e0tDTPvpCQEKWlpWnr1q0BrAx1QUlJiaSLTcXVqry8XMuXL1dpaan69OkT6HICYtKkSRo2bFiFvyeuNvv371erVq3UoUMHjR07VocOHQp0SX63atUqpaSk6P7771eLFi3Us2dPvfrqq4EuC/XEVdNYFRcXq7y8XHFxcRX2x8XFqaCgIEBVoS5wu92aOnWqbrnlFnXt2jXQ5fjd7t271bhxYzmdTv34xz/WihUrlJSUFOiy/G758uXauXOnsrOzA11KwKSmpuq1117TmjVrtGDBAh08eFB9+/bVqVOnAl2aX3355ZdasGCBrr32Wq1du1Y/+clP9PDDD+v3v/99oEtDPdAg0AUAgTZp0iTt2bPnqpxLIkmdO3fWrl27VFJSorffflvp6enatGnTVdVc5efna8qUKVq3bp3Cw8MDXU7A3HHHHZ7/7tatm1JTU9WuXTv96U9/uqoeDbvdbqWkpOjJJ5+UJPXs2VN79uzRwoULlZ6eHuDqUNddNYlV8+bNFRoaqsLCwgr7CwsLFR8fH6CqEGiZmZl6//33tWHDBrVp0ybQ5QREWFiYOnXqpOTkZGVnZ6t79+56/vnnA12WX+Xl5amoqEg33nijGjRooAYNGmjTpk164YUX1KBBA5WXlwe6xIBo0qSJrrvuOh04cCDQpfhVy5YtK/3D4vrrr78qH4vCe1dNYxUWFqbk5GTl5OR49rndbuXk5Fy180muZoZhKDMzUytWrND69evVvn37QJdUZ7jdbpWVlQW6DL8aOHCgdu/erV27dnm2lJQUjR07Vrt27VJoaGigSwyI06dP64svvlDLli0DXYpf3XLLLZWWX/n888/Vrl27AFWE+uSqehSYlZWl9PR0paSkqHfv3po3b55KS0uVkZER6NL86vTp0xX+BXrw4EHt2rVLTZs2Vdu2bQNYmf9MmjRJy5Yt07vvvqvIyEjPPLvo6GhFREQEuDr/mT59uu644w61bdtWp06d0rJly7Rx40atXbs20KX5VWRkZKX5dY0aNVKzZs2uqnl3P/vZzzR8+HC1a9dOR48e1cyZMxUaGqoxY8YEujS/euSRR3TzzTfrySef1MiRI5Wbm6tFixZp0aJFgS4N9YFxlXnxxReNtm3bGmFhYUbv3r2Njz76KNAl+d2GDRsMSZW29PT0QJfmN1WNX5KxZMmSQJfmVw8++KDRrl07IywszIiNjTUGDhxo/OUvfwl0WXVC//79jSlTpgS6DL8aNWqU0bJlSyMsLMxo3bq1MWrUKOPAgQOBLisg3nvvPaNr166G0+k0unTpYixatCjQJaGeuKrWsQIAAPClq2aOFQAAgK/RWAEAANiExgoAAMAmNFYAAAA2obECAACwCY0VAACATWisAAAAbEJjBQAAYBMaKwAAAJvQWAEAANiExgoAAMAmNFYAAAA2+f8wEe3o+Q7XAAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "imshow_zero_center(j[:, 0, :, 0])\n", "_ = plt.title('A (batch, batch) slice')" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.716722Z", "iopub.status.busy": "2022-12-14T22:53:34.716084Z", "iopub.status.idle": "2022-12-14T22:53:34.938202Z", "shell.execute_reply": "2022-12-14T22:53:34.937510Z" }, "id": "g4ZoRJcJNmy5" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAggAAAIQCAYAAADkeZaVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABG/klEQVR4nO3deXwU9f3H8fcmmA0CSTgCMRIORUHkqkEQBAGJIJfQqgWL5Sh4AoqoFWzLodTgzwNUEKwXtYWioCBaDQKCiKBAEAsoKgoSgXBKAlEC2czvD8qWZZOdWTKb7O68no/HPh5kZnbmM192k89+5jufdRmGYQgAAOAMMRUdAAAACD8kCAAAwA8JAgAA8EOCAAAA/JAgAAAAPyQIAADADwkCAADwQ4IAAAD8kCAAAAA/JAg2crlcmjhxovfn2bNny+VyaefOnZaef/fdd+u6667z/rxy5Uq5XC4tWLDA5kjPncvl0siRI23dZ4MGDdS7d29b91kWQ4YMUdWqVU23O3TokKpUqaL33nvPtmNPnDhRLpfLZ1mDBg00ZMgQ244R7c4er9Pvo5UrV1ZYTKEU7eeHikOCYNHzzz8vl8ultm3bhmT/O3bs0EsvvaSHH344JPs/7b333vNJYqLRnj17NHHiRG3atCmkx6lZs6aGDx+uv/zlLyE9DgBUBBIEi+bMmaMGDRpo3bp12r59u+37f+aZZ9SwYUN16dLF9n2f6b333tOkSZNCeoyKtmfPHk2aNCnkCYIk3Xnnndq4caM+/PDDkB3j66+/1osvvhiy/Ue7a665Rr/88ouuueaaig4FiCgkCBbs2LFDa9as0dNPP63k5GTNmTPH1v2fPHlSc+bM0W9/+1tb94vQu+yyy9SsWTPNnj07ZMdwu90677zzQrb/8mYYhn755ZdyO15MTIzi4+MVE8OvOyAYvGMsmDNnjqpXr65evXrppptusj1BWL16tQ4ePKiMjIwS13s8Hj388MNKSUlRlSpVdMMNNygnJ8dnm48//lg333yz6tWrJ7fbrbS0NN13330+v4iHDBmiGTNmSDo1l+D047Ti4mI988wzat68ueLj45WcnKzrr79eGzZs8Itp0aJFatasmdxuty6//HJlZWWVeRw++OADtWrVSvHx8WratKneeustn/WHDx/WAw88oObNm6tq1apKSEhQjx499MUXX3i3Wblypa688kpJ0tChQ73neOYf8M8++0w9e/ZU9erVVaVKFbVo0ULPPPOMXzy7d+9Wv379VLVqVSUnJ+uBBx6Qx+Px2+66667TO++8I7MvRj158qQmTZqkSy65RPHx8apZs6Y6dOigpUuXBnxeSXMQjhw5ovvuu08NGjSQ2+1W3bp1NWjQIB08eNC7TWFhoSZMmKBGjRp5XxN//OMfVVhY6LOvpUuXqkOHDkpKSlLVqlXVuHFjS5e6Xn31VV177bWqXbu23G63mjZtqpkzZ5YYf+/evbVkyRK1bt1alStX1gsvvOA9j9GjRystLU1ut1uNGjXS448/ruLiYtPjG4ahyZMnq27dujr//PPVpUsXbd261W+7kq7RW3m/nDZ//nw1bdpU8fHxatasmRYuXKghQ4aoQYMGPtsVFBTo/vvv955L48aN9eSTT/q9Lk7P4zF7D/3www+6++671bhxY1WuXFk1a9bUzTffbHlOE1BWlSo6gEgwZ84c/eY3v1FcXJxuueUWzZw5U+vXr/f+ISqrNWvWyOVy6Ve/+lWJ6//617/K5XLpoYce0v79+zVt2jRlZGRo06ZNqly5sqRTv8R+/vln3XXXXapZs6bWrVun5557Tj/++KPmz58vSbrjjju0Z88eLV26VP/4xz/8jjNs2DDNnj1bPXr00PDhw1VUVKSPP/5Yn376qVq3bu3dbvXq1Xrrrbd09913q1q1anr22Wd14403ateuXapZs+Y5jcG3336r/v37684779TgwYP16quv6uabb1ZWVpZ34ub333+vRYsW6eabb1bDhg21b98+vfDCC+rUqZO+/PJLpaam6rLLLtMjjzyi8ePH6/bbb1fHjh0lSe3bt5d06o9h7969dcEFF+jee+9VSkqKvvrqK7377ru69957vfF4PB51795dbdu21ZNPPqlly5bpqaee0sUXX6y77rrLJ/b09HRNnTpVW7duVbNmzUo9x4kTJyozM1PDhw9XmzZtlJ+frw0bNmjjxo0+k1PNHDt2TB07dtRXX32lP/zhD7riiit08OBBLV68WD/++KNq1aql4uJi3XDDDVq9erVuv/12XXbZZdq8ebOmTp2qb775RosWLZIkbd26Vb1791aLFi30yCOPyO12a/v27frkk09M45g5c6Yuv/xy3XDDDapUqZLeeecd3X333SouLtaIESN8tv366691yy236I477tBtt92mxo0b6+eff1anTp20e/du3XHHHapXr57WrFmjcePGae/evZo2bVrA448fP16TJ09Wz5491bNnT23cuFHdunXTiRMnTGO38n6RpH//+9/q37+/mjdvrszMTP30008aNmyYLrzwQp/9GYahG264QStWrNCwYcPUqlUrLVmyRA8++KB2796tqVOn+mxv5T20fv16rVmzRgMGDFDdunW1c+dOzZw5U507d9aXX36p888/3/Q8gTIxENCGDRsMScbSpUsNwzCM4uJio27dusa9997rt60kY8KECd6fX331VUOSsWPHjoDHuPXWW42aNWv6LV+xYoUhybjwwguN/Px87/I33njDkGQ888wz3mU///yz3/MzMzMNl8tl/PDDD95lI0aMMEr6b//www8NScY999zjt664uNjnHOPi4ozt27d7l33xxReGJOO5554LeJ6lqV+/viHJePPNN73L8vLyjAsuuMD41a9+5V12/Phxw+Px+Dx3x44dhtvtNh555BHvsvXr1xuSjFdffdVn26KiIqNhw4ZG/fr1jZ9++qnUcxw8eLAhyWefhmEYv/rVr4z09HS/+NesWWNIMl5//fWA59myZUujV69eAbeZMGGC3/9P/fr1jcGDB3t/Hj9+vCHJeOutt/yef/o8/vGPfxgxMTHGxx9/7LN+1qxZhiTjk08+MQzDMKZOnWpIMg4cOBAwrpKU9Jrr3r27cdFFF/nFL8nIysryWf7oo48aVapUMb755huf5WPHjjViY2ONXbt2lXrs/fv3G3FxcUavXr18/u8efvhhQ5LPeJ1+H61YsSJg7CW9X5o3b27UrVvXOHr0qHfZypUrDUlG/fr1vcsWLVpkSDImT57ss8+bbrrJcLlcPu8Xq++hkmJcu3atIcl47bXXAp4fYAcuMZiYM2eO6tSp45086HK51L9/f82bN6/EcvO5OHTokKpXr17q+kGDBqlatWren2+66SZdcMEFPrfXna4kSKdKnQcPHlT79u1lGIY+//xz0xjefPNNuVwuTZgwwW/d2bfdZWRk6OKLL/b+3KJFCyUkJOj77783PU5pUlNT9etf/9r7c0JCggYNGqTPP/9cubm5kk5diz99Hdnj8ejQoUPekvjGjRtNj/H5559rx44dGj16tJKSknzWnX2O0qkJiGfq2LFjied4+v/uzPJ+SZKSkrR161Z9++23prEG8uabb6ply5Y+43Xa6fOYP3++LrvsMjVp0kQHDx70Pq699lpJ0ooVK7wxSdLbb79tqax/pjNfc3l5eTp48KA6deqk77//Xnl5eT7bNmzYUN27d/dZNn/+fHXs2FHVq1f3iTEjI0Mej0erVq0q9djLli3TiRMnNGrUKJ//u9GjRwcde2nvlz179mjz5s0aNGiQz22vnTp1UvPmzX3299577yk2Nlb33HOPz/L7779fhmHo/fff91lu5T10ZownT57UoUOH1KhRIyUlJVl6vQNlRYIQgMfj0bx589SlSxft2LFD27dv1/bt29W2bVvt27dPy5cvt+1YRoDr15dcconPzy6XS40aNfK5Frlr1y4NGTJENWrU8F4z79SpkyT5/bIuyXfffafU1FTVqFHDdNt69er5Latevbp++ukn0+eWplGjRn5/pC+99FJJ8p5ncXGxpk6dqksuuURut1u1atVScnKy/vOf/1g+R0kBLwOcdnoOxplKO8fT/3clJRlneuSRR3TkyBFdeumlat68uR588EH95z//MY3lbN99953pOXz77bfaunWrkpOTfR6nx3T//v2SpP79++vqq6/W8OHDVadOHQ0YMEBvvPGGpWThk08+UUZGhqpUqaKkpCQlJyd75y6UlCCUFGNWVpZfjKfn4pyOsSQ//PCDJP/3RnJycsBk+zQr75fTx2jUqJHf889e9sMPPyg1NdUnkZdOTWI9c1+nWXkP/fLLLxo/frx3TsPp1/uRI0csvd6BsmIOQgAffvih9u7dq3nz5mnevHl+6+fMmaNu3bqV+Tg1a9Ys0x9Xj8ej6667TocPH9ZDDz2kJk2aqEqVKtq9e7eGDBkS9CdDM7GxsSUuD5Tk2OGxxx7TX/7yF/3hD3/Qo48+qho1aigmJkajR48ut3Msyen/u1q1agXc7pprrtF3332nt99+Wx988IFeeuklTZ06VbNmzdLw4cPLFO/ZiouL1bx5cz399NMlrk9LS5N06lPqqlWrtGLFCv373/9WVlaWXn/9dV177bX64IMPSh2H7777Tl27dlWTJk309NNPKy0tTXFxcXrvvfc0depUv/+PMz8Nnxnjddddpz/+8Y8lHuN0MmO38n6/lMTKe2jUqFF69dVXNXr0aLVr106JiYlyuVwaMGBAucQIkCAEMGfOHNWuXds78/9Mb731lhYuXKhZs2aV+MsvGE2aNNGcOXOUl5enxMREv/Vnl6QNw9D27dvVokULSdLmzZv1zTff6O9//7sGDRrk3a6k2fGlfcq9+OKLtWTJEh0+fNhSFcFu27dvl2EYPvF98803kuSdLb5gwQJ16dJFL7/8ss9zjxw54vPHOdA5StKWLVtKvWPkXOzYsUPS/z4tBlKjRg0NHTpUQ4cO1bFjx3TNNddo4sSJQSUIF198sbZs2WK6zRdffKGuXbuaVjZiYmLUtWtXde3aVU8//bQee+wx/elPf9KKFStKHad33nlHhYWFWrx4sc+n4dOXLqyex7Fjx87p/6J+/fqSTr03LrroIu/yAwcOmCbbVt8vp49RUt+Ts5fVr19fy5Yt09GjR32qCNu2bfPZVzAWLFigwYMH66mnnvIuO378uI4cORL0voBzwSWGUvzyyy9666231Lt3b910001+j5EjR+ro0aNavHhxmY/Vrl07GYah7OzsEte/9tprOnr0qPfnBQsWaO/everRo4ek/30aOfPTh2EYJd66V6VKFUny+yVz4403yjCMEpsohboyIJ263rtw4ULvz/n5+XrttdfUqlUrpaSkSDp1nmfHMn/+fO3evdtnWWnneMUVV6hhw4aaNm2a37qynGN2drYSExN1+eWXB9zu0KFDPj9XrVpVjRo18rvt0MyNN96oL774wme8Tjt9Hr/97W+1e/fuEhss/fLLLyooKJB06tbRs7Vq1UqSAsZV0msuLy9Pr776quXz+O1vf6u1a9dqyZIlfuuOHDmioqKiUp+bkZGh8847T88995xPDGZ3PpQWe0nvl9TUVDVr1kyvvfaajh075l3+0UcfafPmzT7b9uzZUx6PR9OnT/dZPnXqVLlcLu97NRglvd6fe+452+Y+AWaoIJRi8eLFOnr0qG644YYS11911VXepkn9+/cv07E6dOigmjVratmyZd5JZGeqUaOGOnTooKFDh2rfvn2aNm2aGjVqpNtuu03SqQrExRdfrAceeEC7d+9WQkKC3nzzzRI/SaWnp0uS7rnnHnXv3l2xsbEaMGCAunTpot///vd69tln9e233+r6669XcXGxPv74Y3Xp0uWcvn/B5XKpU6dOlnrEX3rppRo2bJjWr1+vOnXq6JVXXtG+fft8/uD07t1bjzzyiIYOHar27dtr8+bNmjNnjs8nSOnUJ9OkpCTNmjVL1apVU5UqVdS2bVs1bNhQM2fOVJ8+fdSqVSsNHTpUF1xwgbZt26atW7eW+IfKiqVLl6pPnz6mn9SbNm2qzp07Kz09XTVq1NCGDRu0YMGCoMf2wQcf1IIFC3TzzTfrD3/4g9LT03X48GEtXrxYs2bNUsuWLfX73/9eb7zxhu68806tWLFCV199tTwej7Zt26Y33njD25PgkUce0apVq9SrVy/Vr19f+/fv1/PPP6+6deuqQ4cOpcbQrVs3xcXFqU+fPrrjjjt07Ngxvfjii6pdu7b27t1r+TwWL16s3r17a8iQIUpPT1dBQYE2b96sBQsWaOfOnaVetjndlyIzM1O9e/dWz5499fnnn+v99983vdQTzPvlscceU9++fXX11Vdr6NCh+umnnzR9+nQ1a9bMJ2no06ePunTpoj/96U/auXOnWrZsqQ8++EBvv/22Ro8e7TMh0arevXvrH//4hxITE9W0aVOtXbtWy5YtO+dbiYGglectE5GkT58+Rnx8vFFQUFDqNkOGDDHOO+884+DBg4ZhnPttjoZhGPfcc4/RqFEjn2Wnb1/617/+ZYwbN86oXbu2UblyZaNXr14+t2IZhmF8+eWXRkZGhlG1alWjVq1axm233ea9derM2/2KioqMUaNGGcnJyYbL5fK5pa6oqMh44oknjCZNmhhxcXFGcnKy0aNHDyM7O9u7jSRjxIgRfvGffSve0aNHDUnGgAEDTM+9fv36Rq9evYwlS5YYLVq0MNxut9GkSRNj/vz5PtsdP37cuP/++40LLrjAqFy5snH11Vcba9euNTp16mR06tTJZ9u3337baNq0qVGpUiW/MVi9erVx3XXXGdWqVTOqVKlitGjRwuf2ssGDBxtVqlTxi7OkWxC/+uorQ5KxbNky0/OcPHmy0aZNGyMpKcmoXLmy0aRJE+Ovf/2rceLEiYDHOHtsDcMwDh06ZIwcOdK48MILjbi4OKNu3brG4MGDva9FwzCMEydOGI8//rhx+eWXG26326hevbqRnp5uTJo0ycjLyzMMwzCWL19u9O3b10hNTTXi4uKM1NRU45ZbbvG79bAkixcvNlq0aGHEx8cbDRo0MB5//HHjlVde8XvNn/7/LcnRo0eNcePGGY0aNTLi4uKMWrVqGe3btzeefPJJn3EpicfjMSZNmuR9PXTu3NnYsmWL33iVdBug1feLYRjGvHnzjCZNmhhut9to1qyZsXjxYuPGG280mjRp4ncu9913n5Gammqcd955xiWXXGI88cQTPrdhGob199BPP/1kDB061KhVq5ZRtWpVo3v37sa2bdssnR9gB5dhlEP9GKa+//57NWnSRO+//766du1a0eGU2XvvvafevXvriy++8LslLJqMHj1aq1atUnZ2tmkFAdGjVatWSk5ONu2CCUQy5iCEiYsuukjDhg3TlClTKjoUW6xYsUIDBgyI6uTg0KFDeumllzR58mSSgyh18uRJv7kQK1eu1BdffKHOnTtXTFBAOaGCAACl2LlzpzIyMnTrrbcqNTVV27Zt06xZs5SYmKgtW7YwHwBRjUmKAFCK6tWrKz09XS+99JIOHDigKlWqqFevXpoyZQrJAaIeFQQAAOCHOQgAAMAPCQIAAPBT7nMQiouLtWfPHlWrVo2Z3wDgMIZh6OjRo0pNTfV+O2t5On78uE6cOBGSfcfFxSk+Pj4k+64I5Z4g7Nmzx/tFMQAAZ8rJyVHdunXL9ZjHjx9XcuXKOma+6TlJSUnRjh07oiZJKPcE4X9fZNK1Ig4PAKhQRZKW+301dnk4ceKEjkm6T5Lb5n0XSpqam6sTJ06QIJyr/11WqCTpvPI+PAAgDFTkJebKkuz+Ex6NE/qi8ZwAAEAZUeMHADhKjOz/dByNn7aj8ZwAAEAZUUEAADgKFQRrSBAAAI5CgmBNNJ4TAAAoo7CtIIwb907A9Y9NLi6nSELPFdvXlv0Yh/4eeIOkJFuOEw7sGrOPPgr8OrumA6+zs339deAxu7QRY3a2KVMCj9lDDzJm5YkKgjXReE4AAKCMwraCAABAKFBBsCYazwkAAJQRFQQAgKNQQbAmGs8JAACUERUEAICjUEGwJhrPCQAAlFHYVhDM+hz87lbz3GbuP6Pn3mJLzPoc5Oaa7yMlxZZQIoVZnwNX7FzTfRie39kVTkQw63Pw83Hz9+b58c56b5r1OZg7z3zMfjfAWWMWSi7Z/+m44r68OnTCNkEAACAUXLL/D3o0JghcYgAAAH6oIAAAHCX2vw+79xltqCAAAAA/VBAAAI7CbY7WROM5AQCAMqKCAABwFCoI1kTjOQEAgDKK2AqClSZIrthfAq43PJXtCicyWGmCNHx44PUvvWRPLBHCShMkV+wnJvu42q5wIoKVJkhmzZSc1kjJShOkVasDj5lZ0y/8DxUEayI2QQAA4FyQIFgTjecEAADKiAoCAMBRqCBYE43nBAAAyogKAgDAUaggWBON5wQAAMqICgIAwFH4umdrzilBmDFjhp544gnl5uaqZcuWeu6559SmTRu7Yyszsz4Haz41L6C0v8ph9xab9Dn4zxbzMWvRzFljZtbnoM1V5mO27lNnjZlZn4OH/2w+Zo9NdtaYmfU52JNrPmapKc4aM5RN0JcYXn/9dY0ZM0YTJkzQxo0b1bJlS3Xv3l379+8PRXwAANgqRv/7yme7HtF4vT7oc3r66ad12223aejQoWratKlmzZql888/X6+88koo4gMAwFYxIXpEm6DO6cSJE8rOzlZGRsb/dhATo4yMDK1du9b24AAAQMUIag7CwYMH5fF4VKdOHZ/lderU0bZt20p8TmFhoQoLC70/5+fnn0OYAADYg9scrQn5OWVmZioxMdH7SEtLC/UhAQBAGQWVINSqVUuxsbHat2+fz/J9+/YppZRvChw3bpzy8vK8j5ycnHOPFgCAMmIOgjVBnVNcXJzS09O1fPly77Li4mItX75c7dq1K/E5brdbCQkJPg8AABDegu6DMGbMGA0ePFitW7dWmzZtNG3aNBUUFGjo0KGhiA8AAFsxB8GaoBOE/v3768CBAxo/frxyc3PVqlUrZWVl+U1cjARWmiB9tj7wf3vbK53VeMRKE6Rnpwces3tGOmvMrDRBcsUGvvRmeJw1d8dKE6S7RgR+nc2c4azXmZUmSPnHAo9ZQlVnjRkCO6dOiiNHjtTIkSPtjgUAgJCjgmAN38UAAHAUEgRrovGcAABAGVFBAAA4Ct/maA0VBAAA4IcKAgDAUU5/A6Pd+4w2VBAAAIAfKggmzPoctLnKPMeych98NDHrc3D4iPmY1Uhy1piZ9Tlwxd5jYR/P2hVORDDrc3BDP/PX2eJFznqdmfY52LnTfCcNGtgRSoVyyf5Px8xBAAAAjkAFAQDgKPRBsIYEAQDgKCQI1kTjOQEAgDKiggAAcBQqCNZE4zkBAIAyooIAAHAUKgjWROM5AQCAMqKCUEZWmiC170AediZLTZAGDAh9IBHEShMkV+z75RBJ5LDSBGnfAd6bPiw0QYqG32dUEKyJxnMCAABlRAUBAOAofN2zNSQIAABH4dscreESAwAA8EMFAQDgKExStCYazwkAAJQRFQQAgKO4ZP+nYyYpliNXbN+KDiHiMGbBY8yCx5gFjzFDJArbBAEAgFBgDoI10XhOAACEvRkzZqhBgwaKj49X27ZttW7dOkvPmzdvnlwul/r16xfS+EgQAACOEhOiRzBef/11jRkzRhMmTNDGjRvVsmVLde/eXfv37w/4vJ07d+qBBx5Qx44dgzxi8EgQAACOEg4JwtNPP63bbrtNQ4cOVdOmTTVr1iydf/75euWVV0p9jsfj0cCBAzVp0iRddNFFQR4xeCQIAACUoxMnTig7O1sZGRneZTExMcrIyNDatWtLfd4jjzyi2rVra9iwYeURJpMUAQDOEspJivn5+T7L3W633G63z7KDBw/K4/GoTp06Psvr1Kmjbdu2lbj/1atX6+WXX9amTZvsCtkUFQQAAGySlpamxMRE7yMzM7PM+zx69Kh+//vf68UXX1StWrVsiNIaKggAAEcJ5bc55uTkKCEhwbv87OqBJNWqVUuxsbHat2+fz/J9+/YpJSXFb/vvvvtOO3fuVJ8+fbzLiouLJUmVKlXS119/rYsvvtiGs/BFBQEAAJskJCT4PEpKEOLi4pSenq7ly5d7lxUXF2v58uVq166d3/ZNmjTR5s2btWnTJu/jhhtuUJcuXbRp0yalpaWF5FyoIAAAHCUcvu55zJgxGjx4sFq3bq02bdpo2rRpKigo0NChQyVJgwYN0oUXXqjMzEzFx8erWbNmPs9PSkqSJL/ldiJBAACgnPXv318HDhzQ+PHjlZubq1atWikrK8s7cXHXrl2KianYIr/LMAyjPA+Yn5+vxMRESd0lnVeehwYAVLiTkpYoLy/P51p9eTj992eDpKo27/uYpNZShZxXqFBBAAA4Ct/FYE00nhMAACgjKggAAEdxyf5Px3bfNhkOqCAAAAA/VBAAAI7CHARrwjZBaNfunYDr16wuLqdIQs8V29eW/Tz4YOAx+78pjNnZqlULPGb5RxgzfzcFXGt4fm/TcSqeXWNmfPHXwBuE8F728mbf6wwVLWwTBAAAQoEKgjXReE4AAKCMqCAAAByFCoI10XhOAACgjKggAAAcJZRf9xxNSBAAAI4SDt/mGAm4xAAAAPyEbQXBrM/BmAfMc5unn4yee9itMOtz4Ip9ynQfhud+u8KJCGZ9Dlq0Mn+d/WeTs15nZn0OXLE/WNhHfbvCiQxmfQ4OHjTfR61a9sQCJilaFI3nBAAAyihsKwgAAIQCFQRrovGcAABAGVFBAAA4ChUEa6LxnAAAQBlRQQAAOIpL9n86plESAAARjksM1kTjOQEAgDKK2AqClSZIv7kpcP7z1gKnNbgxb4JUr0HgMdu101ljZqUJ0qAhgcfstdnOGjMrTZBcsatM9nGNXeFEBgtNkPYdCPw6q5PsrNdZWVBBsCYazwkAAJRRxFYQAAA4F1QQrInGcwIAAGVEBQEA4ChUEKyJxnMCAABlRAUBAOAoVBCsIUEAADgKCYI1QScIq1at0hNPPKHs7Gzt3btXCxcuVL9+/UIQWtmZ9Tl4/Anz/9KHHnTWvcVmfQ7eWmQ+Zr/p56wxM+tz8Pws8zG7+05njZlZnwOz3hKS8/pLmPU5WPOp+Zi1v8pZY4ayCTrpKSgoUMuWLTVjxoxQxAMAQEjFhOgRbYKuIPTo0UM9evQIRSwAACBMMAcBAOAozEGwJuQJQmFhoQoLC70/5+fnh/qQAACgjEKe9GRmZioxMdH7SEtLC/UhAQAolUuSy+Wy91HRJxUCIU8Qxo0bp7y8PO8jJycn1IcEAABlFPJLDG63W263O9SHAQDAmkqVJJfNn/kNQyoqsnefFSzoBOHYsWPavn279+cdO3Zo06ZNqlGjhurVq2drcAAA2I4EwZKgE4QNGzaoS5cu3p/HjBkjSRo8eLBmz55tW2DlwUoTpN8OCHwV5o15zmo8YqUJkit2a8D1hudyu8KJCFaaIHW7PvDr7IMsZ73OrDRBcsX+EnC94alsVzgRwUoTpPxjgV9nCVWd9TpDYEEnCJ07d5ZhGKGIBQCA0AtVBSHKROOtmwAAoIxolAQAcBYqCJZQQQAAAH6oIAAAnCU2Voqx+fNxcfRN8KSCAAAA/FBBAAA4S6VKVBAsIEEwYdbnYNAQ8xeZlXu6o4lZn4PHnzAfMys9KqKJWZ+Dx6aYj9nDY501ZmZ9Dtp3MB+zNaudNWZmfQ5e+6f5mA26NQrGjATBEi4xAAAAP1QQAADOQgXBEioIAADADxUEAICzxMaeetjJ47F3f2GACgIAAPBDBQEA4CyVKtlfQbC7dXMYoIIAAAD8UEEAADgLFQRLSBDKyEoTpMsup1BzJitNkK5ozZidyUoTpIsaMWZnstIEyRU7vxwiiRxWmiD97aUoeJ2RIFgSBf/TAADAblQQAADOQgXBEioIAADADxUEAICzxMaeqiIgICoIAADADykUAMBZKlWigmABFQQAAOAnbFMoV2zfig4h4jBmwWPMgseYBY8xCzNUECxhhAAAzkKCYAmXGAAAgB9SKACAs4TiNkfDsHd/YYAKAgAAFWDGjBlq0KCB4uPj1bZtW61bt67UbV988UV17NhR1atXV/Xq1ZWRkRFwezuQIAAAnOX0HAS7H0F4/fXXNWbMGE2YMEEbN25Uy5Yt1b17d+3fv7/E7VeuXKlbbrlFK1as0Nq1a5WWlqZu3bpp9+7ddoxIiVyGUb51kfz8fCUmJkrqLum88jw0AKDCnZS0RHl5eUpISCjXI5/++5N3/fVKOM/evz/5J08qMSvL8nm1bdtWV155paZPny5JKi4uVlpamkaNGqWxY8eaPt/j8ah69eqaPn26Bg0aVOb4S8IcBACAs4TiLob/ftbOz8/3Wex2u+V2u32WnThxQtnZ2Ro3bpx3WUxMjDIyMrR27VpLh/v555918uRJ1ahRo4yBl45LDAAA2CQtLU2JiYneR2Zmpt82Bw8elMfjUZ06dXyW16lTR7m5uZaO89BDDyk1NVUZGRm2xF0SKggAAGcJYQUhJyfH5xLD2dUDO0yZMkXz5s3TypUrFR8fb/v+TyNBAAA4SwgThISEBNM5CLVq1VJsbKz27dvns3zfvn1KSUkJ+Nwnn3xSU6ZM0bJly9SiRYuyxWyCSwwAAJSjuLg4paena/ny5d5lxcXFWr58udq1a1fq8/7v//5Pjz76qLKystS6deuQx0kFAQDgLKFolFRcHNTmY8aM0eDBg9W6dWu1adNG06ZNU0FBgYYOHSpJGjRokC688ELvHIbHH39c48eP19y5c9WgQQPvXIWqVauqatWq9p7Lf5EgAABQzvr3768DBw5o/Pjxys3NVatWrZSVleWduLhr1y7FxPyvyD9z5kydOHFCN910k89+JkyYoIkTJ4YkRvogAADKURj0QRg4UAlxcfbu+8QJJc6ZUyHnFSrMQQAAAH64xAAAcJZQ3MUQ5ByESEAFAQAA+AnbCsK8ee8EXN//5ujJ1lyxfW3Zj9Hqx8AbZGfbcpxwYNeYLVkS+HXWLYPX2dmuuy7wmH2QxZidbc6cwGP2uwGMWbmigmAJFQQAAOAnbCsIAACEBBUES0gQAADOEopGSR6PvfsLA1xiAAAAfqggAACcJRSXGKggAAAAJ6CCAABwFioIloRtgmDW52DfAfPiR53k6JtVGpBZn4PZs833MWSIHZFEDNM+BwcPmu+kVi17gokQZn0OThSZvzfjKjnrvWna52DLFvOdNGtmTzCARWGbIAAAEBJUECxhDgIAAPBDBQEA4Cyh6INQVGTv/sIACQIAwFlCcYnB7v2FAS4xAAAAP9GX8gAAEAgVBEuoIAAAAD/Rl/IAABAIFQRLIvaMrDRBWvxu4ALJDb2d1azFUhOkW28NvP6f/7QllIhhoQnSmk8Dv87aX+Ws15mVJkg1agUes8MHnTVmVpogvbUo8Jj9pp/DxgwhF7EJAgAA54QKgiXMQQAAAH6iL+UBACCQUDRKio21d39hgAQBAOAsXGKwhEsMAADAT/SlPAAABEIFwRIqCAAAwE9QKU9mZqbeeustbdu2TZUrV1b79u31+OOPq3HjxqGKr0zM+hzUTjHPj/bnOuzeYpM+B3ssTMRJjcLvRQ/ErM/BiSLz15mV3gHRxKzPwbPTzcfsnpHOGjOzPgdmfRKs7MMxqCBYElQF4aOPPtKIESP06aefaunSpTp58qS6deumgoKCUMUHAAAqQFApT1ZWls/Ps2fPVu3atZWdna1rrrnG1sAAAAgJbnO0pExzEPLy8iRJNWrUsCUYAAAQHs45hSouLtbo0aN19dVXq1mAPuKFhYUqLCz0/pyfn3+uhwQAoOyYg2DJOZ/RiBEjtGXLFq1evTrgdpmZmZo0adK5HgYAAHuRIFhyTpcYRo4cqXfffVcrVqxQ3bp1A247btw45eXleR85OTnnFCgAACg/QaU8hmFo1KhRWrhwoVauXKmGDRuaPsftdsvtdp9zgAAA2IoKgiVBndGIESM0d+5cvf3226pWrZpyc3MlSYmJiapcuXJIAgQAAOUvqARh5syZkqTOnTv7LH/11Vc1ZMgQu2IqN1aaID09LfBVmDGjndV4xFITpIyMwOuXLbMnmAhhqQnShg2B17dubU8wEcJSE6RNmwKvb9XKjlAihpUmSPsOBP59VifZIb/PqCBYEvQlBgAAEP2iL+UBACAQGiVZwpc1AQAAP1QQAADOwhwES6LvjAAACIQEwRIuMQAAAD/Rl/IAABAIFQRLou+MbGbW5+D5WeZFmLvvdMi9xaeZ9TkYPdp8H9Om2RFJ5DDpc3CiyPx1ZqnfQjQx6XPw+BPmY/bQg84aM7M+Bx8sMx+zbhnOGjMnI0EAADgLtzlawhwEAADghwoCAMBZmINgCRUEAADgJ/pSHgAAAqGCYEn0nREAAIGQIFjCJQYAAOAn+lIeAAACoYJgSfSdUTmz0gRpzAMUanxYaIJUZHpPcW9bQokUVpog7fqR19mZrDRBumc0Y3YmK02Qvt/JmDkFCQIAwFlolGQJqSAAAPBDBQEA4CzMQbCECgIAAPATfSkPAACBUEGwJPrOCACAQEgQLOESAwAA8BO2KY8rtm9FhxBxomvMyqfPQXSNWflgzILHmIUZbnO0hAoCAADwE7YVBAAAQoI5CJZQQQAAAH6iL+UBACAQKgiWUEEAAKACzJgxQw0aNFB8fLzatm2rdevWBdx+/vz5atKkieLj49W8eXO99957IY2PBAEA4CynKwh2P4Lw+uuva8yYMZowYYI2btyoli1bqnv37tq/f3+J269Zs0a33HKLhg0bps8//1z9+vVTv379tGXLFjtGpEQuwzCMkO29BPn5+UpMTJTUXdJ55XloAECFOylpifLy8pSQkFCuRz799yfvp59sP3Z+fr4Sq1e3fF5t27bVlVdeqenTp0uSiouLlZaWplGjRmns2LF+2/fv318FBQV69913vcuuuuoqtWrVSrNmzbLvRM5ABQEAAJvk5+f7PAoLC/22OXHihLKzs5WRkeFdFhMTo4yMDK1du7bE/a5du9Zne0nq3r17qdvbgQQBAOAoxYoJyUOS0tLSlJiY6H1kZmb6Hf/gwYPyeDyqU6eOz/I6deooNze3xJhzc3OD2t4O0TftEgCACpKTk+NzicHtdldgNGVDggAAcJSiolMPu/cpSQkJCaZzEGrVqqXY2Fjt27fPZ/m+ffuUkpJS4nNSUlKC2t4OXGIAAKAcxcXFKT09XcuXL/cuKy4u1vLly9WuXbsSn9OuXTuf7SVp6dKlpW5vByoIAABHCWUFwaoxY8Zo8ODBat26tdq0aaNp06apoKBAQ4cOlSQNGjRIF154oXcOw7333qtOnTrpqaeeUq9evTRv3jxt2LBBf/vb3+w9kTOQIAAAUM769++vAwcOaPz48crNzVWrVq2UlZXlnYi4a9cuxcT8r8jfvn17zZ07V3/+85/18MMP65JLLtGiRYvUrFmzkMVIHwQAQDmq+D4IOTn2Hzs/P19paYkVcl6hQgUBAOAo4XCJIRIwSREAAPgJ2wrCJ5+8E3B9+6uKyymS0HPF9rVlP/fdF3jMnn6SMTvbjBmBx+zuOxmzsz31VOAxGzOaMTub0cMTeIMz2udGOrvGLJQ8Hvs/8XtM/osjERUEAADgJ2wrCAAAhAJzEKyhggAAAPxQQQAAOAoVBGuoIAAAAD9UEAAAjkIFwRoqCAAAwE/YVhDM+hy4Yo+Z7sPwVLUrnIhg1udg/ETzfPCRidFzD7sVZn0OXLHfm+7D8FxkVzgRwazPwcZN5q+zK1o563Vm1ufg6dhY012MicYb7SsIfRCsCdsEAQCAUOASgzVcYgAAAH6oIAAAHIUKgjVUEAAAgB8qCAAAR6GCYA0VBAAA4IcKAgDAUaggWEMFAQAA+InYCoKVJkh/eylw/nP7cGc1a7HSBKleg8Bjtmuns8bMShMkV2zgDimGx7wJTjSx0gTpd7cGfp3N/aezXmdWmiD9aNJMqW40duoJERolWROxCQIAAOeCSwzWcIkBAAD4oYIAAHAUKgjWUEEAAAB+qCAAAByFCoI1VBAAAIAfKggAAEfhNkdrgkoQZs6cqZkzZ2rnzp2SpMsvv1zjx49Xjx49QhFbmZn1OXDF7jPdh+GpY1c4EcGsz8Hid82LTjf0dtY97GZ9Dtp3MB+zNaudNWamfQ7+9S/zndxyiz3BRAizPgf/MemTIEktovGvGEImqAShbt26mjJlii655BIZhqG///3v6tu3rz7//HNdfvnloYoRAADbMAfBmqAShD59+vj8/Ne//lUzZ87Up59+SoIAAIgIJAjWnPMcBI/Ho/nz56ugoEDt2rWzMyYAAFDBgk4QNm/erHbt2un48eOqWrWqFi5cqKZNm5a6fWFhoQoLC70/5+fnn1ukAADYgAqCNUHf5ti4cWNt2rRJn332me666y4NHjxYX375ZanbZ2ZmKjEx0ftIS0srU8AAACD0gk4Q4uLi1KhRI6WnpyszM1MtW7bUM888U+r248aNU15enveRk5NTpoABACiL0xUEux/Rpsx9EIqLi30uIZzN7XbL7XaX9TAAAKAcBZUgjBs3Tj169FC9evV09OhRzZ07VytXrtSSJUtCFR8AALaiUZI1QSUI+/fv16BBg7R3714lJiaqRYsWWrJkia677rpQxRdSVpoguWKzTfaRblc4EcFKE6TnZwW+cnX3nc5qCmSlCVK9BoHHzKyBVdSx0gTpyJHA65OS7IgkYlhpgvSGSTOl30bjXzmcs6AShJdffjlUcQAAUC64i8EavosBAOAoJAjW8G2OAADADxUEAICjUEGwhgoCAADwQwUBAOAo3OZoDRUEAADghwqCCbM+Bx8sM8+xumU46x52sz4HH640H7NrOztrzMz6HLwy23zM/jDEWWNm1ufgRJH5mMVVctaYmfY5eOkl850MH25PMBWIOQjWUEEAAAB+qCAAAByFCoI1JAgAAEchQbCGSwwAAMAPFQQAgKNQQbCGCgIAAPBDBQEA4Cg0SrKGCgIAAPBDBaGMrDRBcsVGYWpZBlaaIH2/k9z1TFaaILlij5VDJJHDUhOkbdtCH0gksdAE6enYWJMtetsTSwgVFUmmp3EO+4w2/BYGAAB+qCAAAByFCoI1JAgAAEchQbCGSwwAAMAPFQQAgKNwm6M1VBAAAIAfKggAAEcpKpJibP54HI1zEMI2QXDF9q3oECIOYxY8xix4jFnwomvMwr/PAewRtgkCAAChQAXBGuYgAAAAP1QQAACOQgXBGhIEAICjkCBYwyUGAADghwQBAOAopxsl2fkIZaOkw4cPa+DAgUpISFBSUpKGDRumY8dK//bWw4cPa9SoUWrcuLEqV66sevXq6Z577lFeXl5QxyVBAAAgjA0cOFBbt27V0qVL9e6772rVqlW6/fbbS91+z5492rNnj5588klt2bJFs2fPVlZWloYNGxbUcV2GYRhlDT4Y+fn5SkxMlNRd0nnleWgAQIU7KWmJ8vLylJCQUK5HPv33p1+/PJ13nr3HPnkyX4sWJdp+Xl999ZWaNm2q9evXq3Xr1pKkrKws9ezZUz/++KNSU1Mt7Wf+/Pm69dZbVVBQoEqVrE0/pIIAAECYWrt2rZKSkrzJgSRlZGQoJiZGn332meX9nE5crCYHEncxAAAcpqhIcrns36d0qkpxJrfbLbfbfc77zc3NVe3atX2WVapUSTVq1FBubq6lfRw8eFCPPvpowMsSJaGCAACATdLS0pSYmOh9ZGZmlrjd2LFj5XK5Aj62bdtW5njy8/PVq1cvNW3aVBMnTgzquVQQAACOEsoKQk5Ojs8chNKqB/fff7+GDBkScJ8XXXSRUlJStH///rOOVaTDhw8rJSUl4POPHj2q66+/XtWqVdPChQt13nnBzfsjQQAAOEooE4SEhARLkxSTk5OVnJxsul27du105MgRZWdnKz09XZL04Ycfqri4WG3bti31efn5+erevbvcbrcWL16s+Ph4aydyBi4xAAAQpi677DJdf/31uu2227Ru3Tp98sknGjlypAYMGOC9g2H37t1q0qSJ1q1bJ+lUctCtWzcVFBTo5ZdfVn5+vnJzc5WbmytPEA0bqCAAABzF47G/ghDKRklz5szRyJEj1bVrV8XExOjGG2/Us88+611/8uRJff311/r5558lSRs3bvTe4dCoUSOffe3YsUMNGjSwdFwSBAAAwliNGjU0d+7cUtc3aNBAZ7Y06ty5s+xocUSCAABwlFB8sVI0fllT2CYIxt9vDrzBrbeWTyDlwBXb15b9GK1+DLxBdrYtxwkHdo3ZCy+8E3D97cOLbTlOOLBrzH74IfCY1avLmJ1t1KjAY/bsNMYM4SdsEwQAAEKBCoI13MUAAAD8UEEAADgKFQRrqCAAAAA/VBAAAI4Sip4FoeyDUFFIEAAAjlJUJNnQJsBHNCYIXGIAAAB+wreCYNLnID821nQXCdGY0gVi1udg+nTzfYwcaU8sEcKsz4ErdpLpPgzPBLvCiQhmfQ4OHzH/3FEjKXru+7fCrM/BPaPNxyyaeiVUNCoI1lBBAAAAfsK3ggAAQAhQQbCGCgIAAPBDBQEA4ChUEKyhggAAAPxQQQAAOIrHY38FoTgKbzIhQQAAOEpRkRRjc/08GhMELjEAAAA/EVtBsNQEaezYwOunTLEnmEhhoQnSLpMGVPWicSZOAFaaILliA29jeMybLUUTK02QHpsS+LPJw2Oj8ONYAFaaIK1aHXjMrungrDErCyoI1lBBAAAAfiK2ggAAwLmggmANFQQAAOCHCgIAwFE8Hvs/8dt922Q4oIIAAAD8UEEAADhKUZHkctm7z2isIJAgAAAchQTBmjIlCFOmTNG4ceN07733atq0aTaFZCOTPgfFFq6wxCgKp6YGYNrnICPDfCfLltkTTIQw63MwaIj56+y12c56nZn1OfhsvfmYtb3SWWNm1ucgta75mO350VljhrI55wRh/fr1euGFF9SiRQs74wEAIKSoIFhzTpMUjx07poEDB+rFF19U9erV7Y4JAABUsHNKEEaMGKFevXopw0q5GQCAMFJUFJpHtAn6EsO8efO0ceNGrV+/3tL2hYWFKiws9P6cn58f7CEBAEA5C6qCkJOTo3vvvVdz5sxRfHy8pedkZmYqMTHR+0hLSzunQAEAsEexDMPeh6JwQntQCUJ2drb279+vK664QpUqVVKlSpX00Ucf6dlnn1WlSpXkKWEG/Lhx45SXl+d95OTk2BY8AAAIjaAuMXTt2lWbN2/2WTZ06FA1adJEDz30kGJL+Kpgt9stt9tdtigBALCN578Pu/cZXYJKEKpVq6ZmzZr5LKtSpYpq1qzptxwAgPBEgmCFozspWmqCNH164PUjR9oTTKSw0gSpd+/A6999155YIoSVJkiPPxH4at9DD0bf9c1ALDVBWrQo8Pp+/ewIJWJYaYLkiv0l4HrDU9mucBAFypwgrFy50oYwAAAoL1QQrODbHAEAgB9HX2IAADhRKG5LjL7LgFQQAACAHyoIAACHYQ6CFVQQAACAHyoIAACHKZb9n/ijbw4CCYIZkz4Hz043L8LcMzL6XjgBmfQ52FZCx82zNSmhbXc0M+tz4IpdYroPw9PdrnAig0mfg7cWmb83f9PPWe9Nsz4Hba4yH7N1n0bDmHGJwQouMQAAAD9UEAAADkMFwQoqCAAAwA8VBACAw9AoyQoqCAAAwA8VBACAwzAHwQoqCAAAwA8VBACAw1BBsIIEoYwsNUEaMCD0gUQQS02QZs8OeRyRxEoTJFfskNAHEkGsNEFKSKKIeiYrTZBuvzMaxowEwYpo+J8GAAA2o4IAAHAYvovBCioIAADADxUEAIDD0CjJCioIAADADxUEAIDDcBeDFVQQAACAn7CtILhi+1Z0CBGHMQseYxY8xix4jFm4oYJgRdgmCAAAhAYJghVcYgAAAH6oIAAAHIYKghVUEAAAgB8qCAAAhzFkf2Mjw+b9VTwqCAAAwA8JAgDAYTwheoTG4cOHNXDgQCUkJCgpKUnDhg3TsWPHLD3XMAz16NFDLpdLixYtCuq4JAgAAISxgQMHauvWrVq6dKneffddrVq1Srfffrul506bNk0ul+ucjsscBACAw0TOXQxfffWVsrKytH79erVu3VqS9Nxzz6lnz5568sknlZqaWupzN23apKeeekobNmzQBRdcEPSxqSAAABwmci4xrF27VklJSd7kQJIyMjIUExOjzz77rNTn/fzzz/rd736nGTNmKCUl5ZyOTQUBAACb5Ofn+/zsdrvldrvPeX+5ubmqXbu2z7JKlSqpRo0ays3NLfV59913n9q3b6++fc+9zTcVBACAw4SugpCWlqbExETvIzMzs8QIxo4dK5fLFfCxbdu2czq7xYsX68MPP9S0adPO6fmnUUEAAMAmOTk5SkhI8P5cWvXg/vvv15AhQwLu66KLLlJKSor279/vs7yoqEiHDx8u9dLBhx9+qO+++05JSUk+y2+88UZ17NhRK1euND0PiQQBAOA4xbK/UdKp/SUkJPgkCKVJTk5WcnKy6Xbt2rXTkSNHlJ2drfT0dEmnEoDi4mK1bdu2xOeMHTtWw4cP91nWvHlzTZ06VX369DE95mkkCAAAhKnLLrtM119/vW677TbNmjVLJ0+e1MiRIzVgwADvHQy7d+9W165d9dprr6lNmzZKSUkpsbpQr149NWzY0PKxmYMAAHCYyLmLQZLmzJmjJk2aqGvXrurZs6c6dOigv/3tb971J0+e1Ndff62ff/7Z1uNSQQAAIIzVqFFDc+fOLXV9gwYNZBiBvwvCbH1JwjhBeDzgWsPTpJziCD1X7LnfhnKmvXo34PoUT/R8HaldY2Y83jHwBg88YMtxwoFdY/byy+8EXP+HIXZf2604do2Z1DjgWsPzfzYdp+LZN2ahVCz7P/FHz+v+NC4xAAAAP2FcQQAAIBQip9VyRSJBAAA4TOhuc4wmXGIAAAB+qCAAAByGSwxWUEEAAAB+qCAAAByGCoIVYZsgmPU5eGyKefHj4bHRN2kkENM+BzNnmu/krrvsCSZSmPQ52BUba7qLelHUX8IK0z4Hx46Z76RqVXuCiRBmfQ6s9A4wPG/bFQ5gSdgmCAAAhAYVBCuYgwAAAPxQQQAAOAytlq0gQQAAOAyNkqzgEgMAAPBDBQEA4DBMUrSCCgIAAPBDBQEA4DBUEKyI2ATBShOk390auEAy95/RN6kkICtNkIYPD7z+pZfsiSVCWGqCNHly4PV//rM9wUQKC02Qdv0Y+L1Zr66z3ptWmiC5YgO/Fg2PeVMvIBgRmyAAAHBuqCBYwRwEAADghwoCAMBhqCBYQYIAAHAYGiVZwSUGAADghwoCAMBh+C4GK6ggAAAAP0FVECZOnKhJkyb5LGvcuLG2bdtma1B2Metz8No/zfOjQbdGX1YYkFmfgwULzPdx0032xBIpzPocFBWZ76OSs4p5Zn0OXLHLTPdheDLsCicimPU5iKlk/vusuMhhv89KxSRFK4L+rXT55Zdr2bL/vXkrOewXGwAAThD0X/dKlSopJSUlFLEAAFAOqCBYEfQchG+//Vapqam66KKLNHDgQO3atSsUcQEAgAoUVAWhbdu2mj17tho3bqy9e/dq0qRJ6tixo7Zs2aJq1aqV+JzCwkIVFhZ6f87Pzy9bxAAAlAkVBCuCShB69Ojh/XeLFi3Utm1b1a9fX2+88YaGDRtW4nMyMzP9JjYCAFBxSBCsKNNtjklJSbr00ku1ffv2UrcZN26c8vLyvI+cnJyyHBIAAJSDMiUIx44d03fffacLLrig1G3cbrcSEhJ8HgAAVJziED2iS1AJwgMPPKCPPvpIO3fu1Jo1a/TrX/9asbGxuuWWW0IVHwAAqABBzUH48ccfdcstt+jQoUNKTk5Whw4d9Omnnyo5OTlU8YWU45og2cFCE6TP1gfOO9te6bBxt9IrZOXKwOs7d7YjkohhpQmSK/YXk31UtiuciGClCZIrNvB1crNmTNGDVstWBJUgzJs3L1RxAACAMEIbRACAw3AXgxV8WRMAAPBDBQEA4DBUEKwgQQAAOEwobkuMvkmKXGIAAAB+qCAAAByGSwxWkCCU0bUZ5kWYD5dFX+kpENM+B3/+s/lOJk+2J5hIYdbn4OBB833UqmVLKJHCrM+BK3adhX20sSuciGDW58AVa1jYh8uucBDmSBAAAA5DBcEK5iAAAAA/VBAAAA5DBcEKKggAAMAPFQQAgMPwZU1WkCAAAByGRklWcIkBAAD4oYIAAHAYJilaQYJQRlaaIK1aTaHGh5UmSP/6V+jjiCRWmiAtWxb6OCKIlSZIe3J5b57JShOk393KmDkFCQIAwGGoIFhBKggAAPxQQQAAOAwVBCuoIAAAAD9UEAAADkMFwQoSBACAw9BJ0QouMQAAAD9hW0Fwxfat6BAiDmMWPMYseIxZ8BizcEOrZSuoIAAAAD9hW0EAACA0PLL/83H0TVKkggAAAPxQQQAAOAwVBCuoIAAAAD9UEAAADkMFwQoSBACAw3CboxXlniAYhvHffxWV96EBABXu1O/+//0tqLgYwn+fFavcE4SjR4/+91/Ly/vQAIAwcfToUSUmJpbrMePi4pSSkqLc3GUh2X9KSori4uJCsu+K4DLKOY0rLi7Wnj17VK1aNblcrvI8dED5+flKS0tTTk6OEhISKjqciMCYBY8xCx5jFrxwHjPDMHT06FGlpqYqJqb858kfP35cJ06cCMm+4+LiFB8fH5J9V4RyryDExMSobt265X1YyxISEsLuDRXuGLPgMWbBY8yCF65jVt6VgzPFx8dH1R/xUOI2RwAA4IcEAQAA+CFB+C+3260JEybI7XZXdCgRgzELHmMWPMYseIwZ7FDukxQBAED4o4IAAAD8kCAAAAA/JAgAAMAPCQIAAPBDgvBfM2bMUIMGDRQfH6+2bdtq3bp1FR1S2Fq1apX69Omj1NRUuVwuLVq0qKJDCnuZmZm68sorVa1aNdWuXVv9+vXT119/XdFhhbWZM2eqRYsW3mY/7dq10/vvv1/RYUWMKVOmyOVyafTo0RUdCiIUCYKk119/XWPGjNGECRO0ceNGtWzZUt27d9f+/fsrOrSwVFBQoJYtW2rGjBkVHUrE+OijjzRixAh9+umnWrp0qU6ePKlu3bqpoKCgokMLW3Xr1tWUKVOUnZ2tDRs26Nprr1Xfvn21devWig4t7K1fv14vvPCCWrRoUdGhIIJxm6Oktm3b6sorr9T06dMlnfq+iLS0NI0aNUpjx46t4OjCm8vl0sKFC9WvX7+KDiWiHDhwQLVr19ZHH32ka665pqLDiRg1atTQE088oWHDhlV0KGHr2LFjuuKKK/T8889r8uTJatWqlaZNm1bRYSECOb6CcOLECWVnZysjI8O7LCYmRhkZGVq7dm0FRoZolpeXJ+nUHzyY83g8mjdvngoKCtSuXbuKDiesjRgxQr169fL5nQaci3L/sqZwc/DgQXk8HtWpU8dneZ06dbRt27YKigrRrLi4WKNHj9bVV1+tZs2aVXQ4YW3z5s1q166djh8/rqpVq2rhwoVq2rRpRYcVtubNm6eNGzdq/fr1FR0KooDjEwSgvI0YMUJbtmzR6tWrKzqUsNe4cWNt2rRJeXl5WrBggQYPHqyPPvqIJKEEOTk5uvfee7V06VK+rRC2cHyCUKtWLcXGxmrfvn0+y/ft26eUlJQKigrRauTIkXr33Xe1atWqsP7a83ARFxenRo0aSZLS09O1fv16PfPMM3rhhRcqOLLwk52drf379+uKK67wLvN4PFq1apWmT5+uwsJCxcbGVmCEiDSOn4MQFxen9PR0LV++3LusuLhYy5cv51onbGMYhkaOHKmFCxfqww8/VMOGDSs6pIhUXFyswsLCig4jLHXt2lWbN2/Wpk2bvI/WrVtr4MCB2rRpE8kBgub4CoIkjRkzRoMHD1br1q3Vpk0bTZs2TQUFBRo6dGhFhxaWjh07pu3bt3t/3rFjhzZt2qQaNWqoXr16FRhZ+BoxYoTmzp2rt99+W9WqVVNubq4kKTExUZUrV67g6MLTuHHj1KNHD9WrV09Hjx7V3LlztXLlSi1ZsqSiQwtL1apV85vTUqVKFdWsWZO5LjgnJAiS+vfvrwMHDmj8+PHKzc1Vq1atlJWV5TdxEads2LBBXbp08f48ZswYSdLgwYM1e/bsCooqvM2cOVOS1LlzZ5/lr776qoYMGVL+AUWA/fv3a9CgQdq7d68SExPVokULLVmyRNddd11FhwY4An0QAACAH8fPQQAAAP5IEAAAgB8SBAAA4IcEAQAA+CFBAAAAfkgQAACAHxIEAADghwQBAAD4IUEAAAB+SBAAAIAfEgQAAOCHBAEAAPj5f+vYL7myExc5AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def plot_as_patches(j):\n", " # Reorder axes so the diagonals will each form a contiguous patch.\n", " j = tf.transpose(j, [1, 0, 3, 2])\n", " # Pad in between each patch.\n", " lim = tf.reduce_max(abs(j))\n", " j = tf.pad(j, [[0, 0], [1, 1], [0, 0], [1, 1]],\n", " constant_values=-lim)\n", " # Reshape to form a single image.\n", " s = j.shape\n", " j = tf.reshape(j, [s[0]*s[1], s[2]*s[3]])\n", " imshow_zero_center(j, extent=[-0.5, s[2]-0.5, s[0]-0.5, -0.5])\n", "\n", "plot_as_patches(j)\n", "_ = plt.title('All (batch, batch) slices are diagonal')" ] }, { "cell_type": "markdown", "metadata": { "id": "OXpTBKyeK84z" }, "source": [ "要获取所需结果，您可以对重复的 `batch` 维度求和，或者使用 `tf.einsum` 选择对角线：" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.942587Z", "iopub.status.busy": "2022-12-14T22:53:34.942002Z", "iopub.status.idle": "2022-12-14T22:53:34.948481Z", "shell.execute_reply": "2022-12-14T22:53:34.947786Z" }, "id": "v65OAjEgLQwl" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(7, 6, 5)\n", "(7, 6, 5)\n" ] } ], "source": [ "j_sum = tf.reduce_sum(j, axis=2)\n", "print(j_sum.shape)\n", "j_select = tf.einsum('bxby->bxy', j)\n", "print(j_select.shape)" ] }, { "cell_type": "markdown", "metadata": { "id": "zT_VfR6lcwxD" }, "source": [ "没有额外维度时，计算会更加高效。`tf.GradientTape.batch_jacobian` 方法就是如此运作的：" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:34.952042Z", "iopub.status.busy": "2022-12-14T22:53:34.951485Z", "iopub.status.idle": "2022-12-14T22:53:35.102117Z", "shell.execute_reply": "2022-12-14T22:53:35.101534Z" }, "id": "YJLIl9WpHqYq" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 5 calls to .f at 0x7f85281da0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "data": { "text/plain": [ "TensorShape([7, 6, 5])" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "jb = tape.batch_jacobian(y, x)\n", "jb.shape" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:35.105361Z", "iopub.status.busy": "2022-12-14T22:53:35.104834Z", "iopub.status.idle": "2022-12-14T22:53:35.110521Z", "shell.execute_reply": "2022-12-14T22:53:35.109932Z" }, "id": "-5t_q5SfHw7T" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.0\n" ] } ], "source": [ "error = tf.reduce_max(abs(jb - j_sum))\n", "assert error < 1e-3\n", "print(error.numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "IUeY2ZCiL31I" }, "source": [ "小心：`tf.GradientTape.batch_jacobian` 只验证源和目标的第一维是否匹配，并不会检查梯度是否独立。用户需要确保仅在合理条件下使用 `batch_jacobian`。例如，添加 `tf.keras.layers.BatchNormalization` 将破坏独立性，因为它在 `batch` 维度进行了归一化：" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:35.113759Z", "iopub.status.busy": "2022-12-14T22:53:35.113287Z", "iopub.status.idle": "2022-12-14T22:53:35.456923Z", "shell.execute_reply": "2022-12-14T22:53:35.456301Z" }, "id": "tnDugVc-L4fj" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:6 out of the last 6 calls to .f at 0x7f85a003df70> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "j.shape: (7, 6, 7, 5)\n" ] } ], "source": [ "x = tf.random.normal([7, 5])\n", "\n", "layer1 = tf.keras.layers.Dense(8, activation=tf.nn.elu)\n", "bn = tf.keras.layers.BatchNormalization()\n", "layer2 = tf.keras.layers.Dense(6, activation=tf.nn.elu)\n", "\n", "with tf.GradientTape(persistent=True, watch_accessed_variables=False) as tape:\n", " tape.watch(x)\n", " y = layer1(x)\n", " y = bn(y, training=True)\n", " y = layer2(y)\n", "\n", "j = tape.jacobian(y, x)\n", "print(f'j.shape: {j.shape}')" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:35.460383Z", "iopub.status.busy": "2022-12-14T22:53:35.459793Z", "iopub.status.idle": "2022-12-14T22:53:35.681924Z", "shell.execute_reply": "2022-12-14T22:53:35.681313Z" }, "id": "SNyZ1WhJMVLm" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhEAAAIkCAYAAACgKXkMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABnkklEQVR4nO3de1xU1doH8N8AchMGRG4iKCqmoiIJR8T7UQLUTEtPWnoU82gXTT12Uc9balp5rexi2k3N0lfTo5WeDt7RvCtmpqmFSeIFFBRGIK6z3z96nZyAtYZhD4zu3/fzmU8xz+LZa29mcPHsvZ/RKYqigIiIiKiaHOp6AkRERHR34iKCiIiIrMJFBBEREVmFiwgiIiKyChcRREREZBUuIoiIiMgqXEQQERGRVbiIICIiIqtwEUFERERW4SLiLpaSkgKdTocNGzbU9VRUtXLlSuh0OqSnp5ue69WrF3r16lVnc6Lq0eLP8F7fP6LKcBFhZ3Q6nUWPlJSUup4q3YMKCwsxa9Ysvr6IyCJOdT0BMvfZZ5+Zfb1q1Sps3769wvNt2rTBmTNnanNqdWrbtm11PQVNKCwsxCuvvAIAqv9VzZ8h0b2Hiwg7M2LECLOvDx06hO3bt1d4HoCmFhHOzs51PQXVFRYWwt3dva6nUWvuxZ8hkdbxdMY9wGg04rXXXkNwcDBcXV3Rp08fpKWlVRh3+PBhJCYmwsvLC+7u7ujZsyf2799vNubWrVuYPHkyQkND4eLiAn9/fzzwwAM4fvx4tXNV5d1330Xbtm3h7u6OBg0aIDo6GmvWrBF+T2Xnm4uKijBr1izcd999cHV1RaNGjfDII4/g/PnzZsdm8eLFaNu2LVxdXREQEIAnn3wSN2/eNMt17NgxJCQkwNfXF25ubmjWrBmeeOIJ6b589dVX6N+/P4KCguDi4oIWLVpgzpw5KC8vrzD/du3aITU1FT169IC7uzv+9a9/AQCKi4sxc+ZMhIWFwcXFBSEhIXjxxRdRXFws3f7tvD/++CP++te/wt3dHY0bN8aCBQsqjL127RrGjBmDgIAAuLq6okOHDvj0009N8fT0dPj5+QEAXnnlFdOps1mzZgnncPr0afTu3Rtubm4IDg7Gq6++CqPRWOlc7/wZlpSUYMaMGYiKioKXlxfq16+P7t27Y/fu3RW+NycnB3//+9+h1+vh7e2NUaNG4fvvv4dOp8PKlSvNxu7atQvdu3dH/fr14e3tjYEDB1ZYcM+aNQs6nQ5paWlISkqCt7c3vLy8MHr0aBQWFpqNXbFiBXr37g1/f3+4uLggPDwcS5cuFR4TIq1gJeIeMG/ePDg4OOD5559HXl4eFixYgOHDh+Pw4cOmMbt27ULfvn0RFRWFmTNnwsHBwfTL8dtvv0WnTp0AAE899RQ2bNiACRMmIDw8HDk5Odi3bx/OnDmDjh07VitXZT766CNMnDgRQ4YMwaRJk1BUVISTJ0/i8OHDePzxxy3e5/Lycjz44IPYuXMnhg0bhkmTJuHWrVvYvn07Tp06hRYtWgAAnnzySaxcuRKjR4/GxIkTceHCBbz33nv47rvvsH//ftSrVw/Xrl1DfHw8/Pz8MG3aNHh7eyM9PR0bN26UzmPlypXw8PDAlClT4OHhgV27dmHGjBkwGAxYuHCh2dicnBz07dsXw4YNw4gRIxAQEACj0YiHHnoI+/btw7hx49CmTRv88MMPeOutt/DTTz/hyy+/lM7h5s2bSExMxCOPPIJHH30UGzZswNSpU9G+fXv07dsXAPDbb7+hV69eSEtLw4QJE9CsWTOsX78eSUlJyM3NxaRJk+Dn54elS5fi6aefxsMPP4xHHnkEABAREVHltjMzM/HXv/4VZWVlmDZtGurXr48PP/wQbm5u0nkbDAZ8/PHHeOyxxzB27FjcunULn3zyCRISEnDkyBFERkYC+H0hOGDAABw5cgRPP/00Wrduja+++gqjRo2qkHPHjh3o27cvmjdvjlmzZuG3337Du+++i65du+L48eMIDQ01G//oo4+iWbNmmDt3Lo4fP46PP/4Y/v7+mD9/vmnM0qVL0bZtWzz00ENwcnLC5s2b8cwzz8BoNGL8+PHS/SS6pylk18aPH69U9WPavXu3AkBp06aNUlxcbHr+7bffVgAoP/zwg6IoimI0GpWWLVsqCQkJitFoNI0rLCxUmjVrpjzwwAOm57y8vJTx48dXOZ/q5KrMwIEDlbZt2wrHrFixQgGgXLhwwfRcz549lZ49e5q+Xr58uQJAefPNNyudo6IoyrfffqsAUFavXm0WT05ONnt+06ZNCgDl6NGjwnlVprCwsMJzTz75pOLu7q4UFRWZzR+AsmzZMrOxn332meLg4KB8++23Zs8vW7ZMAaDs379fuP3beVetWmV6rri4WAkMDFQGDx5sem7x4sUKAOXzzz83PVdSUqLExsYqHh4eisFgUBRFUa5fv64AUGbOnCnfeUVRJk+erABQDh8+bHru2rVripeXl/RnWFZWZva6VRRFuXnzphIQEKA88cQTpuf+/e9/KwCUxYsXm54rLy9XevfurQBQVqxYYXo+MjJS8ff3V3JyckzPff/994qDg4MycuRI03MzZ85UAJhtR1EU5eGHH1YaNmxo9lxlP+OEhASlefPmZs/9ef+ItICnM+4Bo0ePNjvf3L17dwDAL7/8AgA4ceIEfv75Zzz++OPIyclBdnY2srOzUVBQgD59+mDv3r2m8rO3tzcOHz6MK1euVLqt6uSqjLe3Ny5duoSjR4/WaJ///e9/w9fXF88++2yFmE6nAwCsX78eXl5eeOCBB0zzzM7ORlRUFDw8PExlc29vbwDAli1bUFpaWq153PkX961bt5CdnY3u3bujsLAQZ8+eNRvr4uKC0aNHmz23fv16tGnTBq1btzabY+/evQGg0tL+n3l4eJhdM+Ps7IxOnTqZfv4A8M033yAwMBCPPfaY6bl69eph4sSJyM/Px549e6q133fm7dy5s1n1yc/PD8OHD5d+r6Ojo+l1azQacePGDZSVlSE6Otrs9FlycjLq1auHsWPHmp5zcHCoUAW4evUqTpw4gaSkJPj4+Jiej4iIwAMPPIBvvvmmwhyeeuops6+7d++OnJwcGAwG03N3/ozz8vKQnZ2Nnj174pdffkFeXp50P4nuZTydcQ9o0qSJ2dcNGjQAANN5/59//hkAKi3/3paXl4cGDRpgwYIFGDVqFEJCQhAVFYV+/fph5MiRaN68ebVzVWbq1KnYsWMHOnXqhLCwMMTHx+Pxxx9H165dLdzb350/fx6tWrWCk1PVL+Gff/4ZeXl58Pf3rzR+7do1AEDPnj0xePBgvPLKK3jrrbfQq1cvDBo0CI8//jhcXFyE8zh9+jReeukl7Nq1y+wfHgAV/oFp3LhxhYsLf/75Z5w5c8Z0LUJVcxQJDg42LZxua9CgAU6ePGn6+tdff0XLli3h4GD+d0ObNm1McWv8+uuviImJqfB8q1atLPr+Tz/9FG+88QbOnj1rtoBr1qyZ2TYaNWpU4SLUsLCwCnOpattt2rTB1q1bUVBQgPr165ueF7139Ho9AGD//v2YOXMmDh48WOF6iby8PHh5eVm0r0T3Ii4i7gGOjo6VPq8oCgCYKgMLFy40nWf+Mw8PDwC/nyPu3r07Nm3ahG3btmHhwoWYP38+Nm7ciL59+1YrV2XatGmDc+fOYcuWLUhOTsa///1vvP/++5gxY4bp1kK1GI1G+Pv7Y/Xq1ZXGb//Dfbth16FDh7B582Zs3boVTzzxBN544w0cOnSoyv3Jzc1Fz549odfrMXv2bLRo0QKurq44fvw4pk6dWqEiU9l1AkajEe3bt8ebb75Z6TZCQkKk+yn7+durzz//HElJSRg0aBBeeOEF+Pv7w9HREXPnzjW7ONaWZMfu/Pnz6NOnD1q3bo0333wTISEhcHZ2xjfffIO33npLWHUj0gIuIjTg9kWGer0ecXFx0vGNGjXCM888g2eeeQbXrl1Dx44d8dprr6Fv377VzlWZ+vXrY+jQoRg6dChKSkrwyCOP4LXXXsP06dPh6upq8T4dPnwYpaWlqFevXpVjduzYga5du1p0oV/nzp3RuXNnvPbaa1izZg2GDx+OtWvX4h//+Eel41NSUpCTk4ONGzeiR48epucvXLhg0T7cnuP333+PPn36VKgmqKlp06Y4efIkjEajWTXi9imXpk2bAkC159C0aVNTdepO586dk37vhg0b0Lx5c2zcuNFsuzNnzqywjd27d1e4JfbPdyDd3ofKtn327Fn4+vqaVSEssXnzZhQXF+Prr782q1pYcpqJSAt4TYQGREVFoUWLFli0aBHy8/MrxK9fvw7g9zse/lyC9/f3R1BQkOl2Q0tzVSUnJ8fsa2dnZ4SHh0NRlGpdjzB48GBkZ2fjvffeqxC7/Vfko48+ivLycsyZM6fCmLKyMuTm5gL4vXT957/ab1dZRLdZ3v4r9s7vLSkpwfvvv2/xfjz66KO4fPkyPvroowqx3377DQUFBRbnEunXrx8yMzOxbt0603NlZWV499134eHhgZ49ewKA6R/p28fGkryHDh3CkSNHTM9dv369yurPnSo7focPH8bBgwfNxiUkJKC0tNTsGBmNRixZssRsXKNGjRAZGYlPP/3UbP6nTp3Ctm3b0K9fP4v2STbHvLw8rFixotq5iO5FrERogIODAz7++GP07dsXbdu2xejRo9G4cWNcvnwZu3fvhl6vx+bNm3Hr1i0EBwdjyJAh6NChAzw8PLBjxw4cPXoUb7zxRrVyVSU+Ph6BgYHo2rUrAgICcObMGbz33nvo378/PD09Ld6nkSNHYtWqVZgyZQqOHDmC7t27o6CgADt27MAzzzyDgQMHomfPnnjyyScxd+5cnDhxAvHx8ahXrx5+/vlnrF+/Hm+//TaGDBmCTz/9FO+//z4efvhhtGjRArdu3cJHH30EvV4v/IenS5cuaNCgAUaNGoWJEydCp9Phs88+q9ZphL///e/44osv8NRTT2H37t3o2rUrysvLcfbsWXzxxRfYunUroqOjLc5XlXHjxuGDDz5AUlISUlNTERoaig0bNmD//v1YvHix6di7ubkhPDwc69atw3333QcfHx+0a9cO7dq1qzTviy++iM8++wyJiYmYNGmS6RbP25UPkQcffBAbN27Eww8/jP79++PChQtYtmwZwsPDzRaogwYNQqdOnfDcc88hLS0NrVu3xtdff40bN24AMK+eLFy4EH379kVsbCzGjBljusXTy8tL2u+iMvHx8XB2dsaAAQPw5JNPIj8/Hx999BH8/f1x9erVaucjuufU1W0hZBlLbvFcv3692fMXLlyocOuboijKd999pzzyyCNKw4YNFRcXF6Vp06bKo48+quzcuVNRlN9vDXzhhReUDh06KJ6enkr9+vWVDh06KO+//36FbctyVeWDDz5QevToYfq+Fi1aKC+88IKSl5dnGmPJLZ6K8vutd//zP/+jNGvWTKlXr54SGBioDBkyRDl//rzZuA8//FCJiopS3NzcFE9PT6V9+/bKiy++qFy5ckVRFEU5fvy48thjjylNmjRRXFxcFH9/f+XBBx9Ujh07JtwXRVGU/fv3K507d1bc3NyUoKAg5cUXX1S2bt2qAFB2795tNv+qbm0tKSlR5s+fr7Rt21ZxcXFRGjRooERFRSmvvPKK2XGpTFV5R40apTRt2tTsuaysLGX06NGKr6+v4uzsrLRv377Ca0RRFOXAgQNKVFSU4uzsbNHtnidPnlR69uypuLq6Ko0bN1bmzJmjfPLJJ9KfodFoVF5//XWladOmiouLi3L//fcrW7ZsqXTu169fVx5//HHF09NT8fLyUpKSkpT9+/crAJS1a9eajd2xY4fStWtXxc3NTdHr9cqAAQOUH3/80WzM7Vs8r1+/bvZ8Za+9r7/+WomIiFBcXV2V0NBQZf78+aZbjGWvUaJ7nU5R7PzqKyKiSnz55Zd4+OGHsW/fvmrf3UNE6uAigojs3m+//WZ2cWx5eTni4+Nx7NgxZGZmWnThLBGpj9dEEJHde/bZZ/Hbb78hNjYWxcXF2LhxIw4cOIDXX3+dCwiiOsRKBBHZvTVr1uCNN95AWloaioqKEBYWhqeffhoTJkyo66kRaRoXEURERGQV9okgIiIiq3ARQURERFap9QsrjUYjrly5Ak9PT5u2+SUiIvujKApu3bqFoKCgCh8IVxuKiopQUlJik9zOzs4Wt+6/V9T6IuLKlSsWfagQERHduzIyMhAcHFyr2ywqKoKfmxsqNuxXR2BgIC5cuKCphUStLyL+aG3cpy42T0REdaoMwM5qtblXS0lJCfIB/BOAi8q5iwG8lZmJkpISLiJs6Y9TGE4AKv/0RSIiurfV5elsNwBq/zOv1QsMtbrfREREVEM8n0BERJriAPX/gtbqX+RcRBARkaZwEaEere43ERER1RArEUREpCmsRKhHq/tNRERENWS3lYhDhzYL4zF/McqT5OYKwyUePtIUzrnX5NuRMLj6C+NeXgNqvA0AmDdPfMymPvyTPElgoDguOaYAANk90k7yl53BSfyzUeuYffCB+JiNG1EoT1JWJgwbPfTSFA75BvGAzEz5PHx9hWFdw1HyHBbIyREfM0tukZe9BCSHFADg7ir5HZCdLU8i2ZCu8ZPyHBZQ/j1aGDcOekSaI1/SIUnvKu/CeCPfWRi35JDJ+kPVr6/Oe9OWWIlQj1b3m4iIiGrIbisRREREtsBKhHq0ut9ERERUQ6xEEBGRprASoR6t7jcRERHVECsRRESkKaxEqIeLCCIi0hQd1P9Hv+4+k7Ru2e0iIuZ+8T3PH34svt8ZAMY9WCSMZ0ruuwaAJmUWDJLQe3jUOIclpvY9KYz/a2WENMerr4rjDpcuySfi7S2OW3AzukeovIeHGsYNE/dn+HqHvMfDQ50lvUQs6BOBIvFrVXpMARi9a+eYyV7Osl0BAHeI+2/kF7nLc8jemxa87woh344aSh4U94Fw3vK1NEdu5EPCuN5D/rsqN1f8GrHkV5UlfUBIO+x2EUFERGQLOqhfOdBqJUKrp3GIiIiohliJICIiTXH8/4faObWIlQgiIiKyCisRRESkKbzFUz1a3W8iIiKqIVYiiIhIU1iJUA8XEUREpClcRKjHfhcR+eLGKbJGUgDw0FNBwvjXXxrl80grE8eDg+U5du+Wj1FDaKgwLGskBQDTponjC16yYH/T0sTxwEBpCoezP8q3o4Yy8c/3oW43pClGTvYXxj/+WD4NZ8nr3ZIuQA779so3pALZVH1cxY2kAOBKrrjJUxCuyCciacAl24bF21GB7Jihm7iRFAA0GRYvjBuTt0lzFGWK4xb0NENysnwMaYf9LiKIiIhsgJUI9Wh1v4mIiKiGWIkgIiJNYSVCPVrdbyIiIqohViKIiEhTWIlQj1b3m4iIiGqIlQgiItIUfhS4eqxaRCxZsgQLFy5EZmYmOnTogHfffRedOnVSe25CxkBxDwjAgj4Qly7JNyTraZCbK03xY7P+khHL5POwhGQuDtnZ0hQLXhX3gfhmh16ao1+v1uIB0pvmAUNwuHSMKpxqvo5euVIcd8g3yJPIXmeSfhYA8EtwD8mIhfJ5WMDVVRwvhAX9GTzEx8QA+ftb7yp+f1vQjgRGC7ajBtnLTHZMAaBki7gPxM3r8hyytjYWvMzQL+wn+SA75wD1P3VTq2X9au/3unXrMGXKFMycORPHjx9Hhw4dkJCQgGvXrtlifkRERGSnqr2IePPNNzF27FiMHj0a4eHhWLZsGdzd3bF8+XJbzI+IiEhVDjZ6aFG19rukpASpqamIi4v7I4GDA+Li4nDw4EHVJ0dERET2q1onhLOzs1FeXo6AgACz5wMCAnD27NlKv6e4uBjFxcWmrw0GC84PExER2Qhv8VSPzfd77ty58PLyMj1CQkJsvUkiIiKqBdVaRPj6+sLR0RFZWVlmz2dlZSGwikuhp0+fjry8PNMjIyPD+tkSERHVEK+JUE+19tvZ2RlRUVHYuXOn6Tmj0YidO3ciNja20u9xcXGBXq83exAREWndkiVLEBoaCldXV8TExODIkSNVju3Vqxd0Ol2FR//+f7QQSEpKqhBPTEy06T5U+yb5KVOmYNSoUYiOjkanTp2wePFiFBQUYPTo0baYHxERkars4ZqI2+0Sli1bhpiYGCxevBgJCQk4d+4c/P39K4zfuHEjSkpKTF/n5OSgQ4cO+Nvf/mY2LjExEStWrDB97eLiUs2ZVU+1FxFDhw7F9evXMWPGDGRmZiIyMhLJyckVLras+czEU3PItqAvhaypUWioNMUjQ8QvjY0r5dMIP7tLPkgNso41Hh7yHOnpwnC/bvIOPj0SxdWmvZ/Lm17pc2upoY2sC5AF3XccLl0UD5B1+AHwr5fEr7PXp8kvSG5e9KN0jBpkh8y5yIKLpyXvTb0FL9WLl8SvM29veQ69a4l8kApkx6yoSJ5D1huvtaTHGwA4DHlEGDes3CjNcSD7PvmG7Jw9LCLubJcAAMuWLcN//vMfLF++HNOmTasw3sfHx+zrtWvXwt3dvcIiwsXFpcrLC2zBquM4YcIE/PrrryguLsbhw4cRExOj9ryIiIjuOgaDwexx592Jt6nRLuGTTz7BsGHDUL9+fbPnU1JS4O/vj1atWuHpp59GTk5OzXZIQqvXghARkUbZ8sLKkJAQszsS586dW2H7onYJmZmZ0vkfOXIEp06dwj/+8Q+z5xMTE7Fq1Srs3LkT8+fPx549e9C3b1+Ul5dbcliswg/gIiIiUklGRobZDQS2uCbhk08+Qfv27St8ZtWwYcNM/9++fXtERESgRYsWSElJQZ8+fVSfB8BKBBERaYzORg8AFe5GrGwRYU27hNsKCgqwdu1ajBkzRrqfzZs3h6+vL9LS0qRjrcVFBBERUS2ypl3CbevXr0dxcTFGjBgh3c6lS5eQk5ODRo0a1XjOVeEigoiINMXRRo/qmDJlCj766CN8+umnOHPmDJ5++mmzdgkjR47E9OnTK3zfJ598gkGDBqFhw4Zmz+fn5+OFF17AoUOHkJ6ejp07d2LgwIEICwtDQkJCNWdnOV4TQUREVMtk7RIuXrwIBwfzv/PPnTuHffv2Ydu2bRXyOTo64uTJk/j000+Rm5uLoKAgxMfHY86cOTbtFaFTFEWxWfZKGAwGeHl5AUgAUK/KcUrBOnEiC+7flzF6yLtnOuTeEMY3pvgI4wDwSGKhMK6rP1SawxJK3mrxAEtuRpfd0C6LA9IeAPM/C5KmmPpP8f37OpfB8nlYQLn8gXiAJc0GJK9Fi15n+eLeCm9+LM8xZYi4X4Wu6XhpDktIX2cW9CMpLBIXQWUtTywhaxMDAPoy8ftb13BUzScC4PLlzcK4Ci8zi9rAyI6Jfu2H0hxZA8cJ44GBAyQZSgFsRV5eXq13ML7978+/AdSXjq6eAgCDgTrZr7rESgQREWmKPTSbuldodb+JiIiohliJICIiTWElQj1a3W8iIiKqIVYiiIhIU1iJUI9W95uIiIhqiJUIIiLSFFYi1KPV/SYiIqIasttKxI0id2HcJ1/cWAcAcPasMJzfOV6aQn/pkjCemChvNrVus3hf1FLoJG5w4uQtb4DifOq4eButO0pzuKefEsanjpV3Ejp+Sn5cVSHpamR0lf/sHP77H2E8N7a/NIdPtvjjf6cMyZXmmP+/TaRjVCF5T0DyAUIAUOYk/vnKmm8BkP7s9PnZ0hTbTskbn6lB1kzKkj5wJ06I472j5cfs0iXx74DgYeJGUgAQsGyBdIy9YyVCPXa7iCAiIrKFOz91U82cWqTVxRMRERHVECsRRESkKdZ86qYlObWIlQgiIiKyCisRRESkKbywUj1a3W8iIiKqIVYiiIhIU3RQ/y9ord6doVMURanNDRoMBnh5eQFIAFCvNjdNRER1rhTAVuTl5UGvl/euUdPtf392AfBQOXc+gN5AnexXXWIlgoiINIXXRKiHiwgiItIULiLUo9X9JiIiohpiJYKIiDSFlQj1aHW/iYiIqIZYiSAiIk1hJUI9Wt1vIiIiqiFWIoiISFP4UeDqYSWCiIiIrMJKBBERaQo/Clw9XEQQEZGm8MJK9Wh1v4mIiKiGWIkgIiJNYSVCPVrdbyIiIqohViKIiEhTdFD/L2je4klERERUDaxEEBGRpvCaCPXY7SLiP//ZLIz3U/4jzXGxfX9hvInHDflEioqEYYNHkDRFWZk43rDhAPk8LJCBLcK4Pk+R5jh2TByX7QsAxMWJ459/Ls8RGSmOd+igzjH79Vfx66yJ0xVpjp/yxa8BDw/5PIICjeIBktchACAtTRjWdfgfeQ6LjBBGldSW8hSy/Tl1Sp5jyBBxfMMGeQ7Ji1XXYpI8hwXOSd6b+any92bjxuJ4To58Hg0biuM7d8pz+PqK4wkJ6rw36e5gt4sIIiIiW2AlQj1a3W8iIiKqIVYiiIhIU1iJUA8XEUREpCn8FE/1aHXxRERERDXESgQREWkKP8VTPaxEEBERkVXsthLRL65EGL+SLe4BAQBNpPfeu0pzGL19hHF9mXieAGB0cpaOUUNQufhecwcL5tq7l/glYVRh3TlyhOTnotJ2LNEkWDwXg6QHBADc5ys/rjJGiF8jDk7yt6qxXUSN52EJpfhhYfxGvvz17u0tjjtER1djRlX4xz9qnkMlocXi96ZztrwfidFP/Fr09JTPw1XyK0/WegMALHgp2j1eWKkere43ERER1dA9sKYkIiKyHCsR6tHqfhMREVENsRJBRESawkqEeriIICIiTdFB/X/02WyKiIiIqBpYiSAiIk3h6Qz1aHW/iYiIqIbsthJhKBI3rAmCvDnLjVxxc5bkZHdpjseV1cL4xe7DpTmalP0iHaOG5GRxvF+3ImmOX7y8hPEdH4ib5gDyHj8zZsnXrpMnS4eo4kqmeC5B2SflScrKhOH3D3WUpngm7idh/ED2fdIcXcKuSceo4tgxYbgotIs0haPjJmFceXq7fB4jRojjknkCAOLi5GNUcPasOO7hIW9q1txbL4wXpRukOVJSxPGGDaUp0LKlfIy9s5dKxJIlS7Bw4UJkZmaiQ4cOePfdd9GpU6dKx65cuRKjR482e87FxQVFRX/8XlcUBTNnzsRHH32E3NxcdO3aFUuXLkVLG/7QWIkgIiKqZevWrcOUKVMwc+ZMHD9+HB06dEBCQgKuXav6jwG9Xo+rV6+aHr/++qtZfMGCBXjnnXewbNkyHD58GPXr10dCQoLZQkNtXEQQEZGmONjoUR1vvvkmxo4di9GjRyM8PBzLli2Du7s7li9fXuX36HQ6BAYGmh4BAQGmmKIoWLx4MV566SUMHDgQERERWLVqFa5cuYIvv/yymrOzHBcRREREKjEYDGaP4uLiCmNKSkqQmpqKuDtOpzk4OCAuLg4HDx6sMnd+fj6aNm2KkJAQDBw4EKdPnzbFLly4gMzMTLOcXl5eiImJEeasKS4iiIhIU2xZiQgJCYGXl5fpMXfu3Arbz87ORnl5uVklAQACAgKQmZlZ6ZxbtWqF5cuX46uvvsLnn38Oo9GILl264NKlSwBg+r7q5FSD3V5YSUREZAu2vLAyIyMDev0fF8G6uLiokj82NhaxsbGmr7t06YI2bdrggw8+wJw5c1TZhjVYiSAiIlKJXq83e1S2iPD19YWjoyOysrLMns/KykJgYKBF26lXrx7uv/9+pKWlAYDp+2qS0xpcRBARkabU9YWVzs7OiIqKws6dO03PGY1G7Ny506zaIFJeXo4ffvgBjRo1AgA0a9YMgYGBZjkNBgMOHz5scU5rVPt0xt69e7Fw4UKkpqbi6tWr2LRpEwYNGqT6xPSQ3PMsuTcfAEpLxfHHBxXKJ+L0N2E42IIjaMhvLh+kgs6dxfESV/F95gDgelncB2LcoY3SHEY8Iow/9ZQ0Bby95WPUIF2guwZLc1wp8hHGn+l8XD6RsEhhuJ0Ff0gY4C8fpIbQUGE4yPWGNIVyrq0w/uLHD0tzLOhsFA8Ilv/s5C8AdcimYskdeCXZ4t+JPmk/SnP06hUu35CEe5m8HwXJTZkyBaNGjUJ0dDQ6deqExYsXo6CgwNQLYuTIkWjcuLHpmorZs2ejc+fOCAsLQ25uLhYuXIhff/0V//j/xjw6nQ6TJ0/Gq6++ipYtW6JZs2Z4+eWXERQUZJN/o2+r9iKioKAAHTp0wBNPPIFHHhH/Y0FERGRv7KHZ1NChQ3H9+nXMmDEDmZmZiIyMRHJysunCyIsXL8LB4Y+sN2/exNixY5GZmYkGDRogKioKBw4cQHj4HwvDF198EQUFBRg3bhxyc3PRrVs3JCcnw9XVVY1drFS1FxF9+/ZF3759bTEXIiIizZgwYQImTJhQaSzlT+1F33rrLbz11lvCfDqdDrNnz8bs2bPVmqIU784gIiJNsYdKxL3C5ouI4uJis2YbBgPPpxEREd0LbL54mjt3rlnjjZCQEFtvkoiIqEo6/F76V/VR1ztVR2y+iJg+fTry8vJMj4yMDFtvkoiIqGpOTrZ5aJDN99rFxUW1jl1ERERkP6q9iMjPzzd1yAJ+/9CPEydOwMfHB02aNFF1ckRERKpzcgJ0Kp+AUBSL+hfda6q9iDh27Bj++te/mr6eMmUKAGDUqFFYuXKlahOT/TAKfeULloATB4RxY+cu0hwjRojja6adlObQ11JDG1k1zZLPYGkSWCKMGwfJe4M47NsrjAfJumIBuJLpLB2jBul73kPcSAoAgrJ/EsZL2nWU5siW/GwsaeBU6CqfqypkL7TcXHkOyYFfME2+v73jxPu7K9mC990dfxDZkqzxnSXOnhXHQ0PljaT0Q/oJ4xeXfSPN4eEhb1pH2lHtRUSvXr2gKOKuhkRERHbLVpUIDdLqra1ERERUQ9q8nJSIiLSLlQjVsBJBREREVmElgoiItMXREXBQ+W9oo+RTZe9RXEQQEZG2ODlxEaESns4gIiIiq9htJUJ2z7v7vm3SHMd944Vx30vyeayZIO41MftLea+JGUkX5RtSgaxNx8Qk+YefzZ4nvgfc11c+j2ee6iaM73N0lObQf187Fyk550v6EVjQ82Bb+n3CuJMFr7Pe0eKfzd4T8h4QPTJWyzekho8+EsfHjpXn2LJFHL8kP2i7Pp8sjOtc3pTmULJelI5Rw7p14nhcnDyHh4c4bkl7jszF4j4QJ5rKLza8MO8euICQlQjVsBJBREREVrHbSgQREZFNsBKhGlYiiIiIyCqsRBARkbY4Ov7+UFN5ubr57hKsRBAREZFVWIkgIiJtcXJSvxKhdhvtuwQXEUREpC1cRKiGpzOIiIjIKnZbiZD1mnFtLW4kBQAdD30hjB8ufVSao4mT+BDNGPKjNEdJYLh0jBpat5YMcHWV5hgxQhwPDJTP4/gJ8dq03iF5s5qIyb0lI+rLJ2KJ/Hxh+Bc0l6aId9oljJ/0le0LUOIqbvLVI3ujNMdGt+GSEWulOSzSsqU4bsHrDE89JY5b0jlJ0rBK2dlPmqLTg/7y7aggOFgcD28tvz3w8FHx+yrG7xdpjuP54tfz2Vfk780ZK9sI49MQJs1R51iJUA0rEURERGQVu61EEBER2QQrEaphJYKIiIiswkoEERFpi6Pj79UIqjFWIoiIiMgqXIoREZG2ODmxEqESHkUiItIWLiJUo1MURX5jsIoMBgO8vLwAJACoV5ubJiKiOlcKYCvy8vKg14v7o6jt9r8/ed27Q6/yIsJQVgavb7+tk/2qS1yKERGRtrASoRpeWElERERW4VKMiIi0xRa3eNbulQF2g5UIIiIisgorEUREpC22uCaClQgiIiIiy7ESQURE2sJKhGq4iCAiIm3hIkI1PJ1BREREVmElgoiItIWVCNWwEkFERERWYSWCiIi0xRbNpoxGdfPdJViJICIiIquwEkFERNpii2siWIkgIiIishwrEUREpC2sRKiGlQgiIiKyit1WIv72t83C+BfvXZMnKSsTx3Nz5TlCQ8Xx9PQa59DVHyrPYYGxY8XH7MN5N+RJZMesqEieIzBQHE9Lk+fw9RWGdQFj5Dks8MEH4mM2bogKxyw7W55D9jqz4LgbnHyEcS+vAfJ5WEB5+wHxPJImSnP89lvN5+HmVvNt+PmJ446O6hyznBzx68zDo+bbkL0MAfkf385FBnkSye9NXdPx8hx1jZUI1djtIoKIiMgmuIhQDU9nEBER1YElS5YgNDQUrq6uiImJwZEjR6oc+9FHH6F79+5o0KABGjRogLi4uArjk5KSoNPpzB6JiYk23QcuIoiISFtuN5tS8+HoWK0prFu3DlOmTMHMmTNx/PhxdOjQAQkJCbh2rfJT9SkpKXjsscewe/duHDx4ECEhIYiPj8fly5fNxiUmJuLq1aumx//+7/9afZgswUUEERFRLXvzzTcxduxYjB49GuHh4Vi2bBnc3d2xfPnySsevXr0azzzzDCIjI9G6dWt8/PHHMBqN2Llzp9k4FxcXBAYGmh4NGjSw6X5wEUFERNqidhXijmssDAaD2aO4uLjC5ktKSpCamoq4uDjTcw4ODoiLi8PBgwct2oXCwkKUlpbCx8f8guqUlBT4+/ujVatWePrpp5GTk1ODAyXHRQQREZFKQkJC4OXlZXrMnTu3wpjs7GyUl5cjICDA7PmAgABkZmZatJ2pU6ciKCjIbCGSmJiIVatWYefOnZg/fz727NmDvn37ory8vGY7JcC7M4iISFtscXfG//9DnZGRAb1eb3raxcVF3e0AmDdvHtauXYuUlBS4urqanh82bJjp/9u3b4+IiAi0aNECKSkp6NOnj+rzAOx4EfHFP7YJ47qAc9IcymnJQbOk58GxYzXPceGCfIwKPhyxVxh38O0hzWHMldwnfumSfCL5+eL4HS/6KlnSw0MF41qLj5muYXtpDqVAcpO/JX9ZyPqNWPA605dZ0H9DBSd7iftARCTESnPo9+wRxq9kO0tznDkjjstabwDA0aPyMWqQ9YFwThH/vgMAY1y8OIeT/BbDwiJx8fnmb3phHACKIR+jZXq93mwRURlfX184OjoiKyvL7PmsrCwESvrsLFq0CPPmzcOOHTsQEREhHNu8eXP4+voiLS3NZosIns4gIiJtseE1EZZwdnZGVFSU2UWRty+SjI2tehG+YMECzJkzB8nJyYiOjpZu59KlS8jJyUGjRo0snlt12W0lgoiIyCZu3+KpJktaht5hypQpGDVqFKKjo9GpUycsXrwYBQUFGD16NABg5MiRaNy4semaivnz52PGjBlYs2YNQkNDTddOeHh4wMPDA/n5+XjllVcwePBgBAYG4vz583jxxRcRFhaGhIQEdff1DlxEEBER1bKhQ4fi+vXrmDFjBjIzMxEZGYnk5GTTxZYXL16Eg8MfJwuWLl2KkpISDBkyxCzPzJkzMWvWLDg6OuLkyZP49NNPkZubi6CgIMTHx2POnDk2uS7jNi4iiIhIW2xxYaUV+SZMmIAJEyZUGktJSTH7Ol1y3ZSbmxu2bt1a7TnUFK+JICIiIquwEkFERNpiJ5WIewErEURERGQVbS6diIhIu1iJUI397rWkIZFyTn7Liq6VnzjHnh/k8/D1FcctaSTUrZtkwDJ5DktIOtoYs29IUzSP9BHGf1lrwW1Mss46aRY0RercWT5GDd7ewrCSJ/9kvjZR7sL4mdUqHDNLmm+1bi0fUwuM++W9/x2eGieMnxryoTRHWJg4fv68NAW6tJa/J9Qgu/vP+Y7WxVVxSPtJGDeG3VfjeXh6SlPAxp/nRHcZ+11EEBER2QIrEarR5l4TEZF22aLZlKO8ankv4oWVREREZBVWIoiISFt4OkM1rEQQERGRVbS5dCIiIu1iJUI1rEQQERGRVaq1dJo7dy42btyIs2fPws3NDV26dMH8+fPRqlUrG8xMMjXZffUAlHPXhfEXP+4hzbHg1RLxAAvu38+6Je4joBpJbw0UFUlT/LLjF2F89uedpDlmTJMcs/x8aY5C1NIxk904b8ExO7M1VxifvbKjNMdLL4njDtnZ0hyFruIeH2qRtNaw5MeL9AniPhDxX86W5sjqMEMYDw6Wz8OS3yNqkL2M3CVvXUDeB8KhqFCaQ+8h3tCNXPnflT4ekvf33YCVCNVUqxKxZ88ejB8/HocOHcL27dtRWlqK+Ph4FBQU2Gp+REREZKeqtXRKTk42+3rlypXw9/dHamoqevSQ/1VPRERU59gnQjU1Oop5eXkAAB+f2imjEhER1RhPZ6jG6r02Go2YPHkyunbtinbt2lU5rri4GMXFxaavDQaDtZskIiIiO2L1ImL8+PE4deoU9u3bJxw3d+5cvPLKK9ZuhoiISF2sRKjGqls8J0yYgC1btmD37t0IllwCPX36dOTl5ZkeGRkZVk2UiIiI7Eu1lk6KouDZZ5/Fpk2bkJKSgmbNmkm/x8XFBS4uLlZPkIiISFWsRKimWns9fvx4rFmzBl999RU8PT2RmZkJAPDy8oKbm5tNJkhERET2qVqLiKVLlwIAevXqZfb8ihUrkJSUpNacfufrK46fPSvPIel689xz4uYtAKBzMQrjSlaYNEfA9tXSMaqQNc65dKnGOaSNpAB06eUsjB9IlncBck87KR2jCtnrzJLOSZJGUElJTaQpQkPF8YuHLDhma5dLx6hB1vfKkkMmO+w/DRM3kgIA/0CdMF5wXpHmuJgpfq2qRfpHqqzpGYCiMvFc3S1oSIZvvxXH+w6XpjhwrHaOmU2xEqGaap/OICIiIgL4AVxERKQ1bDalGi4iiIhIW3g6QzX8FE8iIiKyijaXTkREpF2sRKiGlQgiIiKyijaXTkREpF2sRKjGfvda9gMRfOiXieTe64C8n6QplOJQYXzi8/7SHO+8OkAyYq00h0Vk95pbcsxyc8VxC+5FP5AibgIw41W9NMfsCYHSMarw9hbHLbh/X9Zbo0n+L9IUF9PEfSDWbAiS5hg07AnxgDGbpDksIetpIWtXAgA3b4rjXl7yHA554lvOm3+5SpqjZNhI+YZUoIf4gwdLIH9PyH4lFvrK+5E4/U3cB8LVgpd7l9Ar8kGkGfa7iCAiIrIF3uKpGl4TQURERFZhJYKIiLSF10SoRpt7TURE2sVFhGp4OoOIiIisos2lExERaRcrEaphJYKIiIisos2lExERaRcrEaqx370uKhLHjx2T5+jcWRj+sew+aYrw7f8Rxt9JaiTN0bFXR+kYVaSlieOWvMglzZWy6jeXpgg4dkAYnx2ZKc3RMfER6Rg1XMwVN/lpkrZLniQsTLwNJ/kxa7JlozDerZv8eBw6JB2iCtlb8+ef5TliGkgavUmOKQD8eFZSSI2WN5IK32fBz1cNmzcLw2n3i5tAAfJD4l50Q5rjYr6PMN7k29XSHMuL5XMl7bDfRQQREZEtsNmUanhNBBEREVmFlQgiItIWXhOhGm3uNRERaRcXEarh6QwiIiKyijaXTkREpF2sRKiGlQgiIiKyik5RFKU2N2gwGODl5QUgAUC92tw0ERHVuVIAW5GXlwe9XtynRW23//3Ju3BB9W0bDAZ4NWtWJ/tVl1iJICIiIqto8yQOERFpF6+JUA0rEURERGQVbS6diIhIu1iJUA0rEUREpC23FxFqP6ppyZIlCA0NhaurK2JiYnDkyBHh+PXr16N169ZwdXVF+/bt8c0335jFFUXBjBkz0KhRI7i5uSEuLg4/W/KJeDXARQQREVEtW7duHaZMmYKZM2fi+PHj6NChAxISEnDt2rVKxx84cACPPfYYxowZg++++w6DBg3CoEGDcOrUKdOYBQsW4J133sGyZctw+PBh1K9fHwkJCSiSffRuDfAWTyIiqkV2cIvnzZu2ucWzQQOL9ysmJgZ/+ctf8N577wEAjEYjQkJC8Oyzz2LatGkVxg8dOhQFBQXYsmWL6bnOnTsjMjISy5Ytg6IoCAoKwnPPPYfnn38eAJCXl4eAgACsXLkSw4YNU2lPzbESQUREpBKDwWD2KC4urjCmpKQEqampiIuLMz3n4OCAuLg4HDx4sNK8Bw8eNBsPAAkJCabxFy5cQGZmptkYLy8vxMTEVJlTDVxEEBGRphjhYJMHAISEhMDLy8v0mDt3boXtZ2dno7y8HAEBAWbPBwQEIDMzs9I5Z2ZmCsff/m91cqpBm5eTEhER2UBGRobZ6QwXF5c6nI3tcRFBRESaUlb2+0PtnACg1+ul10T4+vrC0dERWVlZZs9nZWUhMDCw0u8JDAwUjr/936ysLDRq1MhsTGRkZHV2pVp4OoOIiKgWOTs7IyoqCjt37jQ9ZzQasXPnTsTGxlb6PbGxsWbjAWD79u2m8c2aNUNgYKDZGIPBgMOHD1eZUw2sRBARkabYshJhqSlTpmDUqFGIjo5Gp06dsHjxYhQUFGD06NEAgJEjR6Jx48amayomTZqEnj174o033kD//v2xdu1aHDt2DB9++CEAQKfTYfLkyXj11VfRsmVLNGvWDC+//DKCgoIwaNAgNXfVDBcRRESkKfawiBg6dCiuX7+OGTNmIDMzE5GRkUhOTjZdGHnx4kU4OPxxsqBLly5Ys2YNXnrpJfzrX/9Cy5Yt8eWXX6Jdu3amMS+++CIKCgowbtw45Obmolu3bkhOToarq6sq+1gZ9okgIqJaVPd9IjIy1N+2wWBASIiX5j4KnJUIIiLSFHuoRNwreGElERERWcVuKxEPPLBZGN+28oo8iWxpaMnS0cNDHM/OlucIDRWGdfWHynNY4O23xcfsH/+Q53DPvige4O0tTyI7Zmlp8hyS7egCxshzWODTT8XHbOSQQnmS3NyaT0SFc5YGJx9h3MtrQI23AQDKq52F8ZIX/keaQ9b7xpK3puylaMnHBQQFGoVxneNAeRIL5OSIX2eWfHaT7CViyTGTjdGX3ZAnkfzO07V6Tp6jjpWXq185KC9XN9/dgpUIIiIisordViKIiIhsgddEqIeVCCIiIrIKKxFERKQprESoh5UIIiIisgorEUREpCmsRKiHiwgiItIU3uKpHrtdRGx7fpsw3qZPvDTH4cPiuP7Sj/KJXLokjoeF1TyHSiZMEMdTUuQ5ener/GNob/sp3Vma474y8XEtDA2X5nA/e1w6Rg0je4n7Yqza0ESaIyHBXRj385PPw+HYEfEACz7KV59mwetZBTeeFveB8EnsLc0RvGOXMO6Qb5Dm+GafuLVwZ3E7i99zJNfOGd3SUnHcx1vcrwKAtD/DTcVfmsLNTRwvdBX3GgEA9zBv6RjSDrtdRBAREdkCT2eohxdWEhERkVVYiSAiIk1hJUI9rEQQERGRVViJICIiTWElQj2sRBAREZFVWIkgIiJNYSVCPVxEEBGRprDZlHrsdxERHCwMnz4tT9G/vzj+3+ni5i0AgEBx8yWkpclzREfLx6jAIf0XYbx3tK80x5vviRv4TIk7Kc1RGBYhjLuny5siFbbuKB1TGwYNko9ZulQcnxq7V56kdWtx3ILXmbG1vImXGnJzxXFvSSMpAHCY+5owvi5M3NAKAHr1EsclvZkAAP0ir8gHqaBBA8kASyYrGePXWt5sKjNTHPfwkE8j6zrPgtMf7HcRQUREZAM8naEeLimJiIjIKqxEEBGRprASoR5WIoiIiMgqrEQQEZGmsBKhHlYiiIiIyCqsRBARkaawT4R6qrWIWLp0KZYuXYr09HQAQNu2bTFjxgz07dtX/Znl5wvDsp4IAPDfF9KF8eZJvaU5ftkh2Y6rqzSHocxdOkYVkpu8DRD3gACAKb2OC+NrTsn7NzzezigdYzckv0ksuW9+6v3bhPFHFsdLc2x8T9KvwI5qpbKpOOQbpDlWhYj7QIx8TdxrBACudBf3LLl5U5oCCJP0gVGJ5NcZXD3kPR6cvMVjnE/Je7h4hMqPq0xAg5Ia56hrPJ2hnmqdzggODsa8efOQmpqKY8eOoXfv3hg4cCBOW9L5iYiIiO4p1apEDBgwwOzr1157DUuXLsWhQ4fQtm1bVSdGRERkC6xEqMfqayLKy8uxfv16FBQUIDY2Vs05ERER0V2g2ouIH374AbGxsSgqKoKHhwc2bdqE8PCqe/YXFxejuLjY9LXBID9fSkREZCusRKin2rd4tmrVCidOnMDhw4fx9NNPY9SoUfjxx6o/UGnu3Lnw8vIyPUJCQmo0YSIiIrIP1V5EODs7IywsDFFRUZg7dy46dOiAt99+u8rx06dPR15enumRkZFRowkTERHVxO1KhNoPLapxnwij0Wh2uuLPXFxc4OLiUtPNEBERkZ2p1iJi+vTp6Nu3L5o0aYJbt25hzZo1SElJwdatW201PyIiIlWx2ZR6qrWIuHbtGkaOHImrV6/Cy8sLERER2Lp1Kx544AH1Zyb7CWdny3MEihvJ/LL2iDSFrsVFYVw5XfVFpbfp08QNnFQjaXxlyTyyGoubST3eTt5oprDIWRi/1VB+zALKaukC3KIiYdiSpmaIjhaGNy6S59A1DhDGlSz5W9XBgmZDteF4mrypWa9e4viVOPm+BLVwE8a9c36T5vgprXY6/5eWiuM+ZdekObIUcbMpv3byRlL61vcJ4z9t+Uma41S2+P19N+CFleqp1iLik08+sdU8iIiI6C7Dz84gIiJNYSVCPfwUTyIiIrIKKxFERKQprESoh5UIIiIisgorEUREpCm8xVM9rEQQERGRVey3EuHhIY6HhspzpKWJ497e0hTK5WBh/L5eQdIcP6VckY5RRWamMFzSTtwDAgAC0iX3iReLe1EAgFNgE2F85UppCkz9e758kBokvUSkr0MASE8Xx53kbzMlp+qurwDw/lpxjwAAGDFCPkYNknYk0kMKyNu8WNIGRtYHwn3RbGmO+55/Xr4hFQToxH0gSrzlPzs3cUsTWcsTAMCtb8Xv7zA/eY77yqr+rKS7Ba+JUI/9LiKIiIhsgIsI9fB0BhEREVmFlQgiItIUViLUw0oEERERWYWVCCIi0hRWItTDSgQRERFZhYsIIiLSlNvNptR82LLZ1I0bNzB8+HDo9Xp4e3tjzJgxyM+v+jb4Gzdu4Nlnn0WrVq3g5uaGJk2aYOLEicjLyzMbp9PpKjzWrl1brbnxdAYREZEdGz58OK5evYrt27ejtLQUo0ePxrhx47BmzZpKx1+5cgVXrlzBokWLEB4ejl9//RVPPfUUrly5gg0bNpiNXbFiBRITE01fe1vQP+lO9ruIkJ1gEqzCTGQNqSw5WFu2CMM/vSdvvvT0nHj5dtQg6dDjbEE3GkNohDCu9zBKczhDPGbqw5ImYAD+9d590jGqkL0GTpyQpvjJQ9zE6z7vG/J5SBpWJSX5SFPk5so3owbZW/Pjj+U5pji9I4wbkiZKc8j21/WlGdIcDmdrqXHSoUPCsPODD0pTZOeLC8dBgfL35g8/iHMEfL9NmmNNdi39PrOhsjLA0VH9nLZw5swZJCcn4+jRo4iOjgYAvPvuu+jXrx8WLVqEoKCKDQ/btWuHf//736avW7Rogddeew0jRoxAWVkZnO5ogOft7Y1ASzrEVYGnM4iISFPUPpVx54WaBoPB7FFcLO5GK3Pw4EF4e3ubFhAAEBcXBwcHBxw+fNjiPHl5edDr9WYLCAAYP348fH190alTJyxfvhyKolRrflxEEBERqSQkJAReXl6mx9y5c2uULzMzE/7+5m3RnZyc4OPjg0zJRx3clp2djTlz5mDcuHFmz8+ePRtffPEFtm/fjsGDB+OZZ57Bu+++W6352e/pDCIiIhuw5emMjIwM6PV60/MuLi6Vjp82bRrmz58vzHnmzJkaz8tgMKB///4IDw/HrFmzzGIvv/yy6f/vv/9+FBQUYOHChZg4UX468TYuIoiIiFSi1+vNFhFVee6555CUlCQc07x5cwQGBuLaNfMPcCsrK8ONGzek1zLcunULiYmJ8PT0xKZNm1CvXj3h+JiYGMyZMwfFxcVVLn7+jIsIIiLSlNu3eKqdszr8/Pzg5yf/2NTY2Fjk5uYiNTUVUVFRAIBdu3bBaDQiJiamyu8zGAxISEiAi4sLvv76a7jKPn4XwIkTJ9CgQQOLFxAAFxFERER2q02bNkhMTMTYsWOxbNkylJaWYsKECRg2bJjpzozLly+jT58+WLVqFTp16gSDwYD4+HgUFhbi888/N13kCfy+eHF0dMTmzZuRlZWFzp07w9XVFdu3b8frr7+O559/vlrz4yKCiIg0pawMcFD5tgJbtr1evXo1JkyYgD59+sDBwQGDBw/GO+/8cZt0aWkpzp07h8LCQgDA8ePHTXduhIWFmeW6cOECQkNDUa9ePSxZsgT//Oc/oSgKwsLC8Oabb2Ls2LHVmptOqe79HDVkMBjg5eUFIAGA+PwMERHda0oBbDXdclibbv/7M2ZMHpyd1d12SYkBn3ziVSf7VZdYiSAiIk252yoR9oyLCCIi0hQuItTDZlNERERkFVYiiIhIU1iJUA8rEURERGQVViKIiEhT7KHZ1L2ClQgiIiKyCisRRESkKWVlgE6nfk4tYiWCiIiIrMJKBBERaQorEerhIoKIiDSFiwj18HQGERERWYWVCCIi0hRWItTDSgQRERFZhZUIIiLSlPJy9SsRbDZFREREVA2sRBARkabY4voFrV4TYceLiOeEUeV7H3mKzExx3MmC3ff2FsfT0uQ5hgwRhnWOA+U5LPDyy5uF8fHj5TkCTu8SDwgNlSeRjTl1SpqiMCxCGK9ff4B8HhZ5VRhVLvvJU5w9K457eMhzyMZYcMzQq5cwrAsYI89hgZvYIoy7FyvSHIcOieNFRfJ5tGsnjp84Ic/RubM43rChOq8zJfKSMH7y0++kOYKDxXFLjplszPXr8hwtW4rjah0zujvY8SKCiIhIfaxEqIfXRBAREZFVWIkgIiJNYSVCPVxEEBGRptjidkze4klERERUDaxEEBGRppSVAYr8JqJqYSWCiIiIqBrsthKhbC0RxkcuEvcRAICFC8VjAgp+kU8kPV0cl9ybDwDIzZWPUcHzz4vjehikOQo79xbGf/hBPo+Ysp+E8awA+c8uIO2kfEMqUE7XE8afnhMkzfHGG+Ix7k7i1zIAICVFHE9MlOfIzpaPUYExR/wnnHOi+DUEAD127BAPyM+X5th2SC+MW/LWlE1DLQeWiPtAdHlvnDSHcdmHwrglrURk7UgaNJDn8HEtlA+yc6xEqIeVCCIiIrKK3VYiiIiIbIGVCPWwEkFERERWYSWCiIg0hZUI9XARQUREmlJerv4iwmhUN9/dgqcziIiIyCqsRBARkaaUlQEOKv8JzUoEERERUTXYbyUiMlIY/riXPMW8eeL4jBEWzKN1a3HckkZS3t4WbKjm9GU3hHGjt480xz5J8534dlekOUp87xPGA7LlOaTHXS2BgcLwkiXyFB9/LI6P65YmTxIdLY5b8jqT7ItaSkvFceOOXdIcDiniMYfryxtWyZpJ3bwpTYGHuonfM2qR/XjReZk0h+yYhUkaxQHArVviuKenNAWybrnLB9k5ViLUw0oEERERWcV+KxFEREQ2wEqEeliJICIiIquwEkFERJpSXq5+5UDtvhN3Cy4iiIhIU8rKAJ1O3ZxaXUTwdAYRERFZhZUIIiLSFFYi1FOjRcS8efMwffp0TJo0CYsXL1ZpSpZxLjJIx8z4R74wvu7b5tIcQ/vKtyNT6OFf4xwWcXUVhouK5Cnio8X3zf+SGyTNESp7VTnJX3ZGJ2fpGFVI5uJQViJNMS7ukjC+7mi4NMfQYMnrTPKzBYASp9q5f9/NTRy35Jj9GCjuaRBzbJU0R8n9I6VjZCzpnaKGsjJxvKhIXhR27SY+Zu7pP8knEizu4WKJAM/CGuege4fVi4ijR4/igw8+QEREhJrzISIisilWItRj1TUR+fn5GD58OD766CM0aNBA7TkRERHRXcCqRcT48ePRv39/xMXFqT0fIiIimyors81Di6p9OmPt2rU4fvw4jh49atH44uJiFBcXm742GGp+jQERERHVvWpVIjIyMjBp0iSsXr0arhZc6AUAc+fOhZeXl+kREhJi1USJiIjUYYSiqPsAtNn3ulqLiNTUVFy7dg0dO3aEk5MTnJycsGfPHrzzzjtwcnJCeXl5he+ZPn068vLyTI+MjAzVJk9ERFR95TZ6aE+1Tmf06dMHP/zwg9lzo0ePRuvWrTF16lQ4OjpW+B4XFxe4uLjUbJZERERkd6q1iPD09ES7du3Mnqtfvz4aNmxY4XkiIiL7ZIvKASsR9uXUKXHcw0OeQ3K57NCQdGkKndcbwrjy/UxpDvctW6RjVCHZX/dMeTOanyBuRhMYKJ/G9evieHm5vPlW0KED8g2p4exZcTxf3LAMAODrKwwPbbRXmkLndUYYVy4PkOZwPiTfjhouiXtrIT1d3igsMlIc/6WbvJFU826dhPGAffukObKu105TM9kxs+TXWUqKOB4WJm8klV1f3BzB+3t5s4Pc3NppakZ/uHHjBp599lls3rwZDg4OGDx4MN5++214CF44vXr1wp49e8yee/LJJ7Fs2TLT1xcvXsTTTz+N3bt3w8PDA6NGjcLcuXPhZEFDwNtqvIhIkb2yiYiI7MrdVYkYPnw4rl69iu3bt6O0tBSjR4/GuHHjsGbNGuH3jR07FrNnzzZ97e7+xwKwvLwc/fv3R2BgIA4cOICrV69i5MiRqFevHl5//XWL52a/lQgiIiKNO3PmDJKTk3H06FFER0cDAN59913069cPixYtQlBQ1R9F4O7ujsAqysfbtm3Djz/+iB07diAgIACRkZGYM2cOpk6dilmzZsHZ2bIqHT/Fk4iINMZoo8fvvZDufNzZJ8kaBw8ehLe3t2kBAQBxcXFwcHDA4cOHhd+7evVq+Pr6ol27dpg+fToKC//43JODBw+iffv2CAgIMD2XkJAAg8GA06dPWzw/ViKIiIhU8udeSDNnzsSsWbOszpeZmQl/f/PryJycnODj44PMzMwqv+/xxx9H06ZNERQUhJMnT2Lq1Kk4d+4cNm7caMp75wICgOlrUd4/4yKCiIg0xnbXRGRkZECv15uerarFwbRp0zB//nxhxjNnxBdci4wbN870/+3bt0ejRo3Qp08fnD9/Hi1atLA6759xEUFERBpjhPqLiN9PZ+j1erNFRFWee+45JCUlCcc0b94cgYGBuHbtmtnzZWVluHHjRpXXO1QmJiYGAJCWloYWLVogMDAQR44cMRuTlZUFANXKy0UEERFRLfPz84Ofn590XGxsLHJzc5GamoqoqCgAwK5du2A0Gk0LA0ucOHECANCoUSNT3tdeew3Xrl0znS7Zvn079Ho9wsPDLc5rv4uItDRxfNAgeY5588Tx1q2lKZStTwrjug7/I89x+QPJiM3SHBaRHLOsxh2lKe77dIF4QOPG8nkMGC4MjxghT/HRR13kg9Qg+yC5xx6T51i5UhyX9JEAACX1L8K4rvF78hynLTiwKpC1X3jwQXmOb78Vx7//Xp7jpZQjwvgvLuKeCABwX7G8L4IaZKeYg4PlOcLCxPGGDeU5cveL99ejg/yY/Xaodo6Zbd09t3i2adMGiYmJGDt2LJYtW4bS0lJMmDABw4YNM92ZcfnyZfTp0werVq1Cp06dcP78eaxZswb9+vVDw4YNcfLkSfzzn/9Ejx49EBERAQCIj49HeHg4/v73v2PBggXIzMzESy+9hPHjx1eryzTvziAiIrJjq1evRuvWrdGnTx/069cP3bp1w4cffmiKl5aW4ty5c6a7L5ydnbFjxw7Ex8ejdevWeO655zB48GBs3vzHH6yOjo7YsmULHB0dERsbixEjRmDkyJFmfSUsYb+VCCIiIpu4eyoRAODj4yNsLBUaGgpF+aNCFBISUqFbZWWaNm2Kb775pkZzYyWCiIiIrMJKBBERacwfzaHUzak9rEQQERGRVViJICIijbm7romwZ1xEEBGRxnARoRaeziAiIiKr2G8lIjFRHPfwkOfo1Uscj4uT53j1VWFY+f41aYouQ6r+qFY1GSPFzaQCTp2UJ5kwocbz0OdeEcbnzZMfj/37azwNy7RsKY6np0tTFD41RRh3z78mjAOQdnBS9kjeDwDiJ1veZa4mZI2Rli2T55idO1EY7/U/70hzFBWJ4/pf5U2RnNevlo5Rg6yLcHNfgzTH1yniVsrhYSXSHDvTxR/vnL9Vfszil4yUjrF/rESohZUIIiIisor9ViKIiIhswnYfwKU1rEQQERGRVViJICIijWGzKbWwEkFERERWYSWCiIg0hndnqIWLCCIi0hguItSiU+78/NBaYDAY4OXlBSABQL3a3DQREdW5UgBbkZeXB71e3PtCbX/8+7MPgAW9hqolH0C3OtmvusRKBBERaQwrEWrhhZVERERkFVYiiIhIY1iJUAsrEURERGQVViKIiEhjFKjfHKpW71GwG6xEEBERkVVYiSAiIo3hNRFq4SKCiIg0hosItfB0BhEREVmFlQgiItIYViLUwkoEERERWYWVCCIi0hhWItTCSgQRERFZhZUIIiLSGCPUbzaldr67AysRREREZBVWIoiISGN4TYRaWIkgIiIiq9htJUKZ3l4Y/ynpdWmO0FBxvKhIPo/sbHHc21uew9VVHK9ff4A8iQW2bt0sjMe3uyJP4usrjl+6JM8RGCgfI1EId2FcrWOm7PynMG6I7i3N4eEhjjtYcK704iXxel72YwEAJ8m72cVFpWNW/pUqeYQseXPm5grDBo+gmqZA06bqHLOcHPF706fsmjyJ7JdNfr48R1mZMFzi7S9NIfsV0KKFOsfMtoxQv3KgzWsi7HYRQUREZBs8naEWns4gIiIiq7ASQUREGsNbPNXCSgQRERFZhZUIIiLSGF4ToRZWIoiIiMgqrEQQEZHGsBKhFrtdRPw4QtwHIvzEF/IkwQ8Kw+mZ4l4EgLzlgeS2awCAu2vtXHAj6wPxU778vvlQST8CZ0t2WNawwIIcshRqkfWB0F/6UZ6kdWthuKRMXvBzcZFvRqa2jplRUsDcsUOeIz5O/J4wlMnfm/oycRMXfdkNeQ4nC/pRqMAH4rkYfeX9GWStM9wt6a0haWrijBJpisBAZ/l2SDPsdhFBRERkG6xEqIWLCCIi0hh2rFQLL6wkIiIiq7ASQUREGsNmU2phJYKIiIiswkoEERFpDC+sVAsrEURERGQVViKIiEhjWIlQi90uIkJDZQPEjaQAAMeOCcMeYT2kKdLTxfHgYPk0LGk2pApfcaeoQHGfGQBASoo4Ht/Ngh3OFjcBgqurNIUzaqcJkKT3jgUvRACZmcJwugVNvmQC/Cy4aMuSRmC1IC7OgkGSxkj6UyfkOSRvvhvwkabIr61D5u0tDF+6JE+RliaO9+4m6RQHyH+hSeYJAO528joj+2C3iwgiIiLbYCVCLbwmgoiINKbcRg/buHHjBoYPHw69Xg9vb2+MGTMG+fn5VY5PT0+HTqer9LF+/XrTuMria9eurdbcWIkgIiKyY8OHD8fVq1exfft2lJaWYvTo0Rg3bhzWrFlT6fiQkBBcvXrV7LkPP/wQCxcuRN++fc2eX7FiBRITE01fe1twSutOXEQQEZHG3D3Nps6cOYPk5GQcPXoU0dHRAIB3330X/fr1w6JFixAUVPGaK0dHRwT+6dMjN23ahEcffRQef7oQzNvbu8LY6uDpDCIiIpUYDAazR3FxcY3yHTx4EN7e3qYFBADExcXBwcEBhw8ftihHamoqTpw4gTFjxlSIjR8/Hr6+vujUqROWL18ORVGqNT9WIoiISGNs9wFcISEhZs/OnDkTs2bNsjprZmYm/P3NPyreyckJPj4+yJTcGXbbJ598gjZt2qBLly5mz8+ePRu9e/eGu7s7tm3bhmeeeQb5+fmYOHGixfPjIoKIiEglGRkZ0Ov1pq9dXFwqHTdt2jTMnz9fmOvMmTM1ns9vv/2GNWvW4OWXX64Qu/O5+++/HwUFBVi4cKHtFhGzZs3CK6+8YvZcq1atcPbs2eqkscitW+J4eo67NIe3pA9E0MezpTnyh80QxqV9BgA4lxXKB6khN1cY1kvuzQeA+M7ewvjytXphHACeGCF5WQmuKjap5sU91nIoKxHGS5zkr7Oz2eIxESdWSXP8GD1SPMCSe/Od7ONvAkt6HjRJF/dwef+UvIfLM4G/COM+HuKf7e9jpEPUIXnvNfGQvzebRIrju/bJ+2L0bi3ZYQvedyVwlo6xf7a7xVOv15stIqry3HPPISkpSTimefPmCAwMxLVr18yeLysrw40bNyy6lmHDhg0oLCzEyJGS3zEAYmJiMGfOHBQXF1e5+Pmzav/Wadu2LXbs2PFHAjv5xUVERHS38PPzg5+fn3RcbGwscnNzkZqaiqioKADArl27YDQaERMTI/3+Tz75BA899JBF2zpx4gQaNGhg8QICsGIR4eTkVKMrOYmIiOrW3dNsqk2bNkhMTMTYsWOxbNkylJaWYsKECRg2bJjpzozLly+jT58+WLVqFTp16mT63rS0NOzduxfffPNNhbybN29GVlYWOnfuDFdXV2zfvh2vv/46nn/++WrNr9qLiJ9//hlBQUFwdXVFbGws5s6diyZNmlQ3DRERUR25exYRALB69WpMmDABffr0gYODAwYPHox33nnHFC8tLcW5c+dQWGh+6nz58uUIDg5GfHx8hZz16tXDkiVL8M9//hOKoiAsLAxvvvkmxo4dW625VWsRERMTg5UrV6JVq1a4evUqXnnlFXTv3h2nTp2Cp6dnpd9TXFxsdouLwWCo1gSJiIi0zMfHp8rGUgAQGhpa6a2Zr7/+Ol5//fVKvycxMdGsyZS1qrWIuLPTVUREBGJiYtC0aVN88cUXld5/CgBz586tcDEmERFR3bm7KhH2rEbNpry9vXHfffchTfDxctOnT0deXp7pkZGRUZNNEhERkZ2o0SIiPz8f58+fR6NGjaoc4+LiYrrlxdJbX4iIiGzHaKOH9lRrEfH8889jz549SE9Px4EDB/Dwww/D0dERjz32mK3mR0RERHaqWtdEXLp0CY899hhycnLg5+eHbt264dChQxbdf6q20FD5GFnDql9GiBtJAcB9rz4hjF95dbk0R2CgvGGRKiSNYixpEuOcf0MYl/RGAQB8kyzeTr9uFrzsJI2z1GIoEs/V1VWeIzhYHL/oLW/yEv6x+LV4Y7K8MZqPd+38JeRQJG6e1iTQgp+va2th+JlI+QXYH65tLoyPG2FBkzc76XNT6CpvFOVeJH5v9u4mb641e17FD2u604ynrgnjAOBsSeMzu2e7ttdaU613UHU/Z5yIiIjuXfaxDCciIqo1vDtDLVxEEBGRxnARoZYa3Z1BRERE2sVKBBERaYwtbsnU5oWVrEQQERGRVViJICIijeE1EWqx20VEvXriuCW3KlfxmWAmAX7y8tNP08R9IO5771/SHFcmVP4BKKqTHBRnWHDQJPfNO2TL7yPv103cXGHbIXnX0s6dpUNUIWsTYMnrTNafwcdbnuPKU+I+EEGn9kpzFEb3kG9IBYUQ9z1xz5W/Rgyu/sK4vkzcEwEAxg2RjCmT/3r7JbN2eriUOEmOWVnt9LSYkXRRGP8pV/6JzJb0TiHtsNtFBBERkW2wEqEWXhNBREREVmElgoiINIaVCLVwEUFERBrDz85QC09nEBERkVVYiSAiIo1hsym1sBJBREREVmElgoiINIYXVqrFbhcRsoYm7k4l0hxGJ2dx3IJCjK+vOJ41Sd5IKsjNIB2jBqOruKGNJY2TnCFuemP0FTcJAgCHXHEToPg4edmvpKx2imSyY2JRf5/sbGG4xFt+zGSvM4OHvJGU3oL3hBqkzYYy86U50jPFxySitYc0h+z9XVQkTYHmTuLmS2qRzkXy3gUAZ9nP15IXq4f4uAZLXocA4C75HUHaYreLCCIiIttgJUItvCaCiIiIrMJKBBERaQwrEWrhIoKIiDSGiwi18HQGERERWYWVCCIi0hi2vVYLKxFERERkFbutRNSvP6Cup3DXcXTkMasuLy8es+ri66z6+DqzN2x7rRZWIoiIiMgqdluJICIiso1yqP83NO/OICIiIrIYKxFERKQxrESohYsIIiLSGC4i1MLTGURERGQVViKIiEhjeIunWmp9EaEoyv//X1ltb5qIiOrc77/7//i3oO7mYP857V+tLyJu3br1//+3s7Y3TUREduLWrVvw8vKq1W06OzsjMDAQmZk7bJI/MDAQzs7ONsltr3RKLS8HjUYjrly5Ak9PT+h0utrctJDBYEBISAgyMjKg1+vrejp3BR6z6uMxqz4es+qz52OmKApu3bqFoKAgODjU/mV5RUVFKCkpsUluZ2dnuLq62iS3var1SoSDgwOCg4Nre7MW0+v1dvems3c8ZtXHY1Z9PGbVZ6/HrLYrEHdydXXV3D/0tsS7M4iIiMgqXEQQERGRVbiI+H8uLi6YOXMmXFxc6noqdw0es+rjMas+HrPq4zGj2lLrF1YSERHRvYGVCCIiIrIKFxFERERkFS4iiIiIyCpcRBAREZFVuIj4f0uWLEFoaChcXV0RExODI0eO1PWU7NbevXsxYMAABAUFQafT4csvv6zrKdm9uXPn4i9/+Qs8PT3h7++PQYMG4dy5c3U9Lbu2dOlSREREmBomxcbG4r///W9dT+uuMW/ePOh0OkyePLmup0L3MC4iAKxbtw5TpkzBzJkzcfz4cXTo0AEJCQm4du1aXU/NLhUUFKBDhw5YsmRJXU/lrrFnzx6MHz8ehw4dwvbt21FaWor4+HgUFBTU9dTsVnBwMObNm4fU1FQcO3YMvXv3xsCBA3H69Om6nprdO3r0KD744ANERETU9VToHsdbPAHExMTgL3/5C9577z0Av3++R0hICJ599llMmzatjmdn33Q6HTZt2oRBgwbV9VTuKtevX4e/vz/27NmDHj161PV07ho+Pj5YuHAhxowZU9dTsVv5+fno2LEj3n//fbz66quIjIzE4sWL63padI/SfCWipKQEqampiIuLMz3n4OCAuLg4HDx4sA5nRveyvLw8AL//o0hy5eXlWLt2LQoKChAbG1vX07Fr48ePR//+/c1+pxHZSq1/AJe9yc7ORnl5OQICAsyeDwgIwNmzZ+toVnQvMxqNmDx5Mrp27Yp27drV9XTs2g8//IDY2FgUFRXBw8MDmzZtQnh4eF1Py26tXbsWx48fx9GjR+t6KqQRml9EENW28ePH49SpU9i3b19dT8XutWrVCidOnEBeXh42bNiAUaNGYc+ePVxIVCIjIwOTJk3C9u3b+SmVVGs0v4jw9fWFo6MjsrKyzJ7PyspCYGBgHc2K7lUTJkzAli1bsHfvXgQHB9f1dOyes7MzwsLCAABRUVE4evQo3n77bXzwwQd1PDP7k5qaimvXrqFjx46m58rLy7F371689957KC4uhqOjYx3OkO5Fmr8mwtnZGVFRUdi5c6fpOaPRiJ07d/LcK6lGURRMmDABmzZtwq5du9CsWbO6ntJdyWg0ori4uK6nYZf69OmDH374ASdOnDA9oqOjMXz4cJw4cYILCLIJzVciAGDKlCkYNWoUoqOj0alTJyxevBgFBQUYPXp0XU/NLuXn5yMtLc309YULF3DixAn4+PigSZMmdTgz+zV+/HisWbMGX331FTw9PZGZmQkA8PLygpubWx3Pzj5Nnz4dffv2RZMmTXDr1i2sWbMGKSkp2Lp1a11PzS55enpWuMamfv36aNiwIa+9IZvhIgLA0KFDcf36dcyYMQOZmZmIjIxEcnJyhYst6XfHjh3DX//6V9PXU6ZMAQCMGjUKK1eurKNZ2belS5cCAHr16mX2/IoVK5CUlFT7E7oLXLt2DSNHjsTVq1fh5eWFiIgIbN26FQ888EBdT42I/h/7RBAREZFVNH9NBBEREVmHiwgiIiKyChcRREREZBUuIoiIiMgqXEQQERGRVbiIICIiIqtwEUFERERW4SKC6E9u3ryJ2bNn49q1a3U9FbKCoih45513cOTIkbqeCtE9j4sIoj9p0KABXFxcMHz4cBiNRrNYSkoKdDodcnNza31eSUlJGDRokNXf36tXL0yePFm1+cikp6dDp9PhxIkTVY6xxfHU6XSIjIzE0KFDK3ywHhGpi4sIDUtKSoJOp4NOp0O9evUQEBCABx54AMuXL6/wj6da25s1a5bp69DQUCxevFj17ahh6tSpuO+++8zmq6ba/gcdADZu3Ig5c+bU6jZlunTpYmprraYePXpg3rx5GD58OMrLy1XNTUR/4GdnaFxiYiJWrFiB8vJyZGVlITk5GZMmTcKGDRvw9ddfw8lJuy+RJUuW1PUUVOXj41PXU6jA2dkZgYGBNsk9dOhQDB061Ca5ieh3rERonIuLCwIDA9G4cWN07NgR//rXv/DVV1/hv//9r9mHaV28eBEDBw6Eh4cH9Ho9Hn30UbNS8axZsxAZGYnPPvsMoaGh8PLywrBhw3Dr1q1Kt9urVy/8+uuv+Oc//2mqhlSmspJ4bm4udDodUlJSAPx+DcPw4cPh5+cHNzc3tGzZEitWrDCNz8jIwKOPPgpvb2/4+Phg4MCBSE9Pt/qYAcD+/fsREREBV1dXdO7cGadOnTLFcnJy8Nhjj6Fx48Zwd3dH+/bt8b//+7+meFJSEvbs2YO3337btO+353P69Gk8+OCD0Ov18PT0RPfu3XH+/HmzbS9atAiNGjVCw4YNMX78eJSWllo05z9XPz777DNER0fD09MTgYGBePzxxytcByKaj9FoxOzZsxEcHAwXFxfTB9f92dmzZ9GlSxe4urqiXbt22LNnjyn259MZsmN3ez8mTpyIF198ET4+PggMDLRZxYiIxLiIoAp69+6NDh06YOPGjQB+/8di4MCBuHHjBvbs2YPt27fjl19+qfBX3vnz5/Hll19iy5Yt2LJlC/bs2YN58+ZVuo2NGzciODgYs2fPxtWrV3H16lWr5/vyyy/jxx9/xH//+1+cOXMGS5cuha+vLwCgtLQUCQkJ8PT0xLfffov9+/fDw8MDiYmJKCkpsXqbL7zwAt544w0cPXoUfn5+GDBggOkf86KiIkRFReE///kPTp06hXHjxuHvf/+76UK/t99+G7GxsRg7dqxp30NCQnD58mX06NEDLi4u2LVrF1JTU/HEE0+grKzMtN3du3fj/Pnz2L17Nz799FOsXLnS6k9OLS0txZw5c/D999/jyy+/RHp6utknisrm8/bbb+ONN97AokWLcPLkSSQkJOChhx7Czz//XOFYPffcc/juu+8QGxuLAQMGICcnp9I5yY7dbZ9++inq16+Pw4cPY8GCBZg9eza2b99u1XEgohpQSLNGjRqlDBw4sNLY0KFDlTZt2iiKoijbtm1THB0dlYsXL5rip0+fVgAoR44cURRFUWbOnKm4u7srBoPBNOaFF15QYmJiqtx+06ZNlbfeeks4xwsXLigAlO+++8703M2bNxUAyu7duxVFUZQBAwYoo0ePrvT7P/vsM6VVq1aK0Wg0PVdcXKy4ubkpW7duFW67Mrt371YAKGvXrjU9l5OTo7i5uSnr1q2r8vv69++vPPfcc6ave/bsqUyaNMlszPTp05VmzZopJSUlleYYNWqU0rRpU6WsrMz03N/+9jdl6NChFs29sm3e6ejRowoA5datWxbNJygoSHnttdfMnvvLX/6iPPPMM4qi/PGzmzdvnileWlqqBAcHK/Pnz1cU5Y/jefPmzSrnVdmx69atW4XtTp06tcocRGQbrERQpRRFMZ1iOHPmDEJCQhASEmKKh4eHw9vbG2fOnDE9FxoaCk9PT9PXjRo1qpXbJJ9++mmsXbsWkZGRePHFF3HgwAFT7Pvvv0daWho8PT3h4eEBDw8P+Pj4oKioqMJpguqIjY01/b+Pjw9atWplOhbl5eWYM2cO2rdvDx8fH3h4eGDr1q24ePGiMOeJEyfQvXt31KtXr8oxbdu2haOjo+nrmhzj1NRUDBgwAE2aNIGnpyd69uwJAKZ5iuZjMBhw5coVdO3a1ez5rl27mr0mAPNj5eTkhOjo6ApjbrP02EVERJh9XVuvNSIyp92r5kjozJkzaNasWbW+58//2Oh0uhrf5eHg8Ps6V1EU03N/vgagb9+++PXXX/HNN99g+/bt6NOnD8aPH49FixYhPz8fUVFRWL16dYXcfn5+NZpbVRYuXIi3334bixcvRvv27VG/fn1MnjxZevrEzc1NmlutY1xQUICEhAQkJCRg9erV8PPzw8WLF5GQkGCapyXzUZulx84WrzUiqj5WIqiCXbt24YcffsDgwYMBAG3atEFGRgYyMjJMY3788Ufk5uYiPDzc6u04OztLb7+7/Q/9nddMVNZ3wM/PD6NGjcLnn3+OxYsX48MPPwQAdOzYET///DP8/f0RFhZm9qjJbYWHDh0y/f/Nmzfx008/oU2bNgB+v+hy4MCBGDFiBDp06IDmzZvjp59+Mvv+yvY9IiIC3377rcUXStbE2bNnkZOTg3nz5qF79+5o3bp1hb/kRfPR6/UICgrC/v37zZ7fv39/hdfEnceqrKwMqamppmP1Z5YcOyKyH1xEaFxxcTEyMzNx+fJlHD9+HK+//joGDhyIBx98ECNHjgQAxMXFoX379hg+fDiOHz+OI0eOYOTIkejZsyeio6Ot3nZoaCj27t2Ly5cvIzs7u9Ixbm5u6Ny5M+bNm4czZ85gz549eOmll8zGzJgxA1999RXS0tJw+vRpbNmyxfSP1PDhw+Hr64uBAwfi22+/xYULF5CSkoKJEyfi0qVLVs999uzZ2LlzJ06dOoWkpCT4+vqaGkG1bNkS27dvx4EDB3DmzBk8+eSTFZoehYaG4vDhw0hPT0d2djaMRiMmTJgAg8GAYcOG4dixY/j555/x2Wef4dy5c1bPsypNmjSBs7Mz3n33Xfzyyy/4+uuvK/SQkM3nhRdewPz587Fu3TqcO3cO06ZNw4kTJzBp0iSzPEuWLMGmTZtw9uxZjB8/Hjdv3sQTTzxR6bwsOXZEZD+4iNC45ORkNGrUCKGhoUhMTMTu3bvxzjvv4KuvvjKde9fpdPjqq6/QoEED9OjRA3FxcWjevDnWrVtXo23Pnj0b6enpaNGihfDUwvLly1FWVoaoqChMnjwZr776qlnc2dkZ06dPR0REBHr06AFHR0esXbsWAODu7o69e/eiSZMmeOSRR9CmTRuMGTMGRUVF0Ov1Vs993rx5mDRpEqKiopCZmYnNmzfD2dkZAPDSSy+hY8eOSEhIQK9evRAYGFih0+Tzzz8PR0dHhIeHm04lNGzYELt27UJ+fj569uyJqKgofPTRR8JrJKzl5+eHlStXYv369QgPD8e8efOwaNEiszGy+UycOBFTpkzBc889h/bt2yM5ORlff/01WrZsWeFYzZs3Dx06dMC+ffvw9ddfm+6e+TNLjh0R2Q+dcufJZiK6Z8XGxqJPnz4VFmFERNZiJYLoHldcXIxjx47h9OnTaNu2bV1Ph4juIbw7g+ge8O2336Jv375Vxh0cHPDQQw9hyJAhtTgrIrrX8XQG0T3gt99+w+XLl6uMh4WF1eJsiEgruIggIiIiq/CaCCIiIrIKFxFERERkFS4iiIiIyCpcRBAREZFVuIggIiIiq3ARQURERFbhIoKIiIiswkUEERERWeX/ANob3p6Nw2X2AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_as_patches(j)\n", "\n", "_ = plt.title('These slices are not diagonal')\n", "_ = plt.xlabel(\"Don't use `batch_jacobian`\")" ] }, { "cell_type": "markdown", "metadata": { "id": "M_x7ih5sarvG" }, "source": [ "在此示例中，`batch_jacobian` 仍然可以运行并返回*某些信息*与预期形状，但其内容具有不明确的含义：" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T22:53:35.685362Z", "iopub.status.busy": "2022-12-14T22:53:35.684863Z", "iopub.status.idle": "2022-12-14T22:53:36.035066Z", "shell.execute_reply": "2022-12-14T22:53:36.034368Z" }, "id": "k8_mICHoasCi" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "jb.shape: (7, 6, 5)\n" ] } ], "source": [ "jb = tape.batch_jacobian(y, x)\n", "print(f'jb.shape: {jb.shape}')" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "advanced_autodiff.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" } }, "nbformat": 4, "nbformat_minor": 0 }