{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Tce3stUlHN0L" }, "source": [ "##### Copyright 2020 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2024-08-15T02:32:10.500640Z", "iopub.status.busy": "2024-08-15T02:32:10.500391Z", "iopub.status.idle": "2024-08-15T02:32:10.504295Z", "shell.execute_reply": "2024-08-15T02:32:10.503718Z" }, "id": "tuOe1ymfHZPu" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "qFdPvlXBOdUN" }, "source": [ "# Advanced automatic differentiation" ] }, { "cell_type": "markdown", "metadata": { "id": "MfBg1C5NB3X0" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "8a859404ce7e" }, "source": [ "The [Introduction to gradients and automatic differentiation](autodiff.ipynb) guide includes everything required to calculate gradients in TensorFlow. This guide focuses on deeper, less common features of the `tf.GradientTape` API." ] }, { "cell_type": "markdown", "metadata": { "id": "MUXex9ctTuDB" }, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:10.507922Z", "iopub.status.busy": "2024-08-15T02:32:10.507705Z", "iopub.status.idle": "2024-08-15T02:32:13.150344Z", "shell.execute_reply": "2024-08-15T02:32:13.149628Z" }, "id": "IqR2PQG4ZaZ0" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-08-15 02:32:10.761137: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", "2024-08-15 02:32:10.782161: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", "2024-08-15 02:32:10.788607: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n" ] } ], "source": [ "import tensorflow as tf\n", "\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "\n", "mpl.rcParams['figure.figsize'] = (8, 6)" ] }, { "cell_type": "markdown", "metadata": { "id": "uGRJJRi8TCkJ" }, "source": [ "## Controlling gradient recording\n", "\n", "In the [automatic differentiation guide](autodiff.ipynb) you saw how to control which variables and tensors are watched by the tape while building the gradient calculation.\n", "\n", "The tape also has methods to manipulate the recording." ] }, { "cell_type": "markdown", "metadata": { "id": "gB_i0VnhQKt2" }, "source": [ "### Stop recording\n", "\n", "If you wish to stop recording gradients, you can use `tf.GradientTape.stop_recording` to temporarily suspend recording.\n", "\n", "This may be useful to reduce overhead if you do not wish to differentiate a complicated operation in the middle of your model. This could include calculating a metric or an intermediate result:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:13.154876Z", "iopub.status.busy": "2024-08-15T02:32:13.154063Z", "iopub.status.idle": "2024-08-15T02:32:15.357653Z", "shell.execute_reply": "2024-08-15T02:32:15.356975Z" }, "id": "mhFSYf7uQWxR" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dz/dx: tf.Tensor(4.0, shape=(), dtype=float32)\n", "dz/dy: None\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", "I0000 00:00:1723689133.642575 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.646496 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.650243 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.653354 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.664545 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.668230 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.671627 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.674592 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.677498 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.680982 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.684370 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689133.687370 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.924735 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.926905 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.928886 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.930883 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.932919 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.934914 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.936798 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.938737 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.940666 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.942634 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.944517 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.946466 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.984712 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.986787 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.988685 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.990637 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.993156 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.995163 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.997026 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689134.998929 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689135.000885 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689135.003378 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689135.005704 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "I0000 00:00:1723689135.008045 116670 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n" ] } ], "source": [ "x = tf.Variable(2.0)\n", "y = tf.Variable(3.0)\n", "\n", "with tf.GradientTape() as t:\n", " x_sq = x * x\n", " with t.stop_recording():\n", " y_sq = y * y\n", " z = x_sq + y_sq\n", "\n", "grad = t.gradient(z, {'x': x, 'y': y})\n", "\n", "print('dz/dx:', grad['x']) # 2*x => 4\n", "print('dz/dy:', grad['y'])" ] }, { "cell_type": "markdown", "metadata": { "id": "DEHbEZ1h4p8A" }, "source": [ "### Reset/start recording from scratch\n", "\n", "If you wish to start over entirely, use `tf.GradientTape.reset`. Simply exiting the gradient tape block and restarting is usually easier to read, but you can use the `reset` method when exiting the tape block is difficult or impossible." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:15.361000Z", "iopub.status.busy": "2024-08-15T02:32:15.360750Z", "iopub.status.idle": "2024-08-15T02:32:15.368010Z", "shell.execute_reply": "2024-08-15T02:32:15.367418Z" }, "id": "lsMHsmrh4pqM" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dz/dx: tf.Tensor(4.0, shape=(), dtype=float32)\n", "dz/dy: None\n" ] } ], "source": [ "x = tf.Variable(2.0)\n", "y = tf.Variable(3.0)\n", "reset = True\n", "\n", "with tf.GradientTape() as t:\n", " y_sq = y * y\n", " if reset:\n", " # Throw out all the tape recorded so far.\n", " t.reset()\n", " z = x * x + y_sq\n", "\n", "grad = t.gradient(z, {'x': x, 'y': y})\n", "\n", "print('dz/dx:', grad['x']) # 2*x => 4\n", "print('dz/dy:', grad['y'])" ] }, { "cell_type": "markdown", "metadata": { "id": "6zS7cLmS6zMf" }, "source": [ "## Stop gradient flow with precision\n", "\n", "In contrast to the global tape controls above, the `tf.stop_gradient` function is much more precise. It can be used to stop gradients from flowing along a particular path, without needing access to the tape itself:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:15.371465Z", "iopub.status.busy": "2024-08-15T02:32:15.370862Z", "iopub.status.idle": "2024-08-15T02:32:15.772901Z", "shell.execute_reply": "2024-08-15T02:32:15.772211Z" }, "id": "30qnZMe48BkB" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dz/dx: tf.Tensor(4.0, shape=(), dtype=float32)\n", "dz/dy: None\n" ] } ], "source": [ "x = tf.Variable(2.0)\n", "y = tf.Variable(3.0)\n", "\n", "with tf.GradientTape() as t:\n", " y_sq = y**2\n", " z = x**2 + tf.stop_gradient(y_sq)\n", "\n", "grad = t.gradient(z, {'x': x, 'y': y})\n", "\n", "print('dz/dx:', grad['x']) # 2*x => 4\n", "print('dz/dy:', grad['y'])" ] }, { "cell_type": "markdown", "metadata": { "id": "mbb-9lnGVngH" }, "source": [ "## Custom gradients\n", "\n", "In some cases, you may want to control exactly how gradients are calculated rather than using the default. These situations include:\n", "\n", "1. There is no defined gradient for a new op you are writing.\n", "2. The default calculations are numerically unstable.\n", "3. You wish to cache an expensive computation from the forward pass.\n", "4. You want to modify a value (for example, using `tf.clip_by_value` or `tf.math.round`) without modifying the gradient.\n", "\n", "For the first case, to write a new op you can use `tf.RegisterGradient` to set up your own (refer to the API docs for details). (Note that the gradient registry is global, so change it with caution.)\n", "\n", "For the latter three cases, you can use `tf.custom_gradient`." ] }, { "cell_type": "markdown", "metadata": { "id": "oHr31kc_irF_" }, "source": [ "Here is an example that applies `tf.clip_by_norm` to the intermediate gradient:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:15.776671Z", "iopub.status.busy": "2024-08-15T02:32:15.776398Z", "iopub.status.idle": "2024-08-15T02:32:16.184748Z", "shell.execute_reply": "2024-08-15T02:32:16.184058Z" }, "id": "Mjj01w4NYtwd" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(2.0, shape=(), dtype=float32)\n" ] } ], "source": [ "# Establish an identity operation, but clip during the gradient pass.\n", "@tf.custom_gradient\n", "def clip_gradients(y):\n", " def backward(dy):\n", " return tf.clip_by_norm(dy, 0.5)\n", " return y, backward\n", "\n", "v = tf.Variable(2.0)\n", "with tf.GradientTape() as t:\n", " output = clip_gradients(v * v)\n", "print(t.gradient(output, v)) # calls \"backward\", which clips 4 to 2" ] }, { "cell_type": "markdown", "metadata": { "id": "n4t7S0scYrD3" }, "source": [ "Refer to the `tf.custom_gradient` decorator API docs for more details." ] }, { "cell_type": "markdown", "metadata": { "id": "v0ODp4Oi--I0" }, "source": [ "### Custom gradients in SavedModel\n", "\n", "Note: This feature is available from TensorFlow 2.6.\n", "\n", "Custom gradients can be saved to SavedModel by using the option `tf.saved_model.SaveOptions(experimental_custom_gradients=True)`.\n", "\n", "To be saved into the SavedModel, the gradient function must be traceable (to learn more, check out the [Better performance with tf.function](function.ipynb) guide)." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:16.188528Z", "iopub.status.busy": "2024-08-15T02:32:16.188273Z", "iopub.status.idle": "2024-08-15T02:32:16.192935Z", "shell.execute_reply": "2024-08-15T02:32:16.192301Z" }, "id": "Q5JBgIBYjN1I" }, "outputs": [], "source": [ "class MyModule(tf.Module):\n", "\n", " @tf.function(input_signature=[tf.TensorSpec(None)])\n", " def call_custom_grad(self, x):\n", " return clip_gradients(x)\n", "\n", "model = MyModule()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:16.196064Z", "iopub.status.busy": "2024-08-15T02:32:16.195809Z", "iopub.status.idle": "2024-08-15T02:32:16.474435Z", "shell.execute_reply": "2024-08-15T02:32:16.473627Z" }, "id": "xZTrgy2q-9pq" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Assets written to: saved_model/assets\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Assets written to: saved_model/assets\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor(2.0, shape=(), dtype=float32)\n" ] } ], "source": [ "tf.saved_model.save(\n", " model,\n", " 'saved_model',\n", " options=tf.saved_model.SaveOptions(experimental_custom_gradients=True))\n", "\n", "# The loaded gradients will be the same as the above example.\n", "v = tf.Variable(2.0)\n", "loaded = tf.saved_model.load('saved_model')\n", "with tf.GradientTape() as t:\n", " output = loaded.call_custom_grad(v * v)\n", "print(t.gradient(output, v))" ] }, { "cell_type": "markdown", "metadata": { "id": "d-LfRs5FbJCk" }, "source": [ "A note about the above example: If you try replacing the above code with `tf.saved_model.SaveOptions(experimental_custom_gradients=False)`, the gradient will still produce the same result on loading. The reason is that the gradient registry still contains the custom gradient used in the function `call_custom_op`. However, if you restart the runtime after saving without custom gradients, running the loaded model under the `tf.GradientTape` will throw the error: `LookupError: No gradient defined for operation 'IdentityN' (op type: IdentityN)`." ] }, { "cell_type": "markdown", "metadata": { "id": "8aENEt6Veryb" }, "source": [ "## Multiple tapes\n", "\n", "Multiple tapes interact seamlessly.\n", "\n", "For example, here each tape watches a different set of tensors:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:16.478620Z", "iopub.status.busy": "2024-08-15T02:32:16.477887Z", "iopub.status.idle": "2024-08-15T02:32:16.546935Z", "shell.execute_reply": "2024-08-15T02:32:16.546147Z" }, "id": "BJ0HdMvte0VZ" }, "outputs": [], "source": [ "x0 = tf.constant(0.0)\n", "x1 = tf.constant(0.0)\n", "\n", "with tf.GradientTape() as tape0, tf.GradientTape() as tape1:\n", " tape0.watch(x0)\n", " tape1.watch(x1)\n", "\n", " y0 = tf.math.sin(x0)\n", " y1 = tf.nn.sigmoid(x1)\n", "\n", " y = y0 + y1\n", "\n", " ys = tf.reduce_sum(y)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:16.550694Z", "iopub.status.busy": "2024-08-15T02:32:16.550376Z", "iopub.status.idle": "2024-08-15T02:32:16.561049Z", "shell.execute_reply": "2024-08-15T02:32:16.560472Z" }, "id": "6ApAoMNFfNz6" }, "outputs": [ { "data": { "text/plain": [ "1.0" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tape0.gradient(ys, x0).numpy() # cos(x) => 1.0" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:16.564124Z", "iopub.status.busy": "2024-08-15T02:32:16.563895Z", "iopub.status.idle": "2024-08-15T02:32:16.570416Z", "shell.execute_reply": "2024-08-15T02:32:16.569822Z" }, "id": "rF1jrAJsfYW_" }, "outputs": [ { "data": { "text/plain": [ "0.25" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tape1.gradient(ys, x1).numpy() # sigmoid(x1)*(1-sigmoid(x1)) => 0.25" ] }, { "cell_type": "markdown", "metadata": { "id": "DK05KXrAAld3" }, "source": [ "### Higher-order gradients\n", "\n", "Operations inside of the `tf.GradientTape` context manager are recorded for automatic differentiation. If gradients are computed in that context, then the gradient computation is recorded as well. As a result, the exact same API works for higher-order gradients as well.\n", "\n", "For example:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:16.573544Z", "iopub.status.busy": "2024-08-15T02:32:16.573317Z", "iopub.status.idle": "2024-08-15T02:32:16.582207Z", "shell.execute_reply": "2024-08-15T02:32:16.581596Z" }, "id": "cPQgthZ7ugRJ" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dy_dx: 3.0\n", "d2y_dx2: 6.0\n" ] } ], "source": [ "x = tf.Variable(1.0) # Create a Tensorflow variable initialized to 1.0\n", "\n", "with tf.GradientTape() as t2:\n", " with tf.GradientTape() as t1:\n", " y = x * x * x\n", "\n", " # Compute the gradient inside the outer `t2` context manager\n", " # which means the gradient computation is differentiable as well.\n", " dy_dx = t1.gradient(y, x)\n", "d2y_dx2 = t2.gradient(dy_dx, x)\n", "\n", "print('dy_dx:', dy_dx.numpy()) # 3 * x**2 => 3.0\n", "print('d2y_dx2:', d2y_dx2.numpy()) # 6 * x => 6.0" ] }, { "cell_type": "markdown", "metadata": { "id": "k0HV-Ah4_76i" }, "source": [ "While that does give you the second derivative of a _scalar_ function, this pattern does not generalize to produce a Hessian matrix, since `tf.GradientTape.gradient` only computes the gradient of a scalar. To construct a [Hessian matrix](https://en.wikipedia.org/wiki/Hessian_matrix), go to the [Hessian example](#hessian) under the [Jacobian section](#jacobians).\n", "\n", "\"Nested calls to `tf.GradientTape.gradient`\" is a good pattern when you are calculating a scalar from a gradient, and then the resulting scalar acts as a source for a second gradient calculation, as in the following example.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "t7LRlcpVKHv1" }, "source": [ "#### Example: Input gradient regularization\n", "\n", "Many models are susceptible to \"adversarial examples\". This collection of techniques modifies the model's input to confuse the model's output. The simplest implementation—such as the [Adversarial example using the Fast Gradient Signed Method attack](https://www.tensorflow.org/tutorials/generative/adversarial_fgsm)—takes a single step along the gradient of the output with respect to the input; the \"input gradient\".\n", "\n", "One technique to increase robustness to adversarial examples is [input gradient regularization](https://arxiv.org/abs/1905.11468) (Finlay & Oberman, 2019), which attempts to minimize the magnitude of the input gradient. If the input gradient is small, then the change in the output should be small too.\n", "\n", "Below is a naive implementation of input gradient regularization. The implementation is:\n", "\n", "1. Calculate the gradient of the output with respect to the input using an inner tape.\n", "2. Calculate the magnitude of that input gradient.\n", "3. Calculate the gradient of that magnitude with respect to the model." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:16.585351Z", "iopub.status.busy": "2024-08-15T02:32:16.585116Z", "iopub.status.idle": "2024-08-15T02:32:16.597575Z", "shell.execute_reply": "2024-08-15T02:32:16.596989Z" }, "id": "tH3ZFuUfDLrR" }, "outputs": [], "source": [ "x = tf.random.normal([7, 5])\n", "\n", "layer = tf.keras.layers.Dense(10, activation=tf.nn.relu)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:16.600741Z", "iopub.status.busy": "2024-08-15T02:32:16.600515Z", "iopub.status.idle": "2024-08-15T02:32:17.414971Z", "shell.execute_reply": "2024-08-15T02:32:17.414200Z" }, "id": "E6yOFsjEDR9u" }, "outputs": [], "source": [ "with tf.GradientTape() as t2:\n", " # The inner tape only takes the gradient with respect to the input,\n", " # not the variables.\n", " with tf.GradientTape(watch_accessed_variables=False) as t1:\n", " t1.watch(x)\n", " y = layer(x)\n", " out = tf.reduce_sum(layer(x)**2)\n", " # 1. Calculate the input gradient.\n", " g1 = t1.gradient(out, x)\n", " # 2. Calculate the magnitude of the input gradient.\n", " g1_mag = tf.norm(g1)\n", "\n", "# 3. Calculate the gradient of the magnitude with respect to the model.\n", "dg1_mag = t2.gradient(g1_mag, layer.trainable_variables)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:17.419534Z", "iopub.status.busy": "2024-08-15T02:32:17.418960Z", "iopub.status.idle": "2024-08-15T02:32:17.423751Z", "shell.execute_reply": "2024-08-15T02:32:17.423140Z" }, "id": "123QMq6PqK_d" }, "outputs": [ { "data": { "text/plain": [ "[TensorShape([5, 10]), TensorShape([10])]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[var.shape for var in dg1_mag]" ] }, { "cell_type": "markdown", "metadata": { "id": "E4xiYigexMtQ" }, "source": [ "## Jacobians\n" ] }, { "cell_type": "markdown", "metadata": { "id": "4-hVHVIeExkI" }, "source": [ "All the previous examples took the gradients of a scalar target with respect to some source tensor(s).\n", "\n", "The [Jacobian matrix](https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant) represents the gradients of a vector valued function. Each row contains the gradient of one of the vector's elements.\n", "\n", "The `tf.GradientTape.jacobian` method allows you to efficiently calculate a Jacobian matrix." ] }, { "cell_type": "markdown", "metadata": { "id": "KzNyIM0QBYIH" }, "source": [ "Note that:\n", "\n", "* Like `gradient`: The `sources` argument can be a tensor or a container of tensors.\n", "* Unlike `gradient`: The `target` tensor must be a single tensor." ] }, { "cell_type": "markdown", "metadata": { "id": "O74K3hlxBC8a" }, "source": [ "### Scalar source" ] }, { "cell_type": "markdown", "metadata": { "id": "B08OKn1Orkuc" }, "source": [ "As a first example, here is the Jacobian of a vector-target with respect to a scalar-source." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:17.427622Z", "iopub.status.busy": "2024-08-15T02:32:17.426995Z", "iopub.status.idle": "2024-08-15T02:32:18.258955Z", "shell.execute_reply": "2024-08-15T02:32:18.258207Z" }, "id": "bAFeIE8EuVIq" }, "outputs": [], "source": [ "x = tf.linspace(-10.0, 10.0, 200+1)\n", "delta = tf.Variable(0.0)\n", "\n", "with tf.GradientTape() as tape:\n", " y = tf.nn.sigmoid(x+delta)\n", "\n", "dy_dx = tape.jacobian(y, delta)" ] }, { "cell_type": "markdown", "metadata": { "id": "BgHbUk3zr-WU" }, "source": [ "When you take the Jacobian with respect to a scalar the result has the shape of the **target**, and gives the gradient of the each element with respect to the source:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.262973Z", "iopub.status.busy": "2024-08-15T02:32:18.262722Z", "iopub.status.idle": "2024-08-15T02:32:18.266543Z", "shell.execute_reply": "2024-08-15T02:32:18.265859Z" }, "id": "iZ6awnDzr_BA" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(201,)\n", "(201,)\n" ] } ], "source": [ "print(y.shape)\n", "print(dy_dx.shape)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.269421Z", "iopub.status.busy": "2024-08-15T02:32:18.269170Z", "iopub.status.idle": "2024-08-15T02:32:18.439623Z", "shell.execute_reply": "2024-08-15T02:32:18.439009Z" }, "id": "siNZaklc0_-e" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAp8AAAINCAYAAAB4RhRAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABdXElEQVR4nO3dd3hUZf7+8XtmMpNegJAChN6lSonBgiWK6KKsZbH8BMEua1lWV7Gxul/FtsquDXUVdV0V3RUbCGIUFEFQigLSCS2QkFDSk0lmzu+PSQKBBBJI5kx5v65rrpk55zlnPjOOw53znPM8FsMwDAEAAABeYDW7AAAAAAQPwicAAAC8hvAJAAAAryF8AgAAwGsInwAAAPAawicAAAC8hvAJAAAAryF8AgAAwGtCzC6gIdxut3bv3q3o6GhZLBazywEAAMARDMNQYWGh2rRpI6u1/uObfhE+d+/erZSUFLPLAAAAwHHs3LlT7dq1q3e9X4TP6OhoSZ43ExMTY3I1AAAAOFJBQYFSUlJqclt9/CJ8Vne1x8TEED4BAAB82PFOkeSCIwAAAHgN4RMAAABeQ/gEAACA1/jFOZ8N4XK5VFFRYXYZAcdmsykkJIQhrgAAQJMIiPBZVFSkXbt2yTAMs0sJSBEREUpOTpbD4TC7FAAA4Of8Pny6XC7t2rVLERERat26NUfompBhGHI6ncrNzVVmZqa6det2zEFjAQAAjsfvw2dFRYUMw1Dr1q0VHh5udjkBJzw8XHa7Xdu3b5fT6VRYWJjZJQEAAD8WMIexOOLZfDjaCQAAmgqpAgAAAF5D+AQAAIDXED4BAADgNYRPAAAAeA3h0yTvvPOOWrVqpfLy8lrLR48ereuuu86kqgAAAJpXo8Pnd999p1GjRqlNmzayWCz65JNPjrvNggULdOqppyo0NFRdu3bVW2+9dQKlNoxhGCpxVppya8wg91deeaVcLpc+++yzmmV79+7V7NmzNWHChOb4aAAAAEzX6HE+i4uL1b9/f02YMEGXXXbZcdtnZmbq4osv1q233qr//Oc/ysjI0I033qjk5GSNGDHihIo+ltIKl3o/Mq/J99sQvz02QhGOhn2k4eHhuuaaazRjxgxdeeWVkqR3331X7du319lnn92MVQIAAJin0eFz5MiRGjlyZIPbT58+XZ06ddLf//53SVKvXr20aNEiPf/8880SPv3JTTfdpCFDhigrK0tt27bVW2+9peuvv54xSwEAQMBq9hmOlixZovT09FrLRowYobvvvrtZXi/cbtNvj5kTasPttka1HzhwoPr376933nlHF1xwgdauXavZs2c3U3UAAPgXwzDkchtyVd1Xug0Zhme5YUhuw5DbkAxVL/csMyS53Z5T4dxHtJWqtjFqr6tu667af11tPTUdVp+M6geH39XZxjiiTfX7q7XsWNvVWlf39rXP/vM8SWkZoVPaxMqXNHv4zM7OVmJiYq1liYmJKigoUGlpaZ1TYpaXl9e6EKegoKDBr2exWBrc9e0LbrzxRk2bNk1ZWVlKT09XSkqK2SUBAIKEYRgqr3SrrMKlEqdLpRUulR52X+J0qazC87y8wiWny60Kl2cbZ9WtwlX1+Mj7qnUVbkPuquDocrs9YbLmuVHrufvw5VWBEyfnutM66G+jgyx8noipU6fq0UcfNbsMr7jmmmt0zz336PXXX9c777xjdjkAAD9T6nQpr6hc+4udOlDiVEFZpQpKK1RYVqmCsgoVllWooLTSc3/YusKyCpVWuBQI+c5q8Rx8slokiyyyWCSLRbJaLLKo6v7wNlX3UvVzTxtr1WlvVqtnP9VtLRbJokNTeR9+clz1mXKWqqV1nTl35HaHtznW9pYjGtX9unW/RnWbDq0iji7IZM0ePpOSkpSTk1NrWU5OjmJiYuo86ilJkydP1qRJk2qeFxQUBOwRwdjYWF1++eWaPXu2Ro8ebXY5AAAfYBiGCsoqtSe/VLsPlmr3wTLtyS9VXqFT+4rLlVfkud9X5FSJ09Ukr2m3WRRutyncYVOEI0RhdpsiHDaF220Ks9sUarfKYau6hVhlr7p3hFjlsFmq7q1yhNhkP+x5iM2qEKtFtsNuRz+3HrW8rjZWa92BEv6l2cNnWlqa5syZU2vZ/PnzlZaWVu82oaGhCg0Nbe7SfEZWVpauvfbaoHrPABDsyipc2r6vRJl5RcrMK9GO/SXafbC0KnCWqai8ssH7coRYFR/pUFyEQzHhIYoJsys6zK6Y8BDPfZhn2eHrosJCFOmwKawqYNptDP0N72h0+CwqKtLmzZtrnmdmZmrVqlVq2bKl2rdvr8mTJysrK6umC/nWW2/Viy++qL/85S+aMGGCvvnmG3344YdcWCPpwIEDWrBggRYsWKCXX37Z7HIAAM2grMKlTTlFWpddoPV7CrUxp1CZecXanV+q4w0P3SLCruTYcLWJC1ebuDC1jgpVq6hQtYpyKD7KoVaRnsdRoSEcAYTfaHT4/Pnnn3XOOefUPK/uHh83bpzeeust7dmzRzt27KhZ36lTJ82ePVt/+tOf9I9//EPt2rXTv/71r6AfZknyXO1+4MABPfXUU+rRo4fZ5QAATlJ5pUtrdxdo5Y6DWrXzoH7bna/MvOJ6z6uMDgtR5/hIdYqPVPtWkWobF1YVNMOVHBvmVxfQAg1lMRozLY9JCgoKFBsbq/z8fMXExNRaV1ZWpszMTHXq1ElhYWEmVRjY+IwBoG4Hip1asnWflmXu18qqsFnhOvqf1RYRdvVKjlGPpGj1TIpWl9ZR6hQfqZaRDo5YImAcK68djj+pAABooFKnSz9t268ftuTph815Wru74Kiu81aRDg1sH6eB7VuoT9tY9UqKVuvoUEImUIXwCQDAMewvdurrdTmatyZb32/Ok7PSXWt998QoDesSr4Ht43Rq+xZq1yKcoAkcA+ETAIAj5BSUad7abM1dk62lmftrDXaeHBum07vG64yu8RrWpZUSYjgdCWgMwicAAJKclW59sz5HM3/aqYUbc2tdJNQ7OUYjTknSiD6J6pEYzZFN4CQQPgEAQW3z3kLN/GmnPl6RpX3Fzprlp7aP08g+yRpxSpLa++AsMYC/InwCAIKOYRj6flOepi/cosVb9tUsbx0dqisGtdOVg9qpc+soEysEAhfh08ecffbZGjBggKZNm9Zk+7RYLJo1axbTdwIIepUut75ck63pC7do7e4CSZLNatG5PRM0ZnCKzu7RWiHM9AM0K8JnAHj00Ue1adMmvfvuu2aXAgA+yVnp1oc/79Rr323Vjv0lkqRwu01XDU3RjWd2Vtu4cJMrBIIH4TMAfPrpp7r//vvNLgMAfI7bbejzX3fr719trAmdLSLsGjeso8aldVSLSIfJFQLBh74FExUXF2vs2LGKiopScnKy/v73v9ese+yxx9SnT5+jthkwYIAefvjhmuc7d+7U2rVrdeGFF0qSNm3apLPOOkthYWHq3bu35s+fX2v7d955R1FRUdq0aVPNsttvv109e/ZUSUlJU79FADDNT9v2a9SLi3TXB6u0Y3+J4qNC9cjveuuH+8/V3endCZ6ASQLvyKdhSBUmhSh7hNSI4TfuvfdeLVy4UJ9++qkSEhL0wAMPaMWKFRowYIAmTJigRx99VD/99JOGDBkiSVq5cqV+/fVXffzxxzX7+Oyzz3T22WcrJiZGbrdbl112mRITE7V06VLl5+fr7rvvrvWaY8eO1RdffKFrr71Wixcv1rx58/Svf/1LS5YsUUQEV3MC8H/Z+WWa+uU6fbpqtyQpOjREtwzvrAlndGKudMAHBN7/hRUl0hNtzHntB3ZLjsgGNS0qKtIbb7yhd999V+edd54k6e2331a7du0kSe3atdOIESM0Y8aMmvA5Y8YMDR8+XJ07d67Zz6effqpLL71UkvT1119r/fr1mjdvntq08XwGTzzxhEaOHFnrtV999VX169dPd955pz7++GP99a9/1aBBg07uvQOAydxuQ/9Zul1Pzd2govJKWSzSVUPa654LuqtVVKjZ5QGoQre7SbZs2SKn06nU1NSaZS1btlSPHj1qnt900016//33VVZWJqfTqffee08TJkyoWV9QUKCFCxfqkksukSStW7dOKSkpNcFTktLS0o567RYtWuiNN97QK6+8oi5dunC+KAC/tyW3SH94dYke/nStisorNbB9nD7/4xmaellfgifgYwLvyKc9wnME0qzXbkKjRo1SaGioZs2aJYfDoYqKCl1xxRU167/88kv17t1bKSkpjd73d999J5vNpj179qi4uFjR0dFNWToAeIVhGHp36Q49Pvs3lVW4FeGw6S8jeui6tI6yWZmFCPBFgRc+LZYGd32bqUuXLrLb7Vq6dKnat28vSTpw4IA2btyo4cOHS5JCQkI0btw4zZgxQw6HQ1dddZXCww8NB3J4l7sk9erVSzt37tSePXuUnJwsSfrxxx+Peu3Fixfrqaee0ueff6777rtPf/zjH/X2228359sFgCaXV1Suv/z3V32zfq8k6Yyu8Xry8r5q14Lz1wFfFnjh009ERUXphhtu0L333qtWrVopISFBDz74oKzW2mdC3HjjjerVq5ck6YcffqhZXllZqS+//FL33HNPzbL09HR1795d48aN0zPPPKOCggI9+OCDtfZXWFio6667TnfeeadGjhypdu3aaciQIRo1alSto6oA4Mt+3rZfE99boZyCcjlCrLrvwp4aP6yjrBztBHwe4dNEzzzzjIqKijRq1ChFR0frz3/+s/Lz82u16datm4YNG6b9+/fXOj904cKFioqK0qmnnlqzzGq1atasWbrhhhs0dOhQdezYUf/85z9rhmGSpLvuukuRkZF64oknJEl9+/bVE088oVtuuUVpaWlq27ZtM79rADhxhmHozR+2aeqcdap0G+rSOlIvXXuqeibFmF0agAayGIZhmF3E8RQUFCg2Nlb5+fmKian9A1NWVqbMzEx16tRJYWFhJlXYfAzDULdu3XT77bdr0qRJNcvvvPNOVVZW6uWXX272GgL9MwbgH8orXZr8v9X6eGWWJOl3/ZL15OX9FBXKcRTAFxwrrx2O/2N9WG5urj744ANlZ2dr/Pjxtdb16dOnzivZASAQ7Ssq1y3/Xq6ftx+QzWrRQxf30vXDOsrSiLGVAfgGwqcPS0hIUHx8vF577TW1aNGi1rqbb77ZpKoAwLu25hZp3Ixl2rm/VNFhIXr52lN1ZrfWZpcF4AQRPn2YH5wRAQDNak1Wvsa9uUz7ip1q3zJCb14/WF0TGBoO8GeETwCAT1qyZZ9ueudnFZVXqk/bGL01fqjiGTAe8HuETwCAz/luY65ueudnlVe6dVrnlnp97GBFh9nNLgtAEwiY8EkXdfPhswXgTd9vOhQ803sl6MVrTlWY3WZ2WQCaiN/P7W6zeX6QnE6nyZUErpKSEkmS3c5RBwDNa9GmPN34dnXwTNTL1w4ieAIBxu+PfIaEhCgiIkK5ubmy2+1HzRCEE2cYhkpKSrR3717FxcXVBH0AaA7Ltx847Ihnol6+9lQ5QvhNBwKN34dPi8Wi5ORkZWZmavv27WaXE5Di4uKUlJRkdhkAAtjGnEJNeOsnlVa4NLx7a4InEMD8PnxKksPhULdu3eh6bwZ2u50jngCa1a4DJRr7xjLll1bo1PZxeuX/ETyBQBYQ4VPyzGvO1I8A4F8Kyio0fsZPyi4oU/fEKL15/RBFOALmnyYAdeBPSwCAKSpdbv3xvZXatLdIiTGhenvCUMVFOMwuC0AzI3wCAEzxty9+03cbcxVut+mNcUOUHBtudkkAvIDwCQDwund/3K63l2yXxSJNu2qA+rSNNbskAF5C+AQAeNXKHQf06OdrJUn3juihEacwmgYQTAifAACv2VdUrtv/s0IVLkMj+yTptuFdzC4JgJcRPgEAXuFyG7rzg5Xak1+mzvGRevqKfrJYLGaXBcDLCJ8AAK948ZvN+mHzPoXbbZp+3SBFhzFlLxCMCJ8AgGa3fPt+/fObTZKkx3/fR90To02uCIBZCJ8AgGZVUFahuz5YJZfb0OgBbXTZqe3MLgmAiQifAIBm9cgna7TrQKnatQjXY6P7mF0OAJMRPgEAzWb2r3v0yardslkt+sdVAxXDeZ5A0CN8AgCaxb6icj3y6RpJ0u1nd9GgDi1MrgiALyB8AgCaxZTP1mpfsVM9k6J1x7ndzC4HgI8gfAIAmtyXq/foi1/3yGa16Jkr+ssRwj83ADz4NQAANKn8kgo9XNXdfuvwzurbjnnbARxC+AQANKmn561XXpFTXVpH6s7z6G4HUBvhEwDQZFbuOKD3lu2QJP3f6L4KDbGZXBEAX0P4BAA0iUqXWw/OWiPDkC4b2FZpXVqZXRIAH0T4BAA0iXeWbNdvewoUExaiBy7uZXY5AHwU4RMAcNL2FZXr+fkbJUn3jeyp+KhQkysC4KsInwCAk/bc/I0qLK/UKW1idNWQ9maXA8CHET4BACdlfXaB3q+6yOiR3/WWzWoxuSIAvozwCQA4YYZh6G9f/Ca3IV3UN0mpnbnICMCxET4BACcsY91e/bB5nxw2qyaP5CIjAMdH+AQAnJBKl1tPzl0vSZpwRieltIwwuSIA/oDwCQA4IR+vzNLmvUWKi7Dr9nO6mF0OAD9B+AQANFpZhUvTqoZWuv3sLooJs5tcEQB/QfgEADTauz9u1+78MiXHhmlsWkezywHgRwifAIBGKSir0EvfbpYk3Z3eTWF25m8H0HCETwBAo8xYtE0HSirUuXWkLj+1ndnlAPAzhE8AQIMVlFXojUVbJUl/Su+uEBv/jABoHH41AAAN9vYP21RQVqmuCVG6qG+y2eUA8EOETwBAgxSWVehfizIlSXec25VpNAGcEMInAKBB3lmyXfmlFerSOlK/69fG7HIA+CnCJwDguIrKK/X6955zPe84txtHPQGcMMInAOC43l+6QwdLKtQ5PlKj+nPUE8CJI3wCAI7JWenWG1Xnet4yvDNHPQGcFMInAOCYPl2VpeyCMiXGhGr0wLZmlwPAzxE+AQD1crsNvfqd51zPCad3UmgIsxkBODmETwBAvTLW79XmvUWKDg3RNantzS4HQAAgfAIA6jV94RZJ0rWndVB0mN3kagAEAsInAKBOK3Yc0PLtB+SwWTXh9I5mlwMgQBA+AQB1mvHDNknSJQPaKCEmzNxiAAQMwicA4Ch78ks1Z/UeSdJ4jnoCaEInFD5feukldezYUWFhYUpNTdWyZcuO2X7atGnq0aOHwsPDlZKSoj/96U8qKys7oYIBAM3v30u2y+U2lNqppU5pE2t2OQACSKPD58yZMzVp0iRNmTJFK1asUP/+/TVixAjt3bu3zvbvvfee7r//fk2ZMkXr1q3TG2+8oZkzZ+qBBx446eIBAE2v1OnSe8t2SJImnNHJ5GoABJpGh8/nnntON910k8aPH6/evXtr+vTpioiI0Jtvvlln+8WLF+v000/XNddco44dO+qCCy7Q1VdffdyjpQAAc8xamaWDJRVKaRmu9F6JZpcDIMA0Knw6nU4tX75c6enph3ZgtSo9PV1Lliypc5thw4Zp+fLlNWFz69atmjNnji666KKTKBsA0BwMw9Bbiz1TaY5L68hUmgCaXEhjGufl5cnlcikxsfZfwomJiVq/fn2d21xzzTXKy8vTGWecIcMwVFlZqVtvvfWY3e7l5eUqLy+veV5QUNCYMgEAJ2hZ5n5tzClSuN2mPwxJMbscAAGo2a92X7BggZ544gm9/PLLWrFihT7++GPNnj1bf/vb3+rdZurUqYqNja25paTwAwgA3vDuUs+5nqMHtlEMg8oDaAaNOvIZHx8vm82mnJycWstzcnKUlJRU5zYPP/ywrrvuOt14442SpL59+6q4uFg333yzHnzwQVmtR+ffyZMna9KkSTXPCwoKCKAA0MxyC8s1d41neKVrUzuYXA2AQNWoI58Oh0ODBg1SRkZGzTK3262MjAylpaXVuU1JSclRAdNms0nynFtUl9DQUMXExNS6AQCa14c/71SFy9CAlDj1acvwSgCaR6OOfErSpEmTNG7cOA0ePFhDhw7VtGnTVFxcrPHjx0uSxo4dq7Zt22rq1KmSpFGjRum5557TwIEDlZqaqs2bN+vhhx/WqFGjakIoAMBcLreh96q63P/faRz1BNB8Gh0+x4wZo9zcXD3yyCPKzs7WgAEDNHfu3JqLkHbs2FHrSOdDDz0ki8Wihx56SFlZWWrdurVGjRqlxx9/vOneBQDgpCzYsFdZB0sVG27X7/olm10OgABmMerr+/YhBQUFio2NVX5+Pl3wANAMbnjrJ2Ws36ubzuykBy/ubXY5APxQQ/Mac7sDQJDLzi/Ttxs8s9RdNbS9ydUACHSETwAIcv9bsUtuQxrSsYW6tI4yuxwAAY7wCQBBzO029OHPOyVJfxjMkHYAmh/hEwCC2NLM/dq+r0RRoSG6mAuNAHgB4RMAglj1Uc9R/ZMV4Wj0ACgA0GiETwAIUvmlFZqz2jOj0ZghXGgEwDsInwAQpD77ZbfKK93qkRit/u2Y0QiAdxA+ASBI/W/5LknSlYPbyWKxmFwNgGBB+ASAILQlt0irdh6UzWrRJQPamF0OgCBC+ASAIDRrRZYk6axu8UqIDjO5GgDBhPAJAEHG7TY0a6UnfF52ajuTqwEQbAifABBklmbuV9bBUkWHhej83olmlwMgyBA+ASDIfLzCc6HR7/olK8xuM7kaAMGG8AkAQaTU6aoZ25MudwBmIHwCQBD56rdsFTtdSmkZrsEdWphdDoAgRPgEgCDy+S+7JUmjB7RlbE8ApiB8AkCQOFji1MKNuZKkSxnbE4BJCJ8AECS+XJOtCpehXskx6poQbXY5AIIU4RMAgsSnqzxje17Sn6OeAMxD+ASAIJCdX6almfslSaP6J5tcDYBgRvgEgCDwxa+7ZRjS4A4t1K5FhNnlAAhihE8ACAKfVV3lzoVGAMxG+ASAALctr1i/7sqXzWrRRX3pcgdgLsInAAS4L9dkS5LSOrdSq6hQk6sBEOwInwAQ4L5c45lOc2TfJJMrAQDCJwAEtF0HSvTrrnxZLdIFvQmfAMxH+ASAADa3qst9aKeWah1NlzsA8xE+ASCAzVld1eXehwuNAPgGwicABKjs/DKt2HFQknRhH7rcAfgGwicABKi5VRcaDerQQokxYSZXAwAehE8ACFDVQyyN5KgnAB9C+ASAAJRbWK5l2zxzuY9kYHkAPoTwCQABaN7abBmG1L9drNrGhZtdDgDUIHwCQACqHmKJo54AfA3hEwACzP5ip5Zs3SeJ8z0B+B7CJwAEmPm/ZcvlNtQ7OUYdWkWaXQ4A1EL4BIAAU32V+0XM5Q7ABxE+ASCA5JdU6IfNeZKkC5nVCIAPInwCQAD5el2OKlyGuidGqWtClNnlAMBRCJ8AEEAODSzPUU8AvonwCQABotTp0qLNuZKkEadwvicA30T4BIAA8cPmPJVVuNU2Lly9kqPNLgcA6kT4BIAAkbE+R5KU3itBFovF5GoAoG6ETwAIAG63oa/X7ZUkndcr0eRqAKB+hE8ACACrs/KVW1iuqNAQpXZuaXY5AFAvwicABICv13m63M/qHq/QEJvJ1QBA/QifABAAqrvc0+lyB+DjCJ8A4Od2HSjRuj0Fslqkc3okmF0OABwT4RMA/Nw36z1HPQd3aKkWkQ6TqwGAYyN8AoCfm/+b53zP83px1BOA7yN8AoAfKyyr0I9b90mS0ntzvicA30f4BAA/9v2mPFW4DHWKj1SX1lFmlwMAx0X4BAA/Vj3EUjpd7gD8BOETAPyUy23o2/UMsQTAvxA+AcBPrdhxQAdKKhQbbtegDi3MLgcAGoTwCQB+6uuqq9zP7ZmgEBs/5wD8A79WAOCnqs/3ZIglAP6E8AkAfigzr1hbcotlt1l0VvfWZpcDAA1G+AQAP5RRddQztVMrxYTZTa4GABqO8AkAfmjBhlxJ0jk96XIH4F8InwDgZ4rLK7Usc78k6ewedLkD8C+ETwDwM0u27JPT5VZKy3B1jo80uxwAaBTCJwD4mQUbPQPLn909QRaLxeRqAKBxCJ8A4EcMw6g535MudwD+iPAJAH5kS26xdh0olcNmVVqXVmaXAwCNRvgEAD+yYIOnyz21c0tFOEJMrgYAGo/wCQB+pLrLfTgDywPwU4RPAPATtYdYYnxPAP6J8AkAfqJ6iKV2LcLVpTVDLAHwT4RPAPAT1UMsndODIZYA+C/CJwD4AYZYAhAoCJ8A4AcYYglAoCB8AoAfYIglAIHihMLnSy+9pI4dOyosLEypqalatmzZMdsfPHhQEydOVHJyskJDQ9W9e3fNmTPnhAoGgGDEEEsAAkWj/3yeOXOmJk2apOnTpys1NVXTpk3TiBEjtGHDBiUkHD30h9Pp1Pnnn6+EhAT997//Vdu2bbV9+3bFxcU1Rf0AEPAYYglAIGl0+Hzuued00003afz48ZKk6dOna/bs2XrzzTd1//33H9X+zTff1P79+7V48WLZ7XZJUseOHU+uagAIIgyxBCCQNKrb3el0avny5UpPTz+0A6tV6enpWrJkSZ3bfPbZZ0pLS9PEiROVmJioPn366IknnpDL5Tq5ygEgSFQPsXR2j9YMsQTA7zXqyGdeXp5cLpcSExNrLU9MTNT69evr3Gbr1q365ptvdO2112rOnDnavHmzbr/9dlVUVGjKlCl1blNeXq7y8vKa5wUFBY0pEwACRq0hlrrT5Q7A/zX71e5ut1sJCQl67bXXNGjQII0ZM0YPPvigpk+fXu82U6dOVWxsbM0tJSWlucsEAJ+UmccQSwACS6PCZ3x8vGw2m3Jycmotz8nJUVJSUp3bJCcnq3v37rLZbDXLevXqpezsbDmdzjq3mTx5svLz82tuO3fubEyZABAwvt+UJ0ka3LGFIkMZYgmA/2tU+HQ4HBo0aJAyMjJqlrndbmVkZCgtLa3ObU4//XRt3rxZbre7ZtnGjRuVnJwsh8NR5zahoaGKiYmpdQOAYPT9Jk+X+5ndGGIJQGBodLf7pEmT9Prrr+vtt9/WunXrdNttt6m4uLjm6vexY8dq8uTJNe1vu+027d+/X3fddZc2btyo2bNn64knntDEiROb7l0AQAByVrq1ZMs+SdKZ3eJNrgYAmkaj+3DGjBmj3NxcPfLII8rOztaAAQM0d+7cmouQduzYIav1UKZNSUnRvHnz9Kc//Un9+vVT27Ztddddd+m+++5runcBAAFo5Y4DKna61CrSod7J9AABCAwWwzAMs4s4noKCAsXGxio/P58ueABB49l5G/Tit5t1Sf82+ufVA80uBwCOqaF5jbndAcBHHTrfky53AIGD8AkAPuhAsVO/ZuVL4mIjAIGF8AkAPuiHLXkyDKl7YpSSYsPMLgcAmgzhEwB80PcbPeN7ctQTQKAhfAKAjzEMQ4s2V4dPzvcEEFgInwDgY7bmFSvroGdKzdROTKkJILAQPgHAx3y/0XOV+5BOLRTusB2nNQD4F8InAPiY6vncOd8TQCAifAKAD3FWurVkK1NqAghchE8A8CErdhxQidOl+CiHeiUxoxuAwEP4BAAfUj2r0Rld42W1WkyuBgCaHuETAHxI9fmeZ3C+J4AARfgEAB+xv9ip1TVTanK+J4DARPgEAB/xw2bPlJo9EqOVGMOUmgACE+ETAHxE9fmeHPUEEMgInwDgAwzD0KLq8T27c74ngMBF+AQAH7Alt1i788vkCLFqaMeWZpcDAM2G8AkAPqC6y31ox5ZMqQkgoBE+AcAHHJpSk/M9AQQ2wicAmKy80qUlW6qn1OR8TwCBjfAJACZbsf2gSitcio8KVc+kaLPLAYBmRfgEAJMdPsQSU2oCCHSETwAwWc2Uml053xNA4CN8AoCJ9hWVa81uptQEEDwInwBgoh+27JNhSD2TopXAlJoAggDhEwBMtKjqfE+63AEEC8InAJjEMIxD43sypSaAIEH4BACTbMkt1p78MjlsTKkJIHgQPgHAJNVd7oM7tmBKTQBBg/AJACZZtLlqiCWucgcQRAifAGCCCpdbP27dL0k6syvnewIIHoRPADDBqp0HVVReqRYRdp3SJsbscgDAawifAGCC6qvcT+/KlJoAggvhEwBMsOiw+dwBIJgQPgHAy/JLK/TLLs+Ummd043xPAMGF8AkAXrZkyz653IY6x0eqbVy42eUAgFcRPgHAyxZtrppSky53AEGI8AkAXrao6mIj5nMHEIwInwDgRTv3l2jbvhLZrBad1qWV2eUAgNcRPgHAi6pnNRqQEqeYMLvJ1QCA9xE+AcCL6HIHEOwInwDgJS63oR+2eMIn43sCCFaETwDwkrW783WwpELRoSHqnxJndjkAYArCJwB4SfWUmqd1aSW7jZ9fAMGJXz8A8JLvmVITAAifAOANJc5KLd9+QBIXGwEIboRPAPCCpZn7VeEy1DYuXJ3iI80uBwBMQ/gEAC84fIgli8VicjUAYB7CJwB4QU345HxPAEGO8AkAzWxvQZk25BTKYpFO53xPAEGO8AkAzax6Ss1T2sSoZaTD5GoAwFyETwBoZtVd7md2a21yJQBgPsInADQjwzD0fdWRzzPpcgcAwicANKcNOYXKLSxXmN2qQR1bmF0OAJiO8AkAzai6y31op1YKDbGZXA0AmI/wCQDNqHo+d7rcAcCD8AkAzaS80qWlmfskMb4nAFQjfAJAM1m+/YDKKtyKjwpVz6Ros8sBAJ9A+ASAZnJoSs1WTKkJAFUInwDQTKoHlz+D8T0BoAbhEwCawYFip1Zn5UuSzuR8TwCoQfgEgGbww5Y8GYbUPTFKiTFhZpcDAD6D8AkAzeDQ+Z50uQPA4QifANDEDMM4NL4nXe4AUAvhEwCa2LZ9Jco6WCq7zaLUzi3NLgcAfArhEwCa2KJNuZKkU9u3UIQjxORqAMC3ED4BoInR5Q4A9SN8AkATqnC5tXiLZ0rNMxnfEwCOQvgEgCa0csdBFZVXqmWkQ33bxppdDgD4HMInADShhRv3SpLO6Bovq5UpNQHgSIRPAGhCCzd6LjYa3p0udwCoC+ETAJpIXlG51mQVSJLO7M7FRgBQF8InADSR76uGWOqdHKOEaKbUBIC6ED4BoIl8t9EzxNJZdLkDQL1OKHy+9NJL6tixo8LCwpSamqply5Y1aLsPPvhAFotFo0ePPpGXBQCf5XYb+o7zPQHguBodPmfOnKlJkyZpypQpWrFihfr3768RI0Zo7969x9xu27Ztuueee3TmmWeecLEA4Kt+21OgfcVORTpsGtShhdnlAIDPanT4fO6553TTTTdp/Pjx6t27t6ZPn66IiAi9+eab9W7jcrl07bXX6tFHH1Xnzp1PqmAA8EXVV7mndYmXI4QzmgCgPo36hXQ6nVq+fLnS09MP7cBqVXp6upYsWVLvdo899pgSEhJ0ww03NOh1ysvLVVBQUOsGAL6sZoilHnS5A8CxNCp85uXlyeVyKTExsdbyxMREZWdn17nNokWL9MYbb+j1119v8OtMnTpVsbGxNbeUlJTGlAkAXlVYVqEV2w9IkoYzpSYAHFOz9g0VFhbquuuu0+uvv674+IaPeTd58mTl5+fX3Hbu3NmMVQLAyVm8ZZ8q3YY6xUeqfasIs8sBAJ8W0pjG8fHxstlsysnJqbU8JydHSUlJR7XfsmWLtm3bplGjRtUsc7vdnhcOCdGGDRvUpUuXo7YLDQ1VaGhoY0oDANNUd7mf1Y2B5QHgeBp15NPhcGjQoEHKyMioWeZ2u5WRkaG0tLSj2vfs2VOrV6/WqlWram6XXHKJzjnnHK1atYrudAB+zzAOG2KJ8z0B4LgadeRTkiZNmqRx48Zp8ODBGjp0qKZNm6bi4mKNHz9ekjR27Fi1bdtWU6dOVVhYmPr06VNr+7i4OEk6ajkA+KOtecXadaBUDptVp3VuZXY5AODzGh0+x4wZo9zcXD3yyCPKzs7WgAEDNHfu3JqLkHbs2CGrlWFGAASHhRs8Rz2HdGqhCEejf1IBIOhYDMMwzC7ieAoKChQbG6v8/HzFxMSYXQ4A1Lh+xjIt2JCrBy7qqZvPOvocdgAIFg3NaxyiBIATVFbh0o9b90liPncAaCjCJwCcoJ+27VdZhVuJMaHqkRhtdjkA4BcInwBwgr5dXz3EUmtZLBaTqwEA/0D4BIAT9O2GvZKkc3smmFwJAPgPwicAnIDMvGJl5hXLbrPoDAaXB4AGI3wCwAn4Zr3nqOeQji0VHWY3uRoA8B+ETwA4Ad+up8sdAE4E4RMAGqmovFJLMz1DLBE+AaBxCJ8A0EiLNuWqwmWoY6sIdW4dZXY5AOBXCJ8A0EjV53uew1FPAGg0wicANILbbejbqvnc6XIHgMYjfAJAI6zdXaDcwnJFOGwa2qml2eUAgN8hfAJAI1R3uZ/RNV6hITaTqwEA/0P4BIBG+IZZjQDgpBA+AaCB8orK9euug5K42AgAThThEwAaaMGGXBmG1KdtjBJjwswuBwD8EuETABqoZlajHhz1BIATRfgEgAaocLn13UbPEEt0uQPAiSN8AkAD/LztgArLK9Uq0qH+7eLMLgcA/BbhEwAa4Nuqq9yH92gtq9VicjUA4L8InwDQANXjezLEEgCcHMInABzHtrxibd5bpBCrRWd2a212OQDg1wifAHAcX/2WLUk6rXMrxYbbTa4GAPwb4RMAjuOrtTmSpAtOSTS5EgDwf4RPADiG3MJyLd9xQJJ0fm/CJwCcLMInABzD1+tyZBhS/3axSo4NN7scAPB7hE8AOIav1nrO97zglCSTKwGAwED4BIB6FJZV6IfN+yRJF9DlDgBNgvAJAPVYuDFXTpdbneMj1TUhyuxyACAgED4BoB7zqq5yP/+URFkszGoEAE2B8AkAdSivdOnbqlmNLujN+Z4A0FQInwBQhx+37ldReaVaR4dqYEqc2eUAQMAgfAJAHeZVXeV+fu9EWa10uQNAUyF8AsAR3G5D83+rmtWIq9wBoEkRPgHgCKt2HVRuYbmiQ0M0rEu82eUAQEAhfALAEaq73M/umSBHCD+TANCU+FUFgMMYhqGv1tLlDgDNhfAJAIfZklukzLxiOWxWnd2jtdnlAEDAIXwCwGGqB5Yf1rWVosPsJlcDAIGH8AkAh/lyzR5JDCwPAM2F8AkAVbbvK9aarALZrBaNOIXzPQGgORA+AaDK7NWeo56ndW6pVlGhJlcDAIGJ8AkAVeZUhc+L+7YxuRIACFyETwAQXe4A4C2ETwDQoS73tM6t6HIHgGZE+AQAHepyv6hvssmVAEBgI3wCCHp0uQOA9xA+AQS9L36lyx0AvIXwCSDoff7LbknSxf3ocgeA5kb4BBDUNmQXan12oew2iy7qQ/gEgOZG+AQQ1D77JUuSNLx7gmIjmMsdAJob4RNA0DIMQ59VdblfMoCB5QHAGwifAILWyp0HtXN/qSIcNqX3SjC7HAAICoRPAEHrs1Weo57n905UhCPE5GoAIDjwawsgKLncRs0QS5d6q8vdVSkV5XhuxXmSI1KKSpCiEqWwGO/UAAAmI3wCCEqLt+Qpr6hccRF2ndG1dfO90MGd0oY50uYMadv3UkVJ3e1a95K6nid1O1/qeJZkpWMKQGAifAIISrNWeK5yv7hvshwhzRD0dq+SFv9TWvuJZLgOLbeGSJEJUmS8J4gW7ZXKC6TcdZ7bkhelVt2kYX+U+l0l2cOavjYAMBHhE0DQKS6v1JdrsiVJlw9q17Q7P7hT+vI+acPsQ8vap0ndR0hdzpMS+xx9VLN4n7T1W2nLN9K6L6R9m6TP75K+nSqNeFzqc7lksTRtnQBgEsIngKAzd022Sitc6hQfqYEpcU2zU1eltOw16Zv/kyqKJYtN6nOZNOwOKbn/sbeNbCX1vcJzG/mUtOIdacnLUsEu6X83SL+8L138d6lFx6apFQBMxElFAILOxyt3SZIuG9hWlqY4oliUK/17tDRvsid4ppwm3faDdPm/jh88jxQaLaVNlO5cIZ39gGRzSJu/ll45Xfrt05OvFQBMRvgEEFR2HyzV4i37JEmjB7Y9+R3u/El69SzPxUSOKGnUP6TxX0oJvU5uvyGh0tn3SbctkdoPk5xF0odjpa8e9hxlBQA/RfgEEFQ+WZUlw5BSO7VUSsuIk9vZrx9JM0ZKhbul+O7STd9Kg65v2ivV47tK4z73dN9LnouY3rtSKi9qutcAAC8ifAIIGoZh6OOqq9wvP/UkLzRa+pr08Y2Su0LqNUq66RupdfcmqLIOthDpgv+Trnxbskd6Lkx651KpZH/zvB4ANCPCJ4Cg8cuufG3eW6TQEKtG9k06sZ0YhrTwaenLez3Ph94iXfmO51zN5nbKaGncZ1J4CynrZ2nGRVLBnuZ/XQBoQoRPAEFj5k87JUkX9U1WdJj9xHby3TPSt497Hp892XN1ujcHhG832HNOaXSyZ1zQt0d5LngCAD9B+AQQFEqclfr8F89c7n8YnHJiO1ny8qHgecHj0tn3mzP+ZkIvacI8KTbFMybov38vlR7wfh0AcAIInwCCwuxf96iovFIdWkXotM4tG7+D5W97hlKSpHMe9MxAZKYWHaSxn3pmS8pZLb17hVReaG5NANAAhE8AQeHDnz1d7n8YnNL4sT03zJW+uNvzeNid0ln3Nm1xJ6pVF08ArT4H9KPxDMMEwOcRPgEEvC25Rfpp2wFZLSdwlfvuVdJ/J0iGWxp4nXT+Y7411WVib+na/0kh4dLm+dKXf/FcFAUAPorwCSDgfVh1odE5PRKUFBvW8A3zd0nvjfHMWtT5bOl3z/tW8KzWbpB0+euSLNLPb0hLXjK7IgCoF+ETQEBzVrr1vxWe6TT/MKQRFxo5S6T3r5KKsqXWPaU/vCPZTvAKeW/oNcozFqgkffWQ51QBAPBBhE8AAW3e2mzlFTmVEB2qc3smNGwjw5A+v1PKXi1FxEvXfCiFxTZvoU0hbaI0eIIkQ/r4Jilvs9kVAcBRCJ8AAtq7P26XJF01JEV2WwN/8n58WVr9kWQN8RzxbNGhGStsQhaLdOFTUvs0qbxAmnktV8AD8DknFD5feukldezYUWFhYUpNTdWyZcvqbfv666/rzDPPVIsWLdSiRQulp6cfsz0ANJVNOYVamrlfVot01dD2Ddso8zvpq4c9j0c8IXU8vfkKbA4hDs80nNHJUu566ZPbuAAJgE9pdPicOXOmJk2apClTpmjFihXq37+/RowYob1799bZfsGCBbr66qv17bffasmSJUpJSdEFF1ygrKysky4eAI7lP0t3SJLO65WoNnHhx9+gMLvqynaX1P9qaejNzVxhM4lOlMa8K9kc0rrPpaXTza4IAGpYDKNxfxKnpqZqyJAhevHFFyVJbrdbKSkpuuOOO3T//fcfd3uXy6UWLVroxRdf1NixYxv0mgUFBYqNjVV+fr5iYmIaUy6AIFXirFTq4xkqLK/U2xOGanj31sfewFUpvXOptH2RlNhHuvFryd6AwOrLlr0uzblHstqlG76S2p5qdkUAAlhD81qjjnw6nU4tX75c6enph3ZgtSo9PV1Llixp0D5KSkpUUVGhli3rn2GkvLxcBQUFtW4A0BifrdqtwqoZjc7sGn/8DRZM9QRPR5Sn29rfg6ckDblR6nWJ5K6Q/jteKss3uyIAaFz4zMvLk8vlUmJiYq3liYmJys7ObtA+7rvvPrVp06ZWgD3S1KlTFRsbW3NLSTnBeZgBBCXDMPTvqguNrhnaXlbrccbm3Jwhff+s5/El/5TiuzZzhV5isUiXvCDFtZcObJM+v4vzPwGYzqtXuz/55JP64IMPNGvWLIWF1T/Q8+TJk5Wfn19z27lzpxerBODvlmXu19rdBQqzW/WHwcf547U4z3NRjuQZpqjP5c1foDeFx0lXzPBcub92lrT8LbMrAhDkGhU+4+PjZbPZlJOTU2t5Tk6OkpKSjrnts88+qyeffFJfffWV+vXrd8y2oaGhiomJqXUDgIaa8cM2SdLvB7ZTi0hH/Q0NQ/rsTqkoxzOQ/IgnvFOgt7UbLJ03xfN47v1Szlpz6wEQ1BoVPh0OhwYNGqSMjIyaZW63WxkZGUpLS6t3u6efflp/+9vfNHfuXA0ePPjEqwWA49i5v0Rf/eY5DWjC6R2P3XjF29KG2Z4Lci57PTDO86xP2h+lrudLlWXSR9dLzmKzKwIQpBrd7T5p0iS9/vrrevvtt7Vu3TrddtttKi4u1vjx4yVJY8eO1eTJk2vaP/XUU3r44Yf15ptvqmPHjsrOzlZ2draKioqa7l0AQJW3F2+T25DO7BavbonR9TfM2yzNrfqtOu8RKfnYPTJ+z2qVfj/dM/5n3kZpzl/MrghAkGp0+BwzZoyeffZZPfLIIxowYIBWrVqluXPn1lyEtGPHDu3Zs6em/SuvvCKn06krrrhCycnJNbdnn3226d4FAEgqKq/UzJ8954hPOL1T/Q1dFZ7pJytKpE5neY4KBoPIeOnyf0mySKveldZ+YnZFAIJQo8f5NAPjfAJoiLcXb9OUz9aqc3ykvp40vP6r3DP+5rm6PSxOum2xFNvWq3WaLuMx6fu/B+/7B9AsmmWcTwDwVZUut17/fqskafzpHesPntuXSIue8zweNS04g9fZk6U2A6Wyg9Int0put9kVAQgihE8AAWH26j3adaBULSMdumJQPcMrleVLH98sGW6p/zXSKb/3bpG+wmaXLvuXZI/wzGW/5EWzKwIQRAifAPyeYRiavtBz1PP6YR0V7rDV3XDOvVL+DimugzTyKS9W6IPiu0oXTvU8znhM2vOLufUACBqETwB+77tNeVq3p0DhdpvGpnWou9Hq/0q/zpQsVs+wSmGcP65Tx0k9f+eZfvN/N0nOErMrAhAECJ8A/N70BVskSVcNTVFcRB2Dyh/cKX0xyfP4rHul9qlerM6HWSzSqH9KUUlS3gZp/sNmVwQgCBA+Afi1X3Ye1JKt+xRitejGMzsf3cDtkmbdKpXnS20He8InDolsJf3+Fc/jn/4lbZhrbj0AAh7hE4Bfe+GbTZKkS/q3Udu4OmYoWvyCtH2RZI+ULnvNc7ENautyrnTaRM/jTydKRXvNrQdAQCN8AvBba7Ly9fW6vbJapInndj26we5V0jf/53k88kmpVRev1udXzntESjhFKsnzBFDfHwIagJ8ifALwW//I8Bz1HNW/jbq0jqq90lnimcXIXeG5qGbgdSZU6EfsYZ7Zj2yh0qavPF3wANAMCJ8A/NLa3fma/1uOLBbpjnO7Hd1g/sOeOcyjkqRLXvBcXINjS+wtnf+Y5/FXD0l715tbD4CARPgE4Jf+WX3Us18bdU044qjnxnmHjtz9/hUpoqWXq/NjqbdIXdOlyjLpfzdKleVmVwQgwBA+AfidtbvzNW+t56jnnecdca5nUa7nnEVJOu12z8U0aDiLRbr0ZSmilZSz2jMAPQA0IcInAL/zzLwNkqqPekYfWmEYnuBZnOu5eOa8KSZV6OeiE6VLqqbcXPKitOVbc+sBEFAInwD8yo9b92nBhlyFWC2adH732it/fkPaNM9z0czlr3suosGJ6XmRNGi85/Ent0kl+82tB0DAIHwC8BuGYejpuZ6LYK4amqKO8ZGHVub8Js170PM4/a9S4ineLzDQjHhcatVNKtwjfX4Xwy8BaBKETwB+4+t1e7Vix0GF2a268/Ar3CtKpf9O8Fwk0zVdSr3VvCIDiSPScwTZGiKt+0xa+a7ZFQEIAIRPAH6h0uXWM/M8Rz0nnN5JCTGHdal/9ZCUu06KTJBGvyJZ+WlrMm0GSuc+5Hn85X3Svi3m1gPA7/ELDcAvzPx5pzbmFCk23K5bhh82U9H62YcNqzRdikowp8BANuxOqeOZUkWxZ+B+V4XZFQHwY4RPAD6voKxCf/9qoyTpT+ndFBteNT97ftahYZXS/ih1Pc+kCgOc1eYJ9mGxUtZyaeHTZlcEwI8RPgH4vBcyNml/sVNdWkfq2tM6eBa6XdKsW6TSA1LyAIZVam6x7aTfTfM8/v5ZafsSU8sB4L8InwB8WmZesd5avE2S9NDvestuq/rZWvS8tO17yR4pXfGmFOIwr8hg0ecyqf/VkuGWZt0sleWbXREAP0T4BODTHp+9ThUuQ2f3aK1zelSdz7l9ifTtE57HFz0jtepS/w7QtEY+LcV1kA7ukD6/m+GXADQa4ROAz5r/W46+XpejEKtFD13cy7OwaK/00fWS4ZL6XCENuMbUGoNOWIx0+b88wy+t/Vha9rrZFQHwM4RPAD6pxFmpv362VpJ045mdPdNouio943kWZUvxPaRR//DMRQ7vShkqnV815/u8B6SdP5lbDwC/QvgE4JP+mbFZWQdL1TYuXHee19Wz8NvHD53nOebfUmiUuUUGs9Nul3pfKrkrPEeii/eZXREAP0H4BOBzNmQX6l/fb5Uk/fWSUxThCJE2fCktes7T4JJ/Sq17mFghZLFIl7woteoqFeySPr7RMwIBABwH4ROAT3G5DU3++FdVug2d3ztR5/dOlPZneoZVkqSht0h9rzC3SHiExUh/+Ldkj5C2fMP4nwAahPAJwKfM+CFTK3YcVFRoiP56ySlSRZn04VjPsD7thkgX/J/ZJeJwib0Pjf+58Clp09emlgPA9xE+AfiMrblFembeBknSgxf3UtvYMGnOPVL2r1JEK+nKtxjP0xf1HyMNniDJ8HS/H9hmdkUAfBjhE4BPcLkN/eW/v6q80q0zusbrqiEp0o8vSyv/LckiXfa6Z5Yd+KYLn5TaDPTMOPXeVVJZgdkVAfBRhE8APuHNRZn6efsBRTpsevLyvrJs+kqa96Bn5QX/x7ztvi4kVBrzHykqScpdJ/3vBi5AAlAnwicA063JytfT89ZLkh68uLfaObd5xvOUIZ06VkqbaGp9aKDYttLV70khYdKmr6SvHja7IgA+iPAJwFQlzkrd+cFKVbgMXdA7UVefEia9N0ZyFkkdzpAu+jsDyfuTtoOk30/3PP7xJWn5W6aWA8D3ED4BmOpvX6zT1txiJcaE6qlLu8sy8/9J+Tuklp09A8lzgZH/OeX30jlVp0zM/rOU+Z259QDwKYRPAKb54tfden/ZDlks0nNX9leLjHuknUul0Fjp6plSREuzS8SJOuteqc8VkrtSmnmdlLfZ7IoA+AjCJwBTbMop1F/++6sk6dbhXXR61hvSrzMli036w9tS6+4mV4iTYrFIl74otR0slR2U3rtSKs4zuyoAPoDwCcDrisordeu7y1XidCmtcyvd0/IHacFUz8qLnpG6nGNugWga9nDpqvek2PbS/q3Sf66QygvNrgqAyQifALzKMAz95b+/aEtusZJiwvTq4CzZvrzHs/Ksv0hDbjC3QDSt6ETpuo+l8JbS7pWeLvhKp9lVATAR4ROAV/0zY7PmrM6W3WbRO+eUKWb2bZLhlk4dJ53zgNnloTnEd5Ou/a9kj5S2fivNupkxQIEgRvgE4DWf/7Jbz3+9UZL08pkV6v7NjZKrXOr5O+ni5xhSKZC1GySNeUey2qW1s6TP7pDcbrOrAmACwicAr1i544Du+egXSdIjA0t1/oqJUkWJ1OVc6Yo3JVuIyRWi2XVNl654w3NR2ar/SHPukQzD7KoAeBnhE0Czy8wr1k3v/KzySrdu6HxQ4zP/LDkLPYPIj/mPZ2pGBIfel1YNQm+Rfn5D+vI+AigQZAifAJpVTkGZrntjqfKKnLosYY8e2ne/LGUHpXZDpWs+kBwRZpcIb+v3B+mSf3oeL3vVMxA9XfBA0CB8Amg2+SUVGvvGMu06UKqL43bo2bIpspQXSO3TpP/3Pyk02uwSYZZTx0qXvKiaI6Bf3EUABYIE4RNAsygsq9D1by3ThpxCXRL5m16ofFRWZ5HU8UzPlc9hMWaXCLOdep2nC95ilVa8I318I8MwAUGA8AmgyRWWVWjsm8u0csdBjQlbpn8YT8laWeq5uOiaD6XQKLNLhK/of5V02euSNURa8z/p/askZ7HZVQFoRoRPAE3q8OB5a9jXelL/kMVdIfW53DNfO+d44kh9r/B8N+wR0pYM6e1LmIoTCGCETwBNZl9Rua55fal+2bFfj4f9W/frTVlkSENvli77lxTiMLtE+Kpu6dLYz6TwFlLWz9K/zpNyN5pdFYBmQPgE0CR2HSjRldOXaEtWjt4Ke17X6kvPivS/SiOflqz83OA4UoZIE76S4jpIB7ZJb6RLWxeaXRWAJsa/BgBO2vrsAl3xyhK59m3RF+FTdJaWSyFh0pVvS2f8iZmL0HCtu0s3feMZiqssX3r3Mmnpq4wFCgQQwieAk/LN+hxd/vJi9Sz6UV+EPqzOxk4pKlEa94V0ymizy4M/ioyXxn0u9b1ScldKX/5F+uQ2qaLU7MoANAHCJ4ATYhiG3liUqVveXqqb3R/oTcczilax54jVzQs9XajAibKHea6CH/GEZzrOX96X3jhf2rfF7MoAnCTCJ4BGK3FW6u6Zq/TaF4v0rv1x3RUyS1YZ0qDx0vWzpZhks0tEILBYpLSJ0nWzpIhWUvZq6dWzpF8/MrsyACeB8AmgUTbvLdKlL/6g8l8/0Zeh9yvVul6GI0q6/A1p1DSuaEfT6zxcunWR1OEMyVnkGYx+1m2ec0IB+B3CJ4AGMQxDHyzboWtfnKfbDjyt6Y5pamkpkpL6yXLLd56xGoHmEtNGGveZNPx+z4xIv7wnvTyMq+EBPxRidgEAfN/+Yqfu/9+v0vov9In9bSVb98uwWGU540+eMMDRTniD1SadM1nqco4061bpQKb0ziXS4AmeIb3CYs2uEEADWAzD98evKCgoUGxsrPLz8xUTw3zQgLcYhqHPf92jlz/9TndX/EsX2n7yLG/ZWZbfvyqlDDW5QgSt8iJp/sPSz296nkclSRc9LfW6hKG9AJM0NK8RPgHUKetgqR6btUKdNr+jO0JmKdJSLsMSIsvpd0rD/yLZw80uEZAyv5e+uFvat9nzvMt50oVPesYLBeBVhE8AJ6SswqXXF27RhoXva5LlPXW2ZkuS3G2HyDpqmpTUx9wCgSNVlEnfPyv98A/J5ZSsIZ4pXc+8R4psZXZ1QNAgfAJoFLfb0JzVu/XtnPc1rvRd9bNmSpIqIxIUcsFjUr8xTJEJ37ZvizTvQWlj1dSujijptNuktD9K4XGmlgYEA8IngAYxDEOLNudpzuf/1e8PztBQ6wZJUqUtXLZhE2U5/S4pjP/v4Ec2fy1lPCbt+cXzPCxWGnanlHqrFBplbm1AACN8AjgmwzC0cMNefTf3Q527732dYVsrSaq0hsoYfKPswyd5pjkE/JFhSOs+l759XMpd71kWEe8ZtH7Q9VJES1PLAwIR4RNAncorXZq7Yqt2LXxLFxTOUjdrliTJZQlRRb//p7Dz7vOMqQgEArdLWvM/6dsnPEMzSVJIuNT/Ks+R0ISe5tYHBBDCJ4Ba9uSX6rOFy+RYOUOj3fPVwlIkSSq3RsjV/xpFnHWH1KKjuUUCzcVVIa3+SPrxZc80ndW6nCul3iZ1Pc8zjiiAE0b4BCCX29DS9du0ZeH76rrnc6Va1slq8fwvnx/WViFptyoydRyDcyN4GIa0fbEnhK6fLanqn8DoZKnvlVL/q6XE3qaWCPgrwicQpAzD0KrtuVr7/edqtXWWznYvVbjFWbM+r/VpanHOHbL1HMmRHgS3A9ukpa9Jq/4jlR08tDypryeE9rlcik4yqzrA7xA+gSDidhtam7lT25d+ptAt8zS08mfFWkpq1ueGtpfR7yolnH6dFNfexEoBH1RZLm36SvrlA2njPMldcWhd28FSz4ukHhdJrXsyexJwDIRPIMDlFzu1csViHVjztZJzFmqQsVZ2i6tmfaEtTgc7/05JZ46XPWUQ/2gCDVGy33OB0q8zpV0/1V7XoqMnhHY5T2p/GsM2AUcgfAIBJr/EqXVrVujgb98oZs9i9Sj7Ra0sBbXa7LG3V1HH85Vy2uUK63Qa3erAySjYI22cK22YI21dKLnKD62zhkhtTpU6nSl1PFNKSZUcEebVCvgAwifgx1xuQ9t37lD2usUq3/6TInN/UZeKDWplKazVrkwO7Y4ZIGvXc9Qm9Qo5EpnPGmgW5UXS1m89YTTzO+ngjtrrrSFS4ilS20GHbvHd+QMQQYXwCfiJwpJS7dy8Rge3rVLF7jUKO7BBbcq2KMWy96i2Ttm1I7KPytqertb90pXY83QpxGFC1UCQO7Bd2va9lPm9574g6+g2jijPxUsJvaWEXp5wmtBLCm/h/XoBLyB8Aj6k3FmunB2bdGDXBpXlbJFr/1Y5CrYrrixLKe4shVoq69wuy9ZOebF9ZEsZrDa9T1fLLoOkkFAvVw/gmAxDyt8pZa2QspZ77nevlCqK624f3UZq1cVza3nYfYuOkj3Mq6UDTYnwCXhJRUWF8vZmKT9nu4pzd6n8QJaMgt2yFecovDRbrZy7lWTsVYjFXe8+ShSmLHtHFcZ2V0jSKWrVeYCSeqbKFsEREsAvuV1S7gYpZ620d62U85u09zdPSD2WyNZSTFsptl3Vfdvaz6MS+AMUPqtZw+dLL72kZ555RtnZ2erfv79eeOEFDR06tN72H330kR5++GFt27ZN3bp101NPPaWLLrqowa9H+IS3lJeXqbjggEoLD6q06IBK83NVlp+ryqI8GcX7ZC3bL1vZAYU6Dyq88qBiXAfUyjh4zGBZs2/Drmxbkg6EtVN5dHvZ47souk03JXbuq5jELpLV6oV3CMBUZfmeULpvi7R/y2H3WyVn4fG3l6TQGCky3hNUI+KrHlc9D2/pmTQiLMZzH1p174jiNwbNrqF5LaSxO545c6YmTZqk6dOnKzU1VdOmTdOIESO0YcMGJSQkHNV+8eLFuvrqqzV16lT97ne/03vvvafRo0drxYoV6tOnT2NfHkHKcLtVWVmhygqnKiqcqigvlbOsWM6yUlWUFauyvMRzc5bKVV4qt7NE7opSGVU3S0WZVFkmVZbI5ixWSGWhHJXFCnUVK8xdonCjRJFGicIsFWr0MQWL5DIsOmCJ08GQeBWHtpYzIlFGVLJCYpMVmdxN8Sk91TKpvTpYberQHB8QAP8QFiulDPXcDmcYUukBKX+X5/zRmvusw57v9oxBWl7gue3f2ogXtngCaWjsoXAaGuO5Qt8eLtkjDt3qWmYPl0LCJJvdc+TV5vDcjnzMBVZogEYf+UxNTdWQIUP04osvSpLcbrdSUlJ0xx136P777z+q/ZgxY1RcXKwvvviiZtlpp52mAQMGaPr06Q16TW8e+SwuPKiNiz9VzcdiGDr0CblV/cSQUfO49n3VeuOw9TIO7a9qvcU4Yplx2HY6Yr8y6nzcmLa12tW8B0mGu2a9xZAMuSW327PccHnu3W5Zqh8bdT+21HpuyGK4ZNHhy901zy2GS1ajUjZ3pWyqlNWoVIhRKVv1TS7Z5VkXYrgUoko5Dhu/0htKDYdKLBEqskarJCRO5fY4VYTGyR3eSpaIlrJFxcsR01oRLZLUIqmD4uLbyBpi92qNAIKMYXhmYireJxXnHrqVHPa8ZL8nmJYVeI6yluXXHjS/uVlsVYHULtlCDwVSa4jnZrHVfl7z+HhtbIfWWayecYstVs9NlqrnlsOeW+t4bj3U7rht6ttv9XjJh42bfOSyRrVp6HZ1Pa9nuyPbtOgoJfeTNzTLkU+n06nly5dr8uTJNcusVqvS09O1ZMmSOrdZsmSJJk2aVGvZiBEj9Mknn9T7OuXl5SovPzSeWkFBQb1tm9qBnF0auOROr70ejuMY46KXG3aVWxwql0NOi0MVllBVWD03V/XNFia3LUzukDAZIWFSSLgUFiNrWLRCwmMUEhErR2SswiLjFBYVp4joFoqMjlO43aFwSa289kYB4DgsFs+V8uEtpPiuDdvGMDy9PtVhtLzAE2DLqo6eVpRKFSWSs6TqcXEdy0o8t8pyyeX03CqdnnFPXc4jXs9V1b7J3z1O1JAbpYv/bnYVtTQqfObl5cnlcikxMbHW8sTERK1fv77ObbKzs+tsn52dXe/rTJ06VY8++mhjSmsy9rBwrbOfIkkyDvsr4tDh4aq/mCQZh/11UdNWlsOWH9a2aj+HlldtY7EcsX3V+qOWWw4LYpZatdXaX63Xqaue+vavmr/uDIvV89elLDKs1X9l2jz7qf6rs2Z59WObLFZrTVtZrLJYa9/LapOl6rk1xCFLiEPWkBBZbQ5ZQxyyhdhltYfKarMrxO55brOHymZ3KMTukD0kVDa7XSEOh+z2MIXabI3vIgeAYGKxVHWhh0vRicdv31iGIbkqqoJoRVVAPeKx21V1q6y6uTwhteZ5pafH7fDnRh3bVD+XUdX7ZtT0wtVeZhzdRoe1rbXMaECbw/cr6bBEUGfvYoPbnMx2jdh3i05Hb2eyRp/z6Q2TJ0+udbS0oKBAKSkpXnntxHZdlPjgYq+8FgAAfs1i8Yw1zHjDaIRGhc/4+HjZbDbl5OTUWp6Tk6OkpKQ6t0lKSmpUe0kKDQ1VaCjHtAAAAAJNo8ZdcDgcGjRokDIyMmqWud1uZWRkKC0trc5t0tLSarWXpPnz59fbHgAAAIGr0d3ukyZN0rhx4zR48GANHTpU06ZNU3FxscaPHy9JGjt2rNq2baupU6dKku666y4NHz5cf//733XxxRfrgw8+0M8//6zXXnutad8JAAAAfF6jw+eYMWOUm5urRx55RNnZ2RowYIDmzp1bc1HRjh07ZD1sINthw4bpvffe00MPPaQHHnhA3bp10yeffMIYnwAAAEGI6TUBAABw0hqa15hrCwAAAF5D+AQAAIDXED4BAADgNYRPAAAAeA3hEwAAAF5D+AQAAIDXED4BAADgNYRPAAAAeA3hEwAAAF5D+AQAAIDXED4BAADgNYRPAAAAeA3hEwAAAF4TYnYBDWEYhiSpoKDA5EoAAABQl+qcVp3b6uMX4bOwsFCSlJKSYnIlAAAAOJbCwkLFxsbWu95iHC+e+gC3263du3crOjpaFoul2V+voKBAKSkp2rlzp2JiYpr99fwFn0v9+GzqxudSPz6buvG51I/Ppm58LvXz9mdjGIYKCwvVpk0bWa31n9npF0c+rVar2rVr5/XXjYmJ4YtcBz6X+vHZ1I3PpX58NnXjc6kfn03d+Fzq583P5lhHPKtxwREAAAC8hvAJAAAAryF81iE0NFRTpkxRaGio2aX4FD6X+vHZ1I3PpX58NnXjc6kfn03d+Fzq56ufjV9ccAQAAIDAwJFPAAAAeA3hEwAAAF5D+AQAAIDXED4BAADgNUEZPh9//HENGzZMERERiouLq7PNjh07dPHFFysiIkIJCQm69957VVlZecz97t+/X9dee61iYmIUFxenG264QUVFRc3wDrxjwYIFslgsdd5++umnerc7++yzj2p/6623erFy7+jYseNR7/PJJ5885jZlZWWaOHGiWrVqpaioKF1++eXKycnxUsXNb9u2bbrhhhvUqVMnhYeHq0uXLpoyZYqcTucxtwvU78xLL72kjh07KiwsTKmpqVq2bNkx23/00Ufq2bOnwsLC1LdvX82ZM8dLlXrH1KlTNWTIEEVHRyshIUGjR4/Whg0bjrnNW2+9ddR3IywszEsVe89f//rXo95nz549j7lNoH9fpLp/Zy0WiyZOnFhn+0D+vnz33XcaNWqU2rRpI4vFok8++aTWesMw9Mgjjyg5OVnh4eFKT0/Xpk2bjrvfxv5ONYWgDJ9Op1NXXnmlbrvttjrXu1wuXXzxxXI6nVq8eLHefvttvfXWW3rkkUeOud9rr71Wa9eu1fz58/XFF1/ou+++080339wcb8Erhg0bpj179tS63XjjjerUqZMGDx58zG1vuummWts9/fTTXqraux577LFa7/OOO+44Zvs//elP+vzzz/XRRx9p4cKF2r17ty677DIvVdv81q9fL7fbrVdffVVr167V888/r+nTp+uBBx447raB9p2ZOXOmJk2apClTpmjFihXq37+/RowYob1799bZfvHixbr66qt1ww03aOXKlRo9erRGjx6tNWvWeLny5rNw4UJNnDhRP/74o+bPn6+KigpdcMEFKi4uPuZ2MTExtb4b27dv91LF3nXKKafUep+LFi2qt20wfF8k6aeffqr1mcyfP1+SdOWVV9a7TaB+X4qLi9W/f3+99NJLda5/+umn9c9//lPTp0/X0qVLFRkZqREjRqisrKzefTb2d6rJGEFsxowZRmxs7FHL58yZY1itViM7O7tm2SuvvGLExMQY5eXlde7rt99+MyQZP/30U82yL7/80rBYLEZWVlaT124Gp9NptG7d2njssceO2W748OHGXXfd5Z2iTNShQwfj+eefb3D7gwcPGna73fjoo49qlq1bt86QZCxZsqQZKvQNTz/9tNGpU6djtgnE78zQoUONiRMn1jx3uVxGmzZtjKlTp9bZ/g9/+INx8cUX11qWmppq3HLLLc1ap5n27t1rSDIWLlxYb5v6fqcDzZQpU4z+/fs3uH0wfl8MwzDuuusuo0uXLobb7a5zfbB8XyQZs2bNqnnudruNpKQk45lnnqlZdvDgQSM0NNR4//33691PY3+nmkpQHvk8niVLlqhv375KTEysWTZixAgVFBRo7dq19W4TFxdX64hgenq6rFarli5d2uw1e8Nnn32mffv2afz48cdt+5///Efx8fHq06ePJk+erJKSEi9U6H1PPvmkWrVqpYEDB+qZZ5455qkZy5cvV0VFhdLT02uW9ezZU+3bt9eSJUu8Ua4p8vPz1bJly+O2C6TvjNPp1PLly2v9t7ZarUpPT6/3v/WSJUtqtZc8vzuB/t2QdNzvR1FRkTp06KCUlBRdeuml9f4O+7tNmzapTZs26ty5s6699lrt2LGj3rbB+H1xOp169913NWHCBFkslnrbBcv35XCZmZnKzs6u9Z2IjY1Vampqvd+JE/mdaiohzbp3P5WdnV0reEqqeZ6dnV3vNgkJCbWWhYSEqGXLlvVu42/eeOMNjRgxQu3atTtmu2uuuUYdOnRQmzZt9Ouvv+q+++7Thg0b9PHHH3upUu+48847deqpp6ply5ZavHixJk+erD179ui5556rs312drYcDsdR5xknJiYGzHfkSJs3b9YLL7ygZ5999pjtAu07k5eXJ5fLVefvyPr16+vcpr7fnUD9brjdbt199906/fTT1adPn3rb9ejRQ2+++ab69eun/Px8Pfvssxo2bJjWrl173N8if5Kamqq33npLPXr00J49e/Too4/qzDPP1Jo1axQdHX1U+2D7vkjSJ598ooMHD+r666+vt02wfF+OVP3fvTHfiRP5nWoqARM+77//fj311FPHbLNu3brjnsAdDE7ks9q1a5fmzZunDz/88Lj7P/w81759+yo5OVnnnXeetmzZoi5dupx44V7QmM9m0qRJNcv69esnh8OhW265RVOnTvW5qcxO1ol8Z7KysnThhRfqyiuv1E033XTMbf35O4MTM3HiRK1Zs+aY5zVKUlpamtLS0mqeDxs2TL169dKrr76qv/3tb81dpteMHDmy5nG/fv2UmpqqDh066MMPP9QNN9xgYmW+44033tDIkSPVpk2betsEy/fF3wVM+Pzzn/98zL+GJKlz584N2ldSUtJRV3tVX5GclJRU7zZHnqBbWVmp/fv317uNWU7ks5oxY4ZatWqlSy65pNGvl5qaKslzFMzXg8TJfI9SU1NVWVmpbdu2qUePHketT0pKktPp1MGDB2sd/czJyfG578iRGvu57N69W+ecc46GDRum1157rdGv50/fmbrEx8fLZrMdNZLBsf5bJyUlNaq9P/vjH/9Yc1FmY49G2e12DRw4UJs3b26m6nxDXFycunfvXu/7DKbviyRt375dX3/9daN7Q4Ll+1L93z0nJ0fJyck1y3NycjRgwIA6tzmR36mmEjDhs3Xr1mrdunWT7CstLU2PP/649u7dW9OVPn/+fMXExKh37971bnPw4EEtX75cgwYNkiR98803crvdNf+Q+orGflaGYWjGjBkaO3as7HZ7o19v1apVklTrfwhfdTLfo1WrVslqtR51+kW1QYMGyW63KyMjQ5dffrkkacOGDdqxY0etv9R9UWM+l6ysLJ1zzjkaNGiQZsyYIau18aeW+9N3pi4Oh0ODBg1SRkaGRo8eLcnTzZyRkaE//vGPdW6TlpamjIwM3X333TXL5s+f7/PfjcYwDEN33HGHZs2apQULFqhTp06N3ofL5dLq1at10UUXNUOFvqOoqEhbtmzRddddV+f6YPi+HG7GjBlKSEjQxRdf3KjtguX70qlTJyUlJSkjI6MmbBYUFGjp0qX1juxzIr9TTaZZL2fyUdu3bzdWrlxpPProo0ZUVJSxcuVKY+XKlUZhYaFhGIZRWVlp9OnTx7jggguMVatWGXPnzjVat25tTJ48uWYfS5cuNXr06GHs2rWrZtmFF15oDBw40Fi6dKmxaNEio1u3bsbVV1/t9ffX1L7++mtDkrFu3bqj1u3atcvo0aOHsXTpUsMwDGPz5s3GY489Zvz8889GZmam8emnnxqdO3c2zjrrLG+X3awWL15sPP/888aqVauMLVu2GO+++67RunVrY+zYsTVtjvxsDMMwbr31VqN9+/bGN998Y/z8889GWlqakZaWZsZbaBa7du0yunbtapx33nnGrl27jD179tTcDm8TDN+ZDz74wAgNDTXeeust47fffjNuvvlmIy4urmYUjeuuu864//77a9r/8MMPRkhIiPHss88a69atM6ZMmWLY7XZj9erVZr2FJnfbbbcZsbGxxoIFC2p9N0pKSmraHPm5PProo8a8efOMLVu2GMuXLzeuuuoqIywszFi7dq0Zb6HZ/PnPfzYWLFhgZGZmGj/88IORnp5uxMfHG3v37jUMIzi/L9VcLpfRvn1747777jtqXTB9XwoLC2vyiiTjueeeM1auXGls377dMAzDePLJJ424uDjj008/NX799Vfj0ksvNTp16mSUlpbW7OPcc881XnjhhZrnx/udai5BGT7HjRtnSDrq9u2339a02bZtmzFy5EgjPDzciI+PN/785z8bFRUVNeu//fZbQ5KRmZlZs2zfvn3G1VdfbURFRRkxMTHG+PHjawKtP7v66quNYcOG1bkuMzOz1me3Y8cO46yzzjJatmxphIaGGl27djXuvfdeIz8/34sVN7/ly5cbqampRmxsrBEWFmb06tXLeOKJJ4yysrKaNkd+NoZhGKWlpcbtt99utGjRwoiIiDB+//vf1wpm/m7GjBl1/r91+N+5wfSdeeGFF4z27dsbDofDGDp0qPHjjz/WrBs+fLgxbty4Wu0//PBDo3v37obD4TBOOeUUY/bs2V6uuHnV992YMWNGTZsjP5e777675jNMTEw0LrroImPFihXeL76ZjRkzxkhOTjYcDofRtm1bY8yYMcbmzZtr1gfj96XavHnzDEnGhg0bjloXTN+X6txx5K36/bvdbuPhhx82EhMTjdDQUOO888476jPr0KGDMWXKlFrLjvU71VwshmEYzXtsFQAAAPBgnE8AAAB4DeETAAAAXkP4BAAAgNcQPgEAAOA1hE8AAAB4DeETAAAAXkP4BAAAgNcQPgEAAOA1hE8AAAB4DeETAAAAXkP4BAAvyc3NVVJSkp544omaZYsXL5bD4VBGRoaJlQGA9zC3OwB40Zw5czR69GgtXrxYPXr00IABA3TppZfqueeeM7s0APAKwicAeNnEiRP19ddfa/DgwVq9erV++uknhYaGml0WAHgF4RMAvKy0tFR9+vTRzp07tXz5cvXt29fskgDAazjnEwC8bMuWLdq9e7fcbre2bdtmdjkA4FUc+QQAL3I6nRo6dKgGDBigHj16aNq0aVq9erUSEhLMLg0AvILwCQBedO+99+q///2vfvnlF0VFRWn48OGKjY3VF198YXZpAOAVdLsDgJcsWLBA06ZN07///W/FxMTIarXq3//+t77//nu98sorZpcHAF7BkU8AAAB4DUc+AQAA4DWETwAAAHgN4RMAAABeQ/gEAACA1xA+AQAA4DWETwAAAHgN4RMAAABeQ/gEAACA1xA+AQAA4DWETwAAAHgN4RMAAABeQ/gEAACA1/x/SWhogTu+XXQAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.plot(x.numpy(), y, label='y')\n", "plt.plot(x.numpy(), dy_dx, label='dy/dx')\n", "plt.legend()\n", "_ = plt.xlabel('x')" ] }, { "cell_type": "markdown", "metadata": { "id": "DsOMSD_1BGkD" }, "source": [ "### Tensor source" ] }, { "cell_type": "markdown", "metadata": { "id": "g3iXKN7KF-st" }, "source": [ "Whether the input is scalar or tensor, `tf.GradientTape.jacobian` efficiently calculates the gradient of each element of the source with respect to each element of the target(s).\n", "\n", "For example, the output of this layer has a shape of `(10, 7)`:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.443323Z", "iopub.status.busy": "2024-08-15T02:32:18.442721Z", "iopub.status.idle": "2024-08-15T02:32:18.455734Z", "shell.execute_reply": "2024-08-15T02:32:18.455165Z" }, "id": "39YXItgLxMBk" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([7, 10])" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = tf.random.normal([7, 5])\n", "layer = tf.keras.layers.Dense(10, activation=tf.nn.relu)\n", "\n", "with tf.GradientTape(persistent=True) as tape:\n", " y = layer(x)\n", "\n", "y.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "tshNRtfKuVP_" }, "source": [ "And the layer's kernel's shape is `(5, 10)`:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.458758Z", "iopub.status.busy": "2024-08-15T02:32:18.458521Z", "iopub.status.idle": "2024-08-15T02:32:18.462766Z", "shell.execute_reply": "2024-08-15T02:32:18.462200Z" }, "id": "CigTWyfPvPuv" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([5, 10])" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "layer.kernel.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "mN96JRpnAjpx" }, "source": [ "The shape of the Jacobian of the output with respect to the kernel is those two shapes concatenated together:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.465678Z", "iopub.status.busy": "2024-08-15T02:32:18.465464Z", "iopub.status.idle": "2024-08-15T02:32:18.585137Z", "shell.execute_reply": "2024-08-15T02:32:18.584542Z" }, "id": "pRLzTTbvEimH" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([7, 10, 5, 10])" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "j = tape.jacobian(y, layer.kernel)\n", "j.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "2Lrv7miMvTll" }, "source": [ "If you sum over the target's dimensions, you're left with the gradient of the sum that would have been calculated by `tf.GradientTape.gradient`:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.588391Z", "iopub.status.busy": "2024-08-15T02:32:18.588166Z", "iopub.status.idle": "2024-08-15T02:32:18.596854Z", "shell.execute_reply": "2024-08-15T02:32:18.596171Z" }, "id": "FJjZpYRnDjVa" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "g.shape: (5, 10)\n", "delta: 2.3841858e-07\n" ] } ], "source": [ "g = tape.gradient(y, layer.kernel)\n", "print('g.shape:', g.shape)\n", "\n", "j_sum = tf.reduce_sum(j, axis=[0, 1])\n", "delta = tf.reduce_max(abs(g - j_sum)).numpy()\n", "assert delta < 1e-3\n", "print('delta:', delta)" ] }, { "cell_type": "markdown", "metadata": { "id": "ZKajuGlk_krs" }, "source": [ " \n", "\n", "#### Example: Hessian" ] }, { "cell_type": "markdown", "metadata": { "id": "NYcsXeo8TDLi" }, "source": [ "While `tf.GradientTape` doesn't give an explicit method for constructing a [Hessian matrix](https://en.wikipedia.org/wiki/Hessian_matrix) it's possible to build one using the `tf.GradientTape.jacobian` method.\n", "\n", "Note: The Hessian matrix contains `N**2` parameters. For this and other reasons it is not practical for most models. This example is included more as a demonstration of how to use the `tf.GradientTape.jacobian` method, and is not an endorsement of direct Hessian-based optimization. A Hessian-vector product can be [calculated efficiently with nested tapes](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/eager/benchmarks/resnet50/hvp_test.py), and is a much more efficient approach to second-order optimization." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.600014Z", "iopub.status.busy": "2024-08-15T02:32:18.599794Z", "iopub.status.idle": "2024-08-15T02:32:18.877192Z", "shell.execute_reply": "2024-08-15T02:32:18.876470Z" }, "id": "ELGTaell_j81" }, "outputs": [], "source": [ "x = tf.random.normal([7, 5])\n", "layer1 = tf.keras.layers.Dense(8, activation=tf.nn.relu)\n", "layer2 = tf.keras.layers.Dense(6, activation=tf.nn.relu)\n", "\n", "with tf.GradientTape() as t2:\n", " with tf.GradientTape() as t1:\n", " x = layer1(x)\n", " x = layer2(x)\n", " loss = tf.reduce_mean(x**2)\n", "\n", " g = t1.gradient(loss, layer1.kernel)\n", "\n", "h = t2.jacobian(g, layer1.kernel)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.880444Z", "iopub.status.busy": "2024-08-15T02:32:18.880203Z", "iopub.status.idle": "2024-08-15T02:32:18.883872Z", "shell.execute_reply": "2024-08-15T02:32:18.883295Z" }, "id": "FVqQuZj4XGjm" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "layer.kernel.shape: (5, 8)\n", "h.shape: (5, 8, 5, 8)\n" ] } ], "source": [ "print(f'layer.kernel.shape: {layer1.kernel.shape}')\n", "print(f'h.shape: {h.shape}')" ] }, { "cell_type": "markdown", "metadata": { "id": "_M7XElgaiMeP" }, "source": [ "To use this Hessian for a [Newton's method](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization) step, you would first flatten out its axes into a matrix, and flatten out the gradient into a vector:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.887299Z", "iopub.status.busy": "2024-08-15T02:32:18.886625Z", "iopub.status.idle": "2024-08-15T02:32:18.899284Z", "shell.execute_reply": "2024-08-15T02:32:18.898703Z" }, "id": "6te7N6wVXwXX" }, "outputs": [], "source": [ "n_params = tf.reduce_prod(layer1.kernel.shape)\n", "\n", "g_vec = tf.reshape(g, [n_params, 1])\n", "h_mat = tf.reshape(h, [n_params, n_params])" ] }, { "cell_type": "markdown", "metadata": { "id": "L9rO8b-0mgOH" }, "source": [ "The Hessian matrix should be symmetric:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.902307Z", "iopub.status.busy": "2024-08-15T02:32:18.902088Z", "iopub.status.idle": "2024-08-15T02:32:18.905558Z", "shell.execute_reply": "2024-08-15T02:32:18.904989Z" }, "id": "8TCHc7Vrf52S" }, "outputs": [], "source": [ "def imshow_zero_center(image, **kwargs):\n", " lim = tf.reduce_max(abs(image))\n", " plt.imshow(image, vmin=-lim, vmax=lim, cmap='seismic', **kwargs)\n", " plt.colorbar()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:18.908508Z", "iopub.status.busy": "2024-08-15T02:32:18.908283Z", "iopub.status.idle": "2024-08-15T02:32:19.182786Z", "shell.execute_reply": "2024-08-15T02:32:19.182052Z" }, "id": "DExOxd7Ok2H0" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAH5CAYAAABQ0DBNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQg0lEQVR4nO3de3SU5b33/0+GOAkxzISYkBAJchJjisAjKMZaypaUg91uUdoHrS4R/eGjJe4qbXelTwse2hVrfayHTfHnrkp3K1trH7Gtv4pVNLitQQqahaiwC8ImCAkgJkOSJiGZ+/dHJRrlcH2ZuWcm4/u11qxFJt9cc133Yebie99zfTM8z/MEAAAA3wWS3QEAAIDPCyZeAAAACcLECwAAIEGYeAEAACQIEy8AAIAEYeIFAACQIEy8AAAAEiQz2R34tGg0qt27d2vAgAHKyMhIdncAAEhpnufp4MGDKikpUSCQ+HxKe3u7Ojs7fWk7GAwqOzvbl7aTJeUmXrt371ZpaWmyuwEAQJ9SX1+vIUOGJPQ129vbVdi/v1p8ar+4uFjbt29Pq8lXyk28BgwYIEmqX7lSoZNPdvujQYPcX+C002wd6upyDo1mBk1NBxT1pR+SpEz3Xbv1Pdv/kEaNMPTbb+3t7rHGEzdquBJv2peS2trd27YMUZLy89z7YhmjZD8Mg5n+9cUi8N/bbX/Q0eEeO3q0rW3rRrQwnPfmA8t4/qx/w31/nnqqrSuDi/w7rqznskVnl3tfLLsyEonotNNKez4/E6mzs1Mtkm6RlBXntjsk/ayhQZ2dnUy8XCxdulQ//elP1dDQoHHjxunBBx/Uueeee9y/O3x5MXTyye4TL8vBFgq5x0qfi4lXbq7tjSkUSqGJV9CwzVNo4pUZdG/bMkTJtn8+NxMv6wfSSSe5x/r4nmJm+bS2HljG8+fkk933p3X3+HmM98WJ12HJvD2nv6R4T43S9SZ0X8b15JNPauHChVqyZIneeOMNjRs3TtOnT9fevXv9eDkAAIA+wZeJ17333qv58+dr3rx5Ki8v10MPPaScnBw9+uijfrwcAABIooBPj3QU93F1dnZqw4YNqqys/PhFAgFVVlaqtrb2M/EdHR2KRCK9HgAAAOko7hOv/fv3q7u7W0VFRb2eLyoqUkNDw2fiq6urFQ6Hex58oxEAgL6FjJe7pI9r0aJFam5u7nnU19cnu0sAAMCAiZe7uH+rsaCgQP369VNjY2Ov5xsbG1VcXPyZ+KysLGVlxftLqAAAAKkn7hPKYDCoCRMmaPXq1T3PRaNRrV69WhUVFfF+OQAAkGRkvNz5so7XwoULNXfuXE2cOFHnnnuu7rvvPrW2tmrevHl+vBwAAECf4MvEa86cOdq3b58WL16shoYGjR8/XqtWrfrMDfcAAKDv8yNDRcbLqKqqSlVVVSfewKBB7ssZW0pfWMtkWBhXrvd1BWuD0aOMKzW3GKpyWZdftq4un53jHBvoshVxDZzI0tGOLIdhbq6PjRu2n3QCq2lbjnE/Kz/4WB7HvjK6gXGDm6ot+Hh8S5LlC+qWtxTJVnLLOsygId6yEv2J9AXpicMAAADEhIyXu3QdFwAAQMoh4wUAAGJCxstduo4LAAAg5ZDxAgAAMclQ/DM5GXFuL1Uw8QIAADHJUPwnSuk68eJSIwAAQIKQ8QIAADHp99Ej3m2mIzJeAAAACULGCwAAxITlJNyl7sTrtNOkUMgt1lIOZMcOWz/KypxDrWUvQnLvd2e247Y4zFCpJZhpLBlkqXuRSjUyjH2xlAOxDjM/z8dyN4YyPaayO7KXSDEW0TKJtLj3JWQ4jyVj6R3jNrQcLPZyRIa++HxulhS798VSAkiStm51jx07xr+SaO2yvS+Hsm1ly1xZy6EhuVLoUxEAAPRFZLzcpeu4AAAAUg4ZLwAAEBMyXu6YeAEAgJgw8XKXruMCAABIOWS8AABATMh4uUvXcQEAAKQcMl4AACAmZLzcpeu4AAAAUg4ZLwAAEJOMjx7xbjMdkfECAABIkNTNeHV1mWrOOTPWbNOuXc6hf8saamo6VJjrHBvctdPU9u5M976UFJualrKzjX/QN1lqWFpr6pnis3NMbVvq9VnqHUpSKLPNFG85Vuy1TiPuwV22Y9a0Vaw1DxsanEObsktMTefn+dMPSWrsZ+tLYaF7rHUTWuovHmiyHeP5hr6Ecm11IBv3uVcvLTrpgHvD1pPHBwFJ/XxoMx2l7sQLAAD0Cdxc7y5dxwUAAJByyHgBAICYkPFyl67jAgAASDlkvAAAQEzIeLlL13EBAACkHDJeAAAgJmS83KXruAAAAFIOGS8AABATMl7umHgBAICYMPFyl7ITr2hmUNFMx/IKrnGyV1awlAEqatxoartz4Fjn2B3ttnJEo7vecQ/OG2ZqO2osYeOnffvcYwsLbadxe7t7rLWKkqUalrWcSmeX+zjNJYCMZbza2g19ye609aXFh5Jihxk2urVclKUMUH6moSySJMm9DNl77bYSQCOydxv74l6LLGj9JDK8kVtKAEnSxh0h59ixZbZjtqjVvQTdzswRzrEHW1P2oxxHwN4CAAAxyfjoEe8201G6ZvIAAABSDhkvAAAQk34fPeLdZjoi4wUAANLG0qVLNWzYMGVnZ2vSpElat27dUWOffvppTZw4UXl5eTr55JM1fvx4/epXv/K1f0y8AABATDL08Tcb4/U4kXu8nnzySS1cuFBLlizRG2+8oXHjxmn69Onau3fvEePz8/P1v//3/1Ztba02btyoefPmad68eXr++edP4NXdMPECAABp4d5779X8+fM1b948lZeX66GHHlJOTo4effTRI8ZPmTJFl156qc4880yNHDlS3/rWtzR27Fi9+uqrvvWRiRcAAIhJvLNdn1wXLBKJ9Hp0dHQcsQ+dnZ3asGGDKisrP+5XIKDKykrV1tYedwye52n16tXasmWLJk+ebNwC7ph4AQCAmPg58SotLVU4HO55VFdXH7EP+/fvV3d3t4qKino9X1RUpIaGhqP2vbm5Wbm5uQoGg/rqV7+qBx98UF/5yldOYCu44VuNAAAgZdXX1ysU+nhh26ysrLi2P2DAANXV1amlpUWrV6/WwoULNWLECE2ZMiWur3MYEy8AABATP0sGhUKhXhOvoykoKFC/fv3U2NjY6/nGxkYVFx+9kkIgENCoUaMkSePHj9e7776r6upq3yZeXGoEAAB9XjAY1IQJE7R69eqe56LRqFavXq2KigrndqLR6FHvI4uHlM14BRRVQFG3YEP9uJAMBfgkhQrda59Zai9KUrDpyF9vPZLRw/JMbXeq3L0fmY7b+SOBLkN9MmuhQSNL/UXn4+kjOZnux1VU7vVCJeM2N9ZHNPXEWGTSUntRknJkqQVpO1basvPd+2E8xi31F63HVX6eJdr9/cdq2DDrX7jXXrSy1BeVpHa511MM5dr2j6X+4oEW23mfO8S9/uJQwzEbidjG6IdUKZK9cOFCzZ07VxMnTtS5556r++67T62trZo3b54k6eqrr9app57ac59YdXW1Jk6cqJEjR6qjo0N//OMf9atf/UrLli2L40h6S9mJFwAAgMWcOXO0b98+LV68WA0NDRo/frxWrVrVc8P9zp07FQh8PKVrbW3VN7/5Te3atUv9+/dXWVmZfv3rX2vOnDm+9THD8zwvng3edtttuv3223s9d8YZZ2jz5s1Ofx+JRBQOh9X84YdO13Ql2TIC7baMl3INGS/j/9osGS/l5Zna7jTkPawZL9P29jnj5WdmwjLOaKbtf76mvhgzXibG/eNrxsvaly73bZ6TnToZr1RhGaPk7zjNGS/D27g142U538wZL0MC0/K+HIlEFB44UM3Nze6fm3Fy+DP7OUknx7ntVkkzpaSMy0++fCp+4Qtf0Isvvvjxi/j84QsAANAX+DIjyszMPOY3CAAAQPrI0ImV+Dlem+nIl281/vWvf1VJSYlGjBihK6+8Ujt37jxqbEdHx2dWpQUAAH1HP58e6SjuE69JkyZp+fLlWrVqlZYtW6bt27frS1/6kg4ePHjE+Orq6l4r0paWlsa7SwAAACkh7hOvmTNn6utf/7rGjh2r6dOn649//KOampr0m9/85ojxixYtUnNzc8+jvr4+3l0CAAA+8rNkULrx/a73vLw8jR49Wlu3bj3i77OysuK+/D8AAEAq8n1C2dLSom3btmnw4MF+vxQAAEiCDMU/28XN9Y6+853vaM2aNdqxY4dee+01XXrpperXr5+uuOKKeL8UAABAnxL3S427du3SFVdcoQ8++ECFhYW64IILtHbtWhUWFtoa6uryZeHIzmzbImzBXUf/Ruan7WgfamrbVAbo1VdNbW/Ku9A5dvx440KKhnXZzIvKGhdztSykaCkBJEnav985tCm7xNR0Xp5huxgXZ7VoabHFh7IN5aIkmd5ijOv9mcoAGRdOthzj5kWCGxqcQ99rtx1XljJAgfXrTG1vzD7XFF9W5h5r3YSW47Bxn+38KWrd5RxrKQEk2d7GL8yrcw+2nsg+SJWSQX1B3CdeTzzxRLybBAAASAssKQ8AAGJCxssdEy8AABATJl7u0nVcAAAAKYeMFwAAiAkZL3fpOi4AAICUQ8YLAADEJEPxX/CUBVQBAAAQEzJeAAAgJv0+esS7zXRExgsAACBByHgBAICY8K1Gd6k78crMtBfwcmEs17c7073+4uiud0xtd6rcOdZSe1GSzu6y1GGbaGrbwroLo8ZTLTvb0ratZpul/mJenqlpk4Bs9SsjLe7bMKSIrTMtthOoLTvfOdZUe1GSdrnX1NOQIba2fdTYz/24GpG929h6sXOktfbi2Nz3TPHRTFsdQ78UnXTAFL/T0O+hxmPWUn/xneyznWNbuoznsQ+YeLlL13EBAACknNTNeAEAgD4hQ/HP5LCcBAAAAGJCxgsAAMSEe7zcpeu4AAAAUg4ZLwAAEBMyXu7SdVwAAAAph4wXAACICRkvd+k6LgAAgJRDxgsAAMQkQ/Ffdytd1/HK8DzPS3YnPikSiSgcDmvDhmbl5oac/mb0KGOpEb+0t9viDfVurKV0TGVmnn3W1LZmzHCP9aPs0yccaHLfLvl5/h0n1v2zb597bFFh6vQ70NVpewEf97+l79Z+H2hxLy/l53HV2WXbP0FDCZu2dlvbOdnGca5f7x47frytbcNxZT3GLW/j5m3S5V5yK5rpfgxGIhENHBhWc3OzQiG3z814OfyZvUXSgDi3fVDSGVJSxuUnLjUCAAAkCJcaAQBATLi53l26jgsAACDlkPECAAAxIePlLl3HBQAAkHLIeAEAgJiQ8XKXruMCAABIOWS8AABATDIU/0xOui6gysQLAADEhEuN7tJ1XAAAACmHjBcAAIgJGS93KTvxGjUiqlDIsQ5WS4t7w9bacZZ6itk5pqYt9eMCftY8tNRelKRdu9xjDdtPkpSbawrPyzPU79q/39YXQ9/bM211xEz1FxsaTG1bjvGAcXubj/GmA+7BxmOlK9O9Ly2G2ouSzyVGDfX6Mg31+qxt5xjH2Nll60vQWn/RJ4H2NlN8juU4NGxva3yX3Le3tRtIrpSdeAEAgL6BjJe7dB0XAABAyiHjBQAAYkLGy126jgsAACDlMPECAAAxCfj0OBFLly7VsGHDlJ2drUmTJmndunVHjf23f/s3felLX9LAgQM1cOBAVVZWHjM+Hph4AQCAmKTKxOvJJ5/UwoULtWTJEr3xxhsaN26cpk+frr179x4xvqamRldccYVefvll1dbWqrS0VNOmTdP7779/Aq/uhokXAABIC/fee6/mz5+vefPmqby8XA899JBycnL06KOPHjH+8ccf1ze/+U2NHz9eZWVl+sUvfqFoNKrVq1f71kcmXgAAICZ+ZrwikUivR0dHxxH70NnZqQ0bNqiysvLjfgUCqqysVG1trdM42tradOjQIeXn5xtGb8PECwAApKzS0lKFw+GeR3V19RHj9u/fr+7ubhUVFfV6vqioSA2OC1F/73vfU0lJSa/JW7yxnAQAAIiJn8tJ1NfXKxT6uDpIVlZWnF/p7+666y498cQTqqmpUba16opBeky8LPU9fK0FYmToS2eX7ZC2DNNcjshyQM6ebWv7+edN4a++6h47eb8hWDKVUnrrLVvTk84xBBv3z+gLBjnH/ledrZxK4Nnfm+L/lP1PzrHTpriX0JKk4OaNzrH5Y8aY2jZ58UVb/JQpzqH79tmaLuo2lMUqKDC1banMJkn5ef691z75lPv74ZyLjY3v2OEc2njyCFPTRf3bnWODXRH32Hb32L4oFAr1mngdTUFBgfr166fGxsZezzc2Nqq4uPiYf3vPPfforrvu0osvvqixY8fG1N/j4VIjAACISYakjIyM+D6MfQgGg5owYUKvG+MP3yhfUVFx1L+7++67deedd2rVqlWaOHHiiW0AgxRK/wAAAJy4hQsXau7cuZo4caLOPfdc3XfffWptbdW8efMkSVdffbVOPfXUnvvEfvKTn2jx4sVasWKFhg0b1nMvWG5urnJzc33poznj9corr+jiiy9WSUmJMjIy9Mwzz/T6ved5Wrx4sQYPHqz+/fursrJSf/3rX+PVXwAAkGoyM/15GM2ZM0f33HOPFi9erPHjx6uurk6rVq3queF+586d2rNnT0/8smXL1NnZqa997WsaPHhwz+Oee+6J26b5NPOoWltbNW7cOF177bW67LLLPvP7u+++Ww888IB++ctfavjw4frhD3+o6dOn65133vH1ZjUAAJAkmZlShvXi4HF4ntTVZf6zqqoqVVVVHfF3NTU1vX7eYbinL17ME6+ZM2dq5syZR/yd53m677779IMf/ECXXHKJJOnf//3fVVRUpGeeeUaXX355bL0FAADow+J6j9f27dvV0NDQa/2LcDisSZMmqba29ogTr46Ojl6LoUUi6f3tDAAA0o5fGa80FNdvNR6+Kc2yeFl1dXWvhdFKS0vj2SUAAICUkfTlJBYtWqTm5uaeR319fbK7BAAALFLk5vq+IK4Tr8MLlFkWL8vKyupZHM11kTQAAIC+KK4Tr+HDh6u4uLjX4mWRSESvv/76MRcvAwAAfVi/fvHPdvXrl+xR+cKcx2tpadHWrVt7ft6+fbvq6uqUn5+voUOH6uabb9aPfvQjnX766T3LSZSUlGjWrFnx7DcAAECfk+F5tq8N1NTU6B/+4R8+8/zcuXO1fPlyeZ6nJUuW6OGHH1ZTU5MuuOAC/fznP9fo0aOd2o9EIgqHw2r+8MM+d9kxakwgBhT1qSe2vpj7YS3a5qNorvsxEmjx7xuzbZm2Y9WypF2g3VZP0VfWtfja3WvTWe/n6FTQOTaYaTzGT2DtID9EM93HKEmBLlu9SwvL9paM9WL76jFuOb6NOjNznGMjkYgKC8Nqbm5O+Odmz2d2UZFCgfjeNh6JRhVubEzKuPxkznhNmTJFx5qrZWRk6I477tAdd9wRU8cAAEAfkZkpxXnipah/yYlkSvq3GgEAAD4v0vO7mgAAIHHIeDkj4wUAAJAgZLwAAEBs+vWL//IP3d3xbS9FkPECAABIEDJeAAAgNn4seBrvotspgowXAABAgpDxAgAAsSHj5YyJFwAAiA0TL2dcagQAAEiQ1M14tbdLQbfaYNFs95pWftq3zxZfWOg+77WWBLOUGzvQZJt/5+W518x69VVT07rgAlt84LbFzrHvXWMrY5WX5x6b/8TDprZ1zTXOodbjO/Ds792D//EfTW0/8K+2Y+WfK3c4x0aGlJvaXr7cPfaGG2z9zjTUSLSf9+6xgR3vmdpuKx7hHJvTstfUdtBYp9NSvzTHWgN0xw732GHDTE13drkfK0HrG3NurnPo1q3uzaZE+VwyXs7IeAEAACRI6ma8AABA39Cv39+zXjguMl4AAAAJwvQUAADEJjOTjJcjMl4AAAAJwvQUAADEhoyXM7YSAACIDRMvZ1xqBAAASBCmpwAAIDZ+LCfhefFtL0WQ8QIAAEiQ1M14ZWc7170JdHW6t+vjNWhLCSBJCijqHJuT2WVqOyr3kif5ee79kCTt3+8cOnm/sWZQS6Up3FIGaMSLxrI+s2Y5h/7XlOtNTY/OdN/mgaYDprb/lP1PzrHT2ttMbVtKAEnSo2vdywBdO8s2zn/+mqFcS2axqW11uZ9vRd3u58Pf2y5wDrWUAJKknAZDiSFjKZ22dtv7W0624X3FsL0lqfFk9+1SZDzGLWWA3mvKN7U9wvBeWz7K/XMtEjF8BvrFj3u8yHgBAAAgFqmb8QIAAH0DGS9nZLwAAAAShIwXAACIDRkvZ0y8AABAbJh4OeNSIwAAQIKQ8QIAALHxYwHVqHGpoz6CjBcAAECCkPECAACx8eMeLzJeAAAAiAUZLwAAEBsyXs5SduIVVUBRx4RcwLCzO7tsSb6goaaeocSXJGP9RUN9RElqyi5xjs3PMzXtXENTkjRjhq1tY822vDxDsKH2oiRp1y7n0My8Qba2LSzbW9K0KZa6bba3gMgQ99qLkrH+omF7S9LOvLHOsQXmc9MQXOBee9Eqp2Wv7Q8M9Rf/a6vtvXB09k5bX4oN9TGN531Rf8sONX7M5eY6h1pqL0q2z5+gZRLjYw1ixB97CwAAxIaMlzPu8QIAAEgQMl4AACA2ZLycMfECAACx8WMB1e7u+LaXIrjUCAAAkCBMvAAAQGwOX2qM9+MELF26VMOGDVN2drYmTZqkdevWHTX27bff1uzZszVs2DBlZGTovvvuO8EN4I6JFwAASAtPPvmkFi5cqCVLluiNN97QuHHjNH36dO3de+SlWdra2jRixAjdddddKrYsgRIDJl4AACA2KZLxuvfeezV//nzNmzdP5eXleuihh5STk6NHH330iPHnnHOOfvrTn+ryyy9XVlZWrFvBCRMvAACQsiKRSK9HR0fHEeM6Ozu1YcMGVVZW9jwXCARUWVmp2traRHX3uJh4AQCA2PiY8SotLVU4HO55VFdXH7EL+/fvV3d3t4qKino9X1RUpIaGBt83gauUXU4i8FHRoHizZi5dyxZJ5souiiroHGspASTZSulYxihJ7Zkh59i33jI1rbPOssXnP/Gwc+x/Tbne1LalDNCIu2xtRx9y73dXZo6p7eDmjc6xnWXuZXckaflyU7j++WvupV0sJYAk6Uc/co99+CFraRf3c7OlxdS0pSKNgsY3lbZ293PZXALI2Jdopvs27DK8F0pSsCviHNtpPH+2bnWPLR9lKc9lKwP0+l/c92Vra3rnUOrr6xUKffy5k6hLgn5J2YkXAADoI/xYx+ujGp6hUKjXxOtoCgoK1K9fPzU2NvZ6vrGxMWE3zrtI72kyAADwXwrcXB8MBjVhwgStXr2657loNKrVq1eroqIi3iM+YeaJ1yuvvKKLL75YJSUlysjI0DPPPNPr99dcc40yMjJ6PWbMmBGv/gIAABzRwoUL9W//9m/65S9/qXfffVc33nijWltbNW/ePEnS1VdfrUWLFvXEd3Z2qq6uTnV1ders7NT777+vuro6bbVcczYy5wVbW1s1btw4XXvttbrsssuOGDNjxgw99thjPT/39euxAADgGPyo1XgC7c2ZM0f79u3T4sWL1dDQoPHjx2vVqlU9N9zv3LlTgcDHOafdu3frf/yP/9Hz8z333KN77rlHX/7yl1VTUxPzEI7EPKqZM2dq5syZx4zJyspyvp7a0dHR66uhkYj7TZMAAACfVFVVpaqqqiP+7tOTqWHDhsnzvAT06mO+3ONVU1OjQYMG6YwzztCNN96oDz744Kix1dXVvb4mWlpa6keXAACAX1LgHq++Iu4TrxkzZujf//3ftXr1av3kJz/RmjVrNHPmTHUfpcr4okWL1Nzc3POor6+Pd5cAAABSQtynk5dffnnPv8866yyNHTtWI0eOVE1NjaZOnfqZ+KysLO4BAwCgL0uRe7z6At+XkxgxYoQKCgp8/YYAAABAX+D7dHLXrl364IMPNHjwYL9fCgAAJIMfC6j26xff9lKEeSu1tLT0yl5t375ddXV1ys/PV35+vm6//XbNnj1bxcXF2rZtm/7lX/5Fo0aN0vTp0+PacQAAkCK41OgswzN+j7Kmpkb/8A//8Jnn586dq2XLlmnWrFl688031dTUpJKSEk2bNk133nnnZ4pWHk0kElE4HNaePc1OJQIkqd29HJzy82w12yx1DD+qbuAsmOneF2s9RYt9+2zxRYXxr6F5mHWcgS5DrTQfT2Jzvxt2uwdbivudSLyBeZyWeqvWE8iwP19ba+t3QYF77OhR/r2n+FGv9rBIi22bhHKNfbG8MVsLXlp2kJXlOLS+pxjattS6jEQiGjgwrOZm98/NeDn8md18330K9e8f37b/9jeFb745KePyk/mTaMqUKcdc8+L555+PqUMAAKCPIePljFqNAAAACZKe00kAAJA4ZLyckfECAABIkPScTgIAgMRhOQlnZLwAAAAShIwXAACIDfd4OUvPUQEAgMRh4uWMS40AAAAJkp7TSQAAkDhkvJyl7Kja26WgY8UEU4UUSxkLScrOcQ41HyOm0hTu5SMkW6mRokJT01JDg3uscaMEjOVuoob9E2g6YGpb2dnOoV2Z7v2QpKBlnE1NprZNrAetYXtLspWCMfbFUlLFWmEmVd7v/SzRZK0s1dll60vQsu8N55pkLLvU3mZq29oXE8N7fpfcj29rtS0kV4q8vQAAgD6LjJcz7vECAABIkPScTgIAgMRhAVVnZLwAAAAShIwXAACIDfd4OUvPUQEAgMRh4uWMS40AAAAJkp7TSQAAkDhkvJyR8QIAAEiQ9JxOAgCAxGE5CWdkvAAAABIkZTNe+XlRhUKOdccs9ReNRa0stc/MtcxM0TaRFve+hHLdxyjJ9L+a0RcMMjX9X3W2umqBZ3/vHPun7H8ytT1tSqdzbHDzRlPbGjPGFm9wWzjsHtvRYWr7qqtsfVnRdLl78LPPmtr+9a/dY6++yniMGyy+zXbe33GboS+bN5vafkflzrHlZbZtEmzab4q3FMg016T8zkL34LvuMrX9zmb3vpS3rDO1bTnvLaUuLbG+4R4vZ2S8AAAAEiQ9p5MAACBxyHg5S89RAQCAxGHi5YxLjQAAAAmSntNJAACQOGS8nJHxAgAASJD0nE4CAIDEYQFVZ2S8AAAAEoSMFwAAiA33eDkj4wUAAJAgGZ7necnuxCdFIhGFw2F9+GGzQqFQ3Nu3lACSjKV3Mm3lbpSd7Us/JCmkiHNsNNe2nQPtxnH6ybANTaWlJNP/tjqNBaCCmYbj0Md+W9vuzLYdK0G5l12y9qUt070vlsPEyvqe0tbufi7nyL/3FEs//O6Lr8e4sZ5OZ26+c6zp+Db2JZLp3o9IJKLS0rCam/353Dzea4fDYTVv3qzQgAHxbfvgQYXLypIyLj+lZx4PAAAkDpcanXGpEQAAIEHSczoJAAASh+UknJHxAgAASBAyXgAAIDbc4+WMjBcAAECCpOd0EgAAJA4ZL2dkvAAAQNpYunSphg0bpuzsbE2aNEnr1q07ZvxTTz2lsrIyZWdn66yzztIf//hHX/vHxAsAAMTmcMYr3g+jJ598UgsXLtSSJUv0xhtvaNy4cZo+fbr27t17xPjXXntNV1xxha677jq9+eabmjVrlmbNmqVNmzbFukWOiokXAACITYpMvO69917Nnz9f8+bNU3l5uR566CHl5OTo0UcfPWL8/fffrxkzZui73/2uzjzzTN155506++yz9a//+q+xbpGjYuIFAABSViQS6fXo6Og4YlxnZ6c2bNigysrKnucCgYAqKytVW1t7xL+pra3tFS9J06dPP2p8PKTsnWtdXX9/uLBMiju7jDUPLfUXXTv8EUuttFC2tSaYe18CXba2o9k57m0/+3tT2/rHfzSFP/Cv7tvwnyt3mNqODCl3jl2+3NS0qqoMx6Fhe0vSVVe5xy5fbqwxecG5pvjf/+DY91Z80qRJtr4UTTnTPfjtt01tRy3/J9282dR2zrBhzrFPr7Lt+8suOPLllCP2o6nJ1PbTm0bb+vKPhvcVYzHNdza7759Ro9xrHkpScPNG59gDQ8aa2s411IEMbXrDvWFjPUo/RBWwnTeObUpSaWlpr+eXLFmi22677TPx+/fvV3d3t4qKino9X1RUpM1HOU8bGhqOGN/Q0BBDz48tZSdeAAAA9fX1vYpkZ2VlJbE3sTNNT6urq3XOOedowIABGjRokGbNmqUtW7b0imlvb9eCBQt0yimnKDc3V7Nnz1ZjY2NcOw0AAFLH4atU8X5IUigU6vU42sSroKBA/fr1+8yco7GxUcXFxUf8m+LiYlN8PJgmXmvWrNGCBQu0du1avfDCCzp06JCmTZum1tbWnphbbrlFf/jDH/TUU09pzZo12r17ty677LK4dxwAAOCwYDCoCRMmaPXq1T3PRaNRrV69WhUVFUf8m4qKil7xkvTCCy8cNT4eTJcaV61a1evn5cuXa9CgQdqwYYMmT56s5uZmPfLII1qxYoUuvPBCSdJjjz2mM888U2vXrtV5550Xv54DAICUYLkv29Km1cKFCzV37lxNnDhR5557ru677z61trZq3rx5kqSrr75ap556qqqrqyVJ3/rWt/TlL39Z/+f//B999atf1RNPPKH169fr4YcfjudQeonpHq/m5mZJUn7+328Y3LBhgw4dOtTrGwJlZWUaOnSoamtrjzjx6ujo6PUNhUgkEkuXAADA59ScOXO0b98+LV68WA0NDRo/frxWrVrVcwP9zp07FQh8fLHv/PPP14oVK/SDH/xA3//+93X66afrmWee0ZgxY3zr4wlPvKLRqG6++WZ98Ytf7OlgQ0ODgsGg8vLyesUe6xsC1dXVuv3220+0GwAAIMlSJeMlSVVVVaqqqjri72pqaj7z3Ne//nV9/etfP7EXOwEn/N3PBQsWaNOmTXriiSdi6sCiRYvU3Nzc86ivr4+pPQAAkFh+3lyfbk4o41VVVaVnn31Wr7zyioYMGdLzfHFxsTo7O9XU1NQr63WsbwhkZWX1+a+GAgAAuDBlvDzPU1VVlVauXKmXXnpJw4cP7/X7CRMm6KSTTur1DYEtW7Zo586dvn5DAAAAJE93d/yzXd3dyR6VP0wZrwULFmjFihX63e9+pwEDBvTctxUOh9W/f3+Fw2Fdd911WrhwofLz8xUKhXTTTTepoqKCbzQCAIDPPdPEa9myZZKkKVOm9Hr+scce0zXXXCNJ+tnPfqZAIKDZs2ero6ND06dP189//vO4dBYAAKSeVLq5PtVleJ7nJbsTnxSJRBQOh9X84Ye9SgQck597x1AI0lJ7UZJyZKgDaazS3tblXvcuJztqaluGGm9/Wm+rkzbtAsM2kaQdO5xDH13rXntRkq6ddcA9uL3d1LYsqyJb67Bdfrl77DPPmJr+/SpbPcV/usrxHJakXbtMbb9S59725AuMx7iBpW6gJJWXGfqyf7+p7de2DnKOPf884zaxHoe5ubZ4i/Xr3WPHjzc1faDF/RjPX/8nU9vRymnOsZbNHYlEVFoaVnNzs/vnZpwc/szevLlZAwbE97UPHoyorCw54/ITtRoBAEBMyHi5i28pcQAAABwVGS8AABATMl7uyHgBAAAkCBkvAAAQk8PreMW7zXTExAsAAMSES43uuNQIAACQIGS8AABATMh4uSPjBQAAkCBkvAAAQEzIeLlL2YlXVAFFXRNymbYyJhaWsg2h7E5j64bNbywZlJPpX4kUZWc7h06b4uM2kRQZ4l4GyFQCSDKVsNmZN9bU9NAuw3Yx7ns9+6x7rLEMzKRJxnPNUgZo61ZT08XFZzvHRlpsyX1LtRtTCSDZSovlGMpzSdL55xU4x75UY9smF+ZuNsWbSvVYP2HHjHGPNR7jubnuZc4sJYAkKdBuKYmWY2obfUfKTrwAAEDfQMbLHfd4AQAAJAgZLwAAEBMWUHXHxAsAAMSES43uuNQIAACQIGS8AABATMh4uSPjBQAAkCBkvAAAQEzIeLkj4wUAAJAgZLwAAEBMWE7CHRkvAACABEmLjFdA7rXSrDXbQoq4B7fYpvtt2e41wcy1Fw018qJDhpqa7sp0ryEW3LzR1HZnma3m4fLl7rH//LV2U9uW+os/+pGpaT38kPupFzXWIv31r91jv/a1kKntoilnmuJf+X/fdY611F6UpJwzMpxjQ8b/Ond2ub9PBJv2m9rOMRSCfHrTaFPblxW7v19Zay/+5OVzTfHfnege2yXbMW4pv5hpeL+SpNCmN5xjI6Nsx6yl/mLoP/8/92bbLDUg/cE9Xu7SYuIFAACSh4mXOy41AgAAJAgZLwAAEBMyXu7IeAEAACQIGS8AABATMl7uyHgBAAAkCBkvAAAQExZQdUfGCwAAIEHIeAEAgJhwj5c7Jl4AACAmTLzcZXie5yW7E58UiUQUDofVXFen0IABbn/UbigFU1Zm65Cfez7TMO+1jFGSsrPdY41jPNDiXt4jP89Y6sjIVNrFWHaprd297ZxsW9uvrXVvu6DA1LRGj3LvS9R4t4GlPJeVuZxXrqEvV11l68xEQ72bm2+2tW3g6/6pqzO13VZmK4+zfr17rOWtUJLOP8+/Y9zydmh9T2nc596XokJDKbxIROGBA9Xc3KxQyFYGLFaHP7Pvu69Z/fvH97X/9reIbr45nJRx+YmMFwAAiAkZL3fcXA8AAJAgZLwAAEBMWE7CHRkvAACABCHjBQAAYsI9Xu7IeAEAACQIGS8AABATMl7umHgBAICYMPFyx6VGAACABGHiBQAAYnI44xXvh58OHDigK6+8UqFQSHl5ebruuuvU0tJyzL95+OGHNWXKFIVCIWVkZKipqcn8uky8AADA586VV16pt99+Wy+88IKeffZZvfLKK7r++uuP+TdtbW2aMWOGvv/975/w66buPV4dHdJJJ7nFGuoS2mufGRgLjln6ErAWMzOw1F6U7HXVTIz/xcnMNPTd2HaOYZydXbZtaKm/6Ov2NvKzdmBurq0vpjqdltqLkr8b3XAc+nneW+vW5qjNFJ+ZmeMcayktK8l2LlveI2Tc9cb3lMJC975Yjm9LrF/8XEA1Eon0ej4rK0tZWVkxtf3uu+9q1apV+stf/qKJH70/PPjgg7rooot0zz33qKSk5Ih/d/NHtVlrampO+LWTv7cAAACOorS0VOFwuOdRXV0dc5u1tbXKy8vrmXRJUmVlpQKBgF5//fWY2z8W08Srurpa55xzjgYMGKBBgwZp1qxZ2rJlS6+YKVOmKCMjo9fjhhtuiGunAQBA6vDzHq/6+no1Nzf3PBYtWhRzfxsaGjRo0KBez2VmZio/P18NDQ0xt38sponXmjVrtGDBAq1du1YvvPCCDh06pGnTpqm1tbVX3Pz587Vnz56ex9133x3XTgMAgM+HUCjU63Gsy4y33nrrZ5I/n35s3rw5gb3/LNMNBKtWrer18/LlyzVo0CBt2LBBkydP7nk+JydHxcXF8ekhAABIaV1dUr9+8W/T6tvf/rauueaaY8aMGDFCxcXF2rt376der0sHDhzwff4S052bzc3NkqT8/Pxezz/++OP69a9/reLiYl188cX64Q9/qJycI99o2dHRoY6Ojp6fP30THQAASG2pMvEqLCxUYWHhceMqKirU1NSkDRs2aMKECZKkl156SdFoVJMmTbK/sMEJ31wfjUZ1880364tf/KLGjBnT8/w3vvEN/frXv9bLL7+sRYsW6Ve/+pWuuuqqo7ZTXV3d66a50tLSE+0SAADAcZ155pmaMWOG5s+fr3Xr1unPf/6zqqqqdPnll/d8o/H9999XWVmZ1q1b1/N3DQ0Nqqur09atWyVJb731lurq6nTgwAHn1z7hjNeCBQu0adMmvfrqq72e/+QaGGeddZYGDx6sqVOnatu2bRo5cuRn2lm0aJEWLlzY83MkEmHyBQBAH+LnchJ+efzxx1VVVaWpU6cqEAho9uzZeuCBB3p+f+jQIW3ZskVtbR8vpfLQQw/p9ttv7/n58G1Wjz322HEvcR52QhOvqqqqnsXGhgwZcszYwym7rVu3HnHiFY/1OAAAACzy8/O1YsWKo/5+2LBh8jyv13O33Xabbrvttphe1zTx8jxPN910k1auXKmamhoNHz78uH9TV1cnSRo8ePAJdRAAAKS2ri4pEOeVQdO1SLZp4rVgwQKtWLFCv/vd7zRgwICetS7C4bD69++vbdu2acWKFbrooot0yimnaOPGjbrllls0efJkjR071pcBAAAA9BUZ3qfzaMcKzsg44vOHr23W19frqquu0qZNm9Ta2qrS0lJdeuml+sEPfqBQKOT0GpFIROFwWM0ffuj8N6bSO4YSJlZ+llPps1580RY/ZYopvPFD9xIcRd27bX0x1PWxll3Kz/Nv3y++zf04vOM2Yz+M69+8o3Ln2PIyY1/273ePtdRosvroJltno0Y5h7a1295TcroM3wo31miylqUJZhr2pzG18c5W9/PNfFwdp0jyJ7Vlun1GHZaT7c95H4lEFB44UM3Nzc6fm3F97XBYc+c2KxiM72t3dkb0y1+GkzIuP5kvNR5LaWmp1qxZE1OHAAAA0lUKld8FAAB9Efd4uWPiBQAAYsLEy12cNxMAAACOhowXAACISV9cQDVZyHgBAAAkCBkvAAAQk64u6SgrTsXUZjoi4wUAAJAgZLwAAEBMyHi5I+MFAACQIGS8AABATMh4uUvdiVdXl/NWN6XtMo1D/qgQuIum7BJT0/l5/vRDkhr7ufelqNBYP8xyNhhrL1oVFhqCu/yr12cse2eu62lhqb9orgU4bJgpvtxQm87cF+tGt7Ac44bai5JMNSa7sgfZ2rZsE0utS0ktmba+5OUZ9memrdappf5ipMV2XFkqAlprL1r6EpKh7qahvqRfmHi541IjAABAgqRuxgsAAPQJ3d3xz3ixgCoAAABiQsYLAADExI/7sbjHCwAAADEh4wUAAGJCxssdGS8AAIAEIeMFAABiQsbLHRkvAACABCHjBQAAYuLHmlvpuo5Xeky8DGWArKVaLGWA8jMNJR4kSe7lPd5rt5UjGpG92zm2s8vWdqahvMe+faambSWAJAV2vOcc21Y8wtR2Tste59hgdrapbUtpF3N5oc2bnUOtJYCeXpVjir/sAvdtmNPUZOvLptHOsbNmmZpWwPCeYi11ZCkDFPrto6a2O6+61jn2Z4/ZSgB9L+NuW19u/hfnWGslN0uJHEsJIEl6Z5f7X5SP6jS1Hdq03jn2la7znWNbW03d8EVXl+R58W0zXSdeXGoEAABIkPTIeAEAgKQh4+WOjBcAAECCkPECAAAxIePljowXAABAgpDxAgAAMSHj5Y6MFwAAQIKQ8QIAADHp7o5/xisajW97qYKJFwAAiElXlxSI8zW0dJ14cakRAAAgQch4AQCAmJDxcpfhefG+KhubSCSicDis5g8/VCjkVjPLUssuoL65J631+nwdZ1eXe+z+/ba2CwpM4W1d7nUjcxrc6zpKkgx1DK31+nKy/ds/72x270t5mbEfxv352lb3eoDnn2fsi6Fen6U2ppmlH5KpL51dtuMq2HLAPTgvz9S2uS+Zhv1peU+R8by3nmuGvhxoce+HJOXn+XPeRyIRhQcOVHNzs/PnZlxfOxxWUVGzAoH4vnY0GlFjYzgp4/ITGS8AABATMl7uuMcLAAAgQch4AQCAmHR3xz9DlVo3QsUPGS8AAIAEIeMFAABi0tUlZWTEt810zXgx8QIAADFh4uWOS40AAAAJQsYLAADEhIyXOzJeAAAACULGCwAAxISMl7vUnXi1t0tBt3IMgUzDMCyxktTQ4Bz6XnuJqWlDRRoF1q8ztb0x+1zn2FGjTE0rx7IJjSWArHJa9roHWza4pP/a6p4QHp2909R2JG+oc6y12o2lDJC51FFTkyn+/PPc9/9LNba+XJi72T3Yet6XlbnHWneQoezSzx5zL7kkSd/7bp57sPF8uOQLtmN8+XL3/VlYaCu9k2MoRxRpsR1XoU3rnWPzzzvP1PbDv3Dvy/UFT7s33NZm6gf+7sCBA7rpppv0hz/8QYFAQLNnz9b999+v3KOc0wcOHNCSJUv0pz/9STt37lRhYaFmzZqlO++8U+Fw2Pl1U3fiBQAA+oioPC/eNX78rRl05ZVXas+ePXrhhRd06NAhzZs3T9dff71WrFhxxPjdu3dr9+7duueee1ReXq7//u//1g033KDdu3frt7/9rfPrMvECAACfK++++65WrVqlv/zlL5o4caIk6cEHH9RFF12ke+65RyUln72CNWbMGP3f//t/e34eOXKkfvzjH+uqq65SV1eXMh0z66Yc7LJlyzR27FiFQiGFQiFVVFToueee6/l9e3u7FixYoFNOOUW5ubmaPXu2GhsbLS8BAAD6nG6fHlIkEun16OjoiLm3tbW1ysvL65l0SVJlZaUCgYBef/1153aam5sVCoWcJ12SceI1ZMgQ3XXXXdqwYYPWr1+vCy+8UJdcconefvttSdItt9yiP/zhD3rqqae0Zs0a7d69W5dddpnlJQAAQJ/j38SrtLRU4XC451FdXR1zbxsaGjRoUO97KDMzM5Wfn68Gx3u79+/frzvvvFPXX3+96bVNlxovvvjiXj//+Mc/1rJly7R27VoNGTJEjzzyiFasWKELL7xQkvTYY4/pzDPP1Nq1a3We8SZEAACA+vp6hUKhnp+zsrKOGnvrrbfqJz/5yTHbe/fdd2PuUyQS0Ve/+lWVl5frtttuM/3tCd/j1d3draeeekqtra2qqKjQhg0bdOjQIVVWVvbElJWVaejQoaqtrT3qxKujo6NX2jASiZxolwAAQFJ8nKGKb5vqub3Jxbe//W1dc801x4wZMWKEiouLtXdv72/Fd3V16cCBAyouLj7m3x88eFAzZszQgAEDtHLlSp100klOfTvMPPF66623VFFRofb2duXm5mrlypUqLy9XXV2dgsGg8vLyesUXFRUdM21XXV2t22+/3doNAACAXgoLC1VYWHjcuIqKCjU1NWnDhg2aMGGCJOmll15SNBrVpEmTjvp3kUhE06dPV1ZWln7/+98rOzvb3EfzyvVnnHGG6urq9Prrr+vGG2/U3Llz9c4775hf+LBFixapubm551FfX3/CbQEAgGSI+vTwx5lnnqkZM2Zo/vz5Wrdunf785z+rqqpKl19+ec83Gt9//32VlZVp3bq/r6MZiUQ0bdo0tba26pFHHlEkElFDQ4MaGhrU3e2e7TNnvILBoEZ9tOLmhAkT9Je//EX333+/5syZo87OTjU1NfXKejU2Nh4zbZeVlXXM67UAAADx9vjjj6uqqkpTp07tWUD1gQce6Pn9oUOHtGXLFrV9tEDtG2+80fONx1GfWnl8+/btGua4KHHM63hFo1F1dHRowoQJOumkk7R69WrNnj1bkrRlyxbt3LlTFRUVsb4MAABIWf7d4+WX/Pz8oy6WKknDhg2T94m6RVOmTOn184kyTbwWLVqkmTNnaujQoTp48KBWrFihmpoaPf/88wqHw7ruuuu0cOFC5efnKxQK6aabblJFRQXfaAQAAJBx4rV3715dffXV2rNnj8LhsMaOHavnn39eX/nKVyRJP/vZz3rSdR0dHZo+fbp+/vOfn1jPsrP//kiyxn7u9RdHZO82tn7sb058kqX2oiSNzX3PPTh7mKntzi73umotLaamzWXvgoZjxFqX0FR/0XishnLd713o7LL1O9jkXgswx7jBn9402hR/WbH7t5RNtRcl/eRl93PipptMTStH7rXvrPunJdO9/uL3Mu42td3Z9S/Osdbai8/1N67JWOheQsV8jBs+uUKyfVP+la7znWMnG+9BstRffPKQ+/ZuO5QKqwFEFf8Mlb8lg5LFNPF65JFHjvn77OxsLV26VEuXLo2pUwAAoC/pe5cak8X8rUYAAACcGIpkAwCAGJHxckXGCwAAIEHIeAEAgBj5seBpet5cT8YLAAAgQch4AQCAGHGPlysyXgAAAAlCxgsAAMSIjJcrJl4AACBGTLxccakRAAAgQTK8eJTajqNIJKJwOKzVq5t18skhp78pLXVvv6TY9vXUqGFuGvDxq6/WWmaZhlxmYP06W2fGj/enI7Jtb0lqb3ePzck27p+uLufQaKZ7/UpJCrS71wI0F7wsKLDFWxi2iSTb/je2bdnmr75qatrU7fPP8+89xbq5g5nufWncZzvXigqN509NjXvsqFG2tocMscUbWN5rLdtbkukNK5qd4xwbiUQ0cGBYzc3NCoXcPjfj5fBntrRekrHY7nG1SJqYlHH5iYwXAABAgnCPFwAAiBELqLoi4wUAAJAgZLwAAECM+FajKzJeAAAACULGCwAAxIiMlysmXgAAIEZMvFxxqREAACBByHgBAIAYkfFyRcYLAAAgQVI243XqqdKAAW6xlooqbe3+ld4JGrempTSFsfKOjaUEkM9MpXQk5WRnuwdb668Y4rtkKxkUtBy0ljHKWObKuL2tfTEx7h/LNreeP6Zhmsso+ddvS18KC23HrLVsWdBSBqihwdS2aQcZj9lgrqHsTVOTqW1fz5+k8xT/BU9TqqJh3JDxAgAASJCUzXgBAIC+gnu8XJHxAgAASBAyXgAAIEZkvFwx8QIAADFi4uWKS40AAAAJQsYLAADEiIyXKzJeAAAACULGCwAAxCiq+C+gGu/2UgMZLwAAgAQh4wUAAGLEPV6uUnbiNbgoqlDILc1oqb+4dautH2PHGFKdlvp7ktoVco4NZXea2jbxsRDkk0/ZkqpzLja+wI4dzqGNJ48wNV3Uv905NtgVMbWtggLnUEvtRUkKfGehe/Bdd5nafmezrS/lLevcg8eMMbVtOd3OP894ycJQ8/Cdrbaah+Vl/r2ntGW6v6fkZNq2ibUWrYYMcY+11jB86in32HnzTE3v3OV+jA9tMdaYLCtzDrWUgYwY336QXCk78QIAAH1FVPHPUHGPFwAAAGJAxgsAAMSIe7xcMfECAAAxYjkJV1xqBAAASBAyXgAAIEZcanRFxgsAACBByHgBAIAYkfFyRcYLAAAgQch4AQCAGJHxcpXheZ6X7E58UiQSUTgc1ocfNisUcit/YajuoaCxTMaBJvekYH6msW5Dbq5zaOM+W3Ky6KQDzrHRvHxT24H2NlO8ryylRtrdSwBZdWbmmOJNx6G135YSUMaSNJ25tmMlKEOpK2NfIpnufTGcamYB41feIy3u53JI/r2nWPrhd1+s+97PY7wtd5BzbI6M74WGvhzIdO9HJBLR8OFhNTe7f27Gy+HPbOkXkmzvg8fXJun/Scq4/MSlRgAAEKPDJYPi+fB3Ha8DBw7oyiuvVCgUUl5enq677jq1HGdy/L/+1//SyJEj1b9/fxUWFuqSSy7R5s2bTa/LxAsAAMQo6tPDP1deeaXefvttvfDCC3r22Wf1yiuv6Prrrz/m30yYMEGPPfaY3n33XT3//PPyPE/Tpk1Td7f7ZVHTxGvZsmUaO3asQqGQQqGQKioq9Nxzz/X8fsqUKcrIyOj1uOGGGywvAQAA4Kt3331Xq1at0i9+8QtNmjRJF1xwgR588EE98cQT2r1791H/7vrrr9fkyZM1bNgwnX322frRj36k+vp67dixw/m1TTfXDxkyRHfddZdOP/10eZ6nX/7yl7rkkkv05ptv6gtf+IIkaf78+brjjjt6/iYnJ97XfAEAQGrx7+b6SKT3/YVZWVnKysqKqeXa2lrl5eVp4sSJPc9VVlYqEAjo9ddf16WXXnrcNlpbW/XYY49p+PDhKi0tdX5tU8br4osv1kUXXaTTTz9do0eP1o9//GPl5uZq7dq1PTE5OTkqLi7ueRzvhriOjg5FIpFeDwAAAEkqLS1VOBzueVRXV8fcZkNDgwYN6v0FhszMTOXn56uhoeGYf/vzn/9cubm5ys3N1XPPPacXXnhBwWDQ+bVP+B6v7u5uPfHEE2ptbVVFRUXP848//rgKCgo0ZswYLVq0SG1tx/7WR3V1da8Napk1AgCAVBDvG+s/zqDV19erubm557Fo0aKj9uLWW2/9zC1Pn35Yb4b/tCuvvFJvvvmm1qxZo9GjR+t//s//qXbDt8/N63i99dZbqqioUHt7u3Jzc7Vy5UqVl5dLkr7xjW/otNNOU0lJiTZu3Kjvfe972rJli55++umjtrdo0SItXLiw5+dIJMLkCwAASFLPfeUuvv3tb+uaa645ZsyIESNUXFysvXv39nq+q6tLBw4cUHFx8TH//nCi6PTTT9d5552ngQMHauXKlbriiiuc+mieeJ1xxhmqq6tTc3Ozfvvb32ru3Llas2aNysvLe30b4KyzztLgwYM1depUbdu2TSNHjjxie/G4VgsAAJIpNRZQLSwsVGFh4XHjKioq1NTUpA0bNmjChAmSpJdeeknRaFSTJk1yfj3P8+R5njo6Opz/xnypMRgMatSoUZowYYKqq6s1btw43X///UeMPdz5rVu3Wl8GAADAF2eeeaZmzJih+fPna926dfrzn/+sqqoqXX755SopKZEkvf/++yorK9O6deskSe+9956qq6u1YcMG7dy5U6+99pq+/vWvq3///rroooucXzvmdbyi0ehRZ3p1dXWSpMGDB8f6MgAAIGX5d4+XXx5//HGVlZVp6tSpuuiii3TBBRfo4Ycf7vn9oUOHtGXLlp571bOzs/Wf//mfuuiiizRq1CjNmTNHAwYM0GuvvfaZG/WPxXSpcdGiRZo5c6aGDh2qgwcPasWKFaqpqdHzzz+vbdu2acWKFbrooot0yimnaOPGjbrllls0efJkjR071vIyAACgT/FjwVN/F1DNz8/XihUrjvr7YcOG6ZNVFUtKSvTHP/4x5tc1Tbz27t2rq6++Wnv27FE4HNbYsWP1/PPP6ytf+Yrq6+v14osv6r777lNra6tKS0s1e/Zs/eAHPzihjgUUda6BFrSMwli3K9/Q9sYdtlpSY8vc69gVte4ytb0zc4RzbIGh3KEk5VjqIxoWlZMkDRtmCu/sck/aBq01Dw215qxX08tHGQqMWra3pHc2u2+TUaOMtRc3bzTFHxji/p+uXGMdyNCmN5xjO8ecbWrbUgrQ+p5ieZd4Z5ftPaV8lPt7SmjTelPbr3Sdb4o/7zz32KCxmObOXe7HeEGBbS3JnFf/5BwbOW+aqe0dDe59GdvymnNsZmurqR9ILtPE65FHHjnq70pLS7VmzZqYOwQAAPqaw7Ua491m+qFWIwAAQIKYl5MAAADoLTWWk+gLyHgBAAAkCBkvAAAQIzJersh4AQAAJAgZLwAAECMyXq6YeAEAgBgx8XLFpUYAAIAEIeMFAABi1PdKBiVLWky8LGVj2k0FO6RQrvuOt5QAkqQDLUHn2Nwh7iWAJGlopo8HbJd7uZvGk239LmpvM8VbygC912QrSTMiz30bWkq1SDLWpLEpb1lniB5vattSAkiS8te7l1+JVtrKr0RGuZcBClnPB8Mx3pZpe0/JyfbvuLK8p+RbavpImuznh2BTkyl8aEuDe3DBMFPbljJAoc2Wc00aO2qUez8y3Us0RSIRUz+QXGkx8QIAAMlEySBX3OMFAACQIGS8AABAjPhWoysyXgAAAAlCxgsAAMSIjJcrJl4AACBGLCfhikuNAAAACULGCwAAxIhLja7IeAEAACQIGS8AABAjMl6uyHgBAAAkSMpmvDq7As41GC1l70LZttpnjfvca58Vte4ytW2pv/jqq6amdWFenXvwmDG2xg117Ir6u9dS/DvjIZmb6xxqqb0o2WqABq21Fw3b0BQr2fZnS4up6dxcW71LS/3FgLFOp5TjHNm4z/Z/zMJC9/M+x1gHMtLi3pfQpvWmti31Fx/+hW2bXF/wtCleM2a4x2Zn29ouK3OP3b/f1PSOBvfjylJ7UZL0zDPOoa8WX+sc22Y9dXxBxssVGS8AAIAESdmMFwAA6Csoku2KiRcAAIgRC6i64lIjAABAgpDxAgAAMeLmeldkvAAAABKEjBcAAIgRGS9XZLwAAAAShIwXAACIERkvV2S8AAAAEiRlM16ZmbZSQH4pOumAc+zOTPcSQJI01FBqxFQCSNI72Wc7x5ZZq93IvZxKsCtiarsz071chyRt3eoeWz7KVi7KUgbo9b/Y/g9zzjnu29CyvSVbFaBM4/YObXrDFB8Z5X4cWkoASVLoP/8/99iZM01t28pFmZpWSO7nxCtd55vanmxY98haAujJQ5eZ4r9urAJk0dRkCM4cZGp7bMtrzrGRTNv+sZQBuqjlN+79SImaQWS8XKXA1AYAAPRtrFzvikuNAAAACULGCwAAxIiSQa7IeAEAACQIGS8AABCjbsU/l5OeN9eT8QIAAEgQMl4AACBGZLxckfECAABIEDJeAAAgRmS8XDHxAgAAMWI5CVcpN/HyPE+SFInYSs24CnTZysZY6q8cbLVtzkjEcFBZ6sBIajGU6rFu6q4u99hgu7FkkH+7R5GIsXFDyaDWVtv/9Czb3LK9JWvJIFvb1uPQr/NYkmQpk2Lsh61kkPHDwbANW1ttTZveU4xlZtoO2bahn7vez7YzDRvdenybDllDcORvf5P08ednchjfqJLWZvKl3MTr4MGDkqTTTitNck8AACfqWveyhIiTgwcPKhwOJ/Q1g8GgiouL1dDwoi/tFxcXKxi01atNdRlecqfInxGNRrV7924NGDBAGRkZPc9HIhGVlpaqvr5eoVAoiT30F+NMH5+HMUqMM918HsaZbmP0PE8HDx5USUmJAoHEf2euvb1dndbLFY6CwaCys32suJ4EKZfxCgQCGjJkyFF/HwqF0uJEOR7GmT4+D2OUGGe6+TyMM53GmOhM1ydlZ2en3eTITywnAQAAkCBMvAAAABKkz0y8srKytGTJEmVlZSW7K75inOnj8zBGiXGmm8/DOD8PY0TqSrmb6wEAANJVn8l4AQAA9HVMvAAAABKEiRcAAECCMPECAABIECZeAAAACdJnJl5Lly7VsGHDlJ2drUmTJmndunXJ7lJc3XbbbcrIyOj1KCsrS3a3YvLKK6/o4osvVklJiTIyMvTMM8/0+r3neVq8eLEGDx6s/v37q7KyUn/961+T09kYHG+c11xzzWf27YwZM5LT2RNUXV2tc845RwMGDNCgQYM0a9YsbdmypVdMe3u7FixYoFNOOUW5ubmaPXu2Ghsbk9TjE+MyzilTpnxmf95www1J6vGJWbZsmcaOHduzcntFRYWee+65nt+nw76Ujj/OdNiX6Hv6xMTrySef1MKFC7VkyRK98cYbGjdunKZPn669e/cmu2tx9YUvfEF79uzpebz66qvJ7lJMWltbNW7cOC1duvSIv7/77rv1wAMP6KGHHtLrr7+uk08+WdOnT1d7e3uCexqb441TkmbMmNFr3/7Hf/xHAnsYuzVr1mjBggVau3atXnjhBR06dEjTpk1Ta2trT8wtt9yiP/zhD3rqqae0Zs0a7d69W5dddlkSe23nMk5Jmj9/fq/9effddyepxydmyJAhuuuuu7RhwwatX79eF154oS655BK9/fbbktJjX0rHH6fU9/cl+iCvDzj33HO9BQsW9Pzc3d3tlZSUeNXV1UnsVXwtWbLEGzduXLK74RtJ3sqVK3t+jkajXnFxsffTn/6057mmpiYvKyvL+4//+I8k9DA+Pj1Oz/O8uXPnepdccklS+uOXvXv3epK8NWvWeJ7393130kkneU899VRPzLvvvutJ8mpra5PVzZh9epye53lf/vKXvW9961vJ65RPBg4c6P3iF79I23152OFxel767kuktpTPeHV2dmrDhg2qrKzseS4QCKiyslK1tbVJ7Fn8/fWvf1VJSYlGjBihK6+8Ujt37kx2l3yzfft2NTQ09Nqv4XBYkyZNSrv9Kkk1NTUaNGiQzjjjDN1444364IMPkt2lmDQ3N0uS8vPzJUkbNmzQoUOHeu3PsrIyDR06tE/vz0+P87DHH39cBQUFGjNmjBYtWqS2trZkdC8uuru79cQTT6i1tVUVFRVpuy8/Pc7D0mlfom/ITHYHjmf//v3q7u5WUVFRr+eLioq0efPmJPUq/iZNmqTly5frjDPO0J49e3T77bfrS1/6kjZt2qQBAwYku3tx19DQIElH3K+Hf5cuZsyYocsuu0zDhw/Xtm3b9P3vf18zZ85UbW2t+vXrl+zumUWjUd1888364he/qDFjxkj6+/4MBoPKy8vrFduX9+eRxilJ3/jGN3TaaaeppKREGzdu1Pe+9z1t2bJFTz/9dBJ7a/fWW2+poqJC7e3tys3N1cqVK1VeXq66urq02pdHG6eUPvsSfUvKT7w+L2bOnNnz77Fjx2rSpEk67bTT9Jvf/EbXXXddEnuGWF1++eU9/z7rrLM0duxYjRw5UjU1NZo6dWoSe3ZiFixYoE2bNvX5exCP52jjvP7663v+fdZZZ2nw4MGaOnWqtm3bppEjRya6myfsjDPOUF1dnZqbm/Xb3/5Wc+fO1Zo1a5Ldrbg72jjLy8vTZl+ib0n5S40FBQXq16/fZ75R09jYqOLi4iT1yn95eXkaPXq0tm7dmuyu+OLwvvu87VdJGjFihAoKCvrkvq2qqtKzzz6rl19+WUOGDOl5vri4WJ2dnWpqauoV31f359HGeSSTJk2SpD63P4PBoEaNGqUJEyaourpa48aN0/333592+/Jo4zySvrov0bek/MQrGAxqwoQJWr16dc9z0WhUq1ev7nWdPt20tLRo27ZtGjx4cLK74ovhw4eruLi4136NRCJ6/fXX03q/StKuXbv0wQcf9Kl963meqqqqtHLlSr300ksaPnx4r99PmDBBJ510Uq/9uWXLFu3cubNP7c/jjfNI6urqJKlP7c8jiUaj6ujoSJt9eTSHx3kk6bIvkeKSfXe/iyeeeMLLysryli9f7r3zzjve9ddf7+Xl5XkNDQ3J7lrcfPvb3/Zqamq87du3e3/+85+9yspKr6CgwNu7d2+yu3bCDh486L355pvem2++6Uny7r33Xu/NN9/0/vu//9vzPM+76667vLy8PO93v/udt3HjRu+SSy7xhg8f7v3tb39Lcs9tjjXOgwcPet/5zne82tpab/v27d6LL77onX322d7pp5/utbe3J7vrzm688UYvHA57NTU13p49e3oebW1tPTE33HCDN3ToUO+ll17y1q9f71VUVHgVFRVJ7LXd8ca5detW74477vDWr1/vbd++3fvd737njRgxwps8eXKSe25z6623emvWrPG2b9/ubdy40bv11lu9jIwM709/+pPneemxLz3v2ONMl32JvqdPTLw8z/MefPBBb+jQoV4wGPTOPfdcb+3atcnuUlzNmTPHGzx4sBcMBr1TTz3VmzNnjrd169ZkdysmL7/8sifpM4+5c+d6nvf3JSV++MMfekVFRV5WVpY3depUb8uWLcnt9Ak41jjb2tq8adOmeYWFhd5JJ53knXbaad78+fP73H8ajjQ+Sd5jjz3WE/O3v/3N++Y3v+kNHDjQy8nJ8S699FJvz549yev0CTjeOHfu3OlNnjzZy8/P97KysrxRo0Z53/3ud73m5ubkdtzo2muv9U477TQvGAx6hYWF3tSpU3smXZ6XHvvS8449znTZl+h7MjzP8xKXXwMAAPj8Svl7vAAAANIFEy8AAIAEYeIFAACQIEy8AAAAEoSJFwAAQIIw8QIAAEgQJl4AAAAJwsQLAAAgQZh4AQAAJAgTLwAAgARh4gUAAJAg/z9NuzESSd9yXAAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "imshow_zero_center(h_mat)" ] }, { "cell_type": "markdown", "metadata": { "id": "13fBswmtQes4" }, "source": [ "The Newton's method update step is shown below:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:19.186423Z", "iopub.status.busy": "2024-08-15T02:32:19.186142Z", "iopub.status.idle": "2024-08-15T02:32:19.192987Z", "shell.execute_reply": "2024-08-15T02:32:19.192387Z" }, "id": "3DdnbynBdSor" }, "outputs": [], "source": [ "eps = 1e-3\n", "eye_eps = tf.eye(h_mat.shape[0])*eps" ] }, { "cell_type": "markdown", "metadata": { "id": "-zPdtyoWeUeV" }, "source": [ "Note: [Don't actually invert the matrix](https://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/)." ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:19.196637Z", "iopub.status.busy": "2024-08-15T02:32:19.195958Z", "iopub.status.idle": "2024-08-15T02:32:19.248580Z", "shell.execute_reply": "2024-08-15T02:32:19.247954Z" }, "id": "k1LYftgmswOO" }, "outputs": [], "source": [ "# X(k+1) = X(k) - (∇²f(X(k)))^-1 @ ∇f(X(k))\n", "# h_mat = ∇²f(X(k))\n", "# g_vec = ∇f(X(k))\n", "update = tf.linalg.solve(h_mat + eye_eps, g_vec)\n", "\n", "# Reshape the update and apply it to the variable.\n", "_ = layer1.kernel.assign_sub(tf.reshape(update, layer1.kernel.shape))" ] }, { "cell_type": "markdown", "metadata": { "id": "pF6qjlHKWxF4" }, "source": [ "While this is relatively simple for a single `tf.Variable`, applying this to a non-trivial model would require careful concatenation and slicing to produce a full Hessian across multiple variables." ] }, { "cell_type": "markdown", "metadata": { "id": "PQWM0uN-GO5t" }, "source": [ "### Batch Jacobian" ] }, { "cell_type": "markdown", "metadata": { "id": "hKtB3rY6EySJ" }, "source": [ "In some cases, you want to take the Jacobian of each of a stack of targets with respect to a stack of sources, where the Jacobians for each target-source pair are independent.\n", "\n", "For example, here the input `x` is shaped `(batch, ins)` and the output `y` is shaped `(batch, outs)`:\n" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:19.252063Z", "iopub.status.busy": "2024-08-15T02:32:19.251837Z", "iopub.status.idle": "2024-08-15T02:32:19.333771Z", "shell.execute_reply": "2024-08-15T02:32:19.333103Z" }, "id": "tQMndhIUHMes" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([7, 6])" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = tf.random.normal([7, 5])\n", "\n", "layer1 = tf.keras.layers.Dense(8, activation=tf.nn.elu)\n", "layer2 = tf.keras.layers.Dense(6, activation=tf.nn.elu)\n", "\n", "with tf.GradientTape(persistent=True, watch_accessed_variables=False) as tape:\n", " tape.watch(x)\n", " y = layer1(x)\n", " y = layer2(y)\n", "\n", "y.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "Ff2spRHEJXBU" }, "source": [ "The full Jacobian of `y` with respect to `x` has a shape of `(batch, ins, batch, outs)`, even if you only want `(batch, ins, outs)`:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:19.337404Z", "iopub.status.busy": "2024-08-15T02:32:19.337109Z", "iopub.status.idle": "2024-08-15T02:32:19.453376Z", "shell.execute_reply": "2024-08-15T02:32:19.452768Z" }, "id": "1zSl2A5-HhMH" }, "outputs": [ { "data": { "text/plain": [ "TensorShape([7, 6, 7, 5])" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "j = tape.jacobian(y, x)\n", "j.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "UibJijPLJrpQ" }, "source": [ "If the gradients of each item in the stack are independent, then every `(batch, batch)` slice of this tensor is a diagonal matrix:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:19.456497Z", "iopub.status.busy": "2024-08-15T02:32:19.456243Z", "iopub.status.idle": "2024-08-15T02:32:19.678736Z", "shell.execute_reply": "2024-08-15T02:32:19.678068Z" }, "id": "ZFl9uj3ueVSH" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAIQCAYAAABDpCBuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA7sUlEQVR4nO3deXxU9b3/8fckIROEJIBkYQmERYFIWUwgorIoKSCIYlGWm15D5IdtJRSNXURvWao1WFzQFkG8F7xWEKsVil4BKbJUQZZQKqAiKJawJCECGQgSIHN+f6SMDklOzsCZmWR4PR+P83iYkzPf8zmDD/Lh/f3mOw7DMAwBAADgsoUFuwAAAIBQQWMFAABgExorAAAAm9BYAQAA2ITGCgAAwCY0VgAAADahsQIAALAJjRUAAIBNaKwAAABsQmOFK5Lb7VbXrl31u9/9znNu+vTpcjgcKikpCWJl31m3bp0cDofeeust28b8+uuv5XA49PTTT9s25uVKTk7W7bffXut1K1euVOPGjXX06FHb7j1gwAANGDDA8/WF9+eVV16x7R4Ariw0Vqi3XnzxRTkcDqWnp/v82tdff10FBQXKycnxQ2XfefHFF0P+h/TGjRs1ffp0nThxwq/3GTJkiDp27Ki8vDy/3gcALgeNFeqtRYsWKTk5WVu2bNG+fft8eu2sWbM0ZswYxcbG+qm6SldKYzVjxgy/N1aS9JOf/EQvvfSSTp486Zfx27Ztq2+//Vb/+Z//6ZfxAYQ+GivUS/v379fGjRv17LPPKi4uTosWLbL82n/84x/65z//qVGjRvmxQvjDyJEjVV5erjfffNMv4zscDkVFRSk8PNwv4wMIfTRWqJcWLVqkpk2batiwYbr77rt9aqyWLVumyMhI9evXr9rvl5SUaNSoUYqJidHVV1+tyZMn68yZM17XLFy4ULfeeqvi4+PldDqVkpKiuXPnel2TnJys3bt3a/369XI4HHI4HF7reU6cOKGHHnpIycnJcjqdat26te69994qa7zcbrd+97vfqXXr1oqKitLAgQN9Tuiq89xzz6lt27Zq2LCh+vfvr127dnl9/5NPPtG4cePUvn17RUVFKTExUffdd5+++eYbzzXTp0/XL3/5S0lSu3btPM/59ddfe6557bXX1Lt3b1111VVq2rSp+vXrp/fff79KPR9++KF69+6tqKgotW/fXq+++mqVa+Lj49WtWzf99a9/rfX5CgsLlZ2drdatW8vpdKpFixa68847vWq7WE1rrD7//HONGjVKcXFxatiwoTp16qTHHnvM65pDhw7pvvvuU0JCgpxOp6677jotWLCg1joBhJaIYBcAXIpFixbpRz/6kSIjIzV27FjNnTtXW7duVa9evWp97caNG9W1a1c1aNCg2u+PGjVKycnJysvL08cff6wXXnhBx48f9/pBP3fuXF133XW64447FBERoXfeeUcPPPCA3G63Jk6cKEmaPXu2Jk2apMaNG3t+CCckJEiSTp06pb59++qzzz7Tfffdp+uvv14lJSVavny5Dh48qObNm3vuNXPmTIWFhekXv/iFSktL9fvf/16ZmZnavHnzJb9/r776qk6ePKmJEyfqzJkzev7553Xrrbdq586dnhpXr16tr776StnZ2UpMTNTu3bs1f/587d69Wx9//LEcDod+9KMf6YsvvtDrr7+u5557zlN3XFycJGnGjBmaPn26brzxRv32t79VZGSkNm/erA8++ECDBg3y1LNv3z7dfffdGj9+vLKysrRgwQKNGzdOqampuu6667xqT01N1bJly2p9xpEjR2r37t2aNGmSkpOTVVxcrNWrV+vAgQNKTk62/F598skn6tu3rxo0aKD7779fycnJ+vLLL/XOO+94fvmhqKhIN9xwgxwOh3JychQXF6cVK1Zo/PjxcrlcevDBBy3fD0A9ZwD1zLZt2wxJxurVqw3DMAy32220bt3amDx5sqXXt27d2hg5cmSV89OmTTMkGXfccYfX+QceeMCQZPzzn//0nDt9+nSV1w8ePNho376917nrrrvO6N+/f5Vrp06dakgy3n777Srfc7vdhmEYxtq1aw1JRpcuXYzy8nLP959//nlDkrFz507zB63G/v37DUlGw4YNjYMHD3rOb9682ZBkPPTQQ6bP+PrrrxuSjA0bNnjOzZo1y5Bk7N+/3+vavXv3GmFhYcZdd91lVFRUVPuMhmEYbdu2rTJmcXGx4XQ6jYcffrhKDU8++aQhySgqKqrxOY8fP25IMmbNmlXjNYZhGP379/f687nw/ixcuNBzrl+/fkZ0dLTxr3/9q8ZnGD9+vNGiRQujpKTE65oxY8YYsbGx1b6XAEITU4GodxYtWqSEhATdcsstkirXxYwePVpLlixRRUVFra//5ptv1LRp0xq/fyFxumDSpEmSpPfee89zrmHDhp7/Li0tVUlJifr376+vvvpKpaWltdbwl7/8Rd27d9ddd91V5XsOh8Pr6+zsbEVGRnq+7tu3ryTpq6++qvU+NRkxYoRatWrl+bp3795KT0+v8RnPnDmjkpIS3XDDDZKk7du313qPZcuWye12a+rUqQoL8/6r5uJnTElJ8TyXVJl4derUqdpnvPBnZ7YtRsOGDRUZGal169bp+PHjtdZak6NHj2rDhg2677771KZNm2qfwTAM/eUvf9Hw4cNlGIZKSko8x+DBg1VaWmrp/QIQGmisUK9UVFRoyZIluuWWW7R//37t27dP+/btU3p6uoqKirRmzRpL4xiGUeP3rrnmGq+vO3TooLCwMK+1OR999JEyMjLUqFEjNWnSRHFxcXr00UclyVJj9eWXX6pr166War34B/qFxuJyGoaLn1GSrr32Wq9nPHbsmCZPnqyEhAQ1bNhQcXFxateunSTrzxgWFqaUlJRar734GaXK56zuGS/82V3cnH2f0+nUU089pRUrVighIUH9+vXT73//exUWFtZay/ddaOzM/qyOHj2qEydOaP78+YqLi/M6srOzJUnFxcU+3RdA/cUaK9QrH3zwgY4cOaIlS5ZoyZIlVb6/aNEir7U71bn66qt9akou/gH+5ZdfauDAgercubOeffZZJSUlKTIyUu+9956ee+45ud1uy2NbUdNvqJk1h3YYNWqUNm7cqF/+8pfq0aOHGjduLLfbrSFDhgT1GS/82X1/HVp1HnzwQQ0fPlzLli3TqlWr9Jvf/EZ5eXn64IMP1LNnz8sv+t8uvBc//vGPlZWVVe013bp1s+1+AOo2GivUK4sWLVJ8fLzmzJlT5Xtvv/22li5dqnnz5nlNY12sc+fO2r9/f43f37t3ryeZkSoXVrvdbs+C53feeUfl5eVavny5V9Kydu3aKmPVlKp06NChym/hBdLevXurnPviiy88z3j8+HGtWbNGM2bM0NSpU01fZ/aMbrdbn376qXr06GFL3VLlVhvNmzf3LJA306FDBz388MN6+OGHtXfvXvXo0UPPPPOMXnvtNUv3at++vSSZ/lnFxcUpOjpaFRUVysjIsPYQAEIWU4GoN7799lu9/fbbuv3223X33XdXOXJycnTy5EktX77cdJw+ffpo165dKi8vr/b7Fzdtf/jDHyRJt912m6Tv0pXvpymlpaVauHBhlbEaNWpU7caZI0eO1D//+U8tXbq0yvf8nURJleufDh065Pl6y5Yt2rx5s+kzSpW/6XixRo0aSVKV5xwxYoTCwsL029/+tkrCdTnPmJ+frz59+phec/r06SpbZHTo0EHR0dE1/rlXJy4uTv369dOCBQt04MABr+9deIbw8HCNHDlSf/nLX6ptwOz8CB4AdR+JFeqN5cuX6+TJk7rjjjuq/f4NN9zg2Sx09OjRNY5z55136vHHH9f69eurnTbcv3+/7rjjDg0ZMkSbNm3Sa6+9pv/4j/9Q9+7dJUmDBg1SZGSkhg8frp/85Cc6deqUXn75ZcXHx+vIkSNeY6Wmpmru3Ll64okn1LFjR8XHx+vWW2/VL3/5S7311lu65557dN999yk1NVXHjh3T8uXLNW/ePM+9rFq3bp1uueUWTZs2TdOnT6/1+o4dO+rmm2/Wz372M5WXl2v27Nm6+uqr9atf/UqSFBMT41mXdO7cObVq1Urvv/9+tUlfamqqJOmxxx7TmDFj1KBBAw0fPlwdO3bUY489pscff1x9+/bVj370IzmdTm3dulUtW7a8pI+mKS4u1ieffFLlFwwu9sUXX2jgwIEaNWqUUlJSFBERoaVLl6qoqEhjxozx6Z4vvPCCbr75Zl1//fW6//771a5dO3399df6v//7P+3YsUNS5ZYYa9euVXp6uiZMmKCUlBQdO3ZM27dv19/+9jcdO3bM52cFUE8F6bcRAZ8NHz7ciIqKMsrKymq8Zty4cUaDBg2q/Nr7xbp162aMHz/e69yF7RY+/fRT4+677zaio6ONpk2bGjk5Oca3337rde3y5cuNbt26GVFRUUZycrLx1FNPGQsWLKiy7UBhYaExbNgwIzo62pDk9av933zzjZGTk2O0atXKiIyMNFq3bm1kZWV5ar+w3cKbb77pde/qtgR45513DEnGvHnzTJ/7wmtnzZplPPPMM0ZSUpLhdDqNvn37em0nYRiGcfDgQeOuu+4ymjRpYsTGxhr33HOPcfjwYUOSMW3aNK9rH3/8caNVq1ZGWFhYlfdgwYIFRs+ePQ2n02k0bdrU6N+/v2erDMOo3G5h2LBhVWq9eCsEwzCMuXPnGldddZXhcrlMn7OkpMSYOHGi0blzZ6NRo0ZGbGyskZ6ebvz5z382vUd1761hGMauXbs870VUVJTRqVMn4ze/+Y3XNUVFRcbEiRONpKQko0GDBkZiYqIxcOBAY/78+aa1AggtDsMIwLwDUMf86U9/0sSJE3XgwAE1adIk2OVctl/96ld6/fXXtW/fPjmdzmCX4zc9e/bUgAED9NxzzwW7FACoFmuscEXKzMxUmzZtql0EXx+tXbtWv/nNb0K6qVq5cqX27t2rKVOmBLsUAKgRiRUAAIBNSKwAAABsQmMFAABgExorAAAAm9BYAQAA2CTgG4S63W4dPnxY0dHRph+iCgAAKnf5P3nypFq2bKmwsMDnIWfOnNHZs2f9MnZkZKSioqL8MnawBLyxOnz4sJKSkgJ9WwAA6rWCggK1bt06oPc8c+aM4ho21Ck/jZ+YmKj9+/eHVHMV8MYqOjr63/81MBi3r1NKjy8JdgkAgDrO5XIpqW3b7/38DJyzZ8/qlKSHJNm9S165pOcKC3X27Fkaq8vx3fRfhKQGgb59nRITExPsEgAA9UQwl880lGR36xOqi7xD9bkAAAAC7sqeiwMAALUKk/1JTKgmO6H6XAAAAAFHYgUAAEyRWFlHYwUAAEzRWFkXqs8FAAAQcCRWAADAFImVdaH6XAAAAAFHYgUAAEyRWFkXqs8FAAAQcCRWAADAFImVdaH6XAAAAAFHYgUAAEyRWFkXqs8FAAAQcCRWAADAlEP2JzEOm8erK2isAACAKYfsb4RCtbFiKhAAAMAmJFYAAMBU+L8Pu8cMRSRWAAAANiGxAgAApthuwbpQfS4AAICAI7ECAACmSKysC9XnAgAACDgSKwAAYIrEyjoaKwAAYIrGyrpQfS4AAICAu6TGas6cOUpOTlZUVJTS09O1ZcsWu+sCAAB1RJifjlDk83O98cYbys3N1bRp07R9+3Z1795dgwcPVnFxsT/qAwAAqDd8bqyeffZZTZgwQdnZ2UpJSdG8efN01VVXacGCBf6oDwAABFldSax8mTF7++23lZaWpiZNmqhRo0bq0aOH/vSnP13CXX3j03OdPXtW+fn5ysjI+G6AsDBlZGRo06ZN1b6mvLxcLpfL6wAAAPCFrzNmzZo102OPPaZNmzbpk08+UXZ2trKzs7Vq1Sq/1ulTY1VSUqKKigolJCR4nU9ISFBhYWG1r8nLy1NsbKznSEpKuvRqAQBAwDn8dPjC1xmzAQMG6K677lKXLl3UoUMHTZ48Wd26ddOHH37o45194/e1Y1OmTFFpaannKCgo8PctAQBACLmUGbPvMwxDa9as0Z49e9SvXz9/lurbPlbNmzdXeHi4ioqKvM4XFRUpMTGx2tc4nU45nc5LrxAAAARVmKRwP4wpqcoSoer6BrMZs88//7zGe5SWlqpVq1YqLy9XeHi4XnzxRf3whz+0pf6a+JRYRUZGKjU1VWvWrPGcc7vdWrNmjfr06WN7cQAAIPj8uXg9KSnJa8lQXl6ebXVHR0drx44d2rp1q373u98pNzdX69ats2386vi883pubq6ysrKUlpam3r17a/bs2SorK1N2drY/6gMAACGsoKBAMTExnq+rm+W6lBkzqXK6sGPHjpKkHj166LPPPlNeXp4GDBhgT/HV8LmxGj16tI4ePaqpU6eqsLBQPXr00MqVK6vEcwAAIDT48yNtYmJivBqr6nx/xmzEiBGSvpsxy8nJsXxPt9ut8vLyS6zYmkv6rMCcnByfHgQAAOBy1DZjdu+996pVq1aeqcS8vDylpaWpQ4cOKi8v13vvvac//elPmjt3rl/r5EOYAQCAqbrwIcy1zZgdOHBAYWHfjVpWVqYHHnhABw8eVMOGDdW5c2e99tprGj16tI1PUZXDMAzDr3e4iMvlUmxsrKTBkhoE8tZ1jlHx12CXAACo41wul2KbNlVpaWmtU2Z+uXdsrF6VdJXNY5+WdK8UlOfyJxIrAABgqi4kVvVFqD4XAABAwJFYAQAAUyRW1tFYAQAAUzRW1oXqcwEAAAQciRUAADDl+Pdh95ihiMQKAADAJiRWAADAVPi/D7vHDEUkVgAAADYhsQIAAKYcsj+JYY0VAAAATJFYAQAAU+xjZR2NFQAAMEVjZV2oPhcAAEDAkVgBAABTJFbWhepzAQAABByJFQAAMEViZV2oPhcAAEDAkVgBAABTJFbWhepzAQAABByJFQAAMOWQ/R9BE6ofaUNjBQAATIX/+7B7zFDEVCAAAIBNSKwAAIApFq9bF7TGqvT4EsXExATr9nXC/P8O1f+tfHP//3MHuwQAAGxBYgUAAEw5ZH/CFKqL14lMAAAAbEJiBQAATLHGyrpQfS4AAICAI7ECAACmSKyso7ECAACmaKysC9XnAgAACDgSKwAAYIrEyrpQfS4AAICAI7ECAACmHLJ/Q082CAUAAIApEisAAGAq/N+H3WOGIhIrAAAAm5BYAQAAU/xWoHU0VgAAwBSNlXWh+lwAAAABR2IFAABMOWR/EsN2CwAAADBFYgUAAEyxxsq6UH0uAACAgCOxAgAApkisrAvV5wIAAAg4EisAAGCKxMq6UH0uAACAgCOxAgAAphyyf9+pUN3HisYKAACYCv/3YfeYoYipQAAAAJuQWAEAAFMsXrfO5+fasGGDhg8frpYtW8rhcGjZsmV+KAsAAKD+8bmxKisrU/fu3TVnzhx/1AMAAOqYMD8dvpozZ46Sk5MVFRWl9PR0bdmypcZrX375ZfXt21dNmzZV06ZNlZGRYXq9XXx+rttuu01PPPGE7rrrLn/UAwAAUMUbb7yh3NxcTZs2Tdu3b1f37t01ePBgFRcXV3v9unXrNHbsWK1du1abNm1SUlKSBg0apEOHDvm1Tr9PcZaXl8vlcnkdAACg/qgLidWzzz6rCRMmKDs7WykpKZo3b56uuuoqLViwoNrrFy1apAceeEA9evRQ586d9d///d9yu91as2aNj3f2jd8bq7y8PMXGxnqOpKQkf98SAACEkLNnzyo/P18ZGRmec2FhYcrIyNCmTZssjXH69GmdO3dOzZo181eZlXX5dXRJU6ZMUWlpqecoKCjw9y0BAICNHLI/rbqwQejFs1rl5eVV7l9SUqKKigolJCR4nU9ISFBhYaGlZ/j1r3+tli1bejVn/uD3xsrpdComJsbrAAAA9Yc/pwKTkpK8Zrby8vJsr3/mzJlasmSJli5dqqioKNvH/z72sQIAAEFTUFDgFbo4nc4q1zRv3lzh4eEqKiryOl9UVKTExETT8Z9++mnNnDlTf/vb39StWzd7ijbhc2J16tQp7dixQzt27JAk7d+/Xzt27NCBAwfsrg0AANQB/kysLp7Vqq6xioyMVGpqqtfC8wsL0fv06VNj3b///e/1+OOPa+XKlUpLS7uMd8A6nxOrbdu26ZZbbvF8nZubK0nKysrSK6+8YlthAAAAF+Tm5iorK0tpaWnq3bu3Zs+erbKyMmVnZ0uS7r33XrVq1cozlfjUU09p6tSpWrx4sZKTkz1rsRo3bqzGjRv7rU6fG6sBAwbIMAx/1AIAAOqguvCRNqNHj9bRo0c1depUFRYWqkePHlq5cqVnQfuBAwcUFvbdqHPnztXZs2d19913e40zbdo0TZ8+/TKrrxlrrAAAQL2Qk5OjnJycar+3bt06r6+//vpr/xdUDRorAABgqi4kVvVFqD4XAABAwJFYAQAAUyRW1tFYAQAAUzRW1oXqcwEAAAQciRUAADBFYmVdqD4XAABAwJFYAQAAUyRW1oXqcwEAAAQciRUAADDlkORwOOwdM0Q/Ho/ECgAAwCYkVgAAwFxEhGRzYiXDkM6ft3fMOoDGCgAAmKOxsoypQAAAAJuQWAEAAHP+SqxCEIkVAACATUisAACAORIry0isAAAAbEJiBQAAzIWHS2E2ZzFut73j1REkVgAAADYhsQIAAOYiIkisLKKxAgAA5misLGMqEAAAwCYkVgAAwByJlWUkVgAAADYhsQqi+/9faHbrvnKEVwS7hDrBqAgPdgkAUL3w8MrDThWh+Xc/iRUAAIBNSKwAAIC5iAj7Eyu7PyKnjiCxAgAAsAmJFQAAMEdiZRmNFQAAMEdjZRlTgQAAADYhsQIAAOZIrCwjsQIAALAJiRUAADAXHl6ZWqFWJFYAAAA2of0EAADmIiJIrCwisQIAALAJ7ScAADBHYmUZ7xIAADBHY2UZU4EAAAA2of0EAADm/LHdgmHYO14dQWIFAABgExIrAABgzh9rrEisAAAAYIbECgAAmCOxsozECgAAwCYkVgAAwByJlWU0VgAAwByNlWVMBQIAANiExAoAAJjzxwahbre949URJFYAAAA2IbECAADm/LHGisQKAAAAZnxqrPLy8tSrVy9FR0crPj5eI0aM0J49e/xVGwAAqAsuJFZ2HyHIp8Zq/fr1mjhxoj7++GOtXr1a586d06BBg1RWVuav+gAAACRJc+bMUXJysqKiopSenq4tW7bUeO3u3bs1cuRIJScny+FwaPbs2QGp0ad2ceXKlV5fv/LKK4qPj1d+fr769etna2EAAKCOqANrrN544w3l5uZq3rx5Sk9P1+zZszV48GDt2bNH8fHxVa4/ffq02rdvr3vuuUcPPfSQXVXX6rLWWJWWlkqSmjVrVuM15eXlcrlcXgcAAIAvnn32WU2YMEHZ2dlKSUnRvHnzdNVVV2nBggXVXt+rVy/NmjVLY8aMkdPpDFidl9xYud1uPfjgg7rpppvUtWvXGq/Ly8tTbGys50hKSrrUWwIAgGDw4xqri8OX8vLyKrc/e/as8vPzlZGR4TkXFhamjIwMbdq0KWBvgxWX3FhNnDhRu3bt0pIlS0yvmzJlikpLSz1HQUHBpd4SAAAEw4UNQu08wsMlSUlJSV4BTF5eXpXbl5SUqKKiQgkJCV7nExISVFhYGJC3wKpLmjDNycnRu+++qw0bNqh169am1zqdzoBGcAAAoP4oKChQTEyM5+v63jP41FgZhqFJkyZp6dKlWrdundq1a+evugAAQF3hj8XrFRWSpJiYGK/GqjrNmzdXeHi4ioqKvM4XFRUpMTHR3rouk09TgRMnTtRrr72mxYsXKzo6WoWFhSosLNS3337rr/oAAMAVLjIyUqmpqVqzZo3nnNvt1po1a9SnT58gVlaVT+3n3LlzJUkDBgzwOr9w4UKNGzfOrpoAAEBd4sfEyqrc3FxlZWUpLS1NvXv31uzZs1VWVqbs7GxJ0r333qtWrVp51midPXtWn376qee/Dx06pB07dqhx48bq2LGjvc/yPT5PBQIAAATa6NGjdfToUU2dOlWFhYXq0aOHVq5c6VnQfuDAAYWFfTcRd/jwYfXs2dPz9dNPP62nn35a/fv317p16/xWZ2juJw8AAOxTBxIrqfKX53Jycqr93sXNUnJyclACIT6EGQAAwCYkVgAAwNyFfazsdP68vePVETRWAADAnD+mAu0er45gKhAAAMAmodkuAgAA+5BYWUZiBQAAYJPQbBcBAIB9SKwsI7ECAACwSWi2iwAAwD4kVpaRWAEAANgkNNtFAABgH39sEBoebu94dQSNFQAAMMdUoGVMBQIAANgkNNtFAABgHxIry0isAAAAbBKa7SIAALAPiZVlJFYAAAA2Cc12EQAA2IftFiwjsQIAALAJiRUAADDHGivLQvOpAACAfWisLGMqEAAAwCah2S4CAAD7kFhZRmIFAABgk9BsFwEAgH1IrCwLzadCvWJUhOZeJr5yhJ8Kdgl1glHRONglAMAlo7ECAADm2CDUMtZYAQAA2ITECgAAmGONlWWh+VQAAMA+NFaWMRUIAABgk9BsFwEAgH1IrCwjsQIAALBJaLaLAADAPmy3YBmJFQAAgE1IrAAAgDnWWFlGYgUAAGCT0GwXAQCAfUisLAvNpwIAAPahsbKMqUAAAACbhGa7CAAA7ENiZRmJFQAAgE1Cs10EAAD2YYNQy0isAAAAbEJiBQAAzLHGyjISKwAAAJuEZrsIAADsQ2JlWWg+FQAAsA+NlWVMBQIAANgkNNtFAABgH7ZbsIzECgAAwCYkVgAAwBxrrCwjsQIAALBJaLaLAADAPiRWlvmUWM2dO1fdunVTTEyMYmJi1KdPH61YscJftQEAAHjMmTNHycnJioqKUnp6urZs2WJ6/ZtvvqnOnTsrKipKP/jBD/Tee+/5vUafGqvWrVtr5syZys/P17Zt23Trrbfqzjvv1O7du/1VHwAACLYLiZXdhw/eeOMN5ebmatq0adq+fbu6d++uwYMHq7i4uNrrN27cqLFjx2r8+PH6xz/+oREjRmjEiBHatWuXHe9IjRyGYRiXM0CzZs00a9YsjR8/3tL1LpdLsbGxKj1+XDExMZdzayCkOMJPBbuEOsGoaBzsEoA6xeVyKbZpU5WWlgb856Y/f2b7+lzp6enq1auX/vjHP0qS3G63kpKSNGnSJD3yyCNVrh89erTKysr07rvves7dcMMN6tGjh+bNm2ffg1zkkhevV1RUaMmSJSorK1OfPn1qvK68vFwul8vrAAAAkFSlRygvL69yzdmzZ5Wfn6+MjAzPubCwMGVkZGjTpk3Vjrtp0yav6yVp8ODBNV5vF58bq507d6px48ZyOp366U9/qqVLlyolJaXG6/Py8hQbG+s5kpKSLqtgAAAQWG6F+eWQpKSkJK8+IS8vr8r9S0pKVFFRoYSEBK/zCQkJKiwsrLbmwsJCn663i89L8jt16qQdO3aotLRUb731lrKysrR+/foam6spU6YoNzfX87XL5aK5AgAAkqSCggKvqUCn0xnEai6fz41VZGSkOnbsKElKTU3V1q1b9fzzz+ull16q9nqn01nv3yQAAK5k589XHnaPKcmz04CZ5s2bKzw8XEVFRV7ni4qKlJiYWO1rEhMTfbreLpe9Qajb7a52PhQAAMAOkZGRSk1N1Zo1azzn3G631qxZU+M67z59+nhdL0mrV682XRduB58SqylTpui2225TmzZtdPLkSS1evFjr1q3TqlWr/FUfAAAIMn8mVlbl5uYqKytLaWlp6t27t2bPnq2ysjJlZ2dLku699161atXKs0Zr8uTJ6t+/v5555hkNGzZMS5Ys0bZt2zR//nx7H+QiPjVWxcXFuvfee3XkyBHFxsaqW7duWrVqlX74wx/6qz4AAACNHj1aR48e1dSpU1VYWKgePXpo5cqVngXqBw4cUFjYdxNxN954oxYvXqz/+q//0qOPPqprrrlGy5YtU9euXf1a52XvY+Ur9rECqsc+VpXYxwrwVhf2sSoosP/elb/MFhuU5/Kn0PygHgAAYJu6MBVYX1z24nUAAABUIrECAACmKirsT5gqKuwdr64gsQIAALAJiRUAADDFGivrSKwAAABsQmIFAABMkVhZR2IFAABgExIrAABgisTKOhIrAAAAm5BYAQAAU+xjZR2NFQAAMMVUoHVMBQIAANiExAoAAJgisbKOxAoAAMAmJFYAAMAUiZV1JFYAAAA2IbECAACmSKysI7ECAACwCYkVAAAwxQah1tFYAQAAU0wFWsdUIAAAgE1IrAAAgCkSK+tIrAAAAGxCYgUAAEyRWFlHYgUAAGATEiugjjAqGge7hDrh+jT+vSdJ27e5g10C4MF2C9bxNxgAAIBNSKwAAIAp1lhZR2MFAABM0VhZx1QgAACATUisAACAKRIr60isAAAAbEJiBQAATJFYWUdiBQAAYBMSKwAAYIoNQq0jsQIAALAJiRUAADDFGivraKwAAIApGivrmAoEAACwCYkVAAAwRWJlHYkVAACATUisAACAKbZbsI7ECgAAwCYkVgAAwBRrrKwjsQIAALAJiRUAADBFYmUdjRUAADBFY2UdU4EAAAA2IbECAACmSKysI7ECAACwCYkVAAAwxQah1pFYAQAA2ITECgAAmDp/XgoPt3/MUERiBQAAQsqxY8eUmZmpmJgYNWnSROPHj9epU6dMXzN//nwNGDBAMTExcjgcOnHixCXd+7Iaq5kzZ8rhcOjBBx+8nGEAAEAdduG3Au0+/CUzM1O7d+/W6tWr9e6772rDhg26//77TV9z+vRpDRkyRI8++uhl3fuSpwK3bt2ql156Sd26dbusAgAAQN1Wn6YCP/vsM61cuVJbt25VWlqaJOkPf/iDhg4dqqefflotW7as9nUXQqJ169Zd1v0vKbE6deqUMjMz9fLLL6tp06aXVQAAAIBdNm3apCZNmniaKknKyMhQWFiYNm/e7Pf7X1JjNXHiRA0bNkwZGRm1XlteXi6Xy+V1AACA+uPCdgt2Hhe2W7i4RygvL7+sWgsLCxUfH+91LiIiQs2aNVNhYeFljW2Fz43VkiVLtH37duXl5Vm6Pi8vT7GxsZ4jKSnJ5yIBAEBoSkpK8uoTauovHnnkETkcDtPj888/D3D1Vfm0xqqgoECTJ0/W6tWrFRUVZek1U6ZMUW5urudrl8tFcwUAQD1y/rwUZvM+AhfWWBUUFCgmJsZz3ul0Vnv9ww8/rHHjxpmO2b59eyUmJqq4uPiie53XsWPHlJiYeFk1W+FTY5Wfn6/i4mJdf/31nnMVFRXasGGD/vjHP6q8vFzhF61uczqdNb5JAADgyhYTE+PVWNUkLi5OcXFxtV7Xp08fnThxQvn5+UpNTZUkffDBB3K73UpPT7/semvjU2M1cOBA7dy50+tcdna2OnfurF//+tdVmioAAFD/+TOxsluXLl00ZMgQTZgwQfPmzdO5c+eUk5OjMWPGeH4j8NChQxo4cKBeffVV9e7dW1Ll2qzCwkLt27dPkrRz505FR0erTZs2atasmeX7+9RYRUdHq2vXrl7nGjVqpKuvvrrKeQAAgGBYtGiRcnJyNHDgQIWFhWnkyJF64YUXPN8/d+6c9uzZo9OnT3vOzZs3TzNmzPB83a9fP0nSwoULa52C/D4+0gYAAJiqT4mVJDVr1kyLFy+u8fvJyckyDMPr3PTp0zV9+vTLvvdlN1aXu5EWAACo2+pbYxVMfFYgAACATZgKBAAApi5sEGr3mKGIxAoAAMAmJFYAAMDU+fOSw2H/mKGIxAoAAMAmJFYAAMAUiZV1JFYAAAA2IbECAACmSKyso7ECAACmaKysYyoQAADAJiRWAADAVEWF/YkVG4QCAADAFIkVAAAw5Y/1UKyxAgAAgCkSKwAAYIrEyjoSKwAAAJuQWAEAAFMkVtaRWAEAANiExAoAAJjyx55TobqPFY0VAAAwdf68ZBj2jhmqjRVTgQAAADYhsQIAAKZIrKwjsQIAALAJiRUAADBFYmUdiRUAAIBNSKwAAIApEivraKwA1Cnbt7mDXUKd4AgvCHYJdYJRkRTsEgCf0FgBAABTFRX2J1buEP03FI0VAAAwdf68FGbzquxQbaxYvA4AAGATEisAAGCKxMo6EisAAACbkFgBAABTJFbWkVgBAADYhMQKAACYqqiwP2Gye/uGuoLECgAAwCYkVgAAwNT585LDYe+YoZpY0VgBAABTNFbWMRUIAABgExIrAABgisTKOhIrAAAAm5BYAQAAUyRW1pFYAQAA2ITECgAA1MItw7D7M2hC8zNtSKwAAABsQmIFAABqUfHvw+4xQw+NFQAAqAWNlVVMBQIAANiExAoAANSCxMoqEisAAACbkFgBAIBauGX/9ghstwAAAAATJFYAAKAWrLGyisQKAACElGPHjikzM1MxMTFq0qSJxo8fr1OnTpleP2nSJHXq1EkNGzZUmzZt9POf/1ylpaU+35vECgAA1MIt+xMm/62xyszM1JEjR7R69WqdO3dO2dnZuv/++7V48eJqrz98+LAOHz6sp59+WikpKfrXv/6ln/70pzp8+LDeeustn+7tMAzrny89ffp0zZgxw+tcp06d9Pnnn1u+ocvlUmxsrEqPH1dMTIz1SgHgCuIILwh2CXWCUZEU7BKCzuVyKbZpU5WWlgb85+aFn9nSPknRNo9+UlJH25/rs88+U0pKirZu3aq0tDRJ0sqVKzV06FAdPHhQLVu2tDTOm2++qR//+McqKytTRIT1HMrnqcDrrrtOR44c8Rwffvihr0MAAAD4xaZNm9SkSRNPUyVJGRkZCgsL0+bNmy2Pc6Hh86Wpki5hKjAiIkKJiYm+vgwAANRb/lu87nK5vM46nU45nc5LHrWwsFDx8fFe5yIiItSsWTMVFhZaGqOkpESPP/647r//fp/v73NitXfvXrVs2VLt27dXZmamDhw4YHp9eXm5XC6X1wEAACBJSUlJio2N9Rx5eXnVXvfII4/I4XCYHr4sTaqJy+XSsGHDlJKSounTp/v8ep8Sq/T0dL3yyivq1KmTjhw5ohkzZqhv377atWuXoqOrn3vNy8ursi4LAADUJ/7bILSgoMBrjVVNadXDDz+scePGmY7Yvn17JSYmqri42Ov8+fPndezYsVpn3E6ePKkhQ4YoOjpaS5cuVYMGDSw8hzefFq9f7MSJE2rbtq2effZZjR8/vtprysvLVV5e7vna5XIpKSmJxesAYILF65VYvF5XFq9/Jv8sXu/it8Xr27ZtU2pqqiTp/fff15AhQ0wXr7tcLg0ePFhOp1Pvvfeerrrqqku6/2XtY9WkSRNde+212rdvX43XOJ1OxcTEeB0AAKA+qfDTYb8uXbpoyJAhmjBhgrZs2aKPPvpIOTk5GjNmjKepOnTokDp37qwtW7ZIqmyqBg0apLKyMv3P//yPXC6XCgsLVVhYqIoK3+q8rMbq1KlT+vLLL9WiRYvLGQYAAMA2ixYtUufOnTVw4EANHTpUN998s+bPn+/5/rlz57Rnzx6dPn1akrR9+3Zt3rxZO3fuVMeOHdWiRQvPUVDgW3rs0xqrX/ziFxo+fLjatm2rw4cPa9q0aQoPD9fYsWN9uikAAKhP6tdH2jRr1qzGzUAlKTk5Wd9fCTVgwABdxsooLz41VgcPHtTYsWP1zTffKC4uTjfffLM+/vhjxcXF2VIMAACoi+pXYxVMPjVWS5Ys8VcdAAAA9R6fFQgAAGpRvz4rMJgua/E6AAAAvkNiBQAAauG/DUJDDYkVAACATUisAABALfitQKtIrAAAAGxCYgUAAGpBYmUVjRUAAKgFjZVVTAUCAADYhMQKAADUgsTKKhIrAAAAm5BYAQCAWhiyf0NPw+bx6gYSKwAAAJuQWAEAgFqwxsoqEisAAACbkFgBAIBakFhZRWMFAABqQWNlFVOBAAAANiGxAgAAtSCxsorECgAAwCYkVgAAoBZu2b9BqN3j1Q0kVgAAADYhsQIAALVgjZVVJFYAAAA2IbECAAC1cMv+hIk1VgAAADBBYgUAdZBRkRTsEuqEsAj+/W8YdeE9YI2VVTRWAACgFmy3YFVdaIMBAABCAokVAACoBVOBVpFYAQAA2ITECgAA1ILEyioSKwAAAJuQWAEAgFqQWFlFYgUAAGATEisAAFALPtLGKhorAABQCzYItYqpQAAAAJuQWAEAgFqweN0qEisAAACbkFgBAIBakFhZRWIFAABgExIrAABQCxIrq0isAAAAbEJiBQAAakFiZRWNFQAAqAUbhFrFVCAAAIBNSKwAAEAt+KxAq0isAAAAbEJiBQAAasHidatIrAAAAGxCYgUAAGpBYmUViRUAAIBNaKwAAEAtKvx0+MexY8eUmZmpmJgYNWnSROPHj9epU6dMX/OTn/xEHTp0UMOGDRUXF6c777xTn3/+uc/3prECAAC1qF+NVWZmpnbv3q3Vq1fr3Xff1YYNG3T//febviY1NVULFy7UZ599plWrVskwDA0aNEgVFb7V6TAMw/DlBYcOHdKvf/1rrVixQqdPn1bHjh21cOFCpaWlWXq9y+VSbGysSo8fV0xMjE/FAgCuLGER/PvfMFySYlVaWhrwn5sXfmZL/yUpyubRz0h6wvbn+uyzz5SSkqKtW7d6epOVK1dq6NChOnjwoFq2bGlpnE8++UTdu3fXvn371KFDB8v392nx+vHjx3XTTTfplltu0YoVKxQXF6e9e/eqadOmvgwDAADqFf99pI3L5fI663Q65XQ6L3nUTZs2qUmTJl6BT0ZGhsLCwrR582bdddddtY5RVlamhQsXql27dkpKSvLp/j79U+Cpp55SUlKSFi5cqN69e6tdu3YaNGiQT50cAADABUlJSYqNjfUceXl5lzVeYWGh4uPjvc5FRESoWbNmKiwsNH3tiy++qMaNG6tx48ZasWKFVq9ercjISJ/u71NjtXz5cqWlpemee+5RfHy8evbsqZdfftn0NeXl5XK5XF4HAACoTy58pI2dR2ViVVBQoNLSUs8xZcqUait45JFH5HA4TI9LWWz+fZmZmfrHP/6h9evX69prr9WoUaN05swZn8bwaSrwq6++0ty5c5Wbm6tHH31UW7du1c9//nNFRkYqKyur2tfk5eVpxowZPhUFAACuDDExMZbWWD388MMaN26c6TXt27dXYmKiiouLvc6fP39ex44dU2JiounrL6Rm11xzjW644QY1bdpUS5cu1dixY2ut7wKfGiu32620tDQ9+eSTkqSePXtq165dmjdvXo2N1ZQpU5Sbm+v52uVy+TxfCQAAgin4G4TGxcUpLi6u1uv69OmjEydOKD8/X6mpqZKkDz74QG63W+np6ZbvZxiGDMNQeXm5T3X6NBXYokULpaSkeJ3r0qWLDhw4UONrnE6npxu12pUCAABcii5dumjIkCGaMGGCtmzZoo8++kg5OTkaM2aM5zcCDx06pM6dO2vLli2SKmfk8vLylJ+frwMHDmjjxo2655571LBhQw0dOtSn+/vUWN10003as2eP17kvvvhCbdu29emmAACgPqlf+1gtWrRInTt31sCBAzV06FDdfPPNmj9/vuf7586d0549e3T69GlJUlRUlP7+979r6NCh6tixo0aPHq3o6Ght3LixykL42vg0FfjQQw/pxhtv1JNPPqlRo0Zpy5Ytmj9/vlexAAAg1PhvuwV/aNasmRYvXlzj95OTk/X9bTxbtmyp9957z5Z7+5RY9erVS0uXLtXrr7+url276vHHH9fs2bOVmZlpSzEAAAD1mU+JlSTdfvvtuv322/1RCwAAqJOCv3i9vuCzAgAAAGzic2IFAACuNCRWVpFYAQAA2ITECgAA1ILEyioSKwAAAJuQWAEAgFpc+BBmu8cMPTRWAACgFvVrg9BgYioQAADAJiRWAACgFixet4rECgAAwCYkVgAAoBYkVlaRWAEAANiExAoAANSCxMoqEisAAACbkFgBAIBakFhZRWMFAABqwc7rVjEVCAAAYBMSKwAAUAs+0sYqEisAAACbkFgBAIBaVMj+LCY0F6+TWAEAANiExAoAANSCxMoqEisAAACbkFgBAIBakFhZRWMFAABqwXYLVgW8sTIMQ5LkcrkCfWsAQD1jGKxYkSp/Xl74+Rkc5+vJmMEX8Mbq5MmTkqSktm0DfWsAAOqtkydPKjY2NqD3jIyMVGJiogoL/+aX8RMTExUZGemXsYPFYQS4BXa73Tp8+LCio6PlcDgCeWsPl8ulpKQkFRQUKCYmJig11AW8D5V4HyrxPlTifajE+1CpLrwPhmHo5MmTatmypcLCAp/gnTlzRmfPnvXL2JGRkYqKivLL2MES8MQqLCxMrVu3DvRtqxUTE3NF/4VxAe9DJd6HSrwPlXgfKvE+VAr2+xDopOr7oqKiQq758ScmrwEAAGxCYwUAAGCTK7KxcjqdmjZtmpxOZ7BLCSreh0q8D5V4HyrxPlTifajE+wBfBXzxOgAAQKi6IhMrAAAAf6CxAgAAsAmNFQAAgE1orAAAAGxyxTVWc+bMUXJysqKiopSenq4tW7YEu6SA27Bhg4YPH66WLVvK4XBo2bJlwS4p4PLy8tSrVy9FR0crPj5eI0aM0J49e4JdVsDNnTtX3bp182x+2KdPH61YsSLYZQXdzJkz5XA49OCDDwa7lICaPn26HA6H19G5c+dglxUUhw4d0o9//GNdffXVatiwoX7wgx9o27ZtwS4L9cAV1Vi98cYbys3N1bRp07R9+3Z1795dgwcPVnFxcbBLC6iysjJ1795dc+bMCXYpQbN+/XpNnDhRH3/8sVavXq1z585p0KBBKisrC3ZpAdW6dWvNnDlT+fn52rZtm2699Vbdeeed2r17d7BLC5qtW7fqpZdeUrdu3YJdSlBcd911OnLkiOf48MMPg11SwB0/flw33XSTGjRooBUrVujTTz/VM888o6ZNmwa7NNQDV9R2C+np6erVq5f++Mc/Sqr83MKkpCRNmjRJjzzySJCrCw6Hw6GlS5dqxIgRwS4lqI4ePar4+HitX79e/fr1C3Y5QdWsWTPNmjVL48ePD3YpAXfq1Cldf/31evHFF/XEE0+oR48emj17drDLCpjp06dr2bJl2rFjR7BLCapHHnlEH330kf7+978HuxTUQ1dMYnX27Fnl5+crIyPDcy4sLEwZGRnatGlTECtDXVBaWiqpsqm4UlVUVGjJkiUqKytTnz59gl1OUEycOFHDhg3z+nviSrN37161bNlS7du3V2Zmpg4cOBDskgJu+fLlSktL0z333KP4+Hj17NlTL7/8crDLQj1xxTRWJSUlqqioUEJCgtf5hIQEFRYWBqkq1AVut1sPPvigbrrpJnXt2jXY5QTczp071bhxYzmdTv30pz/V0qVLlZKSEuyyAm7JkiXavn278vLygl1K0KSnp+uVV17RypUrNXfuXO3fv199+/bVyZMng11aQH311VeaO3eurrnmGq1atUo/+9nP9POf/1z/+7//G+zSUA9EBLsAINgmTpyoXbt2XZFrSSSpU6dO2rFjh0pLS/XWW28pKytL69evv6Kaq4KCAk2ePFmrV69WVFRUsMsJmttuu83z3926dVN6erratm2rP//5z1fU1LDb7VZaWpqefPJJSVLPnj21a9cuzZs3T1lZWUGuDnXdFZNYNW/eXOHh4SoqKvI6X1RUpMTExCBVhWDLycnRu+++q7Vr16p169bBLicoIiMj1bFjR6WmpiovL0/du3fX888/H+yyAio/P1/FxcW6/vrrFRERoYiICK1fv14vvPCCIiIiVFFREewSg6JJkya69tprtW/fvmCXElAtWrSo8g+LLl26XJHTovDdFdNYRUZGKjU1VWvWrPGcc7vdWrNmzRW7nuRKZhiGcnJytHTpUn3wwQdq165dsEuqM9xut8rLy4NdRkANHDhQO3fu1I4dOzxHWlqaMjMztWPHDoWHhwe7xKA4deqUvvzyS7Vo0SLYpQTUTTfdVGX7lS+++EJt27YNUkWoT66oqcDc3FxlZWUpLS1NvXv31uzZs1VWVqbs7OxglxZQp06d8voX6P79+7Vjxw41a9ZMbdq0CWJlgTNx4kQtXrxYf/3rXxUdHe1ZZxcbG6uGDRsGubrAmTJlim677Ta1adNGJ0+e1OLFi7Vu3TqtWrUq2KUFVHR0dJX1dY0aNdLVV199Ra27+8UvfqHhw4erbdu2Onz4sKZNm6bw8HCNHTs22KUF1EMPPaQbb7xRTz75pEaNGqUtW7Zo/vz5mj9/frBLQ31gXGH+8Ic/GG3atDEiIyON3r17Gx9//HGwSwq4tWvXGpKqHFlZWcEuLWCqe35JxsKFC4NdWkDdd999Rtu2bY3IyEgjLi7OGDhwoPH+++8Hu6w6oX///sbkyZODXUZAjR492mjRooURGRlptGrVyhg9erSxb9++YJcVFO+8847RtWtXw+l0Gp07dzbmz58f7JJQT1xR+1gBAAD40xWzxgoAAMDfaKwAAABsQmMFAABgExorAAAAm9BYAQAA2ITGCgAAwCY0VgAAADahsQIAALAJjRUAAIBNaKwAAABsQmMFAABgExorAAAAm/x/Hc7NBxFn+e8AAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "imshow_zero_center(j[:, 0, :, 0])\n", "_ = plt.title('A (batch, batch) slice')" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:19.681818Z", "iopub.status.busy": "2024-08-15T02:32:19.681575Z", "iopub.status.idle": "2024-08-15T02:32:19.898017Z", "shell.execute_reply": "2024-08-15T02:32:19.897397Z" }, "id": "g4ZoRJcJNmy5" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgkAAAIQCAYAAAALu/2rAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLQElEQVR4nO3deXgUZdb38V8nmASBsAZCJBAUJSDbGIQBUUACyCY+ow744AARcSMKoo7g47A4jMFXBFwYcNwYRxgQFETFIKuooEAQBxBQFDQCYREhECWBpN4/HFraVLqqSXXS6fp+rquvy66qrjp1021On7rrtMcwDEMAAAC/EVHeAQAAgNBEkgAAAEyRJAAAAFMkCQAAwBRJAgAAMEWSAAAATJEkAAAAUyQJAADAFEkCAAAwRZLgII/HowkTJnifz549Wx6PR3v37rX1+nvuuUfdu3f3Pl+zZo08Ho8WLlzocKTnz+PxKD093dF9JiUlqW/fvo7uszSGDh2qqlWrWm73ww8/qEqVKlq6dKljx54wYYI8Ho/PsqSkJA0dOtSxY4S7347X2c/RmjVryi2mYAr380P5Ikmw6e9//7s8Ho/at28flP3v2bNHL774oh555JGg7P+spUuX+iQy4Wj//v2aMGGCtmzZEtTj1K5dW7fffrv+8pe/BPU4AFBeSBJsmjNnjpKSkrRhwwbt3r3b8f0//fTTaty4sbp27er4vs+1dOlSTZw4MajHKG/79+/XxIkTg54kSNJdd92lzZs3a9WqVUE7xq5du/TCCy8Ebf/h7pprrtHPP/+sa665prxDASockgQb9uzZo3Xr1mnq1KmKi4vTnDlzHN3/6dOnNWfOHP3xj390dL8IvmbNmqlFixaaPXt20I4RHR2tCy64IGj7L2uGYejnn38us+NFREQoJiZGERH87w4IFJ8aG+bMmaOaNWuqT58+uummmxxPEj766CMdOXJEqamppusLCwv1yCOPKD4+XlWqVNH111+v7Oxsn20+/PBD3XzzzWrYsKGio6OVmJio+++/3+d/xkOHDtWMGTMk/TK34OzjrKKiIj399NNq2bKlYmJiFBcXp+uuu06bNm0qFtPixYvVokULRUdH6/LLL1dmZmapx+H9999XmzZtFBMTo+bNm+vNN9/0WX/06FE9+OCDatmypapWrarY2Fj16tVLn3/+uXebNWvW6Morr5QkpaWlec/x3D/in376qXr37q2aNWuqSpUqatWqlZ5++uli8ezbt0833HCDqlatqri4OD344IMqLCwstl337t319ttvy+oHVU+fPq2JEyfq0ksvVUxMjGrXrq1OnTpp+fLlfl9nNifh2LFjuv/++5WUlKTo6Gg1aNBAgwcP1pEjR7zb5Ofna/z48WrSpIn3PfHnP/9Z+fn5Pvtavny5OnXqpBo1aqhq1apq2rSprcter7zyiq699lrVrVtX0dHRat68uWbOnGkaf9++fbVs2TK1bdtWlStX1vPPP+89j1GjRikxMVHR0dFq0qSJnnjiCRUVFVke3zAMTZo0SQ0aNNCFF16orl27avv27cW2M7tmb+fzctaCBQvUvHlzxcTEqEWLFlq0aJGGDh2qpKQkn+3y8vL0wAMPeM+ladOmmjJlSrH3xdl5PVafoW+//Vb33HOPmjZtqsqVK6t27dq6+eabbc9xApxQqbwDqAjmzJmjP/zhD4qKitItt9yimTNnauPGjd4/RqW1bt06eTwe/e53vzNd/7e//U0ej0cPP/ywDh06pOnTpys1NVVbtmxR5cqVJf3yP7KffvpJd999t2rXrq0NGzbo2Wef1ffff68FCxZIku68807t379fy5cv17/+9a9ixxk2bJhmz56tXr166fbbb9eZM2f04Ycf6pNPPlHbtm2923300Ud68803dc8996hatWp65plndOONN+q7775T7dq1z2sMvvrqKw0YMEB33XWXhgwZoldeeUU333yzMjMzvZM5v/nmGy1evFg333yzGjdurIMHD+r5559X586d9cUXXyghIUHNmjXTY489pnHjxumOO+7Q1VdfLUnq2LGjpF/+IPbt21f169fXyJEjFR8frx07duidd97RyJEjvfEUFhaqZ8+eat++vaZMmaIVK1boqaee0iWXXKK7777bJ/aUlBRNmzZN27dvV4sWLUo8xwkTJigjI0O333672rVrp9zcXG3atEmbN2/2mbBq5eTJk7r66qu1Y8cO3Xbbbbriiit05MgRLVmyRN9//73q1KmjoqIiXX/99froo490xx13qFmzZtq6daumTZumL7/8UosXL5Ykbd++XX379lWrVq302GOPKTo6Wrt379bHH39sGcfMmTN1+eWX6/rrr1elSpX09ttv65577lFRUZFGjBjhs+2uXbt0yy236M4779Tw4cPVtGlT/fTTT+rcubP27dunO++8Uw0bNtS6des0duxYHThwQNOnT/d7/HHjxmnSpEnq3bu3evfurc2bN6tHjx4qKCiwjN3O50WS3n33XQ0YMEAtW7ZURkaGfvzxRw0bNkwXXXSRz/4Mw9D111+v1atXa9iwYWrTpo2WLVumhx56SPv27dO0adN8trfzGdq4caPWrVungQMHqkGDBtq7d69mzpypLl266IsvvtCFF15oeZ5AqRnwa9OmTYYkY/ny5YZhGEZRUZHRoEEDY+TIkcW2lWSMHz/e+/yVV14xJBl79uzxe4xbb73VqF27drHlq1evNiQZF110kZGbm+td/vrrrxuSjKefftq77Keffir2+oyMDMPj8Rjffvutd9mIESMMs3/2VatWGZKM++67r9i6oqIin3OMiooydu/e7V32+eefG5KMZ5991u95lqRRo0aGJOONN97wLjt+/LhRv35943e/+5132alTp4zCwkKf1+7Zs8eIjo42HnvsMe+yjRs3GpKMV155xWfbM2fOGI0bNzYaNWpk/PjjjyWe45AhQwxJPvs0DMP43e9+Z6SkpBSLf926dYYkY/78+X7Ps3Xr1kafPn38bjN+/Phi/z6NGjUyhgwZ4n0+btw4Q5Lx5ptvFnv92fP417/+ZURERBgffvihz/pZs2YZkoyPP/7YMAzDmDZtmiHJOHz4sN+4zJi953r27GlcfPHFxeKXZGRmZvos/+tf/2pUqVLF+PLLL32WjxkzxoiMjDS+++67Eo996NAhIyoqyujTp4/Pv90jjzxiSPIZr7Ofo9WrV/uN3ezz0rJlS6NBgwbGiRMnvMvWrFljSDIaNWrkXbZ48WJDkjFp0iSffd50002Gx+Px+bzY/QyZxbh+/XpDkvHqq6/6PT/AKVxusDBnzhzVq1fPO6HQ4/FowIABmjdvnmnp+Xz88MMPqlmzZonrBw8erGrVqnmf33TTTapfv77PrXdnKwrSL2XPI0eOqGPHjjIMQ5999pllDG+88YY8Ho/Gjx9fbN1vb8lLTU3VJZdc4n3eqlUrxcbG6ptvvrE8TkkSEhL0P//zP97nsbGxGjx4sD777DPl5ORI+uXa/NnryoWFhfrhhx+85fHNmzdbHuOzzz7Tnj17NGrUKNWoUcNn3W/PUfplUuK5rr76atNzPPtvd26p30yNGjW0fft2ffXVV5ax+vPGG2+odevWPuN11tnzWLBggZo1a6bk5GQdOXLE+7j22mslSatXr/bGJElvvfWWrRL/uc59zx0/flxHjhxR586d9c033+j48eM+2zZu3Fg9e/b0WbZgwQJdffXVqlmzpk+MqampKiws1Nq1a0s89ooVK1RQUKB7773X599u1KhRAcde0udl//792rp1qwYPHuxzS2znzp3VsmVLn/0tXbpUkZGRuu+++3yWP/DAAzIMQ++9957PcjufoXNjPH36tH744Qc1adJENWrUsPV+B5xAkuBHYWGh5s2bp65du2rPnj3avXu3du/erfbt2+vgwYNauXKlY8cy/FzPvvTSS32eezweNWnSxOfa5HfffaehQ4eqVq1a3mvonTt3lqRi/8M28/XXXyshIUG1atWy3LZhw4bFltWsWVM//vij5WtL0qRJk2J/qC+77DJJ8p5nUVGRpk2bpksvvVTR0dGqU6eO4uLi9J///Mf2OUrye0ngrLNzMs5V0jme/bczSzTO9dhjj+nYsWO67LLL1LJlSz300EP6z3/+YxnLb3399deW5/DVV19p+/btiouL83mcHdNDhw5JkgYMGKCrrrpKt99+u+rVq6eBAwfq9ddft5UwfPzxx0pNTVWVKlVUo0YNxcXFeecymCUJZjFmZmYWi/Hs3JyzMZr59ttvJRX/bMTFxflNuM+y83k5e4wmTZoUe/1vl3377bdKSEjwSealXya2nruvs+x8hn7++WeNGzfOO8fh7Pv92LFjtt7vgBOYk+DHqlWrdODAAc2bN0/z5s0rtn7OnDnq0aNHqY9Tu3btUv2BLSwsVPfu3XX06FE9/PDDSk5OVpUqVbRv3z4NHTo04G+IViIjI02X+0t0nPD444/rL3/5i2677Tb99a9/Va1atRQREaFRo0aV2TmaOftvV6dOHb/bXXPNNfr666/11ltv6f3339eLL76oadOmadasWbr99ttLFe9vFRUVqWXLlpo6darp+sTEREm/fFtdu3atVq9erXfffVeZmZmaP3++rr32Wr3//vsljsPXX3+tbt26KTk5WVOnTlViYqKioqK0dOlSTZs2rdi/x7nfis+NsXv37vrzn/9seoyzCY3TyvrzYsbOZ+jee+/VK6+8olGjRqlDhw6qXr26PB6PBg4cWCYxAhJJgl9z5sxR3bp1vXcEnOvNN9/UokWLNGvWLNP/AQYiOTlZc+bM0fHjx1W9evVi639bnjYMQ7t371arVq0kSVu3btWXX36pf/7znxo8eLB3O7NZ8yV9273kkku0bNkyHT161FY1wWm7d++WYRg+8X355ZeS5J1FvnDhQnXt2lUvvfSSz2uPHTvm8wfa3zlK0rZt20q8k+R87NmzR9Kv3xr9qVWrltLS0pSWlqaTJ0/qmmuu0YQJEwJKEi655BJt27bNcpvPP/9c3bp1s6xwREREqFu3burWrZumTp2qxx9/XP/3f/+n1atXlzhOb7/9tvLz87VkyRKfb8VnL2PYPY+TJ0+e179Fo0aNJP3y2bj44ou9yw8fPmyZcNv9vJw9hllflN8ua9SokVasWKETJ074VBN27tzps69ALFy4UEOGDNFTTz3lXXbq1CkdO3Ys4H0B54vLDSX4+eef9eabb6pv37666aabij3S09N14sQJLVmypNTH6tChgwzDUFZWlun6V199VSdOnPA+X7hwoQ4cOKBevXpJ+vVbybnfQgzDML2tr0qVKpJU7H80N954owzDMG20FOwKgfTL9d9FixZ5n+fm5urVV19VmzZtFB8fL+mX8/xtLAsWLNC+fft8lpV0jldccYUaN26s6dOnF1tXmnPMyspS9erVdfnll/vd7ocffvB5XrVqVTVp0qTYLYlWbrzxRn3++ec+43XW2fP44x//qH379pk2Yfr555+Vl5cn6ZfbSn+rTZs2kuQ3LrP33PHjx/XKK6/YPo8//vGPWr9+vZYtW1Zs3bFjx3TmzJkSX5uamqoLLrhAzz77rE8MVndElBS72eclISFBLVq00KuvvqqTJ096l3/wwQfaunWrz7a9e/dWYWGhnnvuOZ/l06ZNk8fj8X5WA2H2fn/22WcdmwsF2EEloQRLlizRiRMndP3115uu//3vf+9trDRgwIBSHatTp06qXbu2VqxY4Z1Ydq5atWqpU6dOSktL08GDBzV9+nQ1adJEw4cPl/RLJeKSSy7Rgw8+qH379ik2NlZvvPGG6TeqlJQUSdJ9992nnj17KjIyUgMHDlTXrl31pz/9Sc8884y++uorXXfddSoqKtKHH36orl27ntfvNXg8HnXu3NlWT/nLLrtMw4YN08aNG1WvXj29/PLLOnjwoM8fnb59++qxxx5TWlqaOnbsqK1bt2rOnDk+3ySlX76h1qhRQ7NmzVK1atVUpUoVtW/fXo0bN9bMmTPVr18/tWnTRmlpaapfv7527typ7du3m/6xsmP58uXq16+f5Tf25s2bq0uXLkpJSVGtWrW0adMmLVy4MOCxfeihh7Rw4ULdfPPNuu2225SSkqKjR49qyZIlmjVrllq3bq0//elPev3113XXXXdp9erVuuqqq1RYWKidO3fq9ddf9/YseOyxx7R27Vr16dNHjRo10qFDh/T3v/9dDRo0UKdOnUqMoUePHoqKilK/fv1055136uTJk3rhhRdUt25dHThwwPZ5LFmyRH379tXQoUOVkpKivLw8bd26VQsXLtTevXtLvIRztm9FRkaG+vbtq969e+uzzz7Te++9Z3nZJ5DPy+OPP67+/fvrqquuUlpamn788Uc999xzatGihU/i0K9fP3Xt2lX/93//p71796p169Z6//339dZbb2nUqFE+kxTt6tu3r/71r3+pevXqat68udavX68VK1ac923GwHkpy1spKpJ+/foZMTExRl5eXonbDB061LjggguMI0eOGIZx/rdAGoZh3HfffUaTJk18lp29tenf//63MXbsWKNu3bpG5cqVjT59+vjcpmUYhvHFF18YqampRtWqVY06deoYw4cP995Wde6tgGfOnDHuvfdeIy4uzvB4PD632505c8Z48sknjeTkZCMqKsqIi4szevXqZWRlZXm3kWSMGDGiWPy/vU3vxIkThiRj4MCBlufeqFEjo0+fPsayZcuMVq1aGdHR0UZycrKxYMECn+1OnTplPPDAA0b9+vWNypUrG1dddZWxfv16o3Pnzkbnzp19tn3rrbeM5s2bG5UqVSo2Bh999JHRvXt3o1q1akaVKlWMVq1a+dx6NmTIEKNKlSrF4jS7PXHHjh2GJGPFihWW5zlp0iSjXbt2Ro0aNYzKlSsbycnJxt/+9jejoKDA7zF+O7aGYRg//PCDkZ6eblx00UVGVFSU0aBBA2PIkCHe96JhGEZBQYHxxBNPGJdffrkRHR1t1KxZ00hJSTEmTpxoHD9+3DAMw1i5cqXRv39/IyEhwYiKijISEhKMW265pdhtiWaWLFlitGrVyoiJiTGSkpKMJ554wnj55ZeLvefP/vuaOXHihDF27FijSZMmRlRUlFGnTh2jY8eOxpQpU3zGxUxhYaExceJE7/uhS5cuxrZt24qNl9ktgnY/L4ZhGPPmzTOSk5ON6Ohoo0WLFsaSJUuMG2+80UhOTi52Lvfff7+RkJBgXHDBBcall15qPPnkkz63aBqG/c/Qjz/+aKSlpRl16tQxqlatavTs2dPYuXOnrfMDnOIxjDKoJcPSN998o+TkZL333nvq1q1beYdTakuXLlXfvn31+eefF7tdLJyMGjVKa9euVVZWlmUlAeGjTZs2iouLs+yWCVR0zEkIERdffLGGDRumyZMnl3cojli9erUGDhwY1gnCDz/8oBdffFGTJk0iQQhTp0+fLjY3Ys2aNfr888/VpUuX8gkKKENUEgCgBHv37lVqaqpuvfVWJSQkaOfOnZo1a5aqV6+ubdu2MT8AYY+JiwBQgpo1ayolJUUvvviiDh8+rCpVqqhPnz6aPHkyCQJcgUoCAAAwxZwEAABgiiQBAACYKvM5CUVFRdq/f7+qVavGjHAAcBnDMHTixAklJCR4f9W1LJ06dUoFBQVB2XdUVJRiYmKCsu/yUuZJwv79+70/LgMAcKfs7Gw1aNCgTI956tQpxVWurJPWm56X+Ph47dmzJ6wShTJPEn798ZNu5XF4AEC5OiNpZbGf1S4LBQUFOinpfknRDu87X9K0nBwVFBSQJJTGr5cYKkm6oKwPDwAIAeV5ubmyJKf/jIfrBL9wPS8AAFBK1PsBAK4SIee/IYfrN+5wPS8AAFBKVBIAAK5CJcE+kgQAgKuQJNgXrucFAABKKWQrCXFxb/tdfyinqIwiCT5PZH9H9jNjhv8xu+cuxuy3jFss7tV+7TVHjhMKnBoz6U2/a43CSIeOU/4ce5/1KvS/wTvvOHKcUODc+yx4qCTYF67nBQAASilkKwkAAAQDlQT7wvW8AABAKVFJAAC4CpUE+8L1vAAAQClRSQAAuAqVBPvC9bwAAEAphWwlwaoPwn+2Wec3rVqET18AO6z6IBw9Zj1mtWq4a8ws+yDs3Gm9j+RkZ2KpIKz6IKxaY/0+u7aLy95nVn0QeJ+VKY+c/4Zcfj98HVwhmyQAABAMHjn/Rz1ckwQuNwAAAFNUEgAArhL534fT+wxHVBIAAIApKgkAAFfhFkj7wvW8AABAKVFJAAC4CpUE+8L1vAAAQClV2EqCnUZJTzzpPwd6+CF3NXSx0yjp8cn+x+yRMe4aM1sNbLZt87++RQtnYqkg7DRKGjzU//vs1dm8z4pZvNj/+htucCISV6CSYF+FTRIAADgfJAn2het5AQCAUqKSAABwFSoJ9oXreQEAgFKikgAAcBUqCfaF63kBAIBSopIAAHAVfiravvNKEmbMmKEnn3xSOTk5at26tZ599lm1a9fO6dhKzaoPwmOTrAsp4x511/3aVn0QXp5tPWa3DXXXmFn2QZg923ofQ4c6EUmFYdUHoVcf6/fZe++67H1m1Qfhtdes93HrrY6EAvcI+HLD/PnzNXr0aI0fP16bN29W69at1bNnTx06dCgY8QEA4KgI/fpz0U49wvXafcDnNXXqVA0fPlxpaWlq3ry5Zs2apQsvvFAvv/xyMOIDAMBREUF6hKOAzqugoEBZWVlKTU39dQcREUpNTdX69esdDw4AAJSfgOYkHDlyRIWFhapXr57P8nr16mnnzp2mr8nPz1d+fr73eW5u7nmECQCAM7gF0r6gn1dGRoaqV6/ufSQmJgb7kAAAwAEBJQl16tRRZGSkDh486LP84MGDio+PN33N2LFjdfz4ce8jOzv7/KMFAKCUmJNgX0DnFRUVpZSUFK1cudK7rKioSCtXrlSHDh1MXxMdHa3Y2FifBwAACH0B90kYPXq0hgwZorZt26pdu3aaPn268vLylJaWFoz4AABwFHMS7As4SRgwYIAOHz6scePGKScnR23atFFmZmaxyYwVgZ1GSa++5v+ffvCt7mroYqdR0v/e6n/M5r7mrjGz1Shp8WL/660a6YQZO42SPJGG3/VGYbj2wCuBnUZJmzb5X9+2rTOxIGycV8fF9PR0paenOx0LAABBRyXBPn67AQDgKiQJ9oXreQEAgFKikgAAcBV+BdI+KgkAAMAUSQIAwFWc/gXIs49AzZgxQ0lJSYqJiVH79u21YcMGv9sfO3ZMI0aMUP369RUdHa3LLrtMS5cuPY8j28flBgAAytj8+fM1evRozZo1S+3bt9f06dPVs2dP7dq1S3Xr1i22fUFBgbp37666detq4cKFuuiii/Ttt9+qRo0aQY2TJMGCVR+ERx61LsY8PsldfQGs+iC8udh6zP5wg7vGzLIPglUfBTv7CDNWfRBGP2j9Pps6xWXvM6s+CDNnWu/j7rudiaUceeR8GT3QOQlTp07V8OHDvY0IZ82apXfffVcvv/yyxowZU2z7l19+WUePHtW6det0wQUXSJKSkpJKGbU1LjcAAOCQ3Nxcn8e5v4J8VkFBgbKyspSamupdFhERodTUVK1fv950v0uWLFGHDh00YsQI1atXTy1atNDjjz+uwsLCoJ2LRJIAAHCZYP7AU2Jios8vH2dkZBQ7/pEjR1RYWFisU3G9evWUk5NjGvM333yjhQsXqrCwUEuXLtVf/vIXPfXUU5o0aVIpRsIalxsAAK4SzGZK2dnZPj9kGB0d7cj+i4qKVLduXf3jH/9QZGSkUlJStG/fPj355JMaP368I8cwQ5IAAIBD7PzacZ06dRQZGamDBw/6LD948KDi4+NNX1O/fn1dcMEFioz89T6KZs2aKScnRwUFBYqKiip98Ca43AAAcJVgXm6wIyoqSikpKVq5cqV3WVFRkVauXKkOHTqYvuaqq67S7t27VVT062TbL7/8UvXr1w9agiCRJAAAUOZGjx6tF154Qf/85z+1Y8cO3X333crLy/Pe7TB48GCNHTvWu/3dd9+to0ePauTIkfryyy/17rvv6vHHH9eIESOCGieXGwAArhIKP/A0YMAAHT58WOPGjVNOTo7atGmjzMxM72TG7777ThERv+41MTFRy5Yt0/33369WrVrpoosu0siRI/Xwww87eBbFkSQAAFAO0tPTlZ6ebrpuzZo1xZZ16NBBn3zySZCj8kWSUEp2GiVNnc5VnXPZaZR03yjGzIedRkkPPhj0MCoSO42SelzH+8yHnUZJdhp7hbhQqCRUFOF6XgAAoJSoJAAAXIWfiraPJAEA4Crn+6uNVvsMR1xuAAAApqgkAABchYmL9oXreQEAgFKikgAAcBWPnP+GzMTFMuaJ7F/eIVQ4jFngGLPAMWaBY8xQUYVskgAAQDAwJ8G+cD0vAABQSlQSAACuQiXBPpIEAICrkCTYF67nBQAASolKAgDAVagk2Beu5wUAAEqJSgIAwFX4FUj7qCQAAABTVBIAAK7CT0XbRyUBAACYopIAAHAV7m6wjyQBAOAqJAn2het5AQCAUqKSAABwFY+c/4bMLZAAAMBVqCQAAFyFOQn2hWySYKTF+d/gxRfLJpAy4Ins78h+jIMv+d+gTh1HjhMKHBuz567zv8HddztynFDg1Jh17/623/XvZxY5cpxQ4NSYvfSS/zG7bShjhtAUskkCAADBQCXBvnA9LwAAUEpUEgAArkIlwb5wPS8AAFBKVBIAAK7CT0XbR5IAAHAVfgXSPi43AAAAU6FbSbDqg/Dcc9b7SE93JpaKwqIPwn+2WeeErVqEz/3atlj1Qdi2zXofLVo4E0sFYdUH4bFJ1u+zcY+6631m1QdhyTvWY3Z9X3eNWTAxcdG+cD0vAABQSqFbSQAAIAioJNgXrucFAABKiUoCAMBVqCTYF67nBQAASolKAgDAVTxy/hsyzZQAAAgDXG6wL1zPCwAAlFLFrSTYaZQ0e7b/9UOHOhFJhWGnUdLaj/znjdd0cllDFzuNkk6e9L++alVnYqkg7DRKmr/A//tswM3uep/ZaZT05mL/Y/aHG9w1ZqVBJcG+cD0vAABQShW3kgAAwHmgkmBfuJ4XAAAoJSoJAABXoZJgX7ieFwAAIW3GjBlKSkpSTEyM2rdvrw0bNth63bx58+TxeHTDDTcEN0CRJAAAXCYiSI9AzJ8/X6NHj9b48eO1efNmtW7dWj179tShQ4f8vm7v3r168MEHdfXVVwd4xPNDkgAAcJVQSBKmTp2q4cOHKy0tTc2bN9esWbN04YUX6uWXXy7xNYWFhRo0aJAmTpyoiy++OMAjnp+A5ySsXbtWTz75pLKysnTgwAEtWrSoTEoe58WqD8KUKdb7ePBBR0KpKKz6IFj1UbCzj7Bj1Qdh717rfSQlORFJhWHVB+GxSdbvMzv9GMKJVR+EV1+zHrPBt7przEJVQUGBsrKyNHbsWO+yiIgIpaamav369SW+7rHHHlPdunU1bNgwffjhh2URauBJQl5enlq3bq3bbrtNf/jDH4IREwAAQRPMiYu5ubk+y6OjoxUdHe2z7MiRIyosLFS9evV8lterV087d+403f9HH32kl156SVu2bHEqZFsCThJ69eqlXr16BSMWAAAqtMTERJ/n48eP14QJE0q1zxMnTuhPf/qTXnjhBdWpU6dU+woUt0ACAFwlmJWE7OxsxcbGepf/toogSXXq1FFkZKQOHjzos/zgwYOKj48vtv3XX3+tvXv3ql+/ft5lRUW/XDqqVKmSdu3apUsuucSBsygu6ElCfn6+8vPzvc9/W4oBACBcxMbG+iQJZqKiopSSkqKVK1d65/QVFRVp5cqVSjf5XaLk5GRt3brVZ9mjjz6qEydO6Omnny5WvXBS0JOEjIwMTZw4MdiHAQDAFo8kj8fj7D4NI6DtR48erSFDhqht27Zq166dpk+frry8PKWlpUmSBg8erIsuukgZGRmKiYlRi9/82FyNGjUkqdhypwU9SRg7dqxGjx7tfZ6bmxvUrAcAgFA3YMAAHT58WOPGjVNOTo7atGmjzMxM72TG7777ThER5d+lIOhJgtnMTgAAyk2lSpLDlQQZhnTmTEAvSU9PN728IElr1qzx+9rZs2cHdKzzFXCScPLkSe3evdv7fM+ePdqyZYtq1aqlhg0bOhocAACOC5EkoSIIOEnYtGmTunbt6n1+9lLCkCFDyiyzcYyNRkn/iYz0u75VYaFT0VQIdhol/X2W/xLZPXe5rKGLnUZJ5yTeppo0cSSUisJOoyRPZIHf9UZhlFPhVAh2GiV5In/2u94orOxUOAgTAScJXbp0kRHgBA0AAEJGsCoJYaj8Z0UAAICQRDMlAIC7UEmwjUoCAAAwRSUBAOAukZGS0z0IisJzQjaVBAAAYIpKAgDAXSpVopJgE0mCBcs+CCtWWO8kNdWZYCoIqz4IX+62/nBe1iQ8P3AlsuqD8P331vto0MCZWCoIqz4ISzOt32e9r3PX+8yqD8K6T6zHrOPvw2DMSBJs43IDAAAwRSUBAOAuVBJso5IAAABMUUkAALhLZOQvDyeF6e/4UEkAAACmqCQAANylUiXnKwlOt3kOEVQSAACAKSoJAAB3oZJgG0lCadlplPTaa8GPowKx0yjJTsMlV7HTKOnIkeDHUYHYaZT0/greZ+ey0yhpyTthMGYkCbaFwb82AAAIBioJAAB3oZJgG5UEAABgikoCAMBdIiN/qSbAEpUEAABgilQKAOAulSpRSbCJSgIAADAVsqmUJ7J/eYdQ4TBmgWPMAseYBY4xCzFUEmxjlAAA7kKSYBuXGwAAgClSKQCAuwTjFkjDcHZ/IYJKAgAAMEUlAQDgLsGYk0AlAQAAuAmVBACAu1BJsI1KAgAAMEUlAQDgLlQSbCNJAAC4C0mCbVxuAAAApqgkAADcJRjNlIqKnN1fiKCSAAAATFFJAAC4SzDmJFBJAAAAbkIlAQDgLlQSbKOSAAAATIVwJeFxv2uNwsvLKI7g80T2d2Q/xtdP+98gKcmR44QCx8Ysebf/DbZvd+Q4ocCpMYuJedvv+p/zwucblWPvsysP+t/gk08cOU4ocGrMgopKgm1UEgAAgKkQriQAABAEVBJsI0kAALhLMJopFRY6u78QweUGAABgiiQBAOAuZy83OP0I0IwZM5SUlKSYmBi1b99eGzZsKHHbF154QVdffbVq1qypmjVrKjU11e/2TiFJAACgjM2fP1+jR4/W+PHjtXnzZrVu3Vo9e/bUoUOHTLdfs2aNbrnlFq1evVrr169XYmKievTooX379gU1TpIEAIC7hEAlYerUqRo+fLjS0tLUvHlzzZo1SxdeeKFefvll0+3nzJmje+65R23atFFycrJefPFFFRUVaeXKlU6MSIlCduKiVR+Ev8+yzm/uuSs8Z5uWyKoPwq23Wu/jtdccCaXCsOqDMH269T5GjXIikgrDqg/CT6esP5sXxrjss2nVByE93Xofzz3nTCwIqtzcXJ/n0dHRio6O9llWUFCgrKwsjR071rssIiJCqampWr9+va3j/PTTTzp9+rRq1apV+qD9oJIAAHCXIFYSEhMTVb16de8jIyOj2OGPHDmiwsJC1atXz2d5vXr1lJOTY+sUHn74YSUkJCg1NbX04+FHyFYSAACoaLKzsxUbG+t9/tsqghMmT56sefPmac2aNYqJiXF8/+ciSQAAuEsw+iScOSNJio2N9UkSzNSpU0eRkZE6eNC3XffBgwcVHx/v97VTpkzR5MmTtWLFCrVq1ap0MdvA5QYAgLuU88TFqKgopaSk+Ew6PDsJsUOHDiW+7v/9v/+nv/71r8rMzFTbtm1LNQR2UUkAAKCMjR49WkOGDFHbtm3Vrl07TZ8+XXl5eUpLS5MkDR48WBdddJF3TsMTTzyhcePGae7cuUpKSvLOXahataqqVq0atDhJEgAA7hKM324IcH8DBgzQ4cOHNW7cOOXk5KhNmzbKzMz0Tmb87rvvFBHxa7F/5syZKigo0E033eSzn/Hjx2vChAmlDr8kJAkAAJSD9PR0pZdw++uaNWt8nu/duzf4AZkgSQAAuEsIVBIqigp7VnYaJT0+2f+8zEfGuKyhi51GSVbb2GnIFE5sNEoqspj/GyF3vc/sNEp69TX/Yzb4VneNma1GSbNn+18/dKgTkQA+KmySAADAeaGSYBu3QAIAAFPhmfoAAFCSYDRTiox0dn8hgiQBAOAuXG6wjcsNAADAVHimPgAAlIRKgm1UEgAAgKmAUp+MjAy9+eab2rlzpypXrqyOHTvqiSeeUNOmTYMVX6lY9UG4e4R1jjRzhsvu17bqgzBlivU+HnzQmVgqCMs+CL9po2pq4UJngqkgrPogzJ1n/dn834Eu+2xa9UGw05o3iO17KxQqCbYFVEn44IMPNGLECH3yySdavny5Tp8+rR49eigvLy9Y8QEAgHISUOqTmZnp83z27NmqW7eusrKydM011zgaGAAAQcEtkLaVak7C8ePHJUm1atVyJBgAABA6zjuVKioq0qhRo3TVVVepRYsWJW6Xn5+v/Px87/Pc3NzzPSQAAKXHnATbzvusRowYoW3btumjjz7yu11GRoYmTpx4vocBAMBZJAm2ndflhvT0dL3zzjtavXq1GjRo4HfbsWPH6vjx495Hdnb2eQUKAADKVkCpj2EYuvfee7Vo0SKtWbNGjRs3tnxNdHS0oqOjzztAAAAcRSXBtoDOasSIEZo7d67eeustVatWTTk5OZKk6tWrq3LlykEJEAAAlI+AkoSZM2dKkrp06eKz/JVXXtFQq0YfIchOo6Qe1/m/IvN+pssautholPTlbv9jdlkTl42ZnUZJkyb5X//oo87EUkHYaZTkiVzvd71R2MGpcCoGO42SLOaQqVMnR0IJeVQSbAv4cgMAAHCH8Ex9AAAoCc2UbOMHngAAgCkqCQAAd2FOgm3heVYAAJSEJME2LjcAAABT4Zn6AABQEioJtoXnWTnIqg/CtanWxZhVK9zVF8CyD0JGhvVOxo51JpiKwqoPwr//bb2PW25xJpYKwqoPwrpPrD+bHX/vrs+mZR+EW2+13sdrrzkTCyoEkgQAgLtwC6RtzEkAAACmqCQAANyFOQm2UUkAAACmwjP1AQCgJFQSbAvPswIAoCQkCbZxuQEAAJgKz9QHAICSUEmwLTzPqgzZaZTUsRMFGx82GiUtzWTMfNhplFS7tsUGFo10woydRknXdOF95sNOo6TFi4MeBkIHSQIAwF1opmQbaTQAADBFJQEA4C7MSbCNSgIAADAVnqkPAAAloZJgW3ieFQAAJSFJsI3LDQAAwFTIpj6eyP7lHUKFw5gFLrzGrGz6IITXmJUNxizEcAukbVQSAACAqZCtJAAAEBTMSbCNSgIAADAVnqkPAAAloZJgG5UEAADKwYwZM5SUlKSYmBi1b99eGzZs8Lv9ggULlJycrJiYGLVs2VJLly4NeowkCQAAdzlbSXD6EYD58+dr9OjRGj9+vDZv3qzWrVurZ8+eOnTokOn269at0y233KJhw4bps88+0w033KAbbrhB27Ztc2JESuQxDMMI6hF+Izc3V9WrV5fUU9IFZXloAEC5Oy1pmY4fP67Y2NgyPfLZvz/Hf/zR8WPn5uaqes2ats+rffv2uvLKK/Xcc89JkoqKipSYmKh7771XY8aMKbb9gAEDlJeXp3feece77Pe//73atGmjWbNmOXciv0ElAQAAh+Tm5vo88vPzi21TUFCgrKwspaamepdFREQoNTVV69evN93v+vXrfbaXpJ49e5a4vVNIEgAArlKkiKA8JCkxMVHVq1f3PjIyMood/8iRIyosLFS9evV8lterV085OTmmMefk5AS0vVPCczomAADlIDs72+dyQ3R0dDlGU3okCQAAVzlz5peH0/uUpNjYWMs5CXXq1FFkZKQOHjzos/zgwYOKj483fU18fHxA2zuFyw0AAJShqKgopaSkaOXKld5lRUVFWrlypTp06GD6mg4dOvhsL0nLly8vcXunUEkAALhKMCsJdo0ePVpDhgxR27Zt1a5dO02fPl15eXlKS0uTJA0ePFgXXXSRd07DyJEj1blzZz311FPq06eP5s2bp02bNukf//iHsyfyGyQJAACUsQEDBujw4cMaN26ccnJy1KZNG2VmZnonJ3733XeKiPi12N+xY0fNnTtXjz76qB555BFdeumlWrx4sVq0aBHUOOmTAAAoQ+XfJyE72/lj5+bmKjGxermcVzBRSQAAuEooXG6oKJi4CAAATIVsJeHKK9/2u37DJ0VlFEnweSL7O7If49O/+N+gbVtHjhMKnBqz7dv9v8+aJ/M++y3jg4f8b9CpkyPHCQWOjdnvj/jf4OOPHTlOKHBqzIKpsND5b/6Fhc7uL1RQSQAAAKZCtpIAAEAwMCfBPioJAADAFJUEAICrUEmwj0oCAAAwRSUBAOAqVBLso5IAAABMhWwlwaoPwpJ3rPOb6/uGzz3utlj1QbjkEut9fP21M7FUEJZ9EOx8PagUsh+j4LDqg3D55db72L7dmVgqCqs+CC++aL2P2293JhbQJyEALvu/GwDA7bjcYB+XGwAAgCkqCQAAV6GSYB+VBAAAYIpKAgDAVagk2EclAQAAmKKSAABwFSoJ9lFJAAAApipsJcFOo6S/z/KfA91zl8uaLdlplDRhQunWhxsbjZKsGnu5rqmXnUZJVs2D3NY4yM75MmaOoZmSfRU2SQAA4HxwucE+LjcAAABTVBIAAK5CJcE+KgkAAMAUlQQAgKtQSbCPSgIAADBFJQEA4CrcAmlfQEnCzJkzNXPmTO3du1eSdPnll2vcuHHq1atXMGIrNas+CKvWWBdSru3isnvcrfog3HST9T4WLnQklIrCqg/CT6es32cXxrjsfWZ1T39KivU+srKciaWisBqzmTOt93H33c7EAtcIKElo0KCBJk+erEsvvVSGYeif//yn+vfvr88++0yXX355sGIEAMAxzEmwL6AkoV+/fj7P//a3v2nmzJn65JNPSBIAABUCSYJ95z0nobCwUAsWLFBeXp46dOjgZEwAACAEBJwkbN26VR06dNCpU6dUtWpVLVq0SM2bNy9x+/z8fOXn53uf5+bmnl+kAAA4gEqCfQHfAtm0aVNt2bJFn376qe6++24NGTJEX3zxRYnbZ2RkqHr16t5HYmJiqQIGAABlI+AkISoqSk2aNFFKSooyMjLUunVrPf300yVuP3bsWB0/ftz7yM7OLlXAAACUxtlKgtOPcFTqPglFRUU+lxN+Kzo6WtHR0aU9DAAAKGMBJQljx45Vr1691LBhQ504cUJz587VmjVrtGzZsmDFBwCAo2imZF9AScKhQ4c0ePBgHThwQNWrV1erVq20bNkyde/ePVjxBZWdRkkRlfxfkSk647ImOHYaJZ065X99TIwzsVQQdholvb/C//usR6rL3md2GiVt2uR/fdu2zsRSUdhplJSU5H/9fxvlAWcFlCS89NJLwYoDAIAywd0N9vHbDQAAVyFJsI9fgQQAAKaoJAAAXIVKgn1UEgAAgCkqCQAAV+EWSPuoJAAAAFNUEixY9UF45jnrPOu+dJfd427VB+Guu6z3MWuWM7FUEFZ9EL773vp91rCBy95nVn0QMjKs9zF2rDOxVBRWfRBcMmbMSbCPSgIAADBFJQEA4CpUEuwjSQAAuApJgn1cbgAAIIQdPXpUgwYNUmxsrGrUqKFhw4bp5MmTfre/99571bRpU1WuXFkNGzbUfffdp+PHjwd8bCoJAABXqWiVhEGDBunAgQNavny5Tp8+rbS0NN1xxx2aO3eu6fb79+/X/v37NWXKFDVv3lzffvut7rrrLu3fv18L7fxI3zlIEgAACFE7duxQZmamNm7cqLb/vaPn2WefVe/evTVlyhQlJCQUe02LFi30xhtveJ9fcskl+tvf/qZbb71VZ86cUaVK9v/0c7kBAOAqZ5spOfkIVjOl9evXq0aNGt4EQZJSU1MVERGhTz/91PZ+jh8/rtjY2IASBIlKAgAAjsnNzfV5Hh0drejo6PPeX05OjurWreuzrFKlSqpVq5ZycnJs7ePIkSP661//qjvuuCPg45MklJKdRkk9rqNg48NGo6Tck4zZuWw1Svr+++AHUpHYafqzeHHQw6hQ7IzZVVdZbFDHkVCC6cwZKTLS+X1KUmJios/y8ePHa8KECcW2HzNmjJ544gm/+9yxY0ep48rNzVWfPn3UvHlz0ziskCQAAOCQ7OxsxcbGep+XVEV44IEHNHToUL/7uvjiixUfH69Dhw75LD9z5oyOHj2q+Ph4v68/ceKErrvuOlWrVk2LFi3SBRdcYO8kzkGSAABwlWBWEmJjY32ShJLExcUpLi7OcrsOHTro2LFjysrKUkpKiiRp1apVKioqUvv27Ut8XW5urnr27Kno6GgtWbJEMVbt8ktATRcA4CpOT1oMxi2VZzVr1kzXXXedhg8frg0bNujjjz9Wenq6Bg4c6L2zYd++fUpOTtaGDRsk/ZIg9OjRQ3l5eXrppZeUm5urnJwc5eTkqDDAGZZUEgAACGFz5sxRenq6unXrpoiICN1444165plnvOtPnz6tXbt26aeffpIkbd682XvnQ5MmTXz2tWfPHiUlJdk+NkkCAMBVzt4C6fQ+g6VWrVolNk6SpKSkJBmG4X3epUsXn+elweUGAABgikoCAMBVzpyRIhz+ihyuP/AUskmCJ7J/eYdQ4TBmgWPMAseYBS68xiz0+yDAOSGbJAAAEAxUEuxjTgIAADBFJQEA4CpUEuwjSQAAuApJgn1cbgAAAKaoJAAAXKWiNVMqT1QSAACAKSoJAABXOXNG8nic32c4opIAAABMUUkAALgKlQT7qCQAAABTVBIAAK5CJcE+kgQAgKuQJNjH5QYAAGCKSgIAwFUKC52vJNBMCQAAuAqVBACAqwRj/kC4zkkI2SRh8uS3/a5/+KGiMook+DyR/R3Zj9E93/8GmZmOHCcUODZmE6/0v8GjjzpynFDg1Jh9/bX/z+bFSXw2f8t4qJn/DSZPduQ4ocCpMUNoCNkkAQCAYKCSYB9zEgAAgCkqCQAAV6GSYB+VBAAAYIpKAgDAVYLR0yBc+ySQJAAAXOXMGckwnN1nuCYJXG4AAACmQraSYNUHwROZbbkPozDRqXAqBos+CHMjIy138b/hmg6XxKIPQpGNMYtw2ZhZ9UEoOGP93SOqUvj0UrDFog/CKhvvs2td9j4LJioJ9lFJAAAApkK2kgAAQDBQSbCPSgIAADBFJQEA4CpUEuyjkgAAAExRSQAAuEphofOVhKIwvWGHJAEA4CpnzkgRDtfRwzVJ4HIDAAAwVWErCXYaJXXs5D8HWvdRmKZ+JbDTKGmLRVOXNuE6O6cEthol/fvf/tffcoszwVQQthol5eT4Xx8f70wwFYStRklJSf7X793rRCiuQCXBPioJAADAVIWtJAAAcD6oJNhHJQEAAJiikgAAcJXCQue/+Tt9S2WooJIAAABMUUkAALjKmTOSx+PsPsO1kkCSAABwFZIE+0qVJEyePFljx47VyJEjNX36dIdCco5VH4SIStZXW4rOhOmU1RJY9UFYY9FHQZK6uKyXgmUfhMsvt97H9u3OxFJRuKwPgiMs+iDk2Phsxrvts4lSO+8kYePGjXr++efVqlUrJ+MBACCoqCTYd14TF0+ePKlBgwbphRdeUM2aNZ2OCQAAhIDzShJGjBihPn36KDU11el4AAAIqjNngvMIlqNHj2rQoEGKjY1VjRo1NGzYMJ08edLWaw3DUK9eveTxeLR48eKAjx3w5YZ58+Zp8+bN2rhxo63t8/PzlZ+f732em5sb6CEBAHCtQYMG6cCBA1q+fLlOnz6ttLQ03XHHHZo7d67la6dPny5PKa6tBJQkZGdna+TIkVq+fLliYmJsvSYjI0MTJ048r+AAAHBekQzD6UnpwZnkvmPHDmVmZmrjxo1q27atJOnZZ59V7969NWXKFCUkJJT42i1btuipp57Spk2bVL9+/fM6fkCXG7KysnTo0CFdccUVqlSpkipVqqQPPvhAzzzzjCpVqqRCk5mzY8eO1fHjx72P7Ozs8woUAAC3Wb9+vWrUqOFNECQpNTVVERER+vTTT0t83U8//aT//d//1YwZMxRfiruJAqokdOvWTVu3bvVZlpaWpuTkZD388MOKNLkFJzo6WtHR0ecdIAAAzir878PpfRa/pF7av4E5OTmqW7euz7JKlSqpVq1ayvHzk+v333+/OnbsqP79+5/3saUAk4Rq1aqpRYsWPsuqVKmi2rVrF1sOAEBoCl6SkJiY6LN0/PjxmjBhQrGtx4wZoyeeeMLvHnfs2HFekSxZskSrVq3SZ599dl6vP5erOy7aaZT02CT/V2TGPequZkt2GiVtsGjq0s5tDV3sNEoaM8b/+smTnYkljPzjRf+fzTtud9dn01ajpIED/a+fN8+ZYFwsOztbsbGx3uclVREeeOABDR061O++Lr74YsXHx+vQoUM+y8+cOaOjR4+WeBlh1apV+vrrr1WjRg2f5TfeeKOuvvpqrVmzxvI8zip1khDIwQAAKH/BqyTExsb6JAkliYuLU1xcnOV2HTp00LFjx5SVlaWUlBRJvyQBRUVFat++velrxowZo9tvv91nWcuWLTVt2jT169fP8pjncnUlAQCAUNasWTNdd911Gj58uGbNmqXTp08rPT1dAwcO9N7ZsG/fPnXr1k2vvvqq2rVrp/j4eNMqQ8OGDdW4ceOAjs9PRQMAXKYoSI/gmDNnjpKTk9WtWzf17t1bnTp10j/+8Q/v+tOnT2vXrl366aefHD82lQQAAEJYrVq1/DZOSkpKkmHx4xFW60tCkgAAcJngzUkIN1xuAAAApqgkAABcpkjOf/MPz1tuSRIsWPVB8ERat5k2ChMttwknVn0Qllj0UZCk693WS8GqD4KdX1xdscKZWCoIyz4In3xivZPf/96ZYCoKiz4IX9r4bF4WFp9NLjfYxeUGAABgikoCAMBlqCTYRSUBAACYopIAAHCZYDQ/Cs+Ji1QSAACAKSoJAACXYU6CXVQSAACAKSoJAACXoZJgF0lCKdlplHRxEwo257LTKGmzZVOXvs4EU1HYaZTUpYvFBtWdiKTisNMo6dixoIdRkdhqlFS7tsUGnRyJJbhIEuzirxcAADBFJQEA4DL8doNdVBIAAIApKgkAAJehmZJdVBIAAIApKgkAAJfh7ga7qCQAAABTIVtJ8ET2L+8QKpzwGrOy6YMQXmNWNn0QwmvMykZ4jVlF6INghUqCXSGbJAAAEBwkCXZxuQEAAJiikgAAcBkqCXZRSQAAAKaoJAAAXMaQ882PDIf3FxqoJAAAAFNUEgAALsOcBLuoJAAAAFNUEgAALkMlwS6SBACAy5Ak2MXlBgAAYIpKAgDAZagk2EUlAQAAmKKSAABwmSI530zJ6f2FBioJAADAFJUEAIDLMCfBLioJAADAVAhXEu7xu9Yo7FlGcQSfJ7K/Q3vq4netUXi/Q8cpf06N2f33v+13/dQp4XOd0bn3mX9G4Vtlcpyy4NSYGXnz/W8QE+PIcUJBWb3PSqdIzn/zD5//V5yLSgIAADAVwpUEAACCgTkJdpEkAABchlsg7eJyAwAAMEUlAQDgMlxusItKAgAAMEUlAQDgMlQS7ArZJMGqD8LLs62LILcNDc+JJCWx6oMwf4H1mA242V1jZtUHYck71mN2fV93jZlVH4RXX7Mes8G3umvMLPsgbNtmvY8WLZyJBQhAyCYJAAAEB5UEu5iTAAAATJEkAABc5mxbZicfwbuEdvToUQ0aNEixsbGqUaOGhg0bppMnT1q+bv369br22mtVpUoVxcbG6pprrtHPP/8c0LFJEgAALlMUpEdwDBo0SNu3b9fy5cv1zjvvaO3atbrjjjv8vmb9+vW67rrr1KNHD23YsEEbN25Uenq6IiIC+7PPnAQAAELUjh07lJmZqY0bN6pt27aSpGeffVa9e/fWlClTlJCQYPq6+++/X/fdd5/GjBnjXda0adOAj08lAQDgMk5favh1ImRubq7PIz8/v1SRrl+/XjVq1PAmCJKUmpqqiIgIffrpp6avOXTokD799FPVrVtXHTt2VL169dS5c2d99NFHAR+fJAEAAIckJiaqevXq3kdGRkap9peTk6O6dev6LKtUqZJq1aqlnJwc09d88803kqQJEyZo+PDhyszM1BVXXKFu3brpq6++Cuj4XG4AALhM8G6BzM7OVmxsrHdpdHS06dZjxozRE0884XePO3bsOK9Iiop+mR9x5513Ki0tTZL0u9/9TitXrtTLL78cUOJSYZMEO42SrBouua3Zkp1GSf/Z5n/MWrVw15jZaZT0xU7/Y9Y82V1jZqdRklWTKrc1qLLVKGnTJv/rzylHo/zExsb6JAkleeCBBzR06FC/21x88cWKj4/XoUOHfJafOXNGR48eVXx8vOnr6tevL0lq3ry5z/JmzZrpu+++s4ztXBU2SQAA4PyUfzOluLg4xcXFWW7XoUMHHTt2TFlZWUpJSZEkrVq1SkVFRWrfvr3pa5KSkpSQkKBdu3b5LP/yyy/Vq1evgOJkTgIAACGqWbNmuu666zR8+HBt2LBBH3/8sdLT0zVw4EDvnQ379u1TcnKyNmzYIEnyeDx66KGH9Mwzz2jhwoXavXu3/vKXv2jnzp0aNmxYQMenkgAAcJnyryQEYs6cOUpPT1e3bt0UERGhG2+8Uc8884x3/enTp7Vr1y799NNP3mWjRo3SqVOndP/99+vo0aNq3bq1li9frksuuSSgY5MkAABcJhjNj4I3j6ZWrVqaO3duieuTkpJkGEax5WPGjPHpk3A+uNwAAABMUUkAALjM2d9ucHqf4YdKAgAAMBVQJWHChAmaOHGiz7KmTZtq586djgblFKs+CLXqWOdIR4+EZ3ZYEqs+CIxZcVZ9EKz6KNjZR7ix6oMQFWM9ZgWn3DVmln0Q1qyx3keXLk5EEgYq1sTF8hTw5YbLL79cK1as+HUHlbhiAQBAOAr4L3ylSpVK7PIEAEDoo5JgV8BzEr766islJCTo4osv1qBBgwJu8QgAACqGgCoJ7du31+zZs9W0aVMdOHBAEydO1NVXX61t27apWrVqpq/Jz8/3+anM3Nzc0kUMAECpUEmwK6Ak4dyez61atVL79u3VqFEjvf766yW2eszIyCg22REAgPJDkmBXqW6BrFGjhi677DLt3r27xG3Gjh2r48ePex/Z2dmlOSQAACgjpUoSTp48qa+//tr7s5RmoqOjvT+dafcnNAEACJ6iID3CT0BJwoMPPqgPPvhAe/fu1bp16/Q///M/ioyM1C233BKs+AAAQDkJaE7C999/r1tuuUU//PCD4uLi1KlTJ33yySe2fhM7FNlp+uOJfMHveqNwuFPhVAj2xuxHv+uNwppOhVMh2GmUZNVwyW3Nluw0SvJEPuB3vVH4lFPhVAx2GiUtXOh//U03ORJK6KMts10BJQnz5s0LVhwAACDE0C4RAOAy3N1gFz/wBAAATFFJAAC4DJUEu0gSAAAuE4xbFsNz4iKXGwAAgCkqCQAAl+Fyg10kCRas+iBExVgXY+zc8x1OrPogtPu99Zht+MRdY2bVB+HPY6zH7P9NdteYWfVBaJhkPWbf7XXXmFn2QXjxRet93H67M7GgQiBJAAC4DJUEu5iTAAAATFFJAAC4DJUEu6gkAAAAU1QSAAAuww882UWSAABwGZop2cXlBgAAYIpKAgDAZZi4aBdJQinZaZRkp6mLm9hplOSJXFAGkVQctholvfde8AOpQOw0SvJEHi+DSCoQG42SvtnL/8/chCQBAOAyVBLsIiUEAACmqCQAAFyGSoJdVBIAAIApKgkAAJehkmAXSQIAwGXouGgXlxsAAICpkK0keCL7l3cIFQ5jFjjGLHCMWeAYs1BDW2a7qCQAAABTIVtJAAAgOArl/Hfk8Jy4SCUBAACYopIAAHAZKgl2UUkAAACmqCQAAFyGSoJdJAkAAJfhFki7yjxJMAzjv/91pqwPDQAod7/8v//XvwXlF0Po77P8lXmScOLEif/+18qyPjQAIEScOHFC1atXL9NjRkVFKT4+Xjk5K4Ky//j4eEVFRQVl3+XFY5RxOldUVKT9+/erWrVq8ng8ZXlov3Jzc5WYmKjs7GzFxsaWdzgVAmMWOMYscIxZ4EJ5zAzD0IkTJ5SQkKCIiLKfO3/q1CkVFBQEZd9RUVGKiYkJyr7LS5lXEiIiItSgQYOyPqxtsbGxIfehCnWMWeAYs8AxZoEL1TEr6wrCuWJiYsLuD3kwcQskAAAwRZIAAABMkST8V3R0tMaPH6/o6OjyDqXCYMwCx5gFjjELHGMGp5T5xEUAAFAxUEkAAACmSBIAAIApkgQAAGCKJAEAAJgiSfivGTNmKCkpSTExMWrfvr02bNhQ3iGFrLVr16pfv35KSEiQx+PR4sWLyzukkJeRkaErr7xS1apVU926dXXDDTdo165d5R1WSJs5c6ZatWrlbQjUoUMHvffee+UdVoUxefJkeTwejRo1qrxDQQVGkiBp/vz5Gj16tMaPH6/NmzerdevW6tmzpw4dOlTeoYWkvLw8tW7dWjNmzCjvUCqMDz74QCNGjNAnn3yi5cuX6/Tp0+rRo4fy8vLKO7SQ1aBBA02ePFlZWVnatGmTrr32WvXv31/bt28v79BC3saNG/X888+rVatW5R0KKjhugZTUvn17XXnllXruueck/fL7EomJibr33ns1ZsyYco4utHk8Hi1atEg33HBDeYdSoRw+fFh169bVBx98oGuuuaa8w6kwatWqpSeffFLDhg0r71BC1smTJ3XFFVfo73//uyZNmqQ2bdpo+vTp5R0WKijXVxIKCgqUlZWl1NRU77KIiAilpqZq/fr15RgZwtnx48cl/fJHD9YKCws1b9485eXlqUOHDuUdTkgbMWKE+vTp4/P/NOB8lfkPPIWaI0eOqLCwUPXq1fNZXq9ePe3cubOcokI4Kyoq0qhRo3TVVVepRYsW5R1OSNu6das6dOigU6dOqWrVqlq0aJGaN29e3mGFrHnz5mnz5s3auHFjeYeCMOH6JAEoayNGjNC2bdv00UcflXcoIa9p06basmWLjh8/roULF2rIkCH64IMPSBRMZGdna+TIkVq+fDm/cgjHuD5JqFOnjiIjI3Xw4EGf5QcPHlR8fHw5RYVwlZ6ernfeeUdr164N6Z9MDxVRUVFq0qSJJCklJUUbN27U008/reeff76cIws9WVlZOnTokK644grvssLCQq1du1bPPfec8vPzFRkZWY4RoiJy/ZyEqKgopaSkaOXKld5lRUVFWrlyJdc+4RjDMJSenq5FixZp1apVaty4cXmHVCEVFRUpPz+/vMMISd26ddPWrVu1ZcsW76Nt27YaNGiQtmzZQoKA8+L6SoIkjR49WkOGDFHbtm3Vrl07TZ8+XXl5eUpLSyvv0ELSyZMntXv3bu/zPXv2aMuWLapVq5YaNmxYjpGFrhEjRmju3Ll66623VK1aNeXk5EiSqlevrsqVK5dzdKFp7Nix6tWrlxo2bKgTJ05o7ty5WrNmjZYtW1beoYWkatWqFZvjUqVKFdWuXZu5LzhvJAmSBgwYoMOHD2vcuHHKyclRmzZtlJmZWWwyI36xadMmde3a1ft89OjRkqQhQ4Zo9uzZ5RRVaJs5c6YkqUuXLj7LX3nlFQ0dOrTsA6oADh06pMGDB+vAgQOqXr26WrVqpWXLlql79+7lHRrgGvRJAAAAplw/JwEAAJgjSQAAAKZIEgAAgCmSBAAAYIokAQAAmCJJAAAApkgSAACAKZIEAABgiiQBAACYIkkAAACmSBIAAIApkgQAAGDq/wPyf0/VFCjHuwAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def plot_as_patches(j):\n", " # Reorder axes so the diagonals will each form a contiguous patch.\n", " j = tf.transpose(j, [1, 0, 3, 2])\n", " # Pad in between each patch.\n", " lim = tf.reduce_max(abs(j))\n", " j = tf.pad(j, [[0, 0], [1, 1], [0, 0], [1, 1]],\n", " constant_values=-lim)\n", " # Reshape to form a single image.\n", " s = j.shape\n", " j = tf.reshape(j, [s[0]*s[1], s[2]*s[3]])\n", " imshow_zero_center(j, extent=[-0.5, s[2]-0.5, s[0]-0.5, -0.5])\n", "\n", "plot_as_patches(j)\n", "_ = plt.title('All (batch, batch) slices are diagonal')" ] }, { "cell_type": "markdown", "metadata": { "id": "OXpTBKyeK84z" }, "source": [ "To get the desired result, you can sum over the duplicate `batch` dimension, or else select the diagonals using `tf.einsum`:" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:19.901354Z", "iopub.status.busy": "2024-08-15T02:32:19.901103Z", "iopub.status.idle": "2024-08-15T02:32:19.907632Z", "shell.execute_reply": "2024-08-15T02:32:19.906962Z" }, "id": "v65OAjEgLQwl" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(7, 6, 5)\n", "(7, 6, 5)\n" ] } ], "source": [ "j_sum = tf.reduce_sum(j, axis=2)\n", "print(j_sum.shape)\n", "j_select = tf.einsum('bxby->bxy', j)\n", "print(j_select.shape)" ] }, { "cell_type": "markdown", "metadata": { "id": "zT_VfR6lcwxD" }, "source": [ "It would be much more efficient to do the calculation without the extra dimension in the first place. The `tf.GradientTape.batch_jacobian` method does exactly that:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:19.910527Z", "iopub.status.busy": "2024-08-15T02:32:19.910292Z", "iopub.status.idle": "2024-08-15T02:32:20.066210Z", "shell.execute_reply": "2024-08-15T02:32:20.065609Z" }, "id": "YJLIl9WpHqYq" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 5 calls to .f at 0x7f968c10d700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 5 calls to .f at 0x7f968c10d700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "data": { "text/plain": [ "TensorShape([7, 6, 5])" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "jb = tape.batch_jacobian(y, x)\n", "jb.shape" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:20.069156Z", "iopub.status.busy": "2024-08-15T02:32:20.068902Z", "iopub.status.idle": "2024-08-15T02:32:20.074494Z", "shell.execute_reply": "2024-08-15T02:32:20.073908Z" }, "id": "-5t_q5SfHw7T" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.0\n" ] } ], "source": [ "error = tf.reduce_max(abs(jb - j_sum))\n", "assert error < 1e-3\n", "print(error.numpy())" ] }, { "cell_type": "markdown", "metadata": { "id": "IUeY2ZCiL31I" }, "source": [ "Caution: `tf.GradientTape.batch_jacobian` only verifies that the first dimension of the source and target match. It doesn't check that the gradients are actually independent. It's up to you to make sure you only use `batch_jacobian` where it makes sense. For example, adding a `tf.keras.layers.BatchNormalization` destroys the independence, since it normalizes across the `batch` dimension:" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:20.077545Z", "iopub.status.busy": "2024-08-15T02:32:20.077282Z", "iopub.status.idle": "2024-08-15T02:32:20.705360Z", "shell.execute_reply": "2024-08-15T02:32:20.704672Z" }, "id": "tnDugVc-L4fj" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:6 out of the last 6 calls to .f at 0x7f967c72d430> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:6 out of the last 6 calls to .f at 0x7f967c72d430> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "j.shape: (7, 6, 7, 5)\n" ] } ], "source": [ "x = tf.random.normal([7, 5])\n", "\n", "layer1 = tf.keras.layers.Dense(8, activation=tf.nn.elu)\n", "bn = tf.keras.layers.BatchNormalization()\n", "layer2 = tf.keras.layers.Dense(6, activation=tf.nn.elu)\n", "\n", "with tf.GradientTape(persistent=True, watch_accessed_variables=False) as tape:\n", " tape.watch(x)\n", " y = layer1(x)\n", " y = bn(y, training=True)\n", " y = layer2(y)\n", "\n", "j = tape.jacobian(y, x)\n", "print(f'j.shape: {j.shape}')" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:20.708790Z", "iopub.status.busy": "2024-08-15T02:32:20.708535Z", "iopub.status.idle": "2024-08-15T02:32:20.930861Z", "shell.execute_reply": "2024-08-15T02:32:20.930156Z" }, "id": "SNyZ1WhJMVLm" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgkAAAIkCAYAAACUULgQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABnNUlEQVR4nO3de1gUZf8/8PcCAiIuSJxEUFRKREUSEvHIkyRamVZ+U6NU8rGTlGYn7TE1rdAys4NpJ7OD/vTJ1MwM82weUcxK86wkHkARYQXiuPP7o8etleW+Z2UWFvb9uq69Lp373s/cM8wuN5+Z+YxOURQFRERERNdxqusBEBERkX3iJIGIiIgs4iSBiIiILOIkgYiIiCziJIGIiIgs4iSBiIiILOIkgYiIiCziJIGIiIgs4iSBiIiILOIkoR7bsmULdDodli9fXtdD0dSiRYug0+mQmZlpWhYfH4/4+Pg6GxNZxxF/hg19+8gxcZJgZ3Q6narXli1b6nqo1AAVFxdj2rRpPL6ICADgUtcDIHNffvml2f+/+OILrF+/vsry9u3b4/Dhw7U5tDr1448/1vUQHEJxcTFeeeUVAND8r2L+DInqH04S7MxDDz1k9v/du3dj/fr1VZYDcKhJgqura10PQXPFxcXw8PCo62HUmob4MyRq6Hi6oQEwGo147bXXEBwcDHd3d/Tt2xcnTpyo0m/Pnj3o378/vLy84OHhgT59+mDHjh1mfa5evYrx48cjNDQUbm5u8Pf3xx133IH9+/dbHas67733Hjp06AAPDw80a9YMMTExWLJkifA9ls73lpSUYNq0abjlllvg7u6O5s2b47777sPJkyfN9s3cuXPRoUMHuLu7IyAgAI899hiuXLliFmvfvn1ITEyEr68vGjdujNatW+ORRx6Rbsu3336Lu+66C0FBQXBzc0Pbtm0xY8YMVFZWVhl/x44dkZGRgd69e8PDwwMvvfQSAKC0tBRTp05FWFgY3NzcEBISghdeeAGlpaXS9V+L+/vvv+Nf//oXPDw80KJFC7zxxhtV+l68eBGjR49GQEAA3N3d0blzZ3z++eem9szMTPj5+QEAXnnlFdOprWnTpgnHcOjQIdx+++1o3LgxgoOD8eqrr8JoNFoc6z9/hmVlZZgyZQqio6Ph5eWFJk2aoFevXti8eXOV916+fBkPP/ww9Ho9vL29MXLkSPzyyy/Q6XRYtGiRWd9NmzahV69eaNKkCby9vTFo0KAqE+pp06ZBp9PhxIkTGDVqFLy9veHl5YXk5GQUFxeb9f3ss89w++23w9/fH25uboiIiMD8+fOF+4SooWAmoQGYOXMmnJyc8Nxzz6GgoABvvPEGkpKSsGfPHlOfTZs2YcCAAYiOjsbUqVPh5ORk+vL76aef0LVrVwDA448/juXLlyMlJQURERG4fPkytm/fjsOHD6NLly5WxbLk448/xtNPP40hQ4Zg3LhxKCkpwa+//oo9e/bgwQcfVL3NlZWVuPvuu7Fx40YMGzYM48aNw9WrV7F+/XocPHgQbdu2BQA89thjWLRoEZKTk/H000/j9OnTeP/99/Hzzz9jx44daNSoES5evIh+/frBz88PEydOhLe3NzIzM7FixQrpOBYtWgRPT09MmDABnp6e2LRpE6ZMmQKDwYA333zTrO/ly5cxYMAADBs2DA899BACAgJgNBpxzz33YPv27Xj00UfRvn17/Pbbb3j77bdx7NgxrFq1SjqGK1euoH///rjvvvvwwAMPYPny5XjxxRfRqVMnDBgwAADw559/Ij4+HidOnEBKSgpat26Nr7/+GqNGjUJ+fj7GjRsHPz8/zJ8/H0888QTuvfde3HfffQCAyMjIatednZ2Nf/3rX6ioqMDEiRPRpEkTfPTRR2jcuLF03AaDAZ988gmGDx+OMWPG4OrVq/j000+RmJiI9PR0REVFAfhrojdw4ECkp6fjiSeeQHh4OL799luMHDmySswNGzZgwIABaNOmDaZNm4Y///wT7733Hnr06IH9+/cjNDTUrP8DDzyA1q1bIzU1Ffv378cnn3wCf39/zJo1y9Rn/vz56NChA+655x64uLjgu+++w5NPPgmj0YixY8dKt5OoXlPIro0dO1ap7se0efNmBYDSvn17pbS01LT8nXfeUQAov/32m6IoimI0GpWbb75ZSUxMVIxGo6lfcXGx0rp1a+WOO+4wLfPy8lLGjh1b7XisiWXJoEGDlA4dOgj7fPbZZwoA5fTp06Zlffr0Ufr06WP6/8KFCxUAypw5cyyOUVEU5aefflIAKIsXLzZrT0tLM1u+cuVKBYCyd+9e4bgsKS4urrLsscceUzw8PJSSkhKz8QNQFixYYNb3yy+/VJycnJSffvrJbPmCBQsUAMqOHTuE678W94svvjAtKy0tVQIDA5X777/ftGzu3LkKAOWrr74yLSsrK1Pi4uIUT09PxWAwKIqiKJcuXVIAKFOnTpVvvKIo48ePVwAoe/bsMS27ePGi4uXlJf0ZVlRUmB23iqIoV65cUQICApRHHnnEtOybb75RAChz5841LausrFRuv/12BYDy2WefmZZHRUUp/v7+yuXLl03LfvnlF8XJyUkZMWKEadnUqVMVAGbrURRFuffee5WbbrrJbJmln3FiYqLSpk0bs2XXbx9RQ8DTDQ1AcnKy2fneXr16AQBOnToFADhw4ACOHz+OBx98EJcvX0Zubi5yc3NRVFSEvn37Ytu2bab0sLe3N/bs2YPz589bXJc1sSzx9vbG2bNnsXfv3hpt8zfffANfX1889dRTVdp0Oh0A4Ouvv4aXlxfuuOMO0zhzc3MRHR0NT09PU1rb29sbALBmzRqUl5dbNY5//sV89epV5ObmolevXiguLsaRI0fM+rq5uSE5Odls2ddff4327dsjPDzcbIy33347AFhMvV/P09PT7JoVV1dXdO3a1fTzB4C1a9ciMDAQw4cPNy1r1KgRnn76aRQWFmLr1q1Wbfc/43br1s0se+Tn54ekpCTpe52dnU3HrdFoRF5eHioqKhATE2N2eistLQ2NGjXCmDFjTMucnJyq/BV/4cIFHDhwAKNGjYKPj49peWRkJO644w6sXbu2yhgef/xxs//36tULly9fhsFgMC3758+4oKAAubm56NOnD06dOoWCggLpdhLVZzzd0AC0bNnS7P/NmjUDANN59+PHjwOAxfTsNQUFBWjWrBneeOMNjBw5EiEhIYiOjsadd96JESNGoE2bNlbHsuTFF1/Ehg0b0LVrV4SFhaFfv3548MEH0aNHD5Vb+5eTJ0+iXbt2cHGp/hA+fvw4CgoK4O/vb7H94sWLAIA+ffrg/vvvxyuvvIK3334b8fHxGDx4MB588EG4ubkJx3Ho0CFMnjwZmzZtMvvFAqDKL5AWLVpUuXjv+PHjOHz4sOlagOrGKBIcHGyaGF3TrFkz/Prrr6b///HHH7j55pvh5GT+d0H79u1N7Tfijz/+QGxsbJXl7dq1U/X+zz//HG+99RaOHDliNkFr3bq12TqaN29e5SLPsLCwKmOpbt3t27fHunXrUFRUhCZNmpiWiz47er0eALBjxw5MnToVu3btqnK9QkFBAby8vFRtK1F9xElCA+Ds7GxxuaIoAGD6y/7NN980nee9nqenJ4C/ztH26tULK1euxI8//og333wTs2bNwooVKzBgwACrYlnSvn17HD16FGvWrEFaWhq++eYbfPDBB5gyZYrp1jutGI1G+Pv7Y/HixRbbr/1ivlaQavfu3fjuu++wbt06PPLII3jrrbewe/fuarcnPz8fffr0gV6vx/Tp09G2bVu4u7tj//79ePHFF6tkVCydpzcajejUqRPmzJljcR0hISHS7ZT9/O3VV199hVGjRmHw4MF4/vnn4e/vD2dnZ6SmpppdfGpLsn138uRJ9O3bF+Hh4ZgzZw5CQkLg6uqKtWvX4u233xZmzYgaAk4SHMC1i/j0ej0SEhKk/Zs3b44nn3wSTz75JC5evIguXbrgtddew4ABA6yOZUmTJk0wdOhQDB06FGVlZbjvvvvw2muvYdKkSXB3d1e9TXv27EF5eTkaNWpUbZ8NGzagR48eqi6k69atG7p164bXXnsNS5YsQVJSEpYuXYp///vfFvtv2bIFly9fxooVK9C7d2/T8tOnT6vahmtj/OWXX9C3b98q2QAttWrVCr/++iuMRqNZNuHaKZFWrVoBgNVjaNWqlSm79E9Hjx6Vvnf58uVo06YNVqxYYbbeqVOnVlnH5s2bq9wyev0dPNe2wdK6jxw5Al9fX7MsghrfffcdSktLsXr1arOsg5rTQEQNAa9JcADR0dFo27YtZs+ejcLCwirtly5dAvDXHQPXp8j9/f0RFBRkuh1PbazqXL582ez/rq6uiIiIgKIoVl0PcP/99yM3Nxfvv/9+lbZrfwU+8MADqKysxIwZM6r0qaioQH5+PoC/UsvX/9V9LUsiug3x2l+h/3xvWVkZPvjgA9Xb8cADD+DcuXP4+OOPq7T9+eefKCoqUh1L5M4770R2djaWLVtmWlZRUYH33nsPnp6e6NOnDwCYfglf2zdq4u7evRvp6emmZZcuXao2e/NPlvbfnj17sGvXLrN+iYmJKC8vN9tHRqMR8+bNM+vXvHlzREVF4fPPPzcb/8GDB/Hjjz/izjvvVLVNsjEWFBTgs88+szoWUX3ETIIDcHJywieffIIBAwagQ4cOSE5ORosWLXDu3Dls3rwZer0e3333Ha5evYrg4GAMGTIEnTt3hqenJzZs2IC9e/firbfesipWdfr164fAwED06NEDAQEBOHz4MN5//33cddddaNq0qeptGjFiBL744gtMmDAB6enp6NWrF4qKirBhwwY8+eSTGDRoEPr06YPHHnsMqampOHDgAPr164dGjRrh+PHj+Prrr/HOO+9gyJAh+Pzzz/HBBx/g3nvvRdu2bXH16lV8/PHH0Ov1wl8s3bt3R7NmzTBy5Eg8/fTT0Ol0+PLLL61K8z/88MP473//i8cffxybN29Gjx49UFlZiSNHjuC///0v1q1bh5iYGNXxqvPoo4/iww8/xKhRo5CRkYHQ0FAsX74cO3bswNy5c037vnHjxoiIiMCyZctwyy23wMfHBx07dkTHjh0txn3hhRfw5Zdfon///hg3bpzpFshrmQuRu+++GytWrMC9996Lu+66C6dPn8aCBQsQERFhNgEdPHgwunbtimeffRYnTpxAeHg4Vq9ejby8PADm2Y8333wTAwYMQFxcHEaPHm26BdLLy0ta78GSfv36wdXVFQMHDsRjjz2GwsJCfPzxx/D398eFCxesjkdU79TVbRWkjppbIL/++muz5adPn65ya5iiKMrPP/+s3HfffcpNN92kuLm5Ka1atVIeeOABZePGjYqi/HXr3PPPP6907txZadq0qdKkSROlc+fOygcffFBl3bJY1fnwww+V3r17m97Xtm1b5fnnn1cKCgpMfdTcAqkof92a9p///Edp3bq10qhRIyUwMFAZMmSIcvLkSbN+H330kRIdHa00btxYadq0qdKpUyflhRdeUM6fP68oiqLs379fGT58uNKyZUvFzc1N8ff3V+6++25l3759wm1RFEXZsWOH0q1bN6Vx48ZKUFCQ8sILLyjr1q1TACibN282G391t36WlZUps2bNUjp06KC4ubkpzZo1U6Kjo5VXXnnFbL9YUl3ckSNHKq1atTJblpOToyQnJyu+vr6Kq6ur0qlTpyrHiKIoys6dO5Xo6GjF1dVV1e2Qv/76q9KnTx/F3d1dadGihTJjxgzl008/lf4MjUaj8vrrryutWrVS3NzclFtvvVVZs2aNxbFfunRJefDBB5WmTZsqXl5eyqhRo5QdO3YoAJSlS5ea9d2wYYPSo0cPpXHjxoper1cGDhyo/P7772Z9rt0CeenSJbPllo691atXK5GRkYq7u7sSGhqqzJo1y3QLruwYJarvdIpi51c3ERFZsGrVKtx7773Yvn271XfHEJE6nCQQkd37888/zS4+raysRL9+/bBv3z5kZ2erujCViKzHaxKIyO499dRT+PPPPxEXF4fS0lKsWLECO3fuxOuvv84JApENMZNARHZvyZIleOutt3DixAmUlJQgLCwMTzzxBFJSUup6aEQNGicJREREZBHrJBAREZFFnCQQERGRRbV+4aLRaMT58+fRtGlTm5ahJSIi+6MoCq5evYqgoKAqDxyrDSUlJSgrK7NJbFdXV9Wl5euLWp8knD9/XtVDa4iIqOHKyspCcHBwra6zpKQEfo0bo2pBeW0EBgbi9OnTDWqiUOuThL9L7/ati9UTEVGdqgCw0aoy7FopKytDIYBnAIgfAm+9UgBvZ2ejrKyMk4Sa+PsUgwsAy0/vIyKihq0uTzc3BqD1r/GGeoFfQ90uIiIiqiHm+4mIyKE4Qfu/kBvqX9ycJBARkUPhJEG9hrpdREREVEPMJBARkUNhJkG9hrpdREREVEN2m0k4efI7YbuaGhxnz4rbDxyQx7iv4zFhe3HwLdIYHid+FbbrOv9HPhBVvhC2Kt9slofo1k3cXlEhj5GbK2739ZXHkPzwdD1S5TFUUH55TdheFh6pyXpk1OzWmmrSZKAmcZStzwvbTwX3lsZwkXzzXLggH0dsp2Jxh0J5yZw8F39h+003abPPSkvF32cHD8pjdAnNE7bnlPtIY8hKE2RmyscRkb9T2K7VZ9OWmElQr6FuFxEREdWQ3WYSiIiIbIGZBPUa6nYRERFRDTGTQEREDoWZBPUa6nYRERFRDTGTQEREDoWZBPU4SSAiIoeig/a/1OvumZa2ZbeThDbe4nuCMfcTaQyXYS8I28PD5eMoCxXXQcjNlsdo2bGjvJMGlD3Hhe262EHyGAVFNR+IrE6CGrIb6TUiq4Pguk98TzgAGLt1r/E4ZHUS1NRRqK1H2Oe0E9dBaPP9QmkMw5BHhO3e3irGcdVD2O7nJ24HgEJJLRWtyH5+XTJXSGOUdbxP2N64xJoRWRYYqKJTuKSWCjUodjtJICIisgUdtP/Lv6FmEhrqaRQiIiK7Nm/ePISGhsLd3R2xsbFIT08X9p87dy7atWuHxo0bIyQkBM888wxKSjRIIQkwk0BERA7F+X8vrWNaY9myZZgwYQIWLFiA2NhYzJ07F4mJiTh69Cj8/auWC1+yZAkmTpyIhQsXonv37jh27BhGjRoFnU6HOXPmaLMRFjCTQEREVMvmzJmDMWPGIDk5GREREViwYAE8PDywcKHla3p27tyJHj164MEHH0RoaCj69euH4cOHS7MPNcVJAhERORQnG70AwGAwmL1KS0urrL+srAwZGRlISEj4e0xOTkhISMCuXbssjrl79+7IyMgwTQpOnTqFtWvX4s4776zJrpDiJIGIiEgjISEh8PLyMr1SU6s+FTM3NxeVlZUICAgwWx4QEIDsbMu3zD344IOYPn06evbsiUaNGqFt27aIj4/HSy+9ZJPtuIbXJBARkUOxZTGlrKws6PV603I3NzdN4m/ZsgWvv/46PvjgA8TGxuLEiRMYN24cZsyYgZdfflmTdVjCSQIRETkUW04S9Hq92STBEl9fXzg7OyMnJ8dseU5ODgKrKVbx8ssv4+GHH8a///1vAECnTp1QVFSERx99FP/5z3/g5GSbEwN2O0k4lusj7jBYXCgJAG4ZGidsz/ve8rmffzorKbYSGioNgZ27a+msjq+vsFk5dFQa4p6HIoTtqyfKiwuhmwbFVvLzax5DBVmRm5KO8kJJehiF7cUl8p+/R4VB3EFFNSVDieQzo5GCAnF702HiQkkAoG/bXNj+67oL0hienuJ22TiBWqvZBY/Ci8L24v7iQkkA4LHqv8L2irsfkMaoJpNtoqaI1a8HeZa6plxdXREdHY2NGzdi8ODBAACj0YiNGzciJSXF4nuKi4urTAScnf+6p0JRFJuN1W4nCURERLZgD89umDBhAkaOHImYmBh07doVc+fORVFREZKTkwEAI0aMQIsWLUzXNAwcOBBz5szBrbfeajrd8PLLL2PgwIGmyYItcJJARERUy4YOHYpLly5hypQpyM7ORlRUFNLS0kwXM545c8YsczB58mTodDpMnjwZ586dg5+fHwYOHIjXXnvNpuPkJIGIiByKPWQSACAlJaXa0wtbtmwx+7+LiwumTp2KqVOn3sCabhxPLhEREZFFzCQQEZFDsZdMQn3QULeLiIiIaoiZBCIicih8VLR6NzRJmDdvHt58801kZ2ejc+fOeO+999C1a1dNByarT9CkiTyGYZ24DoJPzC3SGC77jslXJNG2bY1DqKPiXnqZ1eM3CdtfWHW7NMa0KHG7mnvTXeVd7IasDoK0BgKgyQ377u41DqFKYWHNx7H/e3EdhC6J4joKAHA+QxwjN1c+DjV1TjQh2SlXr6qIIamD4LFGXEcBAJr0EceQ/WwBILKjuC5IfeAE7Z8C2VDT8lZv17XHW06dOhX79+9H586dkZiYiIsXxcVCiIiIqH6xepJg7eMtiYiI7IktnwLZ0Fi1XTfyeEsiIiKqn6w6ESp6vOWRI0csvqe0tNTsedoGg4rzs0RERDbCWyDVs/l2paammj1bOyQkxNarJCIiIg1YNUm4kcdbTpo0CQUFBaZXVlbWjY+WiIiohnhNgnpWbdc/H295zbXHW8bFWX4ss5ubm+n52mqes01ERET2weqbs2WPtyQiIrJnvCZBPasnCbLHW2rl9qg8YXuZp480hqy2UNlBeaEkfb64/kOxi780RuPG0i7aqOaUj4maYkvh4cLmN+LlhVSWfS3+uAy97ZR8HLJt0YgWBYicINsnnvIgkio2xS7yDFxtlU/tEl4sbC8u8ZDGkBxmKD4pLpQEAEHZ4uMosGMbaQw1xYO0UOYu/vn5qThEZB9f4xBxoSQACDgh/s4zhskLzMmKh9UHnCSod0PfK6LHWxIREVHDwGc3EBGRQ2EmQb2Gul1ERERUQ8wkEBGRQ+FTINVjJoGIiIgsYiaBiIgcijO0f1S01vHsBTMJREREZFG9zSTk5sr7aHGrvcFdXAdBn/m7NEZxaETNB1JbZDeOqygsMDRkn7D9g7Tu0hhPDj4v7VMbSkrkfTxcJDewu8g/ZgaI76PXF4rrdQBAsae8Zocm8vOFzS6+8joJV67UfBguweI6CK6Z8nocLoHyWgpaUFOipKbUHKsuoeI6CK770qUxPGJi1A7Jbumg/V/IDfWahHo7SSAiIroRvAVSvYa6XURERFRDzCQQEZFDYSZBvYa6XURERFRDzCQQEZFDYSZBvYa6XURERFRDzCQQEZFDYSZBvYa6XURERFRD9ptJ8PQUNgd5G6UhzmeL50CSVahSFiYvlORxQl5wSROyQkhqKrr4+tY8RnCwsPnJUHmhpCdmBMnXowHZ5qioHQUjXIXtTpAfq7L1FLvICyV5ZMuLB2khz138s3FRUdSnUSNxu5r9Lise5BIqL5Tk8cP38hVpwAPFwnaji7wAlaSGFXxUfCfm5Yu/E33UFEpSU7XJzjGToJ79ThKIiIhsgE+BVK+hTn6IiIiohphJICIih8KnQKrHTAIRERFZxEwCERE5FF64qF5D3S4iIiKqIWYSiIjIoeig/V/IDfXuBrudJOjc7q/rIdQ7uhaP1fUQ6h03t4F1PYR656abuM+spWsytK6HQHRD7HaSQEREZAu8JkE9ThKIiMihcJKgXkPdLiIiIrs2b948hIaGwt3dHbGxsUhPTxf2z8/Px9ixY9G8eXO4ubnhlltuwdq1a206RmYSiIjIodhDJmHZsmWYMGECFixYgNjYWMydOxeJiYk4evQo/P2rPqulrKwMd9xxB/z9/bF8+XK0aNECf/zxB7y9vTUZf3U4SSAiIqplc+bMwZgxY5CcnAwAWLBgAb7//nssXLgQEydOrNJ/4cKFyMvLw86dO9Hof09ICw0Ntfk4ebqBiIgcipONXgBgMBjMXqWlpVXWX1ZWhoyMDCQkJPw9JicnJCQkYNeuXRbHvHr1asTFxWHs2LEICAhAx44d8frrr6OysrKGe0OMkwQiIiKNhISEwMvLy/RKTU2t0ic3NxeVlZUICAgwWx4QEIDs7GyLcU+dOoXly5ejsrISa9euxcsvv4y33noLr776qk224xqebiAiIodiy0dFZ2VlQa/Xm5a7ublpEt9oNMLf3x8fffQRnJ2dER0djXPnzuHNN9/E1KlTNVmHJZwkEBERaUSv15tNEizx9fWFs7MzcnJyzJbn5OQgMDDQ4nuaN2+ORo0awdn57+dNtm/fHtnZ2SgrK4Orq2vNB28BTzcQEZFDcbbRSy1XV1dER0dj48aNpmVGoxEbN25EXFycxff06NEDJ06cgNFoNC07duwYmjdvbrMJAsBJAhERORhbXrio1oQJE/Dxxx/j888/x+HDh/HEE0+gqKjIdLfDiBEjMGnSJFP/J554Anl5eRg3bhyOHTuG77//Hq+//jrGjh17YztBJZ5uICIiqmVDhw7FpUuXMGXKFGRnZyMqKgppaWmmixnPnDkDJ6e/px4hISFYt24dnnnmGURGRqJFixYYN24cXnzxRZuOU6coimLTNVzHYDDAy8sLQCKARrW5aiIiqnPlANahoKBAeu5ea9d+/+wH4Klx7EIAXYA62S5b4ukGIiIisoinG4iIyKHooP1fyFrfUmkvmEkgIiIii5hJICIih2IPD3iqL+x2klBZ+Z2w3amirOYrKSyUdinz9BG2u7oYhe0AUFYhPnzc3AZKY6iRnS3eZwHNVOwzF8khUVFR4xiy/QEArhCPVed2v3wcKvzyi3ifRYar2GeyfaJmn7m7y/vUkFb77Nw58T4LCpR/JpCfL24vKZGGKPMNErar+WzK6JwH1TgGACiV34o7qDhGjC7ie+FVfSfKPt8qvhMNEF+U5+WlzfcZ2Qe7nSQQERHZAjMJ6jXU7SIiIqIaYiaBiIgcCjMJ6nGSQEREDsWWT4FsaBrq5IeIiIhqiJkEIiJyKNY+tVFtzIaImQQiIiKyyG4zCU4Q3+M8YJD8+dk/vPV7jcfh6im5b1jF/dyu3t41HocaATm/CttvHx4pjbHpq/PiDrL72wFAsr0lnuL72wHA9cgB+Xo0EOkiPkZ0bjdLYyhHz4o7ZGfLBxIYKG5XcwxlZsr7aMDXV9yuc86SxlDWHBR3uHxZGsO1Z09xBxWfTVU1LGrBrLfl32cvPmEQd8jNla9I9sNTEaPCu/4/vIgXLqrXULeLiIiIashuMwlERES2wEyCeg11u4iIiKiGmEkgIiKHwkyCepwkEBGRQ9FB+1/qLKZEREREDoWZBCIicig83aBeQ90uIiIiqiH7zSTs3i1s/mFZR2kInVeRsF1Zd0U+juBgcbuaYiwHDsj7aEFSkEdaKAnAA+PFhY6WLpUXQnI6KC7qpFezzzrKf76akBQ6Us55S0PoWoj3u5IhKcgFAC6Sj6KawkCyY1Ujru/PEbYrvyRIY+g6HxDHWHebfCCFkv3q6SmPoabQlQYMheK/x158qlgaY8pMcRGj6RNVfJ3LCm65u0tD+GTul6/HzjGToF5D3S4iIiKqIfvNJBAREdkAMwnqNdTtIiIiohpiJoGIiBwKMwnqcZJAREQOhZME9RrqdhEREVENMZNAREQOhZkE9ayeJGzbtg1vvvkmMjIycOHCBaxcuRKDBw/WfmS5ueJ22X3lAJRvLwjbdYkZ8hg7EsUdTpyQxigbNkLS4z1pDFXOnq1xiP8u8ha2L1nqIY0xZEiksF12ezsA+OzbJu+khQviYwTe3tIQyh5x3Qdd9K3SGJWVjaR9ZJx++L7GMVTZt0/cHhUlDaGsEffRJX4sj5EzT9xBxXeEms+vFqSlQfIl33cApj/nLWyfMFlcRwEA5oyX1I5Q8R1i6Nhd2ocaDqsnP0VFRejcuTPmzZN8QImIiOyQk41eDZHVmYQBAwZgwIABthgLERER2RFek0BERA6F1ySoZ/NJQmlpKUpLS03/NxgMtl4lERERacDmk5/U1FR4eXmZXiEhIbZeJRERUbV0AHQ6nbavut4oG7H5JGHSpEkoKCgwvbKysmy9SiIiouq5uNjm1QDZfKvc3Nzg5uZm69UQERGRxqyeJBQWFuLEP+4tPn36NA4cOAAfHx+0bNlS08ERERFpzsUF0Gl8gkBRVBTEqH+sniTs27cP//rXv0z/nzBhAgBg5MiRWLRokWYDW417hO33HNkvDyIphKOs6yEN8cIqceGQV1+VFxZxTVst7aOJjh3F7WqqGEmKy8gKJQFAfLy4feeCX+XjCAuT99GC7HbezEx5DHd3YbNSVC4N0b6DuJjS4W9+l4/jVnnRJi180X+JsH2E+055kD//FDYrv7wsDfHgeH9h+5LJKvaZp6S4kEZ8XCQXbAcGyoNIjsU5z8k/349OE/8ht2CB/A89/UEVn19qMKyeJMTHx0NRFFuMhYiIyPZslUlogBrqrZ1ERERUQ5wkEBGRY7GTuxvmzZuH0NBQuLu7IzY2Funp6aret3TpUuh0Ots8N+k6nCQQERHVsmXLlmHChAmYOnUq9u/fj86dOyMxMREXL14Uvi8zMxPPPfccevXqVSvj5CSBiIgci7Oz9lkEZ2erhjBnzhyMGTMGycnJiIiIwIIFC+Dh4YGFCxdW+57KykokJSXhlVdeQZs2bWq6F1ThJIGIiByLDU83GAwGs9c/H0twTVlZGTIyMpCQkGBa5uTkhISEBOzatavaYU+fPh3+/v4YPXq09vukGpwkEBERaSQkJMTsUQSpqalV+uTm5qKyshIBAQFmywMCApCdnW0x7vbt2/Hpp5/i448/tsm4q2O3dSSlt8l7+sqDyO49PnhQGmLyZHH77NnyYbyUEi/poc0P3VDiKmx39/SRxnB1sXyAXlPN8Wtm52zxffLFYfLaEpLSA9qRFT8JDZXHkOwUQ4WHNMThxeK6H8t+6yKNMfT/jNI+WoiJkfXwlgeR1DBBcLA0xJJpx4TtX+yOkMYY0V98/lczkvojquqCyL7PVNRBef99cfvEifJhTJ4sr5Vi91xcACeN/0Y2/vX5y8rKgl6vNy3WouLw1atX8fDDD+Pjjz+Gr6+K330asttJAhERUX2j1+vNJgmW+Pr6wtnZGTk5OWbLc3JyEGhhMnjy5ElkZmZi4MCBpmXG/01KXFxccPToUbRt21aD0VfFSQIRETkWG2YS1HB1dUV0dDQ2btxouo3RaDRi48aNSElJqdI/PDwcv/32m9myyZMn4+rVq3jnnXds+nRlThKIiIhq2YQJEzBy5EjExMSga9eumDt3LoqKipCcnAwAGDFiBFq0aIHU1FS4u7uj43Vl973/d8ru+uVa4ySBiIgci7Oz1bcsSlVWWtV96NChuHTpEqZMmYLs7GxERUUhLS3NdDHjmTNn4KR1tuMGcJJARERUB1JSUiyeXgCALVu2CN+r5QMVRThJICIix3IDxY+ktH5glJ3gJIGIiBwLJwmq1f0JDyIiIrJLdptJiAgrE3co9JQHWbNG3H733dIQ+oPiwkATJ8oLAz0xVnzPrFb0Z38XdygpkQeRFA9qeWCTPEZ4uLDZw0XyswXQJUZcGEozubk1awek+0y/ZbU8hqRC0dCO8n3m41s7+yzCRVzECJL6VACAI0fE7fv2yWP07ClsHjGkWBpC18S6i81u1I+54mJY8SouUHfdLv7slfW8XR5j9zZh+7RpvaUxPI6IC3/VC8wkqMZMAhEREVlkt5kEIiIim2AmQTVmEoiIiMgiZhKIiMixODubHu1MYswkEBERkUWcShERkWNxcWEmQSXuJSIiciycJKimUxRFqc0VGgwGeHl5AUgE0Kg2V01ERHWuHMA6FBQUQK+vnRoy11z7/VPQqxf0Gk8SDBUV8PrppzrZLlviVIqIiBwLMwmq8cJFIiIisohTKSIiciy2uAWyds/c1xpmEoiIiMgiZhKIiMix2OKaBGYSiIiIyJEwk0BERI6FmQTVOEkgIiLHwkmCajzdQERERBYxk0BERI6FmQTVmEkgIiIii5hJICIix2KLYkpGo7bx7AQzCURERGQRMwlERORYbHFNAjMJRERE5EiYSSAiIsfCTIJqzCQQERGRRXabSfjjj++E7YGB8hiuKBO2n891lcYIChTPDotL5PMsj8zfhe26Di9KY6iRkSHeZ106ivcHABhdxPvEqaRYPpCSEnG7u7s8hqSPznmQPIYKO3aI91n3GPk+Q26uuN3bWxrC6O4hbHc6+Kt8HOHhwmad2/3yGCrIPpuenvIYPsgTd1Cxz8oqxJ891xPizx0AIDRU2KxrMlQeQ4WNG8X7LCxMHqNlvvgYKAuPtGZIFrlmn6lxDF2rsTWOYXPMJKhmt5MEIiIim+AkQTWebiAiIiKLmEkgIiLHYotiSpWV2sazE8wkEBERkUXMJBARkWOxxTUJzCQQERGRI2EmgYiIHAszCarZ7SShpa/kfvySCmmMMne9sL1CHgIoLBTHgHgdAFTd862FLoHnxR3OSuoXACgJbCNs91DxwTJ6+wjb8/OlIeDjoqI+gQZiYsTtL02T19J4fZT4GJHWUQDg5Osr7iC5nx+Auh1bC8rLVXSqlByLZ89KQ7jKiqUEB0tjyOpTaEX242uZ9pE8yKhRwmY1v/Nk33l5ni2lMWrp64zshN1OEoiIiGyCmQTVOEkgIiLHYotbIFWlpusfXrhIREREFnGSQEREjuXa6QatX1aaN28eQkND4e7ujtjYWKSnp1fb9+OPP0avXr3QrFkzNGvWDAkJCcL+WuEkgYiIqJYtW7YMEyZMwNSpU7F//3507twZiYmJuHjxosX+W7ZswfDhw7F582bs2rULISEh6NevH86dO2fTcXKSQEREjsUOMglz5szBmDFjkJycjIiICCxYsAAeHh5YuHChxf6LFy/Gk08+iaioKISHh+OTTz6B0WjExo0btdgj1eIkgYiISCMGg8HsVVpaWqVPWVkZMjIykJCQYFrm5OSEhIQE7Nq1S9V6iouLUV5eDh8f8S3nNcVJAhERORYbZhJCQkLg5eVleqWmplZZfW5uLiorKxEQEGC2PCAgANnZ2ao24cUXX0RQUJDZRMMW6u0tkEZPeREj17NnhO3BwfLCIeezxesJCpQ/QzznzyBpHy0YA8XrkdSFAgDkSo7PNr7ygkxOEPfxUXHUGV1UFKnSgGu2+Bh5faK3NMYTL94ibJ8/T8Vz5mU/HBXFhRAWJu+jgZYukqJdTb2lMdZuER+rWnzvuZbkS/tUSAquaaVNrvgCs7JRj0pjHDwobu8SLilAByA3X1w8KsjF8vnwf/r9iL+0jyPLysqCXv/3ceXm5qb5OmbOnImlS5diy5YtcHd31zz+P9XbSQIREdENsUUxpf/F0+v1ZpMES3x9feHs7IycnByz5Tk5OQiUVBKdPXs2Zs6ciQ0bNiAyMrJmY1aBpxuIiMixXCumpOXL2Vn16l1dXREdHW120eG1ixDj4uKqfd8bb7yBGTNmIC0tDTGymvIaYSaBiIiolk2YMAEjR45ETEwMunbtirlz56KoqAjJyckAgBEjRqBFixamaxpmzZqFKVOmYMmSJQgNDTVdu+Dp6QlPT0+bjZOTBCIiciw2PN2g1tChQ3Hp0iVMmTIF2dnZiIqKQlpamulixjNnzsDJ6e9k//z581FWVoYhQ4aYxZk6dSqmTZtW4+FXh5MEIiKiOpCSkoKUlBSLbVu2bDH7f2Zmpu0HZAEnCURE5FjsIJNQX/DCRSIiIrLIqqlPamoqVqxYgSNHjqBx48bo3r07Zs2ahXbt2mk+sDIX8f28Vy7JYwRIbiVx+n+LpTGcE5KE7YZC+TzLz0/apVaoeZJpm0DxvdY/bpffV94vJk/coURNrYVaIhtLbq40xPzXxDv2viHyimgrlmpwr7OKsWpCdiCp+PnGx4s/32vWyIfRv7+43VXFxVyuhZJjVSuSe9mvXJGH6BIork9xPl9ej0V6Ab63tzRGqO2ukas9zCSoZtV38datWzF27Fjs3r0b69evR3l5Ofr164eioiJbjY+IiIjqiFVTn7S0NLP/L1q0CP7+/sjIyEDv3r01HRgREZFNXKuToHXMBqhGe6mgoAAAbP6ACSIiIs3wdINqN7xVRqMR48ePR48ePdCxY8dq+5WWlpo9BctgMNzoKomIiKgW3fAkYezYsTh48CC2b98u7JeamopXXnnlRldDRESkLWYSVLuhi8hTUlKwZs0abN68GcHBwcK+kyZNQkFBgemVlZV1QwMlIiKi2mXV1EdRFDz11FNYuXIltmzZgtatW0vf4+bmZpNHZRIREd0QZhJUs2qrxo4diyVLluDbb79F06ZNTQ+Y8PLyQuPGjW0yQCIiIqobVk0S5s+fDwCIj483W/7ZZ59h1KhRWo0JgHxSFtBUXPQHAM7nigu2+P6fuFASAATIiq2czZbGKA6NkPbRgtOJY8J2n9BQeZD8QmFzv3j5IVMG8d0uW/bJhyG4FlZT5z1vEbYHQVzABgCQLT4GVswV71MAmD6zpbB9ykMqii1JCvZo5VSFeKxtPMukMTwkn6v74tWMRLK9KqqHGVxq586sU56RwvY2fkZpjJxL4mJJQc3k+x2SBwEZX31dGuK3n+WrsXvMJKhm9ekGIiIicgwNc+pDRERUHRZTUo2TBCIiciw83aAanwJJREREFjXMqQ8REVF1mElQjZkEIiIisqhhTn2IiIiqw0yCana7VU6F4gdB5fypl8YI8lVx37BMfr6w+YynvAZCy+xTNR+HGpIS2YYSV2kIfaHknn5fX2mMgwfE7f1cNklj/Jp7u7SPFoI2fCFsNz40QhrDafducYeePaUxpgz5XbyOMPlxZqyQ32uvhTa7Fos7DB8uD3L2rLhdTaEMybH66HPy74iPFtTSPlvzrrD91N1Py2NUiOugGP3ENT8AoHCiuA7CmqXSEHhwWO3sM7IPdjtJICIisgneAqkar0kgIiIii5hJICIix8JrElRrmFtFRERUHU4SVOPpBiIiIrKoYU59iIiIqsNMgmrMJBAREZFFDXPqQ0REVB1mElSz362qqBA2B1Sel4Y4lhkkbA8LUzGM4DbC9pa58nGsPiCOoRlJ4Se9r4ofd2hojdYBAIGBPsL2Y57yQkmREBeO0cqZeHGxpJaZKgphdesmbs/Olsfw9hY2G0vkhcFWrJIXy9LCqbgkYXubkmJ5EFmxpMxMaQhjqPhztWCBfBhlFbWTTD3WX1ws6RbPi9IY+8+KiyV1yc+TxsjNF382VRVK2rJF3ocaDPudJBAREdkCiympxmsSiIiIyCJmEoiIyLHwmgTVGuZWERERVYeTBNV4uoGIiIgsaphTHyIiouowk6AaMwlERERkkd1OfXQ3jazrIdQ7uhaP1fUQ6p1WrQbW9RDqnbZtuc+s1a4d95ld4S2QqjGTQERERBbZbSaBiIjIJnhNgmrMJBAREZFFDXPqQ0REVB1mElRjJoGIiBzLtUmC1i8rzZs3D6GhoXB3d0dsbCzS09OF/b/++muEh4fD3d0dnTp1wtq1a290D6jGSQIREVEtW7ZsGSZMmICpU6di//796Ny5MxITE3HxouUngu7cuRPDhw/H6NGj8fPPP2Pw4MEYPHgwDh48aNNx6hRFUWy6husYDAZ4eXkBSATQqDZXTUREda4cwDoUFBRAr9fX6pqv/f4puHJF83UbDAZ4NWumertiY2Nx22234f333wcAGI1GhISE4KmnnsLEiROr9B86dCiKioqwZs0a07Ju3bohKioKC9Q8F/0GMZNARESkEYPBYPYqLS2t0qesrAwZGRlISEgwLXNyckJCQgJ27dplMe6uXbvM+gNAYmJitf21wkkCERE5FCOcbPICgJCQEHh5eZleqampVdafm5uLyspKBAQEmC0PCAhAdna2xTFnZ2db1V8rDfNyTCIiojqQlZVldrrBzc2tDkdTc5wkEBGRQ6mo+OuldUwA0Ov10msSfH194ezsjJycHLPlOTk5CAwMtPiewMBAq/prhacbiIiIapGrqyuio6OxceNG0zKj0YiNGzciLi7O4nvi4uLM+gPA+vXrq+2vFWYSiIjIodgyk6DWhAkTMHLkSMTExKBr166YO3cuioqKkJycDAAYMWIEWrRoYbqmYdy4cejTpw/eeust3HXXXVi6dCn27duHjz76SNsNuQ4nCURE5FDsYZIwdOhQXLp0CVOmTEF2djaioqKQlpZmujjxzJkzcHL6O9nfvXt3LFmyBJMnT8ZLL72Em2++GatWrULHjh213IwqWCeBiIhqUd3XScjK0n7dBoMBISFedbJdtsRMAhERORR7yCTUF7xwkYiIiCyy20yCcu5DcYfcXGmMPX9GCttjby2TD6SwUNx+9qw0xNqz4nHcdddA+ThUuII1wvYVn8rPLF1X0KuKCxfk42jWTNx+S6BBGmPtdnG6Tqt9dkiyzzz/kO+z5cvF7Z6e8nEMGSJuX7VKHqNbN3F7hw7a7DOlm+Sz9+mn0hgvLIoQtqvZZ1OGHRO2/1pyizSG7Jk8mu2zQ7PEHY4ckcbYGXifsL27+375QMLDhc2/nvCQx5Do3FmbfWZLlZXa/+VfWaltPHvBTAIRERFZZLeZBCIiIlvgNQnqMZNAREREFjGTQEREDoWZBPWYSSAiIiKLmEkgIiKHwkyCepwkEBGRQ+EtkOrZ7yRB8vjLHw8GSUP0i5fUQZDVQAAAb295H4n4sBqHUOX8IfE9/Y+M7yeNkTf4R2F7SIh8HLIPX5m7vGRpcLB8PVoILhDvM/3sKdIYwx6fLmxXc5iVl0vWMUweIz9f3kcT338vbv/hB2mIxx8X10koKZEPozhYXAchWEUMWZ0Ezfj6CptfPyKugQAAL/UvFneoUPFFI/lwRgbnyWPU2k4je8CfNhERORSeblCPFy4SERGRRcwkEBGRQ2EmQT1mEoiIiMgiZhKIiMihMJOgHjMJREREZBEzCURE5FCYSVCPkwQiInIoLKaknt1OErZtF58J6Rd2Sh7kiLiKze8ukdIQEbnHhO05XuKCLgAQkLlf2kcLnp7i9l9niwslAUDYTTphe+Ef4uJDAHD2rLhdzYeztuq1fPKJuH3Iv8WFkgDgxxbifdZmo3yfyYoH7dghDYEePeR9tPDEf3yE7c88kySNcaKteJ9575DvM9lxlJ0tDVFrRbv2nPYXtr80TP599uP2NsL2ft0M8oEcOSJsPhPYVRqi5YaF8vVQg2G3kwQiIiJb4OkG9XjhIhEREVnETAIRETkUZhLUYyaBiIiILGImgYiIHAozCeoxk0BEREQWMZNAREQOhXUS1LNqkjB//nzMnz8fmZmZAIAOHTpgypQpGDBggOYDi4mRdCjxlsYoDhTfVxxRkieNYfQW10FoKrm/HQDQsaOKTjXn7S1ul9VRAAAUie9Pb5n5uzREy27hKlYklnOpdpJcQ4aI22X7FADuuyzeZz5rvpDGMMaPELaHhcnHIau1oJVnnhG3BwbKYwQWiPeZfrmKe/G7jRI2R6g4DA2FtXOcxd5mFHcoke+0fqHiGGfO6qUxWkZFidtdJOMEgIceErePXimPUcd4ukE9qz4hwcHBmDlzJjIyMrBv3z7cfvvtGDRoEA4dOmSr8REREVEdsSqTMHDgQLP/v/baa5g/fz52796NDh06aDowIiIiW2AmQb0bviahsrISX3/9NYqKihAXF6flmIiIiMgOWD1J+O233xAXF4eSkhJ4enpi5cqViIiIqLZ/aWkpSktLTf83GFTUFyciIrIRZhLUs/qqnXbt2uHAgQPYs2cPnnjiCYwcORK//179xWypqanw8vIyvUJCQmo0YCIiIqodVk8SXF1dERYWhujoaKSmpqJz58545513qu0/adIkFBQUmF5ZWVk1GjAREVFNXMskaP1qiGpcJ8FoNJqdTriem5sb3NzcaroaIiIiqmVWTRImTZqEAQMGoGXLlrh69SqWLFmCLVu2YN26dbYaHxERkaZYTEk9qyYJFy9exIgRI3DhwgV4eXkhMjIS69atwx133KH5wDyO7Be2G8K6SGPoCy8K2/Nc/KUxJj4ubv/ouWPSGHB3l/fRwNmz4varV2u+jthb5VV90p2dhe1HPhcX0gGA0FC1I6qZlr7FwvY9v3lIYxQVidvDJIWSAKBlVKSwPXfNr9IYtZWwu2XvYmH777cmSWPIvqA94x+RxmhzQvzZO+UiLoQGqCv8pIn8fHF7YaE8hqRPSxUfGp9A8fGct11eLA0HD8r72DleuKieVZOETz/91FbjICIiIjvDZzcQEZFDYSZBPT4FkoiIiCxiJoGIiBwKMwnqMZNARERkx/Ly8pCUlAS9Xg9vb2+MHj0ahYILWfPy8vDUU0+hXbt2aNy4MVq2bImnn34aBQUFVq+bmQQiInIo9e0WyKSkJFy4cAHr169HeXk5kpOT8eijj2LJkiUW+58/fx7nz5/H7NmzERERgT/++AOPP/44zp8/j+XLl1u1bk4SiIiI7NThw4eRlpaGvXv3IiYmBgDw3nvv4c4778Ts2bMRFBRU5T0dO3bEN998Y/p/27Zt8dprr+Ghhx5CRUUFXFzU/+q330mCZCP0+WekIfI8Wwrb1dzu+9FD24Tt27J7S2N07ChfjxYiwsqE7YYSV2mMDRskHZplSmN89ri4DsL8Iy9JY+Qkvi7to4UvlovvG7/7bnmMRYvE7bfHG6UxNs0V10GIb6WTxnCSFWzQiGGguA5CyQl5jNxccXu/ePGxDAD7D4rrIHR5X15r4fyrC6V9tDBnkY+wfUK3I9IYeeHdhe0+JXnyGGniL728wK7SGD7e3pIeX0pj1DVbXpNw/UMMa1p1eNeuXfD29jZNEAAgISEBTk5O2LNnD+69915VcQoKCqDX662aIAC8JoGIiByMLZ/dEBISYvZQw9TU1BqNNTs7G/7+5oX/XFxc4OPjg+zsbFUxcnNzMWPGDDz66KNWr99+MwlERET1TFZWFvR6ven/1WURJk6ciFmzZgljHT58uMbjMRgMuOuuuxAREYFp06ZZ/X5OEoiIyKHY8nSDXq83myRU59lnn8WoUaOEfdq0aYPAwEBcvGj+iIGKigrk5eUhUFJX/OrVq+jfvz+aNm2KlStXolGjRtJxXY+TBCIiolrm5+cHPz8/ab+4uDjk5+cjIyMD0dHRAIBNmzbBaDQiNja22vcZDAYkJibCzc0Nq1evhvsNPkOI1yQQEZFDseU1CVpr3749+vfvjzFjxiA9PR07duxASkoKhg0bZrqz4dy5cwgPD0d6ejqAvyYI/fr1Q1FRET799FMYDAZkZ2cjOzsblVbeq8lMAhERkR1bvHgxUlJS0LdvXzg5OeH+++/Hu+++a2ovLy/H0aNHUVz811Nt9+/fjz179gAAwsLMn9x7+vRphFrxmF1OEoiIyKHUt2JKPj4+1RZOAoDQ0FAoyt+3nsfHx5v9vyZ4uoGIiIgssttMwhnvSGG7p6c8hk/uMWF775hgeZBMX3GMcHmhnNqydoO4WNLVq/IYt90mWccRcQEbABg+XNx+PkxeKOlUoKx4kIoqRyo89JC4XVpcCvKCS198JZ+Lx8eL251KS+UDSUiQdPCSx1BB9tlT89mU1eNZslxe+GvYMHF78fvyQklBD90n7aOFCTHiomzo1lMaw2fNanEHNZW/SkrE63AxCNsB4Jb4qhX+6puKCsDZWfuYDZHdThKIiIhsgZME9Xi6gYiIiCxiJoGIiBwKMwnqMZNAREREFjGTQEREDqW+3QJZl5hJICIiIouYSSAiIodSUQE4afwnckO9JkGnaFWWSSWDwQAvLy8AiQCsfyIVERHVZ+UA1qGgoEDV0xK1dO33z+jRBXB11XbdZWUGfPqpV51sly0xk0BERA6FmQT1OEkgIiKHwkmCerxwkYiIiCxiJoGIiBwKMwnqMZNAREREFjGTQEREDoXFlNRjJoGIiIgsYiaBiIgcSkUFoNNpH7MhYiaBiIiILGImgYiIHAozCepxkkBERA6FkwT1eLqBiIiILGImgYiIHAozCeoxk0BEREQWMZNAREQOpbJS+0wCiykRERGRQ2EmgYiIHIotrh9oqNck2O0kQSn9psYxjC6uwnanQoM0RrGLXtjugWL5ONw9hO3OzgOlMdRQxjQXtv8+/iNpjIhwo7DdUChPPuXni9tdVBx1shgdOmi0z7Y+L2w3RPWu8Tqys+V9QkPF7a4H90tjGMK6CNu9vDTaZ3teFraXRXWVxpAdA4WF1ozIMn3+GWmfssCWwnY3N2322dat3wnbe3crkwc5e1bc7u4uDWEMDBK2O+3eKY1h6Nhd2K7VcUb2wW4nCURERLbATIJ6vCaBiIiILGImgYiIHAozCepxkkBERA7FFrcr8hZIIiIicijMJBARkUOpqAAURduYzCQQERGRQ7HbTEIZxDUOXn1VHmP6sN/FHXJzpTE8YmLEHVRcraKmHoMWlsSL6yAYOsjrkFb8Ip5eq7nnv6RE3B4eLo+xe7e8jxZ+9RbXQYj8brE8yPDhwuY/veRzcVdI7pMPDpbGkNWW0MqZQHEdhJa7t8mDdOwobNZ7e8tjSD57xmBxDQQAqJAcq1qJihK3vz5b/H0HAC8Nk3RQ8X3mJCtQISvYAUDvrqKmg51jJkE9ZhKIiIjIIrvNJBAREdkCMwnqMZNAREREFjGTQEREDoWZBPU4SSAiIodSWan9JMEofjZevcXTDURERHYsLy8PSUlJ0Ov18Pb2xujRo1Go8lGpiqJgwIAB0Ol0WLVqldXrZiaBiIgcSkUF4KTxn8i2zCQkJSXhwoULWL9+PcrLy5GcnIxHH30US5Yskb537ty50Onkt79Xh5MEIiIiO3X48GGkpaVh7969iPlf3Z733nsPd955J2bPno2goKBq33vgwAG89dZb2LdvH5o3b35D67fbSYKs5sf0f5+RxrhvfISwfeZM+ThuObJf3MHTUx5EtjEaGTJE3L49UH4SrmNn8YwzvFQe48gRcbuaXRYWJu+jBdlYyv4vSRrDdc1qYXtO6D3SGDk54mI6kb7yol0q6i1pIjBQ3F4WKC5QBQCffCJuv/tu+ThaSsbhtOFHaQyPhAT5ijSg37BC2P5SinwcOq+LwnblFw+rxmSRmipm8fE1X08ds2UmwWAwL57n5uYGNze3G467a9cueHt7myYIAJCQkAAnJyfs2bMH9957r8X3FRcX48EHH8S8efMQKPvQCvCaBCIiIo2EhITAy8vL9EpNTa1RvOzsbPj7+5stc3FxgY+PD7IFJXCfeeYZdO/eHYMGDarR+u02k0BERGQLtswkZGVlQa/Xm5ZXl0WYOHEiZs2aJYx5+PDhGxrL6tWrsWnTJvz888839P5/4iSBiIhII3q93mySUJ1nn30Wo0aNEvZp06YNAgMDcfGi+ammiooK5OXlVXsaYdOmTTh58iS8r3sGyv33349evXphy5Yt0vFdw0kCERE5lMpK7e9GsLbugp+fH/z8/KT94uLikJ+fj4yMDERHRwP4axJgNBoRGxtr8T0TJ07Ev//9b7NlnTp1wttvv42BAwdaNU5OEoiIyKFUVAA1uCvQIq2LM13Tvn179O/fH2PGjMGCBQtQXl6OlJQUDBs2zHRnw7lz59C3b1988cUX6Nq1KwIDAy1mGVq2bInWrVtbtX5euEhERGTHFi9ejPDwcPTt2xd33nknevbsiY8++sjUXl5ejqNHj6K4uFjzdTOTQEREDqU+ZRIAwMfHR1g4KTQ0FIpkALL26tRokjBz5kxMmjQJ48aNw9y5c2sSqgqnijJxh+suyLDkq6/E7bm5KgYSFqWik4TK8pk1VSG5lT48XEWQSvGB5HrimDREZHiouINsoABKgjW451sFFYeRVHGCuA5CZKH4/nYAgK+vuL1QXlyilg4zTVx3urSKK1c0WElUlLRLWUUtJVNl9Rjc3aUhlAJx3ZfVW+QXy90TLvleVVMDQYsPDdUbNzxJ2Lt3Lz788ENERkZqOR4iIiKbqm+ZhLp0Q9PowsJCJCUl4eOPP0azZs20HhMRERHZgRuaJIwdOxZ33XUXEmqppCkREZFWKips82qIrD7dsHTpUuzfvx979+5V1b+0tBSlpaWm/19f15qIiIjsk1WZhKysLIwbNw6LFy+Gu4oLbQAgNTXVrI51SEjIDQ2UiIhIG0YoirYvwIbPiq5DVk0SMjIycPHiRXTp0gUuLi5wcXHB1q1b8e6778LFxQWVlZVV3jNp0iQUFBSYXllZWZoNnoiIyHqVNno1PFadbujbty9+++03s2XJyckIDw/Hiy++CGdn5yrvqeljMomIiKhuWDVJaNq0KTp27Gi2rEmTJrjpppuqLCciIrJPtvjLn5mEWnXqrKuwvU2wPIbH7m3C9pYqJjaz3vQRtr/4hIoLMWvpslfZg71cVPy0jxwRt98eHyaN8YWFjNI/nXhZfkOxmpouWpAVIFJTN6akpIYrAYADB8TtPXtKQ3iqu0yoxvbtE7d376bi3OzZs8LmgPx8eYxKcQGq8wiShghyqZ3zyGfyxYWOWuKMPIjkOLpH/tHEuwvEBZmeHqXiS0JVFTpqKGo8SbDmkZNERER1j5kEtfiAJyIiIrLIbk83EBER2YYtblnkLZBERETkQJhJICIiB8NrEtTiJIGIiByMEdr/UufpBiIiInIgdptJaBNYLO5QoqL2gKQ+QU65uAYCALx4x35h+7IfukhjDBwo7aIJ2eM0glXUlpDdAy/vACz6l7gOwqYZ8ge5n3+8dh7O3nLVu8L2sseflsbQe0r+gthwQBrj9/D7hO3dAqUhYMjMk3fSQHd38Wci55L8MxEgqZMgq6MAAL+GPyBsnzxZGgKrVtXO30ktF00Xd3juOXkQWS2NQPlB8nSK+FjVOZ+Qxigtlf987R9PN6jFTAIRERFZZLeZBCIiIttgJkEtZhKIiIjIImYSiIjIwbCYklrMJBAREZFFzCQQEZGD4TUJanGSQEREDoaTBLV4uoGIiIgssttMgtHdQ9juVCIptgQA3t7C5oBDm+QxOnYUNg8Ml4f46it5Hy1kZ4vbVdSnQVSUuH3hvq7SGBMnituPLZAXSvqxhazg0t3SGGqsDRMXS4q+Io8R0ExctOtYR3GhJACI8BUXQjKcVfFRXb5K3kcD+yEuphPlpyJIqaSyV7du0hCRhQZh+1df6aUxnA6IC0NpZW3MFGH7nbln5EHCwsTtsmpqAHDkiLC5slJeKKmwUL4a+8dMglrMJBAREZFFdptJICIisg0+4EktZhKIiIjIImYSiIjIwbCYklrMJBAREZFFzCQQEZGD4d0NanGSQEREDoaTBLV0iqLIb1rXkMFggJeXF4BEAI1qc9VERFTnygGsQ0FBAfR6eS0LLf39+2c7AE+NoxcC6Fkn22VLzCQQEZGDYSZBLV64SERERBYxk0BERA6GmQS1mEkgIiIii5hJICIiB6NA++JHtXoPQK1hJoGIiIgs4iSBiIgcTKWNXraRl5eHpKQk6PV6eHt7Y/To0ShU8czuXbt24fbbb0eTJk2g1+vRu3dv/Pnnn1atm5MEIiJyMPVrkpCUlIRDhw5h/fr1WLNmDbZt24ZHH31U+J5du3ahf//+6NevH9LT07F3716kpKTAycm6X/sspkRERLXIHoop/QCgicbRiwAM0Hy7Dh8+jIiICOzduxcxMTEAgLS0NNx55504e/YsgoKCLL6vW7duuOOOOzBjxowarZ+ZBCIicjC2yyQYDAazV2lpaY1GumvXLnh7e5smCACQkJAAJycn7Nmzx+J7Ll68iD179sDf3x/du3dHQEAA+vTpg+3bt1u9fk4SiIiINBISEgIvLy/TKzU1tUbxsrOz4e/vb7bMxcUFPj4+yM7OtvieU6dOAQCmTZuGMWPGIC0tDV26dEHfvn1x/Phxq9bPWyCJiMjB2K6YUlZWltnpBjc3N4u9J06ciFmzZgkjHj58+IZGYjT+dXvnY489huTkZADArbfeio0bN2LhwoVWTVw4SSAiItKIXq9XdU3Cs88+i1GjRgn7tGnTBoGBgbh48aLZ8oqKCuTl5SEwMNDi+5o3bw4AiIiIMFvevn17nDlzRjq2f+IkgYiIHIwR2hdTsi6en58f/Pz8pP3i4uKQn5+PjIwMREdHAwA2bdoEo9GI2NhYi+8JDQ1FUFAQjh49arb82LFjGDBggFXj5DUJREREdqp9+/bo378/xowZg/T0dOzYsQMpKSkYNmyY6c6Gc+fOITw8HOnp6QAAnU6H559/Hu+++y6WL1+OEydO4OWXX8aRI0cwevRoq9bPTAIRETmY+vWAp8WLFyMlJQV9+/aFk5MT7r//frz77rum9vLychw9ehTFxcWmZePHj0dJSQmeeeYZ5OXloXPnzli/fj3atm1r1bpZJ4GIiGqRPdRJWA7b1EkYUifbZUt2m0n4DmuE7WXfyOc2Z8+K20+ckI9j/Hhxe2ioPMaGDeL2xMSB8iAqKOc+FLYXe1suuvFPHmv+K+5QXi6N8aNfkrA9N1caAnffLW738tJmn+3Y8Z2wvXtHgzTG72fFXwj5+fJxdPc9Ju7w9dfyIE89JWzWeYl/LmopNx8VtuftlmwLgPXrxe2//CIfx6uvituddu+Uxvjdu7uwvUMHbY6zefPEx5nseAeAllu+EHcID5fGOB/cVdgedDZdPpDgYGGzrsVj8hh1zgjt//LX+hoH+2C3kwQiIiLbqF+nG+oSL1wkIiIii5hJICIiB1P3t0DWF8wkEBERkUXMJBARkYPhNQlqMZNAREREFjGTQEREDoaZBLXsdpLg9L24DsKvd+mkMe7JEMf4x+O5q3XggLhdTa0FWb0GreQ4i+sgVObLY3jEx4s7lJRIY8R4itvd3eXjUNNHC927iS82+nGDvChKv26SWgq+8n2GCslOe+IJeYxa2mlnNojrIHjcJP9s3nZS/Nm87Tb5ODIzxe2XnMU1EACgkYofjRYGD9YgiOyzqUKQd7G4Q6CKL0UV3wHUcNjtJIGIiMg2mElQi5MEIiJyMKy4qBYvXCQiIiKLmEkgIiIHw2JKajGTQERERBYxk0BERA6GFy6qxUwCERERWcRMAhERORhmEtSy20nC7t3i9viN4mIsAHAkWlzU5cDz8hgTJ4rbfTzLpDGWrXSV9tFCwNb/ijv07y+N8eDj/sL2Ja+eksbwKTkv7uDpK41RXFI7++z3I+JkmrRQEoCcP8UFl86dkxdk6hIluehJTdUul9r5OKelidt7HpJ/riLiWwrb333ujDTG04+LP3ttcg9IY+ws6Srto4WgtIXiDqNGSWPonJsK25VDF+QDCQwUt2/fLg1RHNNbvh5qMOx2kkBERGQbzCSoxUkCERE5GE4S1OKFi0RERGQRMwlERORgWExJLWYSiIiIyCJmEoiIyMHwAU9qMZNAREREFlmVSZg2bRpeeeUVs2Xt2rXDkSNHNB0UAJw9K26vqJDHkNVBCH9TXEcBAHxmSmabJfKBNG5cO/f8o6RE3F5YKA2xZGKmsP2FBZHSGDNnitvz86Uh4ONdO7Ny2S6Td5DXQejieUwao6ziFmG7q7e3NIaan68WZCUbevaUx/jiVXEdhMEj5Z9NPF4qbvf0lIZwl/94teErrw0io/ySJWxvHC3/bP55uVjcQcVx5uEirw1j/3h3g1pWn27o0KEDNmzY8HeAWirgQkRERLXL6t/wLi4uCJRV7SIiIrJbzCSoZfU1CcePH0dQUBDatGmDpKQknDkjL59KRERkPypt9Gp4rMokxMbGYtGiRWjXrh0uXLiAV155Bb169cLBgwfRtKnluuKlpaUoLf373KHBIK+FT0RERHXPqknCgAEDTP+OjIxEbGwsWrVqhf/+978YPXq0xfekpqZWudiRiIio7vB0g1o1ugXS29sbt9xyC04ILneeNGkSCgoKTK+sLPEVukRERGQfajRJKCwsxMmTJ9G8efNq+7i5uUGv15u9iIiI6o7RRq+Gx6pJwnPPPYetW7ciMzMTO3fuxL333gtnZ2cMHz7cVuMjIiKiOmLVNQlnz57F8OHDcfnyZfj5+aFnz57YvXs3/Pz8NB/YJ5+I22UFXQBg8mRxu8s0cbElAMBzE4TNe4bOkYYIC5OvRgtn4kcI24NV3LnqJLm9VVYoCQCcSsQFW3w8VRx2KoplaaFLmORCWndveQxPSYEal1BpDNnxHBEmH4fRpXaKdr0xTby95/M9pDEeekjWQcVnc1+6sPlMYFdpiEBv+Wq0cD7mHmF7kOQzA0D6RfJnjoqLwl3chc3GjvKCTA0DyzKrZdUkYenSpbYaBxEREdkZlkskIiIHw7sb1OIkgYiIHAwnCWrxKZBERERkETMJRETkYGxxy2LDvHCRmQQiIiKyiJMEIiJyMPXrAU95eXlISkqCXq+Ht7c3Ro8ejcLCQuF7srOz8fDDDyMwMBBNmjRBly5d8M0331i9brs93eBUUSZsDwyU3xPuLr4lGC4qtj7nRXEdhNiiU9IYxYFt5CvSgK+vuN1JRTrMqMW8UbbjS0pqHkMrFeKCDGpqD0j3mIoDLSJc/LM5c1Y+juBgaRdtSPaZp6c8hJpjUaa4o7gOQkt3+TqKS2rn76QgX/H3max+AQD550bFZ0Z2POfny4fh490w0+r2LCkpCRcuXMD69etRXl6O5ORkPProo1iyZEm17xkxYgTy8/OxevVq+Pr6YsmSJXjggQewb98+3HrrrarXzUwCERE5mPqTSTh8+DDS0tLwySefIDY2Fj179sR7772HpUuX4vz589W+b+fOnXjqqafQtWtXtGnTBpMnT4a3tzcyMjKsWj8nCURERBoxGAxmr9LS0hrF27VrF7y9vRETE2NalpCQACcnJ+zZs6fa93Xv3h3Lli1DXl4ejEYjli5dipKSEsTHx1u1fk4SiIjIwdgukxASEgIvLy/TKzU1tUYjzc7Ohr+/v9kyFxcX+Pj4IDs7u9r3/fe//0V5eTluuukmuLm54bHHHsPKlSsRZuVzAuz2mgQiIiLbsN2zG7Kyssyeduzm5max98SJEzFr1ixhxMOHD9/waF5++WXk5+djw4YN8PX1xapVq/DAAw/gp59+QqdOnVTH4SSBiIhII3q93mySUJ1nn30Wo0aNEvZp06YNAgMDcfHiRbPlFRUVyMvLQ2A1D+Q7efIk3n//fRw8eBAdOnQAAHTu3Bk//fQT5s2bhwULFqjbGHCSQEREDqfuiyn5+fmpeoJyXFwc8vPzkZGRgejoaADApk2bYDQaERsba/E9xcV/PVXUycn8igJnZ2cYjdaNk9ckEBER2an27dujf//+GDNmDNLT07Fjxw6kpKRg2LBhCAoKAgCcO3cO4eHhSE//6/Hp4eHhCAsLw2OPPYb09HScPHkSb731FtavX4/BgwdbtX5OEoiIyMHUn1sgAWDx4sUIDw9H3759ceedd6Jnz5746KOPTO3l5eU4evSoKYPQqFEjrF27Fn5+fhg4cCAiIyPxxRdf4PPPP8edd95p1brt9nTD6jRx0Q81hZCqOV1jVYzcXHF7ZhN5oaQLbXWSHnfLB6KCrNaKh4p6LdIiN5IqXwA0KYRUVlE781eDi4+wXV9oqPlK1OwzSQWiwED5Oc4TJ9QOqGbWbhePpWNHeYzCQvHP19nZmhFZdvas/BgKbVc7n82cK+LvswDdRWE7AHmVqrNn5TFCxd9Xagph5Vzi35a1zcfHR1g4KTQ0FIqimC27+eabb6jC4vXsdpJARERkG3xUtFqcEhIREZFFzCQQEZGDYSZBLU4SiIjIwXCSoBZPNxAREZFFzCQQEZGDsV1Z5oaGmQQiIiKySKdcf3OljRkMBnh5eQFIBNCoNldNRER1rhzAOhQUFKh6xoGW/v79cz+0//1TDuCbOtkuW2ImgYiIiCziNQlERORgKqH938i8u4GIiIgcCDMJRETkYJhJUIuTBCIicjCcJKjF0w1ERERkETMJRETkYIzQvvhRwyymVOuThL/LMlTU9qqJiKjO/fXdX8sleiyOwf5j1r1anyRcvXr1f//aWNurJiIiO3H16tX/FTaqPa6urggMDER29gabxA8MDISrq6tNYteVWq+4aDQacf78eTRt2hQ6na42Vy1kMBgQEhKCrKysBlUty5a4z6zHfWY97jPr2fM+UxQFV69eRVBQEJycav+yuJKSEpSVldkktqurK9zd3W0Su67UeibByckJwcHBtb1a1fR6vd19qOwd95n1uM+sx31mPXvdZ7WdQfgnd3f3BveL3JZ4dwMRERFZxEkCERERWcRJwv+4ublh6tSpcHNzq+uh1BvcZ9bjPrMe95n1uM9IK7V+4SIRERHVD8wkEBERkUWcJBAREZFFnCQQERGRRZwkEBERkUWcJPzPvHnzEBoaCnd3d8TGxiI9Pb2uh2S3tm3bhoEDByIoKAg6nQ6rVq2q6yHZvdTUVNx2221o2rQp/P39MXjwYBw9erSuh2XX5s+fj8jISFNBoLi4OPzwww91Pax6Y+bMmdDpdBg/fnxdD4XqMU4SACxbtgwTJkzA1KlTsX//fnTu3BmJiYm4ePFiXQ/NLhUVFaFz586YN29eXQ+l3ti6dSvGjh2L3bt3Y/369SgvL0e/fv1QVFRU10OzW8HBwZg5cyYyMjKwb98+3H777Rg0aBAOHTpU10Oze3v37sWHH36IyMjIuh4K1XO8BRJAbGwsbrvtNrz//vsA/nq+REhICJ566ilMnDixjkdn33Q6HVauXInBgwfX9VDqlUuXLsHf3x9bt25F796963o49YaPjw/efPNNjB49uq6HYrcKCwvRpUsXfPDBB3j11VcRFRWFuXPn1vWwqJ5y+ExCWVkZMjIykJCQYFrm5OSEhIQE7Nq1qw5HRg1ZQUEBgL9+6ZFcZWUlli5diqKiIsTFxdX1cOza2LFjcdddd5l9pxHdqFp/wJO9yc3NRWVlJQICAsyWBwQE4MiRI3U0KmrIjEYjxo8fjx49eqBjx451PRy79ttvvyEuLg4lJSXw9PTEypUrERERUdfDsltLly7F/v37sXfv3roeCjUQDj9JIKptY8eOxcGDB7F9+/a6Horda9euHQ4cOICCggIsX74cI0eOxNatWzlRsCArKwvjxo3D+vXr+ZRD0ozDTxJ8fX3h7OyMnJwcs+U5OTkIDAyso1FRQ5WSkoI1a9Zg27Ztdv3IdHvh6uqKsLAwAEB0dDT27t2Ld955Bx9++GEdj8z+ZGRk4OLFi+jSpYtpWWVlJbZt24b3338fpaWlcHZ2rsMRUn3k8NckuLq6Ijo6Ghs3bjQtMxqN2LhxI899kmYURUFKSgpWrlyJTZs2oXXr1nU9pHrJaDSitLS0rodhl/r27YvffvsNBw4cML1iYmKQlJSEAwcOcIJAN8ThMwkAMGHCBIwcORIxMTHo2rUr5s6di6KiIiQnJ9f10OxSYWEhTpw4Yfr/6dOnceDAAfj4+KBly5Z1ODL7NXbsWCxZsgTffvstmjZtiuzsbACAl5cXGjduXMejs0+TJk3CgAED0LJlS1y9ehVLlizBli1bsG7duroeml1q2rRplWtcmjRpgptuuonXvtAN4yQBwNChQ3Hp0iVMmTIF2dnZiIqKQlpaWpWLGekv+/btw7/+9S/T/ydMmAAAGDlyJBYtWlRHo7Jv8+fPBwDEx8ebLf/ss88watSo2h9QPXDx4kWMGDECFy5cgJeXFyIjI7Fu3TrccccddT00IofBOglERERkkcNfk0BERESWcZJAREREFnGSQERERBZxkkBEREQWcZJAREREFnGSQERERBZxkkBEREQWcZJAdJ0rV65g+vTpuHjxYl0PhW6Aoih49913kZ6eXtdDIar3OEkguk6zZs3g5uaGpKQkGI1Gs7YtW7ZAp9MhPz+/1sc1atQoDB48+IbfHx8fj/Hjx2s2HpnMzEzodDocOHCg2j622J86nQ5RUVEYOnRolQe3EZF1OElwYKNGjYJOp4NOp0OjRo0QEBCAO+64AwsXLqzyy1Gr9U2bNs30/9DQUMydO1fz9WjhxRdfxC233GI2Xi3V9i9sAFixYgVmzJhRq+uU6d69u6nsspZ69+6NmTNnIikpCZWVlZrGJnIkfHaDg+vfvz8+++wzVFZWIicnB2lpaRg3bhyWL1+O1atXw8XFcQ+RefPm1fUQNOXj41PXQ6jC1dXVZo9kHzp0KIYOHWqT2ESOgpkEB+fm5obAwEC0aNECXbp0wUsvvYRvv/0WP/zwg9nDms6cOYNBgwbB09MTer0eDzzwgFkqd9q0aYiKisKXX36J0NBQeHl5YdiwYbh69arF9cbHx+OPP/7AM888Y8pmWGIpZZ2fnw+dToctW7YA+OsagqSkJPj5+aFx48a4+eab8dlnn5n6Z2Vl4YEHHoC3tzd8fHwwaNAgZGZm3vA+A4AdO3YgMjIS7u7u6NatGw4ePGhqu3z5MoYPH44WLVrAw8MDnTp1wv/7f//P1D5q1Chs3boV77zzjmnbr43n0KFDuPvuu6HX69G0aVP06tULJ0+eNFv37Nmz0bx5c9x0000YO3YsysvLVY35+uzFl19+iZiYGDRt2hSBgYF48MEHq1yHIRqP0WjE9OnTERwcDDc3N9OD0a535MgRdO/eHe7u7ujYsSO2bt1qarv+dINs313bjqeffhovvPACfHx8EBgYaLOMD5Gj4ySBqrj99tvRuXNnrFixAsBfvwwGDRqEvLw8bN26FevXr8epU6eq/JV28uRJrFq1CmvWrMGaNWuwdetWzJw50+I6VqxYgeDgYEyfPh0XLlzAhQsXbni8L7/8Mn7//Xf88MMPOHz4MObPnw9fX18AQHl5ORITE9G0aVP89NNP2LFjBzw9PdG/f3+UlZXd8Dqff/55vPXWW9i7dy/8/PwwcOBA0y/rkpISREdH4/vvv8fBgwfx6KOP4uGHHzZdSPfOO+8gLi4OY8aMMW17SEgIzp07h969e8PNzQ2bNm1CRkYGHnnkEVRUVJjWu3nzZpw8eRKbN2/G559/jkWLFt3wkzfLy8sxY8YM/PLLL1i1ahUyMzPNnkgpG88777yDt956C7Nnz8avv/6KxMRE3HPPPTh+/HiVffXss8/i559/RlxcHAYOHIjLly9bHJNs313z+eefo0mTJtizZw/eeOMNTJ8+HevXr7+h/UBEAgo5rJEjRyqDBg2y2DZ06FClffv2iqIoyo8//qg4OzsrZ86cMbUfOnRIAaCkp6criqIoU6dOVTw8PBSDwWDq8/zzzyuxsbHVrr9Vq1bK22+/LRzj6dOnFQDKzz//bFp25coVBYCyefNmRVEUZeDAgUpycrLF93/55ZdKu3btFKPRaFpWWlqqNG7cWFm3bp1w3ZZs3rxZAaAsXbrUtOzy5ctK48aNlWXLllX7vrvuukt59tlnTf/v06ePMm7cOLM+kyZNUlq3bq2UlZVZjDFy5EilVatWSkVFhWnZ//3f/ylDhw5VNXZL6/ynvXv3KgCUq1evqhpPUFCQ8tprr5ktu+2225Qnn3xSUZS/f3YzZ840tZeXlyvBwcHKrFmzFEX5e39euXKl2nFZ2nc9e/asst4XX3yx2hhEdGOYSSCLFEUxnQI4fPgwQkJCEBISYmqPiIiAt7c3Dh8+bFoWGhqKpk2bmv7fvHnzWrmN8IknnsDSpUsRFRWFF154ATt37jS1/fLLLzhx4gSaNm0KT09PeHp6wsfHByUlJVXS+NaIi4sz/dvHxwft2rUz7YvKykrMmDEDnTp1go+PDzw9PbFu3TqcOXNGGPPAgQPo1asXGjVqVG2fDh06wNnZ2fT/muzjjIwMDBw4EC1btkTTpk3Rp08fADCNUzQeg8GA8+fPo0ePHmbLe/ToYXZMAOb7ysXFBTExMVX6XKN230VGRpr9v7aONSJH47hXpZHQ4cOH0bp1a6vec/0vE51OV+O7JJyc/prHKopiWnb9OfgBAwbgjz/+wNq1a7F+/Xr07dsXY8eOxezZs1FYWIjo6GgsXry4Smw/P78aja06b775Jt555x3MnTsXnTp1QpMmTTB+/Hjp6Y3GjRtLY2u1j4uKipCYmIjExEQsXrwYfn5+OHPmDBITE03jVDMerandd7Y41oioKmYSqIpNmzbht99+w/333w8AaN++PbKyspCVlWXq8/vvvyM/Px8RERE3vB5XV1fp7WnXfpH/85oFS/fd+/n5YeTIkfjqq68wd+5cfPTRRwCALl264Pjx4/D390dYWJjZqya33e3evdv07ytXruDYsWNo3749gL8uahw0aBAeeughdO7cGW3atMGxY8fM3m9p2yMjI/HTTz+pvhCxJo4cOYLLly9j5syZ6NWrF8LDw6v8JS4aj16vR1BQEHbs2GG2fMeOHVWOiX/uq4qKCmRkZJj21fXU7Dsiqj2cJDi40tJSZGdn49y5c9i/fz9ef/11DBo0CHfffTdGjBgBAEhISECnTp2QlJSE/fv3Iz09HSNGjECfPn0QExNzw+sODQ3Ftm3bcO7cOeTm5lrs07hxY3Tr1g0zZ87E4cOHsXXrVkyePNmsz5QpU/Dtt9/ixIkTOHToENasWWP6JZSUlARfX18MGjQIP/30E06fPo0tW7bg6aefxtmzZ2947NOnT8fGjRtx8OBBjBo1Cr6+vqZCRzfffDPWr1+PnTt34vDhw3jssceqFPUJDQ3Fnj17kJmZidzcXBiNRqSkpMBgMGDYsGHYt28fjh8/ji+//BJHjx694XFWp2XLlnB1dcV7772HU6dOYfXq1VVqKMjG8/zzz2PWrFlYtmwZjh49iokTJ+LAgQMYN26cWZx58+Zh5cqVOHLkCMaOHYsrV67gkUcesTguNfuOiGoPJwkOLi0tDc2bN0doaCj69++PzZs3491338W3335rOvet0+nw7bffolmzZujduzcSEhLQpk0bLFu2rEbrnj59OjIzM9G2bVth6n/hwoWoqKhAdHQ0xo8fj1dffdWs3dXVFZMmTUJkZCR69+4NZ2dnLF26FADg4eGBbdu2oWXLlrjvvvvQvn17jB49GiUlJdDr9Tc89pkzZ2LcuHGIjo5GdnY2vvvuO7i6ugIAJk+ejC5duiAxMRHx8fEIDAysUinxueeeg7OzMyIiIkyp/ptuugmbNm1CYWEh+vTpg+joaHz88cfCaxRulJ+fHxYtWoSvv/4aERERmDlzJmbPnm3WRzaep59+GhMmTMCzzz6LTp06IS0tDatXr8bNN99cZV/NnDkTnTt3xvbt27F69WrT3SfXU7PviKj26JR/nuwlogYrLi4Offv2rTLJIiKqDjMJRA1caWkp9u3bh0OHDqFDhw51PRwiqkd4dwNRA/DTTz9hwIAB1bY7OTnhnnvuwZAhQ2pxVERU3/F0A1ED8Oeff+LcuXPVtoeFhdXiaIiooeAkgYiIiCziNQlERERkEScJREREZBEnCURERGQRJwlERERkEScJREREZBEnCURERGQRJwlERERkEScJREREZNH/Bx7RsH2PJqjrAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_as_patches(j)\n", "\n", "_ = plt.title('These slices are not diagonal')\n", "_ = plt.xlabel(\"Don't use `batch_jacobian`\")" ] }, { "cell_type": "markdown", "metadata": { "id": "M_x7ih5sarvG" }, "source": [ "In this case, `batch_jacobian` still runs and returns _something_ with the expected shape, but its contents have an unclear meaning:" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "execution": { "iopub.execute_input": "2024-08-15T02:32:20.934532Z", "iopub.status.busy": "2024-08-15T02:32:20.934278Z", "iopub.status.idle": "2024-08-15T02:32:21.328241Z", "shell.execute_reply": "2024-08-15T02:32:21.327515Z" }, "id": "k8_mICHoasCi" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "jb.shape: (7, 6, 5)\n" ] } ], "source": [ "jb = tape.batch_jacobian(y, x)\n", "print(f'jb.shape: {jb.shape}')" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "advanced_autodiff.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.19" } }, "nbformat": 4, "nbformat_minor": 0 }