{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Tce3stUlHN0L" }, "source": [ "##### Copyright 2021 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2024-03-09T12:19:01.108803Z", "iopub.status.busy": "2024-03-09T12:19:01.108167Z", "iopub.status.idle": "2024-03-09T12:19:01.112421Z", "shell.execute_reply": "2024-03-09T12:19:01.111765Z" }, "id": "IcfrhafzkZbH" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "qFdPvlXBOdUN" }, "source": [ "# Sparse weights using structural pruning" ] }, { "cell_type": "markdown", "metadata": { "id": "MfBg1C5NB3X0" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "FbORZA_bQx1G" }, "source": [ "Structural pruning weights from your model to make it sparse in specific pattern can accelerate model inference time with appropriate HW supports.\n", "\n", "This tutorial shows you how to:\n", "* Define and train a model on the mnist dataset with a specific structural sparsity\n", "* Convert the pruned model to tflite format\n", "* Visualize structure of the pruned weights\n", "\n", "For a general overview of the pruning technique for the model optimization, see the [pruning overview](https://www.tensorflow.org/model_optimization/guide/pruning). For tutorial on general weight pruning, see [Pruning in Keras](https://www.tensorflow.org/model_optimization/guide/pruning/pruning_with_keras)." ] }, { "cell_type": "markdown", "metadata": { "id": "0f4SoBcoXNcb" }, "source": [ "## Structural pruning of weights" ] }, { "cell_type": "markdown", "metadata": { "id": "rn_a9362Wr_B" }, "source": [ "Structural pruning systematically zeroes out model weights at the beginning of the training process. You apply this pruning techniques to regular blocks of weights to speed up inference on supporting HWs, for example: grouping weights in the model by blocks of four and zeroing out two of those weights in each block, known as a _2 by 4_ reduction. This technique applies only to the last dimension of the weight tensor for the model that is converted by TensorFlow Lite. For example, `Conv2D` layer weights in TensorFlow Lite have the structure `[channel_out, height, width, channel_in]` and `Dense` layer weights have the structure `[channel_out, channel_in]`. The sparsity pattern is applied to the weights in the last dimension: `channel_in`.\n", "\n", "Compare to the random sparsity, the structured sparsity generally has lower accuracy due to restrictive structure, however, it can reduce inference time significantly on the supported hardware.\n", "\n", "Pruning can be applied to a model together with other model compression techniques for better compression rate. See quantization and clustering examples in [collaborative optimization technique](https://blog.tensorflow.org/2021/10/Collaborative-Optimizations.html) for more details." ] }, { "cell_type": "markdown", "metadata": { "id": "nuABqZnXVDvO" }, "source": [ "## Setup" ] }, { "cell_type": "markdown", "metadata": { "id": "u9mRDekZEfnR" }, "source": [ "Prepare your development environment and data." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "cellView": "both", "execution": { "iopub.execute_input": "2024-03-09T12:19:01.116528Z", "iopub.status.busy": "2024-03-09T12:19:01.115944Z", "iopub.status.idle": "2024-03-09T12:19:07.979069Z", "shell.execute_reply": "2024-03-09T12:19:07.977946Z" }, "id": "lvpH1Hg7ULFz" }, "outputs": [], "source": [ "! pip install -q tensorflow\n", "! pip install -q tensorflow-model-optimization\n", "! pip install -q matplotlib" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:19:07.983845Z", "iopub.status.busy": "2024-03-09T12:19:07.983097Z", "iopub.status.idle": "2024-03-09T12:19:10.837465Z", "shell.execute_reply": "2024-03-09T12:19:10.836584Z" }, "id": "_hn5e5_gWr_E" }, "outputs": [], "source": [ "import tensorflow as tf\n", "from tensorflow import keras\n", "\n", "import tensorflow_model_optimization as tfmot\n", "prune_low_magnitude = tfmot.sparsity.keras.prune_low_magnitude" ] }, { "cell_type": "markdown", "metadata": { "id": "TZyLYFTER4aP" }, "source": [ "## Download and normalize image data from the [MNIST](https://www.tensorflow.org/datasets/catalog/mnist) dataset" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:19:10.842311Z", "iopub.status.busy": "2024-03-09T12:19:10.841503Z", "iopub.status.idle": "2024-03-09T12:19:11.325781Z", "shell.execute_reply": "2024-03-09T12:19:11.324971Z" }, "id": "hSf4jYKGWr_E" }, "outputs": [], "source": [ "# Load MNIST dataset.\n", "mnist = keras.datasets.mnist\n", "(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n", "\n", "# Normalize the input image so that each pixel value is between 0 and 1.\n", "train_images = train_images / 255.0\n", "test_images = test_images / 255.0" ] }, { "cell_type": "markdown", "metadata": { "id": "LKaL3XH1XO0Q" }, "source": [ "## Define structural pruning parameters" ] }, { "cell_type": "markdown", "metadata": { "id": "s9_33ta-Wr_E" }, "source": [ "Define parameters for pruning and specify the type of structural pruning. Set the parameters for pruning to `(2, 4)`.\n", "These settings mean that in a block of four elements, at least two with the lowest magnitude are set to zero.\n", "\n", "You don't have to set the `pruning_schedule` parameter. By default, the pruning mask is defined at the first step and it is not updated during the training." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:19:11.330209Z", "iopub.status.busy": "2024-03-09T12:19:11.329659Z", "iopub.status.idle": "2024-03-09T12:19:11.333200Z", "shell.execute_reply": "2024-03-09T12:19:11.332552Z" }, "id": "1EXNYAPJWr_F" }, "outputs": [], "source": [ "pruning_params_2_by_4 = {\n", " 'sparsity_m_by_n': (2, 4),\n", "}" ] }, { "cell_type": "markdown", "metadata": { "id": "mMKdsdAUWr_F" }, "source": [ "Define parameters for random pruning with the target sparsity of 50%." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:19:11.336607Z", "iopub.status.busy": "2024-03-09T12:19:11.336031Z", "iopub.status.idle": "2024-03-09T12:19:11.339632Z", "shell.execute_reply": "2024-03-09T12:19:11.338990Z" }, "id": "un24AZUOWr_F" }, "outputs": [], "source": [ "pruning_params_sparsity_0_5 = {\n", " 'pruning_schedule': tfmot.sparsity.keras.ConstantSparsity(target_sparsity=0.5,\n", " begin_step=0,\n", " frequency=100)\n", "}" ] }, { "cell_type": "markdown", "metadata": { "id": "jV4Yt0v5Wr_G" }, "source": [ "Define the model architecture and specify which layers to prune. Structural pruning is applied based on the layers of the model you select.\n", "\n", "In the example below, we prune only some of the layers. We prune the second `Conv2D` layer and the first `Dense` layer.\n", "\n", "Notice that the first `Conv2D` layer cannot be pruned structurally. To be pruned structurally, it should have more than one input channels. Instead, we prune the first `Conv2D` layer with random pruning." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:19:11.342918Z", "iopub.status.busy": "2024-03-09T12:19:11.342398Z", "iopub.status.idle": "2024-03-09T12:19:13.434923Z", "shell.execute_reply": "2024-03-09T12:19:13.434113Z" }, "id": "BDGzC6YlWr_G" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-03-09 12:19:11.497336: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:282] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential\"\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "_________________________________________________________________\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Layer (type) Output Shape Param # \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "=================================================================\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " prune_low_magnitude_prunin (None, 28, 28, 32) 1634 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " g_sparsity_0_5 (PruneLowMa \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " gnitude) \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " max_pooling2d (MaxPooling2 (None, 14, 14, 32) 0 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " D) \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " prune_low_magnitude_struct (None, 14, 14, 64) 102466 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " ural_pruning (PruneLowMagn \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " itude) \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " batch_normalization (Batch (None, 14, 14, 64) 256 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Normalization) \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " re_lu (ReLU) (None, 14, 14, 64) 0 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " max_pooling2d_1 (MaxPoolin (None, 7, 7, 64) 0 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " g2D) \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " flatten (Flatten) (None, 3136) 0 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " prune_low_magnitude_struct (None, 1024) 6423554 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " ural_pruning_dense (PruneL \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " owMagnitude) \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " dropout (Dropout) (None, 1024) 0 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " dense (Dense) (None, 10) 10250 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "=================================================================\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Total params: 6538160 (24.94 MB)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Trainable params: 3274762 (12.49 MB)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Non-trainable params: 3263398 (12.45 MB)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "_________________________________________________________________\n" ] } ], "source": [ "model = keras.Sequential([\n", " prune_low_magnitude(\n", " keras.layers.Conv2D(\n", " 32, 5, padding='same', activation='relu',\n", " input_shape=(28, 28, 1),\n", " name=\"pruning_sparsity_0_5\"),\n", " **pruning_params_sparsity_0_5),\n", " keras.layers.MaxPooling2D((2, 2), (2, 2), padding='same'),\n", " prune_low_magnitude(\n", " keras.layers.Conv2D(\n", " 64, 5, padding='same',\n", " name=\"structural_pruning\"),\n", " **pruning_params_2_by_4),\n", " keras.layers.BatchNormalization(),\n", " keras.layers.ReLU(),\n", " keras.layers.MaxPooling2D((2, 2), (2, 2), padding='same'),\n", " keras.layers.Flatten(),\n", " prune_low_magnitude(\n", " keras.layers.Dense(\n", " 1024, activation='relu',\n", " name=\"structural_pruning_dense\"),\n", " **pruning_params_2_by_4),\n", " keras.layers.Dropout(0.4),\n", " keras.layers.Dense(10)\n", "])\n", "\n", "model.compile(optimizer='adam',\n", " loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n", " metrics=['accuracy'])\n", "\n", "model.summary()" ] }, { "cell_type": "markdown", "metadata": { "id": "U_ddzMppWr_G" }, "source": [ "Train and evaluate the model." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:19:13.450903Z", "iopub.status.busy": "2024-03-09T12:19:13.450261Z", "iopub.status.idle": "2024-03-09T12:20:00.649190Z", "shell.execute_reply": "2024-03-09T12:20:00.648402Z" }, "id": "F4CnppA1Wr_H" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Pruned test accuracy: 0.9897000193595886\n" ] } ], "source": [ "batch_size = 128\n", "epochs = 2\n", "\n", "model.fit(\n", " train_images,\n", " train_labels,\n", " batch_size=batch_size,\n", " epochs=epochs,\n", " verbose=0,\n", " callbacks=tfmot.sparsity.keras.UpdatePruningStep(),\n", " validation_split=0.1)\n", "\n", "_, pruned_model_accuracy = model.evaluate(test_images, test_labels, verbose=0)\n", "print('Pruned test accuracy:', pruned_model_accuracy)" ] }, { "cell_type": "markdown", "metadata": { "id": "bA8EDPHMWr_H" }, "source": [ "Remove the pruning wrapper so that it is not included in the model when you convert it to TensorFlow Lite format." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:00.653242Z", "iopub.status.busy": "2024-03-09T12:20:00.652563Z", "iopub.status.idle": "2024-03-09T12:20:00.700091Z", "shell.execute_reply": "2024-03-09T12:20:00.699294Z" }, "id": "3wn-OQ_gWr_H" }, "outputs": [], "source": [ "model = tfmot.sparsity.keras.strip_pruning(model)" ] }, { "cell_type": "markdown", "metadata": { "id": "eM28m66YWr_H" }, "source": [ "## Convert model to tflite format" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:00.704118Z", "iopub.status.busy": "2024-03-09T12:20:00.703568Z", "iopub.status.idle": "2024-03-09T12:20:02.775948Z", "shell.execute_reply": "2024-03-09T12:20:02.775118Z" }, "id": "EJ7DsA6-Wr_I" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Assets written to: /tmpfs/tmp/tmp04kvq4rj/assets\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Assets written to: /tmpfs/tmp/tmp04kvq4rj/assets\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Saved converted pruned model to: /tmpfs/tmp/tmp218fgsbq.tflite\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", "W0000 00:00:1709986802.425001 13320 tf_tfl_flatbuffer_helpers.cc:390] Ignored output_format.\n", "W0000 00:00:1709986802.425052 13320 tf_tfl_flatbuffer_helpers.cc:393] Ignored drop_control_dependency.\n" ] } ], "source": [ "import tempfile\n", "\n", "converter = tf.lite.TFLiteConverter.from_keras_model(model)\n", "tflite_model = converter.convert()\n", "\n", "_, tflite_file = tempfile.mkstemp('.tflite')\n", "print('Saved converted pruned model to:', tflite_file)\n", "with open(tflite_file, 'wb') as f:\n", " f.write(tflite_model)" ] }, { "cell_type": "markdown", "metadata": { "id": "S44x9Rz3Wr_I" }, "source": [ "## Visualize and check weights" ] }, { "cell_type": "markdown", "metadata": { "id": "_CTu0wxFWr_J" }, "source": [ "Now visualize the structure of weights in the `Dense` layer pruned with 2 by 4 sparsity. Extract the weights from the tflite file." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:02.780418Z", "iopub.status.busy": "2024-03-09T12:20:02.779759Z", "iopub.status.idle": "2024-03-09T12:20:02.786154Z", "shell.execute_reply": "2024-03-09T12:20:02.785470Z" }, "id": "fOIp6QB5Wr_J" }, "outputs": [], "source": [ "# Load tflite file with the created pruned model\n", "interpreter = tf.lite.Interpreter(model_path=tflite_file, experimental_preserve_all_tensors=True)\n", "interpreter.allocate_tensors()\n", "\n", "details = interpreter.get_tensor_details()\n", "\n", "# Weights of the dense layer that has been pruned.\n", "tensor_name = 'structural_pruning_dense/MatMul'\n", "detail = [x for x in details if tensor_name in x[\"name\"]]\n", "\n", "# We need the first layer.\n", "tensor_data = interpreter.tensor(detail[0][\"index\"])()" ] }, { "cell_type": "markdown", "metadata": { "id": "yy0jTs_QWr_K" }, "source": [ "To verify that we selected the correct layer that has been pruned, print the shape of the weight tensor." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:02.789639Z", "iopub.status.busy": "2024-03-09T12:20:02.789072Z", "iopub.status.idle": "2024-03-09T12:20:02.792927Z", "shell.execute_reply": "2024-03-09T12:20:02.792252Z" }, "id": "mCDkwMUPWr_K" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Shape of Dense layer is (1, 1024)\n" ] } ], "source": [ "print(f\"Shape of Dense layer is {tensor_data.shape}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "mvYTILeUWr_K" }, "source": [ "Now we visualize the structure for a small subset of the weight tensor. The structure of the weight tensor is sparse in the last dimension, using the `(2,4)` pattern: two elements out of four are zeros. To make the visualization more clear, we replace all non-zero values with ones." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:02.796358Z", "iopub.status.busy": "2024-03-09T12:20:02.795857Z", "iopub.status.idle": "2024-03-09T12:20:03.467538Z", "shell.execute_reply": "2024-03-09T12:20:03.466512Z" }, "id": "WZfn34bRWr_K" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "# The value 24 is chosen for convenience.\n", "width = height = 24\n", "\n", "subset_values_to_display = tensor_data[0:height, 0:width]\n", "\n", "val_ones = np.ones([height, width])\n", "val_zeros = np.zeros([height, width])\n", "subset_values_to_display = np.where(abs(subset_values_to_display) > 0, val_ones, val_zeros)" ] }, { "cell_type": "markdown", "metadata": { "id": "fOfWvKwKWr_L" }, "source": [ "Define the auxiliary function to draw separation lines to see the structure clearly." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:03.472239Z", "iopub.status.busy": "2024-03-09T12:20:03.471537Z", "iopub.status.idle": "2024-03-09T12:20:03.477811Z", "shell.execute_reply": "2024-03-09T12:20:03.477108Z" }, "id": "LUplruw9Wr_L" }, "outputs": [], "source": [ "def plot_separation_lines(height, width):\n", "\n", " block_size = [1, 4]\n", "\n", " # Add separation lines to the figure.\n", " num_hlines = int((height - 1) / block_size[0])\n", " num_vlines = int((width - 1) / block_size[1])\n", " line_y_pos = [y * block_size[0] for y in range(1, num_hlines + 1)]\n", " line_x_pos = [x * block_size[1] for x in range(1, num_vlines + 1)]\n", "\n", " for y_pos in line_y_pos:\n", " plt.plot([-0.5, width], [y_pos - 0.5 , y_pos - 0.5], color='w')\n", "\n", " for x_pos in line_x_pos:\n", " plt.plot([x_pos - 0.5, x_pos - 0.5], [-0.5, height], color='w')" ] }, { "cell_type": "markdown", "metadata": { "id": "sbyjrRgLWr_L" }, "source": [ "Now visualize the subset of the weight tensor." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:03.481280Z", "iopub.status.busy": "2024-03-09T12:20:03.480780Z", "iopub.status.idle": "2024-03-09T12:20:03.706043Z", "shell.execute_reply": "2024-03-09T12:20:03.705295Z" }, "id": "ATeyf5vCWr_L" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdAAAAGkCAYAAACB7nTXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAzY0lEQVR4nO3deViU5eL/8c+AMqjkUioUkShmapp6UPmaetSisEyzb57MzIXUVlvktGi/EtvQQk1Ly2OZLebl1r5pRnpOJenJwmyxzHI5fgP1mKkYIjPP7w8OcxxhxnkWwXHer+ua65KHZ7lnbvDmXp7n4zIMwxAAADAlqqYLAABAOKIBBQDAAhpQAAAsoAEFAMACGlAAACygAQUAwAIaUAAALKABBQDAAhpQAAAsoAGFn+TkZI0cObKmi2HZyJEjlZycXGPXz83NVYsWLRQdHa2OHTvWWDlOJb1791bv3r1ruhhAJad0A7px40YNGjRIzZo1U2xsrBITE3XJJZfo6aef9tsvJydHb775Zs0U8iQqA+z58MMPde+996p79+6aP3++cnJyTuj1Ro4cKZfL5XvFxcWpRYsWGjRokF577TV5vd4Ten0g0tWq6QKcKGvWrFGfPn10zjnnaMyYMUpISNCOHTv0+eefa+bMmbr99tt9++bk5GjQoEEaOHBgjZX3ZCjDqeC5556rsYbj448/VlRUlObNm6eYmJhquabb7dbzzz8vSfrjjz+0bds2vfPOOxo0aJB69+6tt956S/Xr16+WsgCR5pRtQB977DE1aNBA//znP9WwYUO/7+3atcvyeYuLi1WvXj2bpaseXq9XpaWlio2Nremi+DmRn2Ht2rVPyHlDsWvXLtWpU8exxtMwDJWUlKhOnToB96lVq5auv/56v22PPvqopkyZogkTJmjMmDFavHixI+VBZSfr7xiqxyk7hLtlyxadf/75lRpPSWratKnv3y6XS8XFxXrppZd8Q2EVc4CTJk2Sy+XSd999p+uuu06NGjVSjx49JAWel6lqDs7r9WrmzJlq3769YmNj1aRJE/Xt21dffPHFccsQaE6vomxHc7lcGjt2rF599VWdf/75crvdWr58uSRp6tSpuvDCC3XGGWeoTp06Sk1N1bJly0L4JCvbunWrXC6Xpk6dqieffFLNmjVTnTp11KtXL33zzTeVPo+4uDht2bJFl19+uU477TQNHTpUUuD51mM/29WrV8vlcmnJkiV67LHHdPbZZys2NlYXX3yxfvrpp0rXO/rzOrqsc+fOVUpKitxut7p06aJ//vOfla69dOlStW3bVrGxsWrXrp3eeOONkOZVXS6X5s+fr+LiYl8dvvjii5KksrIyPfLII75rJycn6/7779fhw4f9zpGcnKwrrrhCK1asUOfOnVWnTh397W9/C3rdQMaPH69LL71US5cu1Y8//uj3vQ8++EA9e/ZUvXr1dNppp6lfv3769ttv/fapqLedO3dq4MCBiouLU5MmTXT33XfL4/H47bto0SKlpqbqtNNOU/369dW+fXvNnDnTb599+/bprrvuUlJSktxut1q2bKnHH3/c0mhBaWmpJk6cqNTUVDVo0ED16tVTz549tWrVKt8+hmEoOTlZV155ZaXjS0pK1KBBA910002+bYcPH1Z2drZatmwpt9utpKQk3XvvvZXqKNjvGCLPKdsDbdasmfLz8/XNN9+oXbt2Afd75ZVXNHr0aHXt2lU33nijJCklJcVvn7/85S8699xzlZOTIyvpb6NGjdKLL76oyy67TKNHj1ZZWZk++eQTff755+rcuXNIZQjVxx9/rCVLlmjs2LFq3Lix7z/+mTNnasCAARo6dKhKS0u1aNEi/eUvf9G7776rfv36WbrWyy+/rAMHDui2225TSUmJZs6cqYsuukgbN25UfHy8b7+ysjJlZGSoR48emjp1qurWrWvpelOmTFFUVJTuvvtu/f7773riiSc0dOhQrV279rjHLly4UAcOHNBNN90kl8ulJ554Qv/7v/+rn3/+2ddrfe+99zR48GC1b99ekydP1m+//aZRo0YpMTHxuOd/5ZVXNHfuXK1bt843pHrhhRdKkkaPHq2XXnpJgwYN0l//+letXbtWkydP1vfff6833njD7zw//PCDhgwZoptuukljxozReeedZ/Zj8hk2bJg+/PBDrVy5Uq1atfKVc8SIEcrIyNDjjz+uQ4cO6dlnn1WPHj301Vdf+f2h4PF4lJGRobS0NE2dOlUfffSRpk2bppSUFN1yyy2SpJUrV2rIkCG6+OKL9fjjj0uSvv/+e3322We68847JUmHDh1Sr169tHPnTt10000655xztGbNGk2YMEG//vqrZsyYYep97d+/X88//7yGDBmiMWPG6MCBA5o3b54yMjK0bt06dezYUS6XS9dff72eeOIJ7d27V6effrrv+HfeeUf79+/39dy9Xq8GDBigTz/9VDfeeKPatGmjjRs36sknn9SPP/5YaW1CoN8xRCDjFPXhhx8a0dHRRnR0tNGtWzfj3nvvNVasWGGUlpZW2rdevXrGiBEjKm3Pzs42JBlDhgyp9L1evXoZvXr1qrR9xIgRRrNmzXxff/zxx4Yk44477qi0r9frPW4Zjj3fsWU7miQjKirK+Pbbbyvtf+jQIb+vS0tLjXbt2hkXXXSR3/ZmzZpVWY6j/fLLL4Yko06dOsa//vUv3/a1a9cakoxx48b5lV+SMX78+ErnCXStYz/bVatWGZKMNm3aGIcPH/ZtnzlzpiHJ2Lhxo9/1jv68Ksp6xhlnGHv37vVtf+uttwxJxjvvvOPb1r59e+Pss882Dhw44Nu2evVqQ1KVdXCsESNGGPXq1fPbVlBQYEgyRo8e7bf97rvvNiQZH3/8sd/nIclYvnz5ca8V6HpH++qrr/zq48CBA0bDhg2NMWPG+O1XWFhoNGjQwG97Rb09/PDDfvt26tTJSE1N9X195513GvXr1zfKysoCluORRx4x6tWrZ/z4449+28ePH29ER0cb27dvD/o+j/15KCsr8/s5MAzD+O2334z4+Hjjhhtu8G374YcfDEnGs88+67fvgAEDjOTkZN/v3yuvvGJERUUZn3zyid9+c+bMMSQZn332mW9bsN8xRJ5Tdgj3kksuUX5+vgYMGKANGzboiSeeUEZGhhITE/X222+bOtfNN99suRyvvfaaXC6XsrOzK33v2CFYJ/Tq1Utt27attP3oebTffvtNv//+u3r27Kkvv/zS8rUGDhzo1zvr2rWr0tLS9P7771fat6LHYkdmZqbf/GLPnj0lST///PNxjx08eLAaNWoU8Nj/+7//08aNGzV8+HDFxcX59uvVq5fat29vucwVn0VWVpbf9r/+9a+Synu9R2vevLkyMjIsX+9oFe/jwIEDksp7i/v27dOQIUO0Z88e3ys6OlppaWl+Q6AVjv3Z79mzp9/n3bBhQxUXF2vlypUBy7F06VL17NlTjRo18rtuenq6PB6P/vGPf5h6X9HR0b6fA6/Xq71796qsrEydO3f2+3lu1aqV0tLS9Oqrr/q27d27Vx988IGGDh3q+/1bunSp2rRpo9atW/uV76KLLpKkSp9LoN8xRJ5TdghXkrp06aLXX39dpaWl2rBhg9544w09+eSTGjRokAoKCkL+JWjevLnlMmzZskVnnXWW3xDSiRSorO+++64effRRFRQU+M3r2GnEzz333ErbWrVqpSVLlvhtq1Wrls4++2zL16lwzjnn+H1d0SD+9ttvto/dtm2bJKlly5aVjm3ZsqXlPzS2bdumqKioSudNSEhQw4YNfdetYOdn7VgHDx6UJJ122mmSpM2bN0uSr2E41rGrdSvm64/WqFEjv8/71ltv1ZIlS3TZZZcpMTFRl156qa655hr17dvXt8/mzZv19ddfVzpXBSuL+l566SVNmzZNmzZt0pEjR3zbj/38hg8frrFjx2rbtm1q1qyZli5dqiNHjmjYsGF+5fv+++9DLp+TdYTwdko3oBViYmLUpUsXdenSRa1atVJmZqaWLl1aZa+wKlWtgnS5XFXOhx67wMKuQA1coOtUVdZPPvlEAwYM0J///Gc988wzOvPMM1W7dm3Nnz9fCxcudLS8VXG73YqKqjzYEey9RUdHV9pe1TZJIc1L2znWCaH+oRJsxa1ZFQu6KhrvigU7r7zyihISEirtX6uW/38HgT6zozVt2lQFBQVasWKFPvjgA33wwQeaP3++hg8frpdeesl33UsuuUT33ntvleeomJ8N1YIFCzRy5EgNHDhQ99xzj5o2baro6GhNnjxZW7Zs8dv32muv1bhx4/Tqq6/q/vvv14IFC9S5c2e/uWWv16v27dtr+vTpVV4vKSnJ72sn6wjhLSIa0KN17txZkvTrr7/6tlnphTVq1KjKocNjexQpKSlasWJFpYUMxwpUhkaNGmnfvn3HvU4wr732mmJjY7VixQq53W7f9vnz54d8jqpU9GiO9uOPP4a8qCLYe2vRooWtspnVrFkzSaq0qjfQNjPn9Xq92rx5s9q0aePbXlRUpH379vmueyK88sorcrlcuuSSSyT9d2Fa06ZNlZ6e7th1YmJi1L9/f/Xv319er1e33nqr/va3v+nBBx9Uy5YtlZKSooMHDzp2zWXLlqlFixZ6/fXX/X5vqvqD+PTTT1e/fv306quvaujQofrss88qLVpKSUnRhg0bdPHFF5+QaRWcuk7ZOdBVq1ZV2buomJM6+i/QevXqVfkfeTApKSnatGmTdu/e7du2YcMGffbZZ377XX311TIMQw899FClcxxdvkBlSElJ0e+//66vv/7at+3XX3+ttHozmOjoaLlcLr9e69atW20/+ejNN9/Uzp07fV+vW7dOa9eu1WWXXRbS8SkpKfr8889VWlrq2/buu+9qx44dtsplxVlnnaV27drp5Zdf9g19StLf//53bdy40fJ5L7/8ckmq9J92RW/H6gro45kyZYo+/PBDDR482DfUnpGRofr16ysnJ8dv2LPC0T/Lofr3v//t93VUVJQuuOACSfJNFVxzzTXKz8/XihUrKh2/b98+lZWVmbpmRc/46N+ftWvXKj8/v8r9hw0bpu+++0733HOPoqOjde211/p9/5prrtHOnTv13HPPVTr2jz/+UHFxsanyIXKcsj3Q22+/XYcOHdJVV12l1q1bq7S0VGvWrNHixYuVnJyszMxM376pqan66KOPNH36dJ111llq3ry50tLSgp7/hhtu0PTp05WRkaFRo0Zp165dmjNnjs4//3zt37/ft1+fPn00bNgwPfXUU9q8ebP69u0rr9erTz75RH369NHYsWODluHaa6/Vfffdp6uuukp33HGH77aDVq1ahTwv169fP02fPl19+/bVddddp127dmn27Nlq2bKlX8NsVsuWLdWjRw/dcsstOnz4sGbMmKEzzjgj4FDdsUaPHq1ly5apb9++uuaaa7RlyxYtWLDA8i08duXk5OjKK69U9+7dlZmZqd9++02zZs1Su3bt/BpVMzp06KARI0Zo7ty52rdvn3r16qV169bppZde0sCBA9WnTx9bZS4rK9OCBQskld/fuG3bNr399tv6+uuv1adPH82dO9e3b/369fXss89q2LBh+tOf/qRrr71WTZo00fbt2/Xee++pe/fumjVrlqnrjx49Wnv37tVFF12ks88+W9u2bdPTTz+tjh07+nrc99xzj95++21dccUVGjlypFJTU1VcXKyNGzdq2bJl2rp1qxo3bhzyNa+44gq9/vrruuqqq9SvXz/98ssvmjNnjtq2bVtlPfXr109nnHGGli5dqssuu8zvPnCpvIFdsmSJbr75Zq1atUrdu3eXx+PRpk2btGTJEt99uUAlNbgC+IT64IMPjBtuuMFo3bq1ERcXZ8TExBgtW7Y0br/9dqOoqMhv302bNhl//vOfjTp16hiSfLdWVNwqsnv37iqvsWDBAqNFixZGTEyM0bFjR2PFihVV3nZSVlZm5ObmGq1btzZiYmKMJk2aGJdddpmxfv3645bBMMpvyWnXrp0RExNjnHfeecaCBQsC3sZy2223VVnWefPmGeeee67hdruN1q1bG/Pnz6/yHGZuY8nNzTWmTZtmJCUlGW632+jZs6exYcMGv32Pd6vFtGnTjMTERMPtdhvdu3c3vvjii4C3sSxdurTKcsyfP9/velXdxpKbm1vp2pKM7Oxsv22LFi0yWrdubbjdbqNdu3bG22+/bVx99dVG69atg34mwd7rkSNHjIceesho3ry5Ubt2bSMpKcmYMGGCUVJS4rdfs2bNjH79+h33OkdfT5LvVbduXSM5Odm4+uqrjWXLlhkej6fK41atWmVkZGQYDRo0MGJjY42UlBRj5MiRxhdffHHc93Lsz8yyZcuMSy+91GjatKkRExNjnHPOOcZNN91k/Prrr37HHThwwJgwYYLRsmVLIyYmxmjcuLFx4YUXGlOnTq3y1rKjHfvz4PV6jZycHKNZs2aG2+02OnXqZLz77rsBb/kyDMO49dZbDUnGwoULq/x+aWmp8fjjjxvnn3++4Xa7jUaNGhmpqanGQw89ZPz+++++/YL9jiHyuAyjmlZR4JSxdetWNW/eXLm5ubr77rtrujgnXMeOHdWkSZOgt2rg5DZu3DjNmzdPhYWFlh/kARzrlJ0DBcw6cuRIpfm41atXa8OGDcRphbGSkhItWLBAV199NY0nHHXKzoECZu3cuVPp6em6/vrrddZZZ2nTpk2aM2eOEhISbD1MAzVj165d+uijj7Rs2TL9+9//9j1aEHAKDSjwH40aNVJqaqqef/557d69W/Xq1VO/fv00ZcoUnXHGGTVdPJj03XffaejQoWratKmeeuopAs7hOOZAAQBh7R//+Idyc3O1fv16321+x8tWXr16tbKysvTtt98qKSlJDzzwQJXpUMEwBwoACGvFxcXq0KGDZs+eHdL+v/zyi/r166c+ffqooKBAd911l0aPHl3lvcrB0AMFAJwyXC7XcXug9913n9577z2//OJrr71W+/btM5XvyhwoAMARJSUlfk8Ws8MwjEqPVnS73X6PI7UqPz+/0qMlMzIydNddd5k6T8gN6CVRfzF14qOt+L8Cy8dKUsZZHav/2q46ioovf0pP/7jrVXLo8HEOcJbdz8ySo96zt+gCyfij2otgp66tiq3r1jsHy5/mY/V92y23nfq2cm0n3rNd4VrXUs2UvYKdn5WohMrPr3ZKSUmJmjeLU+EuZwI14uLiKj1ZKjs7W5MmTbJ97sLCQsXHx/tti4+P1/79+/XHH3+EHBhADxQAYFtpaakKd3n0y/pmqn+aveU1+w941Tx1m3bs2OEXs+dE79NJNKAAAMfUPy3KdgPqO1f9+pVyap2QkJCgoqIiv21FRUWqX7++qbg6GlAAgGM8hlcem0tTPYbXmcIE0K1bN18yV4WVK1eqW7dups7DbSwAAMd4ZTjyMuPgwYMqKChQQUGBpPLbVAoKCrR9+3ZJ0oQJEzR8+HDf/jfffLN+/vln3Xvvvdq0aZOeeeYZLVmyROPGjTN1XRpQAEBY++KLL9SpUyd16tRJkpSVlaVOnTpp4sSJksozlCsaU0lq3ry53nvvPa1cuVIdOnTQtGnT9PzzzysjI8PUdRnCBQA4xiuv7A7Amj1D7969FeyRBi+++GKVx3z11Vdmi+aHBhQA4BiPYchj8/k8do+vLgzhAgBgAT1QAIBjrCwCquoc4YAGFADgGK8MeWhAAQAwJ5J6oMyBAgBgQcg90Ni6Np5B6Ar90Ugnz7X/e5zbzvWtsvmZWVPH/9+ugDueMLbq2iL/+rX2vm2X20Z9W7m2E+/ZrnCta6lmyu5TI/83hC6SVuGSBwoAsG3//v1q0KCBNn0fr9NsPgv3wAGvWrcp0u+//35CnoXrFIZwAQCwIOQh3P5x11u+yFtbvrZ8rCRdmXJBDVy7jqLi10qSBjUdpcPVnAdq9zOz5r/v2VuUJqn6MyLt1LVV7rpuLds1T5L192233Hbq28q1nXjPdoVrXUs1U/YKdn5WKvJ+TySPA6tw7R5fXUJuQG0FStsM662Rax81N3L40OFqD9SuiYBj//mgP2qkDNX+OVdi7X3bLreNz9r+Z0Zdm1WjZa+J/xtM8BhyII3FmbKcaAzhAgBgAfeBAgAc4/3Py+45wgENKADAMV655LF5X5S3Ju6rsoAhXAAALKAHCgBwjNcof9k9RzigAQUAOMbjwBCu3eOrCw0oAMAxkdSAMgcKAIAF9EABAI7xGi55DZurcG0eX11oQAEAjmEIFwAABEUeaEDkgZIRGTryQM0L17qWyAMNxqMoeWz2zTwOleVEIw8UAGBbRR5o3sZzVM9mHmjxAa8ubr+dPFAAAE5F5IEGRB4oGZGhIw/UvHCta4k80GAiaREReaCBkAdKRqQJ5IGaF651LZEHGozHiJLHsDkHGiYTiwzhAgBgAfeBAgAc45VLXpt9M6/CowtKAwoAcAxzoAAAWODMHGh49ECZAwUAwAJ6oAAAx5TPgdp8mDxDuACASON14FF+4bKIiCFcAAAsoAcKAHBMJC0iogEFADjGqyjuAz0WcWbVjDizakOcGXVtFnFmkIgzAwA4oCLO7JWv2qvuadG2znXogEfDOm086ePMGMIFADjGmUDt8OjXEWcWEHFmRFyFjjgz88K1riXizFCOOLNAiDMj4soE4szMC9e6logzC8ZrRMlrcxWuN0xmFhnCBQA4hiFcAAAs8EryGHYf5RceeBIRAAAW0AMFADjGmQcphEffjgYUAOAYZx7lFx4NaHiUEgCAkww9UACAY8gDBQDAAoZwAQBAUPRAAQCOceZBCuHRt6MBBQA4xmu45LX7IAWbx1cX8kADIg+UjMjQkQdqXrjWtUQeKMqRBwoAsK0iD3TKP3spNs7e4GbJwTKN7/J38kABAJHDmTSWU2wOlDxQ8kCrQ7hmRJIHal641rVEHmgwHrnksTknYPf46kIeaCDkgZIRaQJ5oOaFa11L5IGiHEO4AADHMIQLAIAFHtkfgvU4U5QTLjyaeQAATjL0QAEAjmEIFwAAC3iYPAAAYWT27NlKTk5WbGys0tLStG7duqD7z5gxQ+edd57q1KmjpKQkjRs3TiUlJaauSQMKAHCM8Z88UDsvw+QipMWLFysrK0vZ2dn68ssv1aFDB2VkZGjXrl1V7r9w4UKNHz9e2dnZ+v777zVv3jwtXrxY999/v6nr0oACABxTMYRr92XG9OnTNWbMGGVmZqpt27aaM2eO6tatqxdeeKHK/desWaPu3bvruuuuU3Jysi699FINGTLkuL3WY9GAAgBOSvv37/d7HT5c+QEWpaWlWr9+vdLT033boqKilJ6ervz8/CrPe+GFF2r9+vW+BvPnn3/W+++/r8svv9xU+VhEBABwjJNxZklJSX7bs7OzNWnSJL9te/bskcfjUXx8vN/2+Ph4bdq0qcrzX3fdddqzZ4969OghwzBUVlamm2++2fQQLnFmARFnRsRV6IgzMy9c61oiziwYJwO1d+zY4ZfG4nY787mvXr1aOTk5euaZZ5SWlqaffvpJd955px555BE9+OCDIZ+HODMAgG0VcWZ3fHql3HG1bZ3r8MEjeqrHWyHFmZWWlqpu3bpatmyZBg4c6Ns+YsQI7du3T2+99ValY3r27Kn/+Z//UW5urm/bggULdOONN+rgwYOKigrtDwDmQAEAYSsmJkapqanKy8vzbfN6vcrLy1O3bt2qPObQoUOVGsno6GhJkpk+JXFmARFnRsRV6IgzMy9c61oiziwYr6Lktdk3M3t8VlaWRowYoc6dO6tr166aMWOGiouLlZmZKUkaPny4EhMTNXnyZElS//79NX36dHXq1Mk3hPvggw+qf//+voY0FMSZBUKcGRFXJhBnZl641rVEnFkwHsMlj81FRGaPHzx4sHbv3q2JEyeqsLBQHTt21PLly30Li7Zv3+7X43zggQfkcrn0wAMPaOfOnWrSpIn69++vxx57zNR1WYULAAh7Y8eO1dixY6v83urVq/2+rlWrlrKzs5WdnW3rmjSgAADHOHkby8mOBhQA4BjDgTQWg4fJAwBw6qIHCgBwjEcueWw+mcPu8dWFBhQA4BivYX8O0xsmj/dhCBcAAAvogQIAHON1YBGR3eOrCw0oAMAxFaHYds8RDmhAAQCOqYknEdUU4swCIs6MiKvQEWdmXrjWtUScGcoRZwYAsK0izuzavOsVExdj61ylB0u16OIFIcWZ1SSGcAEAjvHKgUf5nWpzoMSZEWdWHcI14oo4M/PCta4l4sxQjjizQIgzI+LKBOLMzAvXupaIMwvGcGAVrnGq9UABADieSEpjCY+7VQEAOMnQAwUAOIYnEQEAYAFDuAAAICh6oAAAx/AsXAAALIikIVwaUACAYyKpAWUOFAAAC+iBAgAcE0k9UBpQAIBjaECrQB5oNSMPtNqQB0pdm0UeKCTyQAEADqjIA01//ybVqmfvD4yy4sP66PK/kQcKAIgcDOFWgTxQ8kCrQ7hmRJIHal641rVEHijKkQcaCHmgZESaQB6oeeFa1xJ5oMHQAwUAwIJIakB5kAIAABbQAwUAOCaSeqA0oAAAxxiGS4bNBtDu8dWFBhQA4JhIijNjDhQAAAvogQIAHMMcKAAAFkTSHChDuAAAWEAPFADgGIZwq0CcWTUjzqzaEGdGXZtFnFlgkTSES5wZAMC2ijiz1NfGORJntv7qJ4kzAwBEDsOBIdxw6YESZxYQcWZEXIWOODPzwrWuJeLMgjEk2R3XDJdhUeLMAiHOjIgrE4gzMy9c61oizgzlGMIFADjGK5dcEfIoPxpQAIBjImkVLg0oAMAxXsMlV4TcB8qTiAAAsIAeKADAMYbhwCrcMFmGSwMKAHBMJM2BMoQLAIAF9EABAI6JpB4oDSgAwDGswgUAAEHRAwUAOIZVuFUgD7SakQdabcgDpa7NIg80sPIG1O4cqEOFOcHIAwUA2FaRB3rugvGKrhtr61yeQyXafP0U8kABAJGDVbhVIA+UPNDqEK4ZkeSBmheudS2RBxqMIft5nuEyLEoeaCDkgZIRaQJ5oOaFa11L5IEGE0k9UG5jAQDAAuZAAQDOiaAxXHqgAADn/GcI185LFoZwZ8+ereTkZMXGxiotLU3r1q0Luv++fft022236cwzz5Tb7VarVq30/vvvm7omPVAAQFhbvHixsrKyNGfOHKWlpWnGjBnKyMjQDz/8oKZNm1bav7S0VJdccomaNm2qZcuWKTExUdu2bVPDhg1NXZcGFADgmJp4EtH06dM1ZswYZWZmSpLmzJmj9957Ty+88ILGjx9faf8XXnhBe/fu1Zo1a1S7dm1JUnJysulyMoQLAHCM3eHbo1fx7t+/3+91+HDl1c+lpaVav3690tPTfduioqKUnp6u/Pz8Ksv49ttvq1u3brrtttsUHx+vdu3aKScnRx6Px9R7pQEFAJyUkpKS1KBBA99r8uTJlfbZs2ePPB6P4uPj/bbHx8ersLCwyvP+/PPPWrZsmTwej95//309+OCDmjZtmh599FFT5WMIFwDgHIuLgCqdQ9KOHTv8HuXndjvzDGKv16umTZtq7ty5io6OVmpqqnbu3Knc3FxlZ2eHfB4aUACAY5ycA61fv/5xn4XbuHFjRUdHq6ioyG97UVGREhISqjzmzDPPVO3atRUdHe3b1qZNGxUWFqq0tFQxMTEhlZMhXACAcwyHXiGKiYlRamqq8vLyfNu8Xq/y8vLUrVu3Ko/p3r27fvrpJ3m9Xt+2H3/8UWeeeWbIjadEnFkQxJkRcRU64szMC9e6logzO9lkZWVpxIgR6ty5s7p27aoZM2aouLjYtyp3+PDhSkxM9M2h3nLLLZo1a5buvPNO3X777dq8ebNycnJ0xx13mLoucWYAANsq4szOmTtRUTbjzLyHSrT9xodNxZnNmjVLubm5KiwsVMeOHfXUU08pLS1NktS7d28lJyfrxRdf9O2fn5+vcePGqaCgQImJiRo1apTuu+8+v2Hd46EBBQDY5teA1rHZgP5hvgGtCcSZBUScGRFXoSPOzLxwrWuJODOUI84sEOLMiLgygTgz88K1riXizIKJpDgzbmMBADiHNBYAABAMPVAAgINcsn9fFEO4AIBIwxAuAAAIhh4oAMA5EdQDpQEFADjHwTSWkx0NKADAMU6msZzsmAMFAMACeqAAAOcwB1oZcWbVjDizakOcGXVtFnFmQUTQHChpLAAA2yrSWM5+6mFH0lj+dcfEUyeNBQCA43EZ5S+75wgHxJkFRJwZEVehI87MvHCta4k4s6CYA62MODPizKpDuEZcEWdmXrjWtUScGcoxhAsAcE4ELSKiAQUAOCeChnB5kAIAABbQAwUAOCeCeqA0oAAA59CAAgBgQQQtImIOFAAAC+iBAgAcw5OIAACwIoLmQBnCBQDAAhpQAAAsIA80IPJAyYgMHXmg5oVrXUvkgQbjkgNzoI6U5MQjDxQAYFtFHmizxx9VVKzNPNCSEm277wHyQAEAESSC7gMlDzQg8kDJiAwdeaDmhWtdS+SBBhVBq3DJAw2EPFAyIk0gD9S8cK1riTxQlGMIFwDgHHqgAACYx5OIAACwIoJ6oDxIAQAAC+iBAgCcE0E9UBpQAIBjImkOlCFcAAAsoAcKAHAOTyICAMCCCJoDZQgXAAALiDMLiDgzIq5CR5yZeeFa1xJxZsFE0iIi4swAALZVxJm1mJjjSJzZzw/ff9LHmTGECwCABcSZBUScGRFXoSPOzLxwrWuJOLOgHBjCDZdFRMSZBUKcGRFXJhBnZl641rVEnFlQEbQKl9tYAADOiaAGlDlQAAAsoAcKAHBMJN3GQg8UAAALaEABALCAIVwAgHMiaBERDSgAwDHMgQIAgKDogQIAnBUmPUi7aEABAM6JoDlQhnABALCAPNCAyAMlIzJ05IGaF651LZEHGkwkLSIiDxQAYFtFHui59+Qo2m0vD9RzuESbc0/+PFDmQAEAjomkHih5oAGRB0pGZOjIAzUvXOtaIg8U5cgDDYQ8UDIiTSAP1LxwrWuJPNCgamgV7uzZs5Wbm6vCwkJ16NBBTz/9tLp27Xrc4xYtWqQhQ4boyiuv1JtvvmnqmqzCBQA4x3DoZcLixYuVlZWl7Oxsffnll+rQoYMyMjK0a9euoMdt3bpVd999t3r27Gnugv9BAwoACGvTp0/XmDFjlJmZqbZt22rOnDmqW7euXnjhhYDHeDweDR06VA899JBatGhh6bo0oAAAx1QsIrL7kspX9h79Ony48tB5aWmp1q9fr/T0dN+2qKgopaenKz8/P2A5H374YTVt2lSjRo2y/F5pQAEAznFwCDcpKUkNGjTwvSZPnlzpcnv27JHH41F8fLzf9vj4eBUWFlZZxE8//VTz5s3Tc889Z+utchsLAOCktGPHDr/7QN1u+w+wOHDggIYNG6bnnntOjRs3tnUuGlAAgHMcXIVbv3794z5IoXHjxoqOjlZRUZHf9qKiIiUkJFTaf8uWLdq6dav69+/v2+b1eiVJtWrV0g8//KCUlJSQiskQLgDAMU7OgYYiJiZGqampysvL823zer3Ky8tTt27dKu3funVrbdy4UQUFBb7XgAED1KdPHxUUFCgpKSnka9MDBQCEtaysLI0YMUKdO3dW165dNWPGDBUXFyszM1OSNHz4cCUmJmry5MmKjY1Vu3bt/I5v2LChJFXafjw0oAAA59TAgxQGDx6s3bt3a+LEiSosLFTHjh21fPly38Ki7du3KyrK+QFXGlAAgGNq6lm4Y8eO1dixY6v83urVq4Me++KLL5q/oIgzC4I4MyKuQkecmXnhWtcScWZBRVCgNnFmAADbKuLM2tzmTJzZ97OJMwMARJII6oESZxYQcWZEXIWOODPzwrWuJeLMgnHJ/oxADcwoWEKcWSDEmRFxZQJxZuaFa11LxJmhHEO4AADnMIQLAIB5NXUbS03gUX4AAFhADxQA4ByGcAEAsChMGkC7GMIFAMACeqAAAMdE0iIiGlAAgHOYAwUAwLxI6oEyBwoAgAXEmQVEnBkRV6Ejzsy8cK1riTizoCJoCJc4MwCAbRVxZhfckKPoGJtxZqUl+vqFkz/OjCFcAAAsIM4sIOLMiLgKHXFm5oVrXUvEmQUVQUO4xJkFQpwZEVcmEGdmXrjWtUScWVAR1IAyhAsAgAXcBwoAcEwk3QdKAwoAcA5DuAAAIBh6oAAAx7gMQy6bjxewe3x1oQEFADgngoZwaUABAI6JpEVEzIECAGABPVAAgHMYwgUAwDyGcAEAQFDkgQZEHigZkaEjD9S8cK1riTzQoCJoCJc8UACAbRV5oKmDH3MkD3T94v9HHigAAKci8kADIg+UjMjQkQdqXrjWtUQeaFARNIRLHmgg5IGSEWkCeaDmhWtdS+SBHk+4rKK1iyFcAAAs4D5QAIBzDKP8ZfccYYAGFADgmEh6kAINKADAORG0iIg5UAAALKAHCgBwjMtb/rJ7jnBAAwoAcA5DuAAAIBh6oAAAx7AKFwAAK7gPtDLizKoZcWbVhjgz6tos4swgEWcGAHBARZxZWv9HVKu2vTizsiMlWvvOgyd9nBlDuAAA50TQKlzizAIizoyIq9ARZ2ZeuNa1RJwZyhFnFghxZkRcmUCcmXnhWtcScWbBsAoXAAArWIULAIB5kdQD5UlEAABYQA8UAOAcVuECAGAeQ7gAACAoeqAAAOd4jfKX3XOEARpQAIBzImgOlCFcAAAsoAcKAHCMSw4sInKkJCceDSgAwDk8iagy8kCrGXmg1YY8UOraLPJAIZEHCgBwQEUeaI+LJqlWLZt5oGUl+vTjSabyQGfPnq3c3FwVFhaqQ4cOevrpp9W1a9cq933uuef08ssv65tvvpEkpaamKicnJ+D+gbCICADgHMOhlwmLFy9WVlaWsrOz9eWXX6pDhw7KyMjQrl27qtx/9erVGjJkiFatWqX8/HwlJSXp0ksv1c6dO01dN+QeKHmg5IFWh3DNiCQP1LxwrWuJPNCqVPRAe/bOdqQH+snqh0LugaalpalLly6aNWuWJMnr9SopKUm33367xo8ff9zjPR6PGjVqpFmzZmn48OEhl5M80EDIAyUj0gTyQM0L17qWyAOtLvv37/f72u12y+32n38uLS3V+vXrNWHCBN+2qKgopaenKz8/P6TrHDp0SEeOHNHpp59uqnwM4QIAnON16CUpKSlJDRo08L0mT55c6XJ79uyRx+NRfHy83/b4+HgVFhaGVOT77rtPZ511ltLT0029VW5jAQA4xmUYctlcm1px/I4dO/yGcI/tfTphypQpWrRokVavXq3YWHNDzzSgAICTUv369Y87B9q4cWNFR0erqKjIb3tRUZESEhKCHjt16lRNmTJFH330kS64wPy8NkO4AADnVPMq3JiYGKWmpiovL8+3zev1Ki8vT926dQt43BNPPKFHHnlEy5cvV+fOnU28wf+iBwoAcE4NPIkoKytLI0aMUOfOndW1a1fNmDFDxcXFyszMlCQNHz5ciYmJvjnUxx9/XBMnTtTChQuVnJzsmyuNi4tTXFxcyNelAQUAhLXBgwdr9+7dmjhxogoLC9WxY0ctX77ct7Bo+/btior674Drs88+q9LSUg0aNMjvPNnZ2Zo0aVLI16UBBQA4xmU48DB5C8ePHTtWY8eOrfJ7q1ev9vt669at5i9QBRpQAIBzIuhh8iwiAgDAAnqgAADHuLzlL7vnCAfEmQVEnBkRV6Ejzsy8cK1riTizoCJoCJc4MwCAbRUPk+/d5f858jD51f98zFScWU1gDhQAAAtCHsIlzow4s+oQrhFXxJmZF651LRFnFoyTz8I92RFnFghxZkRcmUCcmXnhWtcScWZBRdAcKEO4AABYwG0sAADnGPLledo6RxigAQUAOCaS5kAZwgUAwAJ6oAAA5xhyYBGRIyU54WhAAQDOYRUuAAAIhh4oAMA5Xtl/tvKp9jB5AACOJ5JW4dKAAgCcE0FzoMSZBUScGRFXoSPOzLxwrWuJODOUI84MAGBbRZzZxW3vVq1oe39glHkOK++7qSd9nBlDuAAA5zCEWxlxZsSZVYdwjbgizsy8cK1riTgzlCPOLBDizIi4MoE4M/PCta4l4syC4jYWAADMi6TbWHgSEQAAFtADBQA4h0VEAABY4DUkl80G0BseDShDuAAAWEAPFADgHIZwAQCwwoEGNEwStWlAAQDOiaAeKHOgAABYQA8UAOAcryHbQ7BhsgqXBhQA4BzDW/6ye44wQB5oQOSBkhEZOvJAzQvXupbIA0U58kABALZV5IGmJ92iWlE280C9h/XRjmfJAwUARBDmQCsjD5Q80OoQrhmR5IGaF651LZEHinLkgQZCHigZkSaQB2peuNa1RB5oUBF0HyhDuAAA5xhyoAF1pCQnHA9SAADAAnqgAADnMIQLAIAFXq8kmw9C8J5iD1IAAOC4IqgHyhwoAAAW0AMFADgngnqgNKAAAOdE0JOIGMIFAMACeqAAAMcYhleGzTgyu8dXF+LMAiLOjIir0BFnZl641rVEnFlQhmF/CDZM5kCJMwMA2FYRZ3Zxw+Gq5Yqxda4yo1R5+14mzgwAEEEMBxYRhUm/jjizgIgzI+IqdMSZmReudS0RZxaU1yu5bM5hnmpzoMSZEWdWHcI14oo4M/PCta4l4sxQjiFcAIBzGMIFAMA8w+uVYXMI95S7jQUAgOOKoB4oTyICAMACeqAAAOd4DckVGT1QGlAAgHMMQ7YDtcOkAWUIFwAAC+iBAgAcY3gNGTaHcMPlCbP0QAEAzjG8zrxMmj17tpKTkxUbG6u0tDStW7cu6P5Lly5V69atFRsbq/bt2+v99983fU0aUABAWFu8eLGysrKUnZ2tL7/8Uh06dFBGRoZ27dpV5f5r1qzRkCFDNGrUKH311VcaOHCgBg4cqG+++cbUdWlAAQCOMbyGIy8zpk+frjFjxigzM1Nt27bVnDlzVLduXb3wwgtV7j9z5kz17dtX99xzj9q0aaNHHnlEf/rTnzRr1ixT16UBBQA4p5qHcEtLS7V+/Xqlp6f7tkVFRSk9PV35+flVHpOfn++3vyRlZGQE3D+QkBcRrfQuNXViJ62s4ac6vXNwQc0WoAZUS2pDFWq6rq2+75ost91rU9fm1XTZT2ZlOmL7QURlOiKpPGP0aG63W263f5j5nj175PF4FB8f77c9Pj5emzZtqvL8hYWFVe5fWFhoqpyswgUA2BYTE6OEhAR9Wmh+MU5V4uLilJSU5LctOztbkyZNcuT8TqABBQDYFhsbq19++UWlpaWOnM8wDLlcfhmLlXqfktS4cWNFR0erqKjIb3tRUZESEhKqPHdCQoKp/QOhAQUAOCI2NlaxsbHVes2YmBilpqYqLy9PAwcOlCR5vV7l5eVp7NixVR7TrVs35eXl6a677vJtW7lypbp162bq2jSgAICwlpWVpREjRqhz587q2rWrZsyYoeLiYmVmZkqShg8frsTERE2ePFmSdOedd6pXr16aNm2a+vXrp0WLFumLL77Q3LlzTV2XBhQAENYGDx6s3bt3a+LEiSosLFTHjh21fPly30Kh7du3KyrqvzedXHjhhVq4cKEeeOAB3X///Tr33HP15ptvql27dqau6zLC5ZlJAACcRLgPFAAAC2hAAQCwgAYUAAALaEABALCABhQAAAtoQAEAsIAGFAAAC2hAAQCwgAYUAAALaEABALCABhQAAAtoQAEAsOD/A8P/nfHMQy1hAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_separation_lines(height, width)\n", "\n", "plt.axis('off')\n", "plt.imshow(subset_values_to_display)\n", "plt.colorbar()\n", "plt.title(\"Structural pruning for Dense layer\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "72f7VlOLWr_M" }, "source": [ "Visualize weights for the `Conv2D` layer. The structural sparsity is applied in the last channel, similar to the `Dense` layer. Only the second `Conv2D` layer is structurally pruned as pointed out above." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:03.709585Z", "iopub.status.busy": "2024-03-09T12:20:03.709296Z", "iopub.status.idle": "2024-03-09T12:20:03.714518Z", "shell.execute_reply": "2024-03-09T12:20:03.713795Z" }, "id": "_Dkbt7eRWr_M" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Shape of the weight tensor is (64, 5, 5, 32)\n" ] } ], "source": [ "# Get weights of the convolutional layer that has been pruned with 2 by 4 sparsity.\n", "op_details = interpreter._get_ops_details()\n", "op_name = 'CONV_2D'\n", "op_detail = [x for x in op_details if op_name in x[\"op_name\"]]\n", "tensor_data = interpreter.tensor(op_detail[1][\"inputs\"][1])()\n", "print(f\"Shape of the weight tensor is {tensor_data.shape}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "m7a6uTdLWr_M" }, "source": [ "Similar to the weights of `Dense` layer, the last dimension of the kernel has a (2, 4) structure." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:03.718092Z", "iopub.status.busy": "2024-03-09T12:20:03.717476Z", "iopub.status.idle": "2024-03-09T12:20:03.888151Z", "shell.execute_reply": "2024-03-09T12:20:03.887355Z" }, "id": "wyvLpfa6Wr_M" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdQAAAGkCAYAAACIBdStAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABAUElEQVR4nO3de3wMV+M/8M9uSCJJiWtCmrqVRt3iCdHQuFQI1aCKFBVS5denDSUvT1WLKN9SpUpd6kHRFg9Fv33aUkrwbVopLaVVpS51bROX1l2y7JzfH7rbrGxkd8/Mzq75vF+vvF5M5nZmZvfknDkzH5MQQoCIiIikmPXeASIiorsBK1QiIiIVsEIlIiJSAStUIiIiFbBCJSIiUgErVCIiIhWwQiUiIlIBK1QiIiIVsEIlIiJSAStUL6hVqxYGDRpk//+2bdtgMpmwbds23fbJX5hMJkyYMEHXfZA5X7Zl16xZo/6OuSA/Px+9evVC5cqVYTKZMHPmTF32w+j4mTcGtyvUH3/8Eb169ULNmjURHByMqKgodOzYEbNnz3aYb/Lkyfj444/V2k+P+MI+ELlqxYoVqld4I0eOxMaNGzFmzBh88MEH6Ny5s6rrd6agoABvvfUWWrZsiQoVKiA4OBj169dHRkYGfvnlF823X5rz589j2rRpaNOmDapWrYrw8HA89NBDWLVqVbF5ly5dCpPJZP8JDg5GjRo1kJycjLfffhuXL1/WoQTkq8q4M/P27dvRvn173HfffRgyZAgiIyNx8uRJfPPNN5g1axaGDRtmn3fy5Mno1asXevToofY+u8wX9oH8X5s2bXD9+nUEBgZqup0VK1Zg3759GDFihGrr3LJlC7p3745Ro0apts47OXfuHDp37oxdu3bhscceQ79+/RAWFoaDBw9i5cqVWLBgASwWi1f2pSS5ubl45ZVX8Oijj2Ls2LEoU6YM1q5diyeffBL79+/Hq6++WmyZiRMnonbt2rhx4wby8vKwbds2jBgxAjNmzMAnn3yCJk2a6FAS8jVuVaivvfYaKlSogG+//Rbh4eEOvztz5ozHO3H16lWEhoZ6vLw3KYoCi8WC4OBgvXdFFTdv3oSiKJpXFv7MbDb77fk+c+ZMsc+qjIKCAgQGBsJsdt65NWjQIHz//fdYs2YNnnjiCYffTZo0Ca+88opq++Kphg0b4tChQ6hZs6Z92nPPPYekpCRMnToVL774YrHvoy5duqB58+b2/48ZMwZbtmzBY489hm7duuHnn39GuXLlvFYGtfnTd7Avc6vL98iRI2jYsKHTD2i1atXs/zaZTLh69Sree+89e1eJ7R7ihAkTYDKZsH//fvTr1w8VK1bEww8/DABo164d2rVrV2zdgwYNQq1atRymKYqCWbNmoXHjxggODkbVqlXRuXNnfPfdd6Xug7P1Fd23okwmEzIyMrB8+XI0bNgQQUFB2LBhAwBg+vTpaNWqFSpXroxy5cohLi7Oo3tlWVlZKFu2LM6ePVvsd0OHDkV4eDgKCgpKXH7QoEEICwvD0aNHkZycjNDQUNSoUQMTJ05E0TChY8eOwWQyYfr06Zg5cybq1q2LoKAg7N+/3961dezYMYd1O7v3065dOzRq1Aj79+9H+/btERISgqioKLzxxhvF9q2wsBBZWVm4//77ERQUhOjoaLz44osoLCwsNt/IkSNRtWpV3HPPPejWrRtOnTpV6rETQqBKlSrIzMy0T1MUBeHh4QgICMCFCxfs06dOnYoyZcrgypUr9mkHDhxAr169UKlSJQQHB6N58+b45JNPSj0GADB37lzUqVMH5cqVQ3x8PHJyckq8hhVFwWuvvYZ7770XwcHB6NChAw4fPuxwTNetW4fjx4/br9ei1+js2bPRsGFDhISEoGLFimjevDlWrFhR4nGxnU8hBObOnWtfp83Ro0fRu3dvVKpUCSEhIXjooYewbt06p+VeuXIlxo4di6ioKISEhODSpUtOt7ljxw6sW7cOgwcPLlaZAkBQUBCmT5/uMG3Lli1ITExEaGgowsPD0b17d/z8888O89g+l4cPH8agQYMQHh6OChUqID09HdeuXbPP16hRI7Rv377YdhVFQVRUFHr16gUAqF27tkNlCtz6nPfo0QOFhYU4evSo0/Ld7pFHHsG4ceNw/PhxLFu2zKVlisrJyUHv3r1x33332T8bI0eOxPXr1+3zLFmyBCaTCd9//32x5SdPnoyAgACcPn3aPm3Hjh3o3LkzKlSogJCQELRt2xZff/21w3J3+g4mOW61UGvWrInc3Fzs27cPjRo1KnG+Dz74AM888wzi4+MxdOhQAEDdunUd5unduzfq1auHyZMnw5MEucGDB2Pp0qXo0qULnnnmGdy8eRM5OTn45ptv0Lx5c5f2wVVbtmzBhx9+iIyMDFSpUsX+RTdr1ix069YN/fv3h8ViwcqVK9G7d2989tln6Nq1q8vrHzBgACZOnIhVq1YhIyPDPt1isdj/0i+thWS1WtG5c2c89NBDeOONN7BhwwZkZWXh5s2bmDhxosO8S5YsQUFBAYYOHYqgoCBUqlTJ9YPxlz///BOdO3dGz5490adPH6xZswajR49G48aN0aVLFwC3vsi6deuGr776CkOHDkWDBg3w448/4q233sIvv/zicH/7mWeewbJly9CvXz+0atUKW7ZscekYmkwmtG7dGl9++aV92g8//ICLFy/CbDbj66+/tq8nJycHzZo1Q1hYGADgp59+QuvWrREVFYWXXnoJoaGh+PDDD9GjRw+sXbsWjz/+eInbfeedd5CRkYHExESMHDkSx44dQ48ePVCxYkXce++9xeZ//fXXYTabMWrUKFy8eBFvvPEG+vfvjx07dgAAXnnlFVy8eBGnTp3CW2+9BQD2/Vy4cCGGDx+OXr164YUXXkBBQQF++OEH7NixA/369XO6f23atMEHH3yAAQMGoGPHjkhLS7P/Lj8/H61atcK1a9cwfPhwVK5cGe+99x66deuGNWvWFCv3pEmTEBgYiFGjRqGwsLDE3gzbHyIDBgwo8bgVtXnzZnTp0gV16tTBhAkTcP36dcyePRutW7fG7t27i/3R26dPH9SuXRtTpkzB7t27sWjRIlSrVg1Tp04FAKSmpmLChAnIy8tDZGSkfbmvvvoKv/32G5588sk77k9eXh4AoEqVKi7tv62sL7/8Mr744gsMGTLE5eUAYPXq1bh27Rr++c9/onLlyti5cydmz56NU6dOYfXq1QCAXr164fnnn8fy5cvRrFkzh+WXL1+Odu3aISoqCsCt76kuXbogLi4OWVlZMJvNWLJkCR555BHk5OQgPj7eYXnZ72ByQrjhiy++EAEBASIgIEAkJCSIF198UWzcuFFYLJZi84aGhoqBAwcWm56VlSUAiL59+xb7Xdu2bUXbtm2LTR84cKCoWbOm/f9btmwRAMTw4cOLzasoSqn7cPv6bt+3ogAIs9ksfvrpp2LzX7t2zeH/FotFNGrUSDzyyCMO02vWrOmwH1u3bhUAxNatW+3TEhISRMuWLR2W++ijj4rN58zAgQMFADFs2DD7NEVRRNeuXUVgYKA4e/asEEKIX3/9VQAQ5cuXF2fOnHFYx5IlSwQA8euvvzpMd7avbdu2FQDE+++/b59WWFgoIiMjxRNPPGGf9sEHHwiz2SxycnIc1jl//nwBQHz99ddCCCH27NkjAIjnnnvOYb5+/foJACIrK+uO5Z82bZoICAgQly5dEkII8fbbb4uaNWuK+Ph4MXr0aCGEEFarVYSHh4uRI0fal+vQoYNo3LixKCgocDhurVq1EvXq1SvxGBQWForKlSuLFi1aiBs3btjnW7p0qQDgcA3blm3QoIEoLCy0T581a5YAIH788Uf7tK5duzq9Lrt37y4aNmx4x2NQEgDi+eefd5g2YsQIAcDhvFy+fFnUrl1b1KpVS1itVod9r1OnTrFr3ZnHH39cABB//vmnS/sWGxsrqlWrJs6fP2+ftnfvXmE2m0VaWpp9mu1z+fTTTxfbXuXKle3/P3jwoAAgZs+e7TDfc889J8LCwu5YhvPnz4tq1aqJxMREh+m2z8W3335b4rIVKlQQzZo1u2NZnX2OnO3PlClThMlkEsePH7dP69u3r6hRo4b9vAghxO7duwUAsWTJEiHEreu2Xr16Ijk52eE78Nq1a6J27dqiY8eO9ml3+g4mOW51+Xbs2BG5ubno1q0b9u7dizfeeAPJycmIiooq1k1Wmmeffdat+Ytau3YtTCYTsrKyiv3u9i5bNbRt2xYPPvhgselF75n8+eefuHjxIhITE7F79263t5GWloYdO3bgyJEj9mnLly9HdHQ02rZt69I6irZubV3VFosFmzdvdpjviSeeQNWqVd3ex6LCwsLw1FNP2f8fGBiI+Ph4h+6y1atXo0GDBoiJicG5c+fsP4888ggAYOvWrQCA9evXAwCGDx/usA1XB+ckJibCarVi+/btAG61RBMTE5GYmIicnBwAwL59+3DhwgUkJiYCAP744w9s2bIFffr0weXLl+37dv78eSQnJ+PQoUMOXWlFfffddzh//jyGDBmCMmX+7uTp378/Klas6HSZ9PR0h5adbT9c6V4MDw/HqVOn8O2337pwNEq3fv16xMfHO3TzhYWFYejQoTh27Bj279/vMP/AgQNduj9o6wq+5557Sp33999/x549ezBo0CCHHpImTZqgY8eO9muiqNu/MxITE3H+/Hn7duvXr4/Y2FiH0bpWqxVr1qxBSkpKiWVQFAX9+/fHhQsXij2t4IqwsDCPRvsW3Z+rV6/i3LlzaNWqFYQQDl28aWlp+O233+yfF+DWd0O5cuXsXet79uzBoUOH0K9fP5w/f95+PV+9ehUdOnTAl19+CUVRHLYv8x1Mzrn92EyLFi3w0Ucf4c8//8TOnTsxZswYXL58Gb169Sr2QbyT2rVru7tpuyNHjqBGjRoedVV6oqR9/eyzz/DQQw8hODgYlSpVQtWqVfHOO+/g4sWLbm8jNTUVQUFBWL58OQDg4sWL+Oyzz9C/f3+X/kgwm82oU6eOw7T69esDQLH7ojLH3ubee+8ttl8VK1bEn3/+af//oUOH8NNPP6Fq1aoOP7b9sg1kO378OMxmc7Eu+QceeMClffnHP/6BkJAQe+Vpq1DbtGmD7777DgUFBfbf2SqRw4cPQwiBcePGFds/2x9qJQ20O378OADg/vvvd5hepkwZp/fmAeC+++5z+L+t4i16vEoyevRohIWFIT4+HvXq1cPzzz9f7L6YO44fP+702DZo0MD++6JcvV7Kly8PAC5VLrZtlLQftsqgKFeOYWpqKr7++mv7H0Pbtm3DmTNnkJqaWuK+DBs2DBs2bMCiRYvQtGnTUvf9dleuXHHpj4jbnThxwv4HRVhYGKpWrWr/47nod0jHjh1RvXp1+3eDoij4z3/+g+7du9u3e+jQIQC3/vi5/XpetGgRCgsLi30vqfE9QI7cuodaVGBgIFq0aIEWLVqgfv36SE9Px+rVq522Gp1x9teibRDF7axWq6e76VRJFVRJ23G2rzk5OejWrRvatGmDefPmoXr16ihbtiyWLFlyx8EiJalYsSIee+wxLF++HOPHj8eaNWtQWFjo0ApUS0nH3pmSjklAQIDT6UXPn6IoaNy4MWbMmOF03ujo6NJ21SVly5ZFy5Yt8eWXX+Lw4cPIy8tDYmIiIiIicOPGDezYsQM5OTmIiYmxt8xtf62PGjUKycnJTtd7e4Upw5XjVZIGDRrg4MGD+Oyzz7BhwwasXbsW8+bNw/jx450+4qE2V0evxsTEALj1rLqtBa4mV45hamoqxowZg9WrV2PEiBH48MMPUaFChRKfv3311Vcxb948vP766y7f+y3q1KlTuHjxotvXitVqRceOHfHHH39g9OjRiImJQWhoKE6fPo1BgwY5tCYDAgLQr18/LFy4EPPmzcPXX3+N3377zeG7wTb/tGnTEBsb63SbtnvyNv48KtlXeVyhFmUbTv7777/bp3nS9VqxYkWnXWC3/8Vct25dbNy4EX/88ccdW6kl7UPFihUdRn+WtJ07Wbt2LYKDg7Fx40YEBQXZpy9ZssTlddwuLS0N3bt3x7fffmsfhNCwYUOXllUUBUePHrW3/gDYH6IvqdVUlO2v/duPizvH5HZ169bF3r170aFDhzteDzVr1oSiKDhy5IhDi+XgwYMubysxMRFTp07F5s2bUaVKFcTExMBkMqFhw4bIyclBTk4OHnvsMfv8ttZ82bJlkZSU5Fa5bCNEDx8+7DCq9ObNmzh27JjHzyTe6RiFhoYiNTUVqampsFgs6NmzJ1577TWMGTPG7Ud6atas6fTYHjhwwP57T6SkpGDKlClYtmxZqRWqbRsl7UeVKlU8eoyjdu3aiI+Ptw/w++ijj9CjRw+Hz6jN3LlzMWHCBIwYMQKjR492e1vArQGYAEr8o6wkP/74I3755Re89957DgPGNm3a5HT+tLQ0vPnmm/j000/x+eefo2rVqg7btPXulC9f3u3rmdTjVpfv1q1bnf5FbbvfUfTLMDQ01GmldSd169bFgQMHHB4f2bt3b7HurSeeeAJCCKd/nRfdv5L2oW7durh48SJ++OEH+7Tff/8d//u//+vyvgYEBMBkMjm04I4dOyb1ZqYuXbqgSpUqmDp1Kv7v//7P7dbpnDlz7P8WQmDOnDkoW7YsOnToUOqytg9k0dGyVqsVCxYscGsfiurTpw9Onz6NhQsXFvvd9evX7V16tlHBb7/9tsM87rw1KDExEYWFhZg5cyYefvhhe+WUmJiIDz74AL/99pvDl3y1atXQrl07/Pvf/3b4Q9DG2SNMNs2bN0flypWxcOFC3Lx50z59+fLlLnXhliQ0NNTp7YLz5887/D8wMBAPPvgghBC4ceOG29t59NFHsXPnTuTm5tqnXb16FQsWLECtWrWcjhdwRUJCAjp37oxFixY5/RxYLBb7CyaqV6+O2NhYvPfeew6f0X379uGLL77Ao48+6tE+ALdaqd988w0WL16Mc+fOOe3uXbVqFYYPH47+/fuX2INSmi1btmDSpEmoXbs2+vfv79ayttZ20e8rIQRmzZrldP4mTZqgSZMmWLRokf0lFEXv38fFxaFu3bqYPn26w2NhNne6nkk9brVQhw0bhmvXruHxxx9HTEwMLBYLtm/fjlWrVqFWrVpIT0+3zxsXF4fNmzdjxowZqFGjBmrXro2WLVvecf1PP/00ZsyYgeTkZAwePBhnzpzB/Pnz0bBhQ4dn39q3b48BAwbg7bffxqFDh9C5c2coioKcnBy0b9/ePjinpH148sknMXr0aDz++OMYPnw4rl27hnfeeQf169d3eUBR165dMWPGDHTu3Bn9+vXDmTNnMHfuXNx///0OFbU7ypYtiyeffBJz5sxBQEAA+vbt6/KywcHB2LBhAwYOHIiWLVvi888/x7p16/Dyyy+7NACpYcOGeOihhzBmzBh7y3/lypUOFYa7BgwYgA8//BDPPvsstm7ditatW8NqteLAgQP48MMPsXHjRjRv3hyxsbHo27cv5s2bh4sXL6JVq1bIzs52eE6zNAkJCShTpgwOHjxof0wKuPX4yDvvvAMAxVpNc+fOxcMPP4zGjRtjyJAhqFOnDvLz85Gbm4tTp05h7969TrcVGBiICRMmYNiwYXjkkUfQp08fHDt2DEuXLkXdunU9HhgXFxeHVatWITMzEy1atEBYWBhSUlLQqVMnREZGonXr1oiIiMDPP/+MOXPmoGvXrh7du3vppZfwn//8B126dMHw4cNRqVIlvPfee/j111+xdu3aEl/a4Ir3338fnTp1Qs+ePZGSkoIOHTogNDQUhw4dwsqVK/H777/bn0WdNm0aunTpgoSEBAwePNj+2EyFChWk3t/cp08fjBo1CqNGjUKlSpWKtdh27tyJtLQ0VK5cGR06dLDfm7Rp1apVsfEIn3/+OQ4cOICbN28iPz8fW7ZswaZNm1CzZk188sknbvcSxMTEoG7duhg1ahROnz6N8uXLY+3atXf8gywtLc3+B8ntf2ybzWYsWrQIXbp0QcOGDZGeno6oqCicPn0aW7duRfny5fHpp5+6tY/kAXeGBH/++efi6aefFjExMSIsLEwEBgaK+++/XwwbNkzk5+c7zHvgwAHRpk0bUa5cOQHA/tiIbci27VGO2y1btkzUqVNHBAYGitjYWLFx40anj7ncvHlTTJs2TcTExIjAwEBRtWpV0aVLF7Fr165S90GIW48ANWrUSAQGBooHHnhALFu2rMTHZm5/7MDm3XffFfXq1RNBQUEiJiZGLFmyxOk6XHlsxmbnzp0CgOjUqZPTbTozcOBAERoaKo4cOSI6deokQkJCREREhMjKynIYam97bGbatGlO13PkyBGRlJQkgoKCREREhHj55ZfFpk2bnD424+wxDmfnyWKxiKlTp4qGDRuKoKAgUbFiRREXFydeffVVcfHiRft8169fF8OHDxeVK1cWoaGhIiUlRZw8edKlx2ZsWrRoIQCIHTt22KedOnVKABDR0dElljktLU1ERkaKsmXLiqioKPHYY4+JNWvW2Ocp6XzZHs8JCgoS8fHx4uuvvxZxcXGic+fOxZZdvXq1w7K2c2F77EEIIa5cuSL69esnwsPDBQD7sfz3v/8t2rRpIypXriyCgoJE3bp1xb/+9S+H41eSkq7fI0eOiF69eonw8HARHBws4uPjxWeffeYwT0n7Xppr166J6dOnixYtWti/J+rVqyeGDRsmDh8+7DDv5s2bRevWrUW5cuVE+fLlRUpKiti/f7/DPCV9Z5T0qJcQQrRu3VoAEM8880yx39mWK+mn6Dm5fd7AwEARGRkpOnbsKGbNmmV/VKs0zq6h/fv3i6SkJBEWFiaqVKkihgwZIvbu3VtsH2x+//13ERAQIOrXr1/idr7//nvRs2dP+7VSs2ZN0adPH5GdnW2fp7TvYPKcSQg+0etL9u7di9jYWLz//vsuD5IYNGgQ1qxZ47Srh7xHURRUrVoVPXv2dNrNTSTj3LlzqF69OsaPH49x48bpvTvkBOPbfMzChQsRFhaGnj176r0rdAcFBQXFxhO8//77+OOPP5y+epBI1tKlS2G1Wj0ajUzeocooX5L36aefYv/+/ViwYAEyMjL4omof980332DkyJHo3bs3KleujN27d+Pdd99Fo0aN0Lt3b713j+4iW7Zswf79+/Haa6+hR48eLo3aJ32wQvURw4YNQ35+Ph599FGvPFtIcmrVqoXo6Gi8/fbb9kFcaWlpeP3115ncQ6qaOHEitm/fjtatW3v0JifyHt5DJSKiu8qXX36JadOmYdeuXfZHIkvLxd62bRsyMzPx008/ITo6GmPHjrUnlLmK91CJiOiucvXqVTRt2hRz5851af5ff/0VXbt2Rfv27bFnzx6MGDECzzzzDDZu3OjWdtlCJSKiu5bJZCq1hTp69GisW7cO+/bts0978sknceHCBXv+tSt4D5WIiDRRUFAAi8WiyrqEEMVemhIUFOT0tZLuys3NLfYCkOTkZJcTr2w8rlA7mvUbybjxtz3e3aCpHMwRt95+lBL2FAquFXq0Gq/vdxHJNWLdXiY4JAifXlkGAFDymwDiute2rRa3j3mRcy1TZlnePmZqnWsZelwnRcttpM+2zSZltXo7cpuCggLUrhmGvDPqhJuEhYUVe9Y+KytL6q1aNnl5eYiIiHCYFhERgUuXLuH69esuBwmwhUpERKqzWCzIO2PFr7tqovw9csN1Ll1WUDvuOE6ePGmPCQSgSutUTaxQiYhIM+XvMUtXqPZ1lS/vUKGqJTIyEvn5+Q7T8vPzUb58ebdi7lihEhGRZqxCgVVy6KtVKKXPJCEhIcGemmazadMmJCQkuLUePjZDRESaUSBU+XHHlStXsGfPHuzZswfArcdi9uzZgxMnTgAAxowZ45BD++yzz+Lo0aN48cUXceDAAcybNw8ffvghRo4c6dZ2WaESEdFd5bvvvkOzZs3QrFkzAEBmZiaaNWuG8ePHA7iVf22rXIFbwfTr1q3Dpk2b0LRpU7z55ptYtGiR28Hx7PIlIiLNKFAg22Hr7hratWtXLLyiqKVLlzpd5vvvv3d31xywQiUiIs1YhYBV8v1Bsst7C7t8iYiIVMAWKhERacaTQUXO1uEPWKESEZFmFAhYWaESERHJMVILlfdQiYiIVOBxCzU4RMd3KJpcfxWUOv7eXpBMub2+33/z5Hw5lrUcYCpxVtW3rRq3j3k5x397WGZZ3j5map1rGXpcJ0XLbaTPNgCPgwDcZaRRvsxDJSIyICWvHsyRhzRb/6VLl1ChQgUc+DkC90i+y/fyZQUxDfJx8eJFTd7lqxZ2+RIREanA4y7flLCnPN7of4/84PGyANC9bhMvb7sczBE7AAC9qg1GoYddJTLllimz59v+u9xKfksA3s/IlC23u4JCgrDmzLsA5Mqsz/nyfNtqlVuGt8814DvnW4bs96nWrCqM8pVd3ls8rlCl+t8lw4u9vu0i95MKrxV6vn2Jckvf75AsN3Bdl9Bpb93ncc7zMutyvtTatiHPNaDr+Zahw7lyh1VAhbQZdfZFa+zyJSIiUgGfQyUiIs0of/3IrsMfsEIlIiLNKDDBKvkclqLX82tuYpcvERGRCthCJSIizSji1o/sOvwBK1QiItKMVYUuX9nlvYUVKhERacZIFSrvoRIREamALVQiItKMIkxQhOQoX8nlvYUVKhERaYZdvkREROQWffJQJbMDvb9t/fNQpbMiJcttlIxMn8mA9fK1wjxUwDiZv3/x0juArTDDKtl2s6q0L1pjHioRkQF1NPfGJmW1Zuu35aFm/3gfQiXzUK9eVtCh8QnmoRIRERmBx12+Sr7n+X9GzYr0fo6r3LaLlts4ObD6Z8ACxs2BlaFn5q+/fba9yUiDkjwf5etv2Z5qbVuvzEQ9M2RhoBxYH8iABfTOBjVQLqhK59ufP9taswozrELyHqqf3Jhkly8REZEK+BwqERFpRoEJimTbTYF/NFFZoRIRkWZ4D5WIiEgF6txD9Y8WKu+hEhERqYAtVCIi0syte6iSL8dnly8RERmdosKrB/1lUBK7fImIiFTAFioREWnGSIOSWKESEZFmFJj5HGqp/C6KzPNt+0TEkw6Rd0XLbZzYOv0j6wADx9bJ0DGi0N8+24Dvv7LQHzG+jYjIgLwV3/bB940Rck+A1LquXbZiQLMffT6+jV2+RESkGXUCxv2j3edxhZoS9pTHG9UzrshfI55kyZab8W3epWd8m8y5liH7veAZ/a9xWb4e32YkHleofhtX5K8RT7Iky834Nu/S81qROtcy9DjWPnCNy/L1e6GKMEORHOWr+MmdSXb5EhGRZtjlS0REpAIFgFXIvnrQP/BNSURERCpgC5WIiDSjzosd/KPtxwqViIg0o86rB/2jQvWPvSQiIvJxbKESEZFmmIdKRESkAnb5EhERkVvYQiUiIs2o82IH/2j7sUIlIiLNKMIERfbFDpLLe4vHFao/5v95vm0fyEyUJVlu5qF6l555qFLnWobk94Jn9L/GZTEP1XcwD5WIyICUvHowRx7SbP22PNTXv22L4DC5ztCCKzfxUov/Yx4qEREZlzppM3f5PVSZPFRZ3s8e1D8jU4/MQ7UyMv0pK9IXckEBXuPe4gvXuPczf73LChOskvdOZJf3Fn3yUGV5O3vQBzIy9b7fYcSsSN1yQQFe4zowTOYvaYZdvkREpBl2+RIREanACvkuW6s6u6I5/6j2iYiIfBxbqEREpBl2+RIREamAL8cnIiLyY3PnzkWtWrUQHByMli1bYufOnXecf+bMmXjggQdQrlw5REdHY+TIkSgoKHBrm6xQiYhIM+KvPFSZH+HmoKZVq1YhMzMTWVlZ2L17N5o2bYrk5GScOXPG6fwrVqzASy+9hKysLPz888949913sWrVKrz88stubZcVKhERacbW5Sv7444ZM2ZgyJAhSE9Px4MPPoj58+cjJCQEixcvdjr/9u3b0bp1a/Tr1w+1atVCp06d0Ldv31JbtbdjhUpERH7h0qVLDj+FhcVfimGxWLBr1y4kJSXZp5nNZiQlJSE3N9fpelu1aoVdu3bZK9CjR49i/fr1ePTRR93aPw5KIiIizagZ3xYdHe0wPSsrCxMmTHCYdu7cOVitVkRERDhMj4iIwIEDB5yuv1+/fjh37hwefvhhCCFw8+ZNPPvss253+eoT3ybL61FJ+kd66XG8VYv08qNoK5+IMQN4jXuJL1zj3o8o/IuXXlmoZsD4yZMnHdJmgoLUuWa2bduGyZMnY968eWjZsiUOHz6MF154AZMmTcK4ceNcXg/j24iIDKijuTc2Kas1W78tvm34V90RFFZWal2FV27g7Yf/61J8m8ViQUhICNasWYMePXrYpw8cOBAXLlzAf//732LLJCYm4qGHHsK0adPs05YtW4ahQ4fiypUrMJtd+4OA91CJiOiuERgYiLi4OGRnZ9unKYqC7OxsJCQkOF3m2rVrxSrNgIAAAIA7bU5d4ttk44Zk4o482zajrWTK7f3z5fm21SqzLG+fb1+IrdMnhuzvz7ZRIgq9TYEZimTbzd3lMzMzMXDgQDRv3hzx8fGYOXMmrl69ivT0dABAWloaoqKiMGXKFABASkoKZsyYgWbNmtm7fMeNG4eUlBR7xeoKfeLbJPvuvb5tRltBptx+e63odK4Bfc+3brF1ehzrIp9tI0YUeoNVmGCVHJTk7vKpqak4e/Ysxo8fj7y8PMTGxmLDhg32gUonTpxwaJGOHTsWJpMJY8eOxenTp1G1alWkpKTgtddec2u7HOVLRER3nYyMDGRkZDj93bZt2xz+X6ZMGWRlZSErK0tqm6xQiYhIM2o+NuPrWKESEZFmhAppM4IvxyciIjIOtlCJiEgzVphglXxTiOzy3sIKlYiINKMI+Xugip+8fohdvkRERCpgC5WIiDSjqDAoSXZ5b2GFSkREmrGFhMuuwx+wQiUiIs3o8aYkvegT3yYZTeX9bTPaSqbc/nStqFVmWd4+3z4RW6dLzN/f2zRKRKGNr7+y0B8xvo2IyICUvHowRx7SbP22+LYns59CYFig1LosVyxY2WGZS/FtemKXLxERaUaBCq8evNvvoeoZ3ybDk6gjRlvpF98my/3jpn9UH+C/8W38bLtPz2NG6vLL+DYZsvcNjBptpVt8myx399kHovoAP45v42fbfToeM28QKozyFXd7C5WIiKg0Rkqb8Y+nZYmIiHwcW6hERKQZvimJiIhIBezyJSIiIrewhUpERJrhu3yJiIhUYKQuX1aoRESkGSNVqLyHSkREpAK2UImISDNGaqGyQiUiIs2wQnWBnhmXMjzZb2ZF/vVvPfJQZbl93PTPvgX8OA+Vn2336XXM7vJ3COuBeahERAbU0dwbm5TVmq3floeatP7/oUyo3B8rN68WYvOj/2YeKhERGRe7fF3APFT3sdzuk80F1SsP1b8yYIGi5TZWLqg65fa3zzZpg3mobmJWpAckyi2dT6lTHqpfZcACDuU2VC6oWuX248+21thCJSIiUoGRKlS+2IGIiEgFbKESEZFmjNRCZYVKRESaEcIEIVkhyi7vLaxQiYhIM0aKb+M9VCIiIhWwhUpERJrhPVQiIiIVGOkeKrt8iYiIVMAWKhERaYZdvi5gfJsHWG63SceY6RTf5l+RdUDRchsrxkylcvvZZxvw3isLjdTly/g2IiIDUvLqwRx5SLP12+Lb4taOVCW+bdcTbzG+jYiIjEuo0OXrLy1UjytUJd/zyCDvR3LJbdsXYsxk+Wu59Yxv0/Ncy5Sb8W3uMGq5vUcAkO0H9ZduVM9bqP4UyaXito0a8WTE+DY9z7XX4xEZ32ascpMm2OVLRESaUWCCySCvHmSFSkREmjHSKF9WqEREpBlFmGAyyHOofFMSERGRCthCJSIizQihwihfPxnmywqViIg0Y6R7qOzyJSIiUgFbqEREpBkjtVBZoRIRkWY4ypeIiIjcwhYqERFphqN8XeFXGZdy2/aFXFBZ/lpuPfNQ9TzX3s8bZh6qscoNr71D+FaFKnsPVaWd0RjzUImIDKijuTc2Kas1W78tD7XespcQEBIstS7rtQIceup15qESEZFxcZSvC1LCnlJzP9zi/fxAo2Ym/l1uJb8lAM+6iLyf7en5ttXKgJXFa9xb1LnGZchm/vo6Afk8U3/pRvW4QtUlN9DG2/mBRs1MdPij8LrH++D1bE+Vtq3buQZ4jXuLSte4DF2/S73ASC1UPjZDRESkAt5DJSIi7Rioz5ctVCIi0s5fXb4yP/Cgy3fu3LmoVasWgoOD0bJlS+zcufOO81+4cAHPP/88qlevjqCgINSvXx/r1693a5tsoRIR0V1l1apVyMzMxPz589GyZUvMnDkTycnJOHjwIKpVq1ZsfovFgo4dO6JatWpYs2YNoqKicPz4cYSHh7u1XVaoRESkGT3elDRjxgwMGTIE6enpAID58+dj3bp1WLx4MV566aVi8y9evBh//PEHtm/fjrJlywIAatWq5fZ+ssuXiIg0I9vdW3SU8KVLlxx+CguLj5C2WCzYtWsXkpKS7NPMZjOSkpKQm5vrdB8/+eQTJCQk4Pnnn0dERAQaNWqEyZMnw2q1ulVWVqhEROQXoqOjUaFCBfvPlClTis1z7tw5WK1WREREOEyPiIhAXl6e0/UePXoUa9asgdVqxfr16zFu3Di8+eab+J//+R+39o9dvkREpB0PBxUVWweAkydPOrx6MChInfcvK4qCatWqYcGCBQgICEBcXBxOnz6NadOmISsry+X1sEIlIiLNqHkPtXz58qW+y7dKlSoICAhAfn6+w/T8/HxERkY6XaZ69eooW7YsAgIC7NMaNGiAvLw8WCwWBAYGurSf7PIlIiLtCJV+XBQYGIi4uDhkZ2fbpymKguzsbCQkJDhdpnXr1jh8+DAURbFP++WXX1C9enWXK1NAooUqHaslw+txR0aNeCrn+G8Pe228H0Xm+bZVi6yTxWvcS9S5xmXo9V16N7/yMDMzEwMHDkTz5s0RHx+PmTNn4urVq/ZRv2lpaYiKirLfg/3nP/+JOXPm4IUXXsCwYcNw6NAhTJ48GcOHD3drux5XqJ9eWebpon7N9uJ0o7G9QNwTn15RcUe8uG2jnmujllvmGpeh1+dDyavnle3o8S7f1NRUnD17FuPHj0deXh5iY2OxYcMG+0ClEydOwGz+u4M2OjoaGzduxMiRI9GkSRNERUXhhRdewOjRo93aLvNQiYgMSMmrB3PkIc3Wb8tDvW/BeJjLyeWhKtcLcGLoxLs3D1XJ1y9yyNtxR0UjvfSKMZPlWTQW49uMGN/Ga9x9/ldu0oLno3z1iFr6i759/zrFmMnyZJ8Z32bI+DZe4+7zu3J7kZHi2/jYDBERaYdpM0REROQOtlCJiEhDJsg/j8QuXyIiMjp2+RIREZE72EIlIiLtGKiFygqViIi0o2LajK9jhUpERJpRM23G1/EeKhERkQrYQiUiIu3wHqoLdIlausXbcUeOcVY6xZjJ8uh8Mb5NNzrGt/Ead5//lRvee2Whge6hMm2GiMiAvJU2c+/bE1VJmzk1fPzdmzZDRERUGpO49SO7Dn/A+DYXqBXppWfMkifHzIjlZlSfcc41YOTz7UW8h+oCg8a3SUV6+fExM2a5DRTnVeQWlTHPNWCo802aYJcvERFpx0CDklihEhGRdgzU5csXOxAREamALVQiItKOgVqorFCJiEg7rFCJiIhUYKBBSbyHSkREpAK2UImISDN8UxIREZEaDHQPlV2+REREKmCFSkREpALmobpAtYxMPztmRiw3s2+Nc64BI59veO0dwCaocA9VlT3RHvNQiYgMyFt5qDWn/g/MwZJ5qAUFOD56LPNQiYjIwAz0HKoueaiy2YEy+X965oLK0CfzUP+MTO9fK3+X2Vj5mPqXW/YaZx6qjzLQKF9d8lClswN13LZUVqQMPTIPfSAj0+vXisMfwgbKx/SFckte48xDJb2xy5eIiLTDFioREZE8vimJiIhIDQZqofLFDkRERCpgC5WIiLRjoBYqK1QiItKMke6hssuXiIhIBWyhEhGRdvimJCIiIhUY6B4qu3yJiIhUoEt8m3TUkZe3rVqMmQxdYrH0j/Ty/rVSzvHfhonz8oFyS17jjG9zk7fi2ww0KInxbUREBuSt+LY64yerEt92dOLLPh/fxi5fIiIiFegS3ybL21FJxo14MnZ8m15RfYAekVxGPNeAL5xvn49fk6VCl6+/DErSJb5Nlq5RSUaKeDJ4fJtuUX2A9z9fRjzXgG+c77s9fs1Ao3z52AwREWnHQBUq76ESERGpgC1UIiLSjJEem2ELlYiISAWsUImIiFTALl8iItKOgQYlsUIlIiLN8B4qERERuYUtVCIi0paftDBlsUIlIiLtGOgeKrt8iYiIVKBLHqosb2cPGjcz0dh5qLpl3wI6fL6MeK4Bnzjfen2XMg9VdcxDJSIyIG/lodb712QEBMnloVoLC3Bomu/nofIeKhERacZILVSPK9SUsKc83qhs/p9MbqJsZqJMHqoMPbJUi+bAGicrUv9cUFmeXCu+kPmrx/eCL5Rb1l2fp+pHPK5QpXIDJfvuvb5th3umnuehytA3A9ZAWZE+kAsqS/486ZT5q+f3AgBDZR17k06jfOfOnYtp06YhLy8PTZs2xezZsxEfH1/qcitXrkTfvn3RvXt3fPzxx25tk6N8iYhIO0KlHzesWrUKmZmZyMrKwu7du9G0aVMkJyfjzJkzd1zu2LFjGDVqFBITE93b4F9YoRIR0V1lxowZGDJkCNLT0/Hggw9i/vz5CAkJweLFi0tcxmq1on///nj11VdRp04dj7bLCpWIiDRjG5Qk+wPcGjlc9KewsHhXu8Viwa5du5CUlGSfZjabkZSUhNzc3BL3c+LEiahWrRoGDx7scVlZoRIRkXZU7PKNjo5GhQoV7D9Tpkwptrlz587BarUiIiLCYXpERATy8vKc7uJXX32Fd999FwsXLpQqKh+bISIiv3Dy5EmH51CDguRfxnH58mUMGDAACxcuRJUqVaTWxQqViIi0o+Io3/Lly5f6YocqVaogICAA+fn5DtPz8/MRGRlZbP4jR47g2LFjSElJsU9TFAUAUKZMGRw8eBB169Z1aTfZ5UtERJpR8x6qKwIDAxEXF4fs7Gz7NEVRkJ2djYSEhGLzx8TE4Mcff8SePXvsP926dUP79u2xZ88eREdHu7xttlCJiOiukpmZiYEDB6J58+aIj4/HzJkzcfXqVaSnpwMA0tLSEBUVhSlTpiA4OBiNGjVyWD48PBwAik0vDStUIiLSjg4vdkhNTcXZs2cxfvx45OXlITY2Fhs2bLAPVDpx4gTMZvU7aFmhEhGRZvR6l29GRgYyMjKc/m7btm13XHbp0qXubxASFapU1JJkXJH3t13O8d8exrfJ0CP6rWiclXGirfSPMZPlybXiExGFOnwv+ES5ZXl63Lz1ykIDBYwzvo2IyIC8Fd/W4Hl14tt+nsv4NiIiMjIDtVA9rlCVfM/jivwv6kid+DZGW3mf+8eN8W3GOdeAL3y2Zfl6fJsJ8nfJdLjL5hHPW6gS/e9+F3WkUnwbo6104O4+M74NhjnXgG98tmX5enybgbDLl4iItMMuXyIiInl6PTajB756kIiISAVsoRIRkXbY5UtERKQSP6kQZbHLl4iISAVsoRIRkWaMNCiJFSoREWmH91CJiIjkGamFynuoREREKvC8hSoRteR/UUfqxLcx2koHbh83xrcZ51wDPvHZlsX4Np/B+DYiIgPyVnxbk6cnIyBQMr7NUoAfFvt+fBu7fImIDCi5Rqzeu3DX8cv4Npm4ItloK5lILxn6RDQZL8qMMWZy5ZahxzFT67PtT9e41xmoy9c/49t03LZUpJcMPSKaDB9lZqA4L5VizGToesxg1GvcCwxUobLLl4iISAV8DpWIiDRjpOdQWaESEZF22OVLRERE7mALlYiINGMSAibJ1x3ILu8trFCJiEg7BuryZYVKRESaMdKgJN5DJSIiUgFbqEREpB12+RIREcljly8RERG5xT/zUL287aJZkVIZmTJ0ydc0XjYoc0H/+reH5ZahxzFT7bPtR9e4jdfeAWygLl/moRIRGZC38lDjUl9TJQ9116pXmIdKRERkBH6ZhypDz6xImXLL5i3qmQPr7fxauW3rf65l+es1Lku23MbKOvYiA3X5+mceqgwdsyKlyi2Zt6hrDqw/Zef6wrmW5a/XuCzJchsq69jL/GWUrix2+RIREamAz6ESEZF2hLj1I7sOP8AKlYiINGOkFzuwQiUiIu0YaFAS76ESERGpgC1UIiLSjEm59SO7Dn/ACpWIiLTDLl8iIiJyB1uoRESkGY7yJSIiUgOfQ3WBnvFtMnSMtpIqt2Q8lK6xdX4V9ecD51qWv17jsiTLbaxoRhjilYfexvg2IiID8lZ8W8uUSShTVi6+7eaNAuz4dJzPx7exy5eIiLRjoFG+fhnf5u04sKIxZoxvc5+e8VTultsXzrUsPWPM/CuqDzBubB1pwT/j23SNjmN8m9t0vFcjV27GmHll23/R5XvBqLF1XsRRvkRERGrgKF8iIiJ5Rmqh8k1JREREKmALlYiItMNRvkRERPLY5UtERERuYQuViIi0o4hbP7Lr8AOsUImISDsGuofKLl8iIiIVsIVKRESaMUGFQUmq7In2WKESEZF2+KYkF/hVxqXcth1zEpmH6ja98h7hfrl94lzL0jMX1O++F4yaAwuffwewP2IeKhGRAXU098YmZbVm67floT78yASUKSOZh3qzAF9tmeBWHurcuXMxbdo05OXloWnTppg9ezbi4+Odzrtw4UK8//772LdvHwAgLi4OkydPLnH+knBQEhERaUeo9OOGVatWITMzE1lZWdi9ezeaNm2K5ORknDlzxun827ZtQ9++fbF161bk5uYiOjoanTp1wunTp93arsct1JSwpzxZDIB/5WMCzMgE5Motw9vHzFfOtZ6ZvzJ5qDL0+V4w3jVuU3Ct0Cst1MR2Waq0UHO2vepyC7Vly5Zo0aIF5syZAwBQFAXR0dEYNmwYXnrppVKXt1qtqFixIubMmYO0tDSX99Pje6h6ZnvK8Ns8VFk6ZkXK0PWY6XmudcwVlcpDlaHH94Lhr3H/cunSJYf/BwUFISjI8f61xWLBrl27MGbMGPs0s9mMpKQk5ObmurSda9eu4caNG6hUqZJb+8cuXyIi0o6i0g+A6OhoVKhQwf4zZcqUYps7d+4crFYrIiIiHKZHREQgLy/PpV0ePXo0atSogaSkJLeKysdmiIhIMyYhYJIc+2pb/uTJkw5dvre3TtXw+uuvY+XKldi2bRuCg93rqmaFSkREfqF8+fKl3kOtUqUKAgICkJ+f7zA9Pz8fkZGRd1x2+vTpeP3117F582Y0aeL+vW12+RIRkXa8PMo3MDAQcXFxyM7Otk9TFAXZ2dlISEgocbk33ngDkyZNwoYNG9C8eXM3Cvg3tlCJiEg7OrwpKTMzEwMHDkTz5s0RHx+PmTNn4urVq0hPTwcApKWlISoqyn4PdurUqRg/fjxWrFiBWrVq2e+1hoWFISwszOXtskIlIqK7SmpqKs6ePYvx48cjLy8PsbGx2LBhg32g0okTJ2A2/91B+84778BisaBXr14O68nKysKECRNc3i4rVCIi0oxJqPByfA+Wz8jIQEZGhtPfbdu2zeH/x44dc38DTrBCJSIi7Rjo5fgclERERKQCtlCJiEgzJuXWj+w6/IHHFaqeUWQy/Da+TZaO0VYyvH3MfOZc6xhRKBXfJkOX7wXjXeM2XnvloYG6fBnfRkRkQEpePZgjD2m2ftvL8du1eEWVl+Nv+/Y1t+Lb9MB7qERERCrwuMtXJr5Nlvdjnv6OeJKJtjJqbJ0Mf41vk6VnuXmNe2fbatHzmLtCzXf5+jp94ttkeTtiqch9FaloK4PG1snw1/g2WXqWm9e4HtuWoOMxd4mB7qGyy5eIiEgFfGyGiIi0I2DPM5Vahx9ghUpERJox0j1UdvkSERGpgC1UIiLSjoAKg5JU2RPNsUIlIiLtcJQvERERuYMtVCIi0o4C+Xck3+0vxyciIiqNkUb5skIlIiLtGOgeqj7xbbK8HvP09/akoq0MGlsnw1/j22TpWW5e497Ztmo8Pea+/spCP8T4NiIiA/JWfFuHB0ehTIDcHxw3rYXI3j/d5+Pb2OVLRETaYZdv6WTi2xjx5H2eHfO/Y+uMU251ovpk6RlRaJxzDRi33KQFfeLbGPHkfZ7ss8P9JIOUW62oPlk6RhQa5lwDxi23N/GxGSIiInlGemyGb0oiIiJSAVuoRESkHQ5KIiIiUoEiAJNkhaj4R4XKLl8iIiIVsIVKRETaYZcvERGRGlSoUP0kYZwVKhERacdALVTeQyUiIlIBW6hERKQdRUC6y9ZPRvmyQiUiIu0I5daP7Dr8gD55qMxM9D6Pjnk5x38botwqZd/K0jHz1zjnGjBuueH77wD2Q8xDJSIyIG/loSZF/xNlzJJ5qEohNp98h3moRERkYLyHWjqZPFRZ/poVKUOPvMWiObB6ZYP667n2v3xM/cste65ls46Nc42TVvTJQ5Xlp1mRMnQ93tAxG9RPz7XffT58odyS51r2mBvmGvc2Az2Hyi5fIiLSjoAKFaoqe6I5vtiBiIhIBWyhEhGRdtjlS0REpAJFASD5YgblLn+xAxERUakM1ELlPVQiIiIVsIVKRETaMVALlRUqERFpx0BvSmKXLxERkQrYQiUiIs0IoUBIxq/JLu8t+sS3yfLTaCsZehzvovFlukWZ+em59r/Phw+UW/Jcy0YzGuca/4u3XnkohHyXrZ/cQ2V8GxGRAXkrvq1DeBrKmAKl1nVTWJB94X2fj2/jPVQiIgNKrhHrnQ3ZRvnK/vgBxre5hPFtRon0UqvMsrx9vtWKMZM5X7Jllo2t06vcsvSMCnSJogAmyXugd/s9VL+Lp5LB+DYYM9JLn3MN6Hu+pWLMJI6XdJklY+v0Krcs/b8byIajfImISDtChedQ7/YuXyIiotIIRYGQ7PK96x+bISIiKpWBWqgc5UtERKQCtlCJiEg7igBMxmihskIlIiLtCAHpgHE/qVDZ5UtERKQCtlCJiEgzQhEQkl2+/vKGXLZQiYhIO0JR58dNc+fORa1atRAcHIyWLVti586dd5x/9erViImJQXBwMBo3boz169e7vU1WqEREdFdZtWoVMjMzkZWVhd27d6Np06ZITk7GmTNnnM6/fft29O3bF4MHD8b333+PHj16oEePHti3b59b22WFSkREmhGKUOXHHTNmzMCQIUOQnp6OBx98EPPnz0dISAgWL17sdP5Zs2ahc+fO+Ne//oUGDRpg0qRJ+Mc//oE5c+a4tV1WqEREpB0vd/laLBbs2rULSUlJ9mlmsxlJSUnIzc11ukxubq7D/ACQnJxc4vwl8XhQ0iZltaeL+jVzhD6pEpt0fvOWTLn13HeZbet1rgF9j9mnV5bpsl29r3G9yi1L7+NWmpu4If2ipJu4AeBWxmpRQUFBCApyDIY/d+4crFYrIiIiHKZHRETgwIEDTtefl5fndP68vDy39pOjfImISHWBgYGIjIzEV3nuD+5xJiwsDNHR0Q7TsrKyMGHCBFXWrwZWqEREpLrg4GD8+uuvsFgsqqxPCAGTySFLs1jrFACqVKmCgIAA5OfnO0zPz89HZGSk03VHRka6NX9JWKESEZEmgoODERwc7NVtBgYGIi4uDtnZ2ejRowcAQFEUZGdnIyMjw+kyCQkJyM7OxogRI+zTNm3ahISEBLe2zQqViIjuKpmZmRg4cCCaN2+O+Ph4zJw5E1evXkV6ejoAIC0tDVFRUZgyZQoA4IUXXkDbtm3x5ptvomvXrli5ciW+++47LFiwwK3tskIlIqK7SmpqKs6ePYvx48cjLy8PsbGx2LBhg33g0YkTJ2A2//2QS6tWrbBixQqMHTsWL7/8MurVq4ePP/4YjRo1cmu7JuEv73QiIiLyYXwOlYiISAWsUImIiFTACpWIiEgFrFCJiIhUwAqViIhIBaxQiYiIVMAKlYiISAWsUImIiFTACpWIiEgFrFCJiIhUwAqViIhIBaxQiYiIVPD/AasDlMxxETaAAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "weights_to_display = tf.reshape(tensor_data, [tf.reduce_prod(tensor_data.shape[:-1]), -1])\n", "weights_to_display = weights_to_display[0:width, 0:height]\n", "\n", "val_ones = np.ones([height, width])\n", "val_zeros = np.zeros([height, width])\n", "subset_values_to_display = np.where(abs(weights_to_display) > 1e-9, val_ones, val_zeros)\n", "\n", "plot_separation_lines(height, width)\n", "\n", "plt.axis('off')\n", "plt.imshow(subset_values_to_display)\n", "plt.colorbar()\n", "plt.title(\"Structurally pruned weights for Conv2D layer\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "1aX2O8w0Wr_M" }, "source": [ "Let's see how those randomly pruned weights look. We extract them and display a subset of the weight tensor." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:03.892086Z", "iopub.status.busy": "2024-03-09T12:20:03.891524Z", "iopub.status.idle": "2024-03-09T12:20:03.896452Z", "shell.execute_reply": "2024-03-09T12:20:03.895711Z" }, "id": "eEHu5nizWr_M" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Shape of the weight tensor is (32, 5, 5, 1)\n" ] } ], "source": [ "# Get weights of the convolutional layer that has been pruned with random pruning.\n", "tensor_name = 'pruning_sparsity_0_5/Conv2D'\n", "detail = [x for x in details if tensor_name in x[\"name\"]]\n", "tensor_data = interpreter.tensor(detail[0][\"index\"])()\n", "print(f\"Shape of the weight tensor is {tensor_data.shape}\")" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:03.899865Z", "iopub.status.busy": "2024-03-09T12:20:03.899283Z", "iopub.status.idle": "2024-03-09T12:20:04.061823Z", "shell.execute_reply": "2024-03-09T12:20:04.061038Z" }, "id": "Cimzp3kVWr_M" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdEAAAGkCAYAAABuLB/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/kElEQVR4nO3deXgUReI+8HcSchBCYpQcHFlAxAUkHBvkUMKxRCJoWBQwoBgIiCgEkCyePyQgCgrCxgUUUIGVQwL49VoRhCCKgKuGQw5xEZDThCAid0Jm6vcHO2OGTJKZrmIqnXk/zzPPA53u6a7umqmp6uO1CCEEiIiIyGN+ujeAiIjIrNiIEhERGcRGlIiIyCA2okRERAaxESUiIjKIjSgREZFBbESJiIgMYiNKRERkEBtRIiIig9iIVhIbN26ExWLBxo0bdW+KNg0aNMDgwYO1bsPPP/8Mi8WCRYsWGV721VdfVb9hbjh//jweeeQRxMTEwGKx4IknntCyHb5Opg6R+XjUiE6cOBEWiwWnTp1y+ffmzZujS5cuKrbLpSlTpuCDDz64bu9vlm0g81u9ejUmTpyo9D2nTJmCRYsW4fHHH8fixYvx8MMPK31/V6xWKxYuXIguXbrgxhtvRFBQEBo0aIC0tDR899131339Fbl48SLmzJmD7t27o3bt2qhZsyZat26NN954A1ar1Wle+w9Z+ysoKAjR0dHo0qULpkyZgoKCAk2loMrMVD3RytCAVYZtoOunfv36uHTp0nVvgFavXo1JkyYpfc8NGzagffv2yMzMxMCBAxEfH6/0/a916dIl3HvvvRgyZAiEEHjuuefwxhtvIDU1FVu3bkXbtm1x7Nix67oNFTl48CBGjRoFIQQyMjLw6quvomHDhhgxYgSGDBnicpnRo0dj8eLFmD9/Pp588knceOONyMzMRNOmTbFhwwYvl4Aqu2q6N+B6uXDhAmrUqKF7MyoVm82GoqIiBAcH696USstisZh2/5w8eRLNmjVT9n7FxcWw2WwIDAx0+fcnn3wSa9aswT/+8Y9SQ8eZmZn4xz/+oWxbjIqJicGuXbtw2223OaYNHz4cQ4YMwcKFC/H888/jlltucVomISEBffv2dZq2c+dOdO/eHX369MHevXtRu3Ztr2z/9cDvRrWua0/UPjyyYsUKvPTSS6hXrx6Cg4PRrVs3/PTTT07z7t+/H3369EFMTAyCg4NRr1499O/fH7///juAq19uFy5cwL/+9S/HcIv9/Jl9mHnv3r148MEHERERgY4dOwIAunTp4nKIefDgwWjQoIHTNJvNhtdeew1xcXEIDg5GZGQk7r77bsewVHnbAADHjx/HkCFDEB0djaCgINx2221YsGBBqXUfO3YMvXv3Ro0aNRAVFYWxY8eisLDQrX1qL+u+ffvwwAMPICwsDDfddBPGjBmDy5cvO81rsViQnp6OpUuX4rbbbkNQUBDWrFlT5vlXV+dyBg8ejNDQUBw/fhy9e/dGaGgoIiMjMW7cuFLDYTabDVlZWbjtttsQHByM6OhoDB8+HL/99pvTfEIIvPjii6hXrx5CQkLQtWtX7Nmzx63y/+Uvf8H999/vNC0uLg4WiwXff/+9Y1p2djYsFgt++OEHxzR3jk9Z57NWrlyJZs2aITg4GM2bN8f777/vsg7ZzZ8/H40aNUJQUBBuv/12fPvtt46/DR48GHPmzAEAp+FDu+XLlyM+Ph41a9ZEWFgY4uLi8Nprr5W5T+zH89ChQ/jkk08c7/fzzz8DuNq4Dh06FNHR0QgODkbLli3xr3/9y2W5X331VWRlZTm2fe/evS7XeezYMcybNw933XWXy3Ov/v7+GDduHOrVq+eYtn37dvTo0QNhYWEIDQ1Ft27d8PXXXzstt2jRIlgsFmzevBkZGRmIjIxEjRo1cN999zkNp9577724+eabXW5bhw4d0KZNGwBArVq1nBpQu/vuuw8AnOpHeVq2bImsrCycOXMGs2fPdmuZkr7//nsMHjwYN998M4KDgxETE4MhQ4bg119/dczz+eefw2Kx4P333y+1/LJly2CxWLB161bHtH379qFv37648cYbERwcjDZt2uCjjz5yWs6+P7/44guMGDECUVFRTseE5HmlJ/ryyy/Dz88P48aNw++//45p06bhoYcewn/+8x8AQFFREZKSklBYWIhRo0YhJiYGx48fx7///W+cOXMG4eHhWLx4MR555BG0bdsWjz76KACgUaNGTuvp168fGjdujClTpsBIwtvQoUOxaNEi9OjRA4888giKi4uxadMmfP3112jTpk2525Cfn4/27ds7Gq7IyEh8+umnGDp0KM6ePev4orl06RK6deuGI0eOYPTo0ahTpw4WL17s8TDRAw88gAYNGmDq1Kn4+uuv8c9//hO//fYb3nnnHaf5NmzYgBUrViA9PR21atVCgwYNcObMGY/WZbVakZSUhHbt2uHVV1/F+vXrMWPGDDRq1AiPP/64Y77hw4dj0aJFSEtLw+jRo3Ho0CHMnj0b27dvx+bNmxEQEAAAmDBhAl588UX07NkTPXv2xLZt29C9e3cUFRVVuC0JCQl49913Hf8/ffo09uzZAz8/P2zatAktWrQAAGzatAmRkZFo2rQpAPePjyuffPIJUlJSEBcXh6lTp+K3337D0KFDUbduXZfzL1u2DOfOncPw4cNhsVgwbdo03H///Th48CACAgIwfPhwnDhxAuvWrcPixYudll23bh0GDBiAbt264ZVXXgFw9Yt+8+bNGDNmjMv1NW3aFIsXL8bYsWNRr149/P3vfwcAREZG4tKlS+jSpQt++uknpKeno2HDhli5ciUGDx6MM2fOlHrPhQsX4vLly3j00UcRFBSEG2+80eU6P/30UxQXF7s97L1nzx4kJCQgLCwMTz31FAICAjBv3jx06dIFX3zxBdq1a+c0/6hRoxAREYHMzEz8/PPPyMrKQnp6OrKzswEAKSkpSE1Nxbfffovbb7/dsdzhw4fx9ddfY/r06eVuT15eHoCrjay7+vbti6FDh+Kzzz7DSy+95PZywNXjevDgQaSlpSEmJgZ79uzB/PnzsWfPHnz99dewWCzo0qULYmNjsXTpUkcjb7d06VI0atQIHTp0AHB1f955552oW7cunnnmGdSoUQMrVqxA79698d5775VafsSIEYiMjMSECRNw4cIFj7adKiA8kJmZKQCIgoICl3+/7bbbROfOnR3///zzzwUA0bRpU1FYWOiY/tprrwkAYteuXUIIIbZv3y4AiJUrV5a7/ho1aohBgwaVuV0DBgwo9bfOnTs7bZPdoEGDRP369R3/37BhgwAgRo8eXWpem81W4TYMHTpU1K5dW5w6dcppev/+/UV4eLi4ePGiEEKIrKwsAUCsWLHCMc+FCxfELbfcIgCIzz//vNR7uyprr169nKaPGDFCABA7d+50TAMg/Pz8xJ49e5zmtR+Xa9d16NAhAUAsXLjQMW3QoEECgHjhhRec5m3durWIj493/H/Tpk0CgFi6dKnTfGvWrHGafvLkSREYGCjuuecep/363HPPCQAu921JK1euFADE3r17hRBCfPTRRyIoKEj06tVLpKSkOOZr0aKFuO+++xz/d/f4uNoHcXFxol69euLcuXOOaRs3bhQAnOqQfdmbbrpJnD592jH9ww8/FADExx9/7Jg2cuRI4erjN2bMGBEWFiaKi4vL3Q+u1K9fX9xzzz1O0+z1bcmSJY5pRUVFokOHDiI0NFScPXvWadvDwsLEyZMnK1zX2LFjBQCxfft2t7atd+/eIjAwUBw4cMAx7cSJE6JmzZqiU6dOjmkLFy4UAERiYqJT/Rg7dqzw9/cXZ86cEUII8fvvv4ugoCDx97//3Wk906ZNExaLRRw+fLjMbSksLBTNmjUTDRs2FFeuXHFMt38uyvseatmypYiIiCi3rK7qkL1+lfTuu+8KAOLLL790THv22WdFUFCQo5xCXP3MVKtWTWRmZjqmdevWTcTFxYnLly87ptlsNnHHHXeIxo0bO6bZ92fHjh0N1SmqmFcuLEpLS3M6r5KQkADg6kl/AAgPDwcArF27FhcvXjS8nscee8zwsu+99x4sFgsyMzNL/a3kUJsrQgi89957SE5OhhACp06dcrySkpLw+++/Y9u2bQCuXlBSu3Ztp3MuISEhjp6tu0aOHOn0/1GjRjnev6TOnTsrOU927b5NSEhwHD/g6nBneHg47rrrLqfyx8fHIzQ0FJ9//jkAYP369SgqKsKoUaOc9qu7t2PY686XX34J4GqP8/bbb8ddd92FTZs2AQDOnDmD3bt3O+b15Phc68SJE9i1axdSU1MRGhrqmN65c2fExcW5XCYlJQURERGltrnk/irLDTfcgAsXLmDdunUVzuuO1atXIyYmBgMGDHBMCwgIwOjRo3H+/Hl88cUXTvP36dMHkZGRFb7v2bNnAQA1a9ascF6r1YrPPvsMvXv3dhqCrV27Nh588EF89dVXjveze/TRR53qR0JCAqxWKw4fPgwACAsLQ48ePbBixQqnUafs7Gy0b98ef/rTn8rcnvT0dOzduxezZ89GtWqeDcaFhobi3LlzHi0DANWrV3f8+/Llyzh16hTat28PAE51LzU1FYWFhVi1apVjWnZ2NoqLizFw4EAAV0dfNmzYgAceeADnzp1z1OVff/0VSUlJ2L9/P44fP+60/mHDhsHf39/j7aaKKW9EXTU411Zo+xeM/VxZw4YNkZGRgbfeegu1atVCUlIS5syZ4zgf6q6GDRsa3GrgwIEDqFOnTpnDV+UpKCjAmTNnMH/+fERGRjq90tLSAFw9LwVcHW665ZZbSu2nP//5zx6ts3Hjxk7/b9SoEfz8/Bznwexk9omd/fxwSREREU7nOvfv34/ff/8dUVFRpfbB+fPnncrvavsjIyOdGp6yREdHo3Hjxo4Gc9OmTUhISECnTp1w4sQJHDx4EJs3b4bNZnM0Xp4cn2vZt/fai0/KmgZUXN/LM2LECNx6663o0aMH6tWrhyFDhmDNmjUVLleWw4cPo3HjxvDzc/6o24e57eWzc7e+hIWFAYBbDUpBQQEuXrzoso43bdoUNpsNR48edZruzj5MSUnB0aNHHecJDxw4gNzcXKSkpJS5LdOnT8ebb76JyZMno2fPnhVu+7XOnz/v1g+Ha50+fRpjxoxBdHQ0qlevjsjISMe+Lvk916RJE9x+++1YunSpY9rSpUvRvn17R3376aefIITA888/X6o+2zsB19ZnFd8D5JpHP8PsVy1eunTJ5d8vXrzo8srGsn4BlfwFOWPGDAwePBgffvghPvvsM4wePdpxvs/dE+Elf+3ZWSwWl+dHr70oRobNZgMADBw4EIMGDXI5j/1c3fVSVm+5rH3iSln7xJ1fsDabDVFRUU4f/pLc6d24q2PHjsjJycGlS5eQm5uLCRMmoHnz5rjhhhuwadMm/PDDDwgNDUXr1q0d2wZ47/i4U9/LEhUVhR07dmDt2rX49NNP8emnn2LhwoVITU0tdTHQ9eCqvrjSpEkTAMCuXbvQqlUr5dvhzj5MTk5GSEgIVqxYgTvuuAMrVqyAn58f+vXr53LZRYsW4emnn8Zjjz2G8ePHe7xNV65cwX//+180b97c42UfeOABbNmyBU8++SRatWqF0NBQ2Gw23H333Y76aZeamooxY8bg2LFjKCwsxNdff+10MZN9/nHjxiEpKcnl+q79gefucSXPedSI1q9fHwDw448/IjY21ulvFy9exNGjR9G9e3fDGxMXF4e4uDiMHz8eW7ZswZ133om5c+fixRdfBFDxsKorERERLofRrv0F3qhRI6xduxanT58utzfqahsiIyNRs2ZNWK1WJCYmlrs99evXx+7duyGEcHqvH3/8saKiONm/f7/Tr8uffvoJNputzKtFS7L/qr/2AqNr94knGjVqhPXr1+POO+8s9wNrr0P79+93GtorKChwq6cGXB3aW7hwIZYvXw6r1Yo77rgDfn5+6Nixo6MRveOOOxxfxJ4cn7K299qrycua5q7y6nJgYCCSk5ORnJwMm82GESNGYN68eS5vx6hI/fr18f3338Nmszn1Rvft2+f4uxE9evSAv78/lixZUuHFRZGRkQgJCXFZx/ft2wc/P79S3yfuqFGjBu69916sXLkSM2fORHZ2NhISElCnTp1S83744Yd45JFHcP/99zuujPbUqlWrcOnSpTIbrrL89ttvyMnJwaRJkzBhwgTH9P3797ucv3///sjIyMC7776LS5cuISAgwKl3bf/cBAQEeFyfST2PhnO7deuGwMBAvPHGG6V+Pc2fPx/FxcXo0aOHxxtx9uxZFBcXO02Li4uDn5+f060fNWrU8PjK0kaNGmHfvn1Ol8fv3LkTmzdvdpqvT58+EEK4vAG+5K9fV9vg7++PPn364L333sPu3btLLV9y3T179sSJEyecznlcvHgR8+fP96hc134RzJo1CwDc2v/169eHv7+/47yi3euvv+7RNpT0wAMPwGq1YvLkyaX+Vlxc7NhniYmJCAgIwKxZs5z2a1ZWltvrsg/TvvLKK2jRooXjnHpCQgJycnLw3XffOeYBPDs+16pTpw6aN2+Od955B+fPn3dM/+KLL7Br1y63t/la9vv0rq1LJW95AAA/Pz9HL9nd26BK6tmzJ/Ly8hxXtQJXj8esWbMQGhqKzp07e/yeABAbG4thw4bhs88+c9S9kmw2G2bMmIFjx47B398f3bt3x4cffuh0uiE/Px/Lli1Dx44dHcPDnkpJScGJEyfw1ltvYefOnS6Hcr/88kv0798fnTp1wtKlS0sNbbtj586deOKJJxAREVHqeoSK2H/MXTsSUVadr1WrFnr06IElS5Zg6dKluPvuu52uIo6KikKXLl0wb948/PLLL6WW55OVvMujnmhUVBQmTJiA8ePHo1OnTujVqxdCQkKwZcsWvPvuu+jevTuSk5M93ogNGzYgPT0d/fr1w6233ori4mIsXrzY8eVnFx8fj/Xr12PmzJmoU6cOGjZsWOrS+GsNGTIEM2fORFJSEoYOHYqTJ09i7ty5uO2225wuZujatSsefvhh/POf/8T+/fsdwyybNm1C165dkZ6eXu42vPzyy/j888/Rrl07DBs2DM2aNcPp06exbds2rF+/HqdPnwZw9QT/7NmzkZqaitzcXNSuXRuLFy9GSEiIR/vs0KFD6NWrF+6++25s3boVS5YswYMPPoiWLVtWuGx4eDj69euHWbNmwWKxoFGjRvj3v/9d5nlBd3Tu3BnDhw/H1KlTsWPHDnTv3h0BAQHYv38/Vq5ciddeew19+/Z13GM6depU3HvvvejZsye2b9+OTz/91O3bDW655RbExMTgxx9/dFxQBQCdOnXC008/DQBOjSgAt4+PK1OmTMHf/vY33HnnnUhLS8Nvv/2G2bNno3nz5k4NqyfsTxMaPXo0kpKS4O/vj/79++ORRx7B6dOn8de//hX16tXD4cOHMWvWLLRq1cpxHtMTjz76KObNm4fBgwcjNzcXDRo0wKpVq7B582ZkZWUZOr9nN2PGDBw4cACjR4/G//3f/+Hee+9FREQEjhw5gpUrV2Lfvn3o378/AODFF1/EunXr0LFjR4wYMQLVqlXDvHnzUFhYiGnTphnehp49e6JmzZoYN25cqe8L4OroSq9evWCxWNC3b1+sXLnS6e8tWrQoNZS/adMmXL58GVarFb/++is2b96Mjz76COHh4Xj//fcRExPj0TaGhYWhU6dOmDZtGq5cuYK6devis88+w6FDh8pcJjU11XHxoasfpnPmzEHHjh0RFxeHYcOG4eabb0Z+fj62bt2KY8eOYefOnR5tI0kwcknvkiVLRPv27UWNGjVEUFCQaNKkiZg0aZLT5dZClH3J+LWXgB88eFAMGTJENGrUSAQHB4sbb7xRdO3aVaxfv95puX379olOnTqJ6tWrO90OUdGtN0uWLBE333yzCAwMFK1atRJr164tdYuLEEIUFxeL6dOniyZNmojAwEARGRkpevToIXJzcyvcBiGEyM/PFyNHjhSxsbEiICBAxMTEiG7duon58+c7refw4cOiV69eIiQkRNSqVUuMGTPGcSuIu7e47N27V/Tt21fUrFlTREREiPT0dHHp0iWneQGIkSNHunyfgoIC0adPHxESEiIiIiLE8OHDxe7du13e4lKjRo0yt+Na8+fPF/Hx8aJ69eqiZs2aIi4uTjz11FPixIkTjnmsVquYNGmSqF27tqhevbro0qWL2L17t6hfv36Ft7jY9evXTwAQ2dnZjmlFRUUiJCREBAYGltoXQrh3fFzdniCEEMuXLxdNmjQRQUFBonnz5uKjjz4Sffr0EU2aNCm17PTp00utG4DTLQrFxcVi1KhRIjIyUlgsFse+XLVqlejevbuIiooSgYGB4k9/+pMYPny4+OWXXyrcJ65ucbGXOy0tTdSqVUsEBgaKuLi4UuUrb9vLU1xcLN566y2RkJAgwsPDRUBAgKhfv75IS0srdfvLtm3bRFJSkggNDRUhISGia9euYsuWLU7z2G/J+Pbbb52ml3VblhBCPPTQQ47bYq5lX66sV8ljcu28AQEBIjIyUnTq1Em89NJLbt36I4TrOnTs2DFx3333iRtuuEGEh4eLfv36iRMnTpTaBrvCwkIREREhwsPDXdZlIYQ4cOCASE1NFTExMSIgIEDUrVtX3HvvvWLVqlWOecran6SORQgDTyUgbSZOnIhJkyahoKDAoxvFSb1WrVohMjJS2e0oRHbFxcWoU6cOkpOT8fbbb+veHCqHqR5AT6TDlStXSp2z37hxI3bu3HldU4vId33wwQcoKChAamqq7k2hClTZB9ATqXL8+HEkJiZi4MCBqFOnDvbt24e5c+ciJiZG6gEfRNf6z3/+g++//x6TJ09G69atDV/4Rd7DRpSoAhEREYiPj8dbb72FgoIC1KhRA/fccw9efvll3HTTTbo3j6qQN954A0uWLEGrVq0Y6m0SPCdKRESm9+WXX2L69OnIzc3FL7/8gvfffx+9e/cud5mNGzciIyMDe/bsQWxsLMaPH++UzOUOnhMlIiLTu3DhAlq2bOn2wzQOHTqEe+65B127dsWOHTvwxBNP4JFHHsHatWs9Wi97okREVKXYc1nL64k+/fTT+OSTT5wewNK/f3+cOXPGo+dV85woEREpdfnyZbfygSsirnk8KgAEBQUhKChI+r23bt1a6rGJSUlJbidK2XnUiNryGlc8UxmS6rQyvCwArD2xQ2p5QyzV4Rf9PQDAlt8CEK4fvF8emXLLltnIuoNDgvDx+SUAjJfZ6LrtfPFY61LyeCeHDsTli54/WlDL8fof2TputMyAXLl11pV1tpUVzyTh8uXLaFg/FHkn5UM+QkNDSz0VLDMzExMnTpR+77y8PERHRztNi46OxtmzZ3Hp0iW3H9rPnigRESlTVFSEvJNWHMqtj7Caxi+7OXvOhobxh3H06FGnZyur6IWqxEaUiIiUC6vpJ9WIOt4nLMxwQEF5YmJikJ+f7zQtPz8fYWFhHkXHsRElIiLlrMIGq8Rlq1Zhq3gmCR06dMDq1audpq1btw4dOnTw6H14iwsRESlng5B+eeL8+fPYsWMHduzYAeDqLSw7duzAkSNHAADPPvus02MUH3vsMRw8eBBPPfUU9u3bh9dffx0rVqzA2LFjPVovG1EiIjK97777Dq1bt0br1q0BABkZGWjdurUjCP2XX35xNKgA0LBhQ3zyySdYt24dWrZsiRkzZuCtt97yOHSdw7lERKScDTbIDMh6unSXLl1KBZ+X5Ooxil26dMH27ds93TQnbESJiEg5qxCwSjzLR2ZZb+JwLhERkUHsiRIRkXJGLg66dnkzYCNKRETK2SBgZSNKRETkOV/pifKcKBERkUGe9UQt7j8K6VrBIZLPO5RYt3HVnf9tKXPGMkmVW7LMRtYd5LSMsTIbXbeDLx5rTUoe7yCj26/leF0lW8cNlxnQ+31oAr5ydS7zRImISJmzZ88iPDwc+36IRk2JZ+eeO2dDk6b5+P3336/Ls3NV4XAuERGRQR4N5yaHDrxe21GhDw98r2Gt1eEX/R8AQN+ooSg0mDtolGyZ/9aohcfLBIUEYdXJtwEAtvx2AIzliRpZtyrG9tsfx9pouWXLLHO8ja675PHWUcdl6TrWsnTWFXtu7vVmlbw6V2ZZb/KoETUaXquEwXBoKSXOixVeLPR++SXLLL+9lwxvg+nqitM5UGPlli6zxPFWsb+11HFZmo61LJ11xVusApIpLuq25XricC4REZFBvE+UiIiUs/3vJbO8GbARJSIi5WywwGr0Hrn/LW8GHM4lIiIyiD1RIiJSziauvmSWNwM2okREpJxVcjhXZllvYiNKRETK+UojynOiREREBrEnSkREytmEBTYhcXWuxLLexEaUiIiU43AuERERlcujnqjWDDzNGZNSuYNG+WqeqCxD+60S5IlqyKdUlq2pi6ZjLcuc+cqescIPVol+mlXhtlxPzBMlIiJl7HmiObv+hBoSeaIXztnQLe4I80SJiIiqKo+Gc235xjPwdOZLGqUia1FPDqoM/VmLsnTmqMrQ8RnRXW49uZr667jO78OPzy/xynp85cIiz67O1Zx3qJPhrEUT5P45qQRZi7J05qjK0P8Z8X65teRqVoI6rv9YX39W4QerkDgnapITjRzOJSIiMoj3iRIRkXI2WGCT6KfZYI6uKBtRIiJSjudEiYiIDJI/J2qOnijPiRIRERnEnigRESl39ZyoxAPoOZxLRES+yib52D+zXFjE4VwiIiKD2BMlIiLlfOXCIjaiRESknA1+vE+0FA1RTTopiYkyQWSRM/0xUbJ0RsDJ0PEZ0V1uPZFg+uu4Gb8PyTVGoRERkTL2KLTF2+MQUtPf8PtcPGfFw613VfooNA7nEhGRcvKh3Obo33nUiCaHDrxe21EhPZFif0Qm+WIUmtEym5HuSDB99MaC+WL8G6ArAu4qv2izfSdVbh41olrje3REcpU4V+KLUWiGy2x65oyAM0RzLJj++mXSKDQT1E+b8INN4upcm0nONHI4l4iIlONwLhERkUE2AFYh89g/c+ATi4iIiAxiT5SIiJSTf9iCOfp4bESJiEg5+cf+maMRNcdWEhERVULsiRIRkXLMEyUiIjKIw7lERERULvZEiYhIOfmHLZijj8dGlIiIlLMJC2wyD1uQWNabPGpEtWbgacnl/GOdvpgnarjMJqQ7V1MfvdmavpihCujKUaXrgXmiRESkjD1P9OVvOyM41Phg5+XzxXjm9i+YJ0pERL5HPsWlCp4TlckTlc3VlMnfM75u+axFHXmJdsbKrTdfEtCfMakrR9WsmblmUxXyRGV8fH6JV9ZjhQVWibFymWW9yXt5opL5d1rWrSBr0XQZrJrzJQH9GZPaclTNmplrar5Zx0kdDucSEZFyHM4lIiIyyAq5IVmruk25rszR1BMREVVC7IkSEZFyHM4lIiIyiA+gJyIiMpk5c+agQYMGCA4ORrt27fDNN9+UO39WVhb+/Oc/o3r16oiNjcXYsWNx+fJlt9fHRpSIiJQT/8sTNfoSBi5Kys7ORkZGBjIzM7Ft2za0bNkSSUlJOHnypMv5ly1bhmeeeQaZmZn44Ycf8PbbbyM7OxvPPfec2+tkI0pERMrZh3NlXp6aOXMmhg0bhrS0NDRr1gxz585FSEgIFixY4HL+LVu24M4778SDDz6IBg0aoHv37hgwYECFvdeS2IgSEVGldfbsWadXYaHrB1UUFRUhNzcXiYmJjml+fn5ITEzE1q1bXS5zxx13IDc319FoHjx4EKtXr0bPnj3d3j5eWERERMqpikKLjY11mp6ZmYmJEyeWmv/UqVOwWq2Ijo52mh4dHY19+/a5XMeDDz6IU6dOoWPHjhBCoLi4GI899phHw7nei0KTjO7Rs275mCjzxcfpjcYC9MdjaYuAM2vcn8lUiSg0E1AVyn306FGnFJegIHX7buPGjZgyZQpef/11tGvXDj/99BPGjBmDyZMn4/nnn3frPRiFRkREytij0EZ/9TcEhQYYfp/C81fwz44fuh2FVlRUhJCQEKxatQq9e/d2TB80aBDOnDmDDz/8sNQyCQkJaN++PaZPn+6YtmTJEjz66KM4f/48/Pwq/hHAc6JERGR6gYGBiI+PR05OjmOazWZDTk4OOnTo4HKZixcvlmoo/f39AQDu9i89Gs615RuP79EZ/WNUZYhM8j410Vh6Yr1kyJfb7HF/usotw8g+qwqxdzJ1xVtRaDb4wSbRTzOybEZGBgYNGoQ2bdqgbdu2yMrKwoULF5CWlgYASE1NRd26dTF16lQAQHJyMmbOnInWrVs7hnOff/55JCcnOxrTinh2YZFEZJD5o3/0RCZ5napoLLPtKxXlNnncn65yy5D9XjFr7J0Zvk+twgKrxIVFRpZNSUlBQUEBJkyYgLy8PLRq1Qpr1qxxXGx05MgRp57n+PHjYbFYMH78eBw/fhyRkZFITk7GSy+95PY6eXUuERFVGenp6UhPT3f5t40bNzr9v1q1asjMzERmZqbh9bERJSIi5VTd4lLZsRElIiLlhGSKi+AD6ImIiKo29kSJiEg5KyywSjzJQmZZb2IjSkREytmE3HlNm0keA8ThXCIiIoPYEyUiIuVskhcWySzrTWxEiYhIOXu4tszyZsBGlIiIlNPxxCIdPGtEJaKazBj9Uxkik7xPUTSWllgvGQrKbfK4P13llmFkn1WF2Dszfp9WVYxCIyIiZexRaP1zBiIwNNDw+xSdL8LybkvcjkLThcO5RESknA2Sj/0zydCfR41ocuhAwyvSGf1jFKPQ9MREyTJW1/4ot65jbdY6braYw8rwuda5z7wVheYrPGpEtUQ1qVi3EoxCMw0jx8npR6+eY61/Xxsrt/7tluGrx/r6E5JX54qq2BMlIiJyh6+kuJjjblYiIqJKiD1RIiJSjk8sIiIiMojDuURERFQu9kSJiEg5PjuXiIjIIF8ZzmUjSkREyvlKI8pzokRERAaxJ0pERMr5Sk+UjSgRESnHRtQFPXmHCtZtEPNETZpZaKiuVXf+t4ZjbdY6brZsy8rwuTbbPqOyMU+UiIiUseeJJq4ejmo1jP9YKL5QiPU95zFPlIiIfA+Hc13QmScqw2h2n+7cQd35krryRPXUFfkcVbPXcTPmx8pmx8qUWeZ4y362ZdbtF62vnlZFpskTlaEmu8/7uYO6Mwe15YnqqCsqclRNXsdNmR8rmR0rVWaJ4y29n02QbcyeKBERkUG+0ojyYQtEREQGsSdKRETK+UpPlI0oEREpJ4QFQqIhlFnWm9iIEhGRcr4ShcZzokRERAaxJ0pERMrxnCgREZFBvnJOlMO5REREBrEnSkREynE41wWdUWgyjG637sgk3dFY2qLQtNQVBRFwJq/jpoy+k4y9kyqzxPGW/mxrrGvu8pXhXEahERGRMvYotPj3xkpHoeX2+Qej0IiIyPcIyeFcs/REvRaFJktH7JDumCjZaC0j5VYV/yYT9WTmcsvQs8/UxIIZZeZjrSOq0M4MUWgCgMw4p1mGSL0XhSZLZ+wQNMVEScYdyW+v8fg3nbF5OsstQ8s+UxULZpSJj7VZvw9JLQ7nEhGRcjZYYPGBx/6xESUiIuV85epcNqJERKScTVhg8YH7RPnEIiIiIoPYEyUiIuWEkLw61ySX57IRJSIi5XzlnCiHc4mIiAxiT5SIiJTzlZ4oG1EiIlKOV+cSERFRudgTJSIi5Xh1rgs68i0dNGT3ac9alMwMNFJuVRmqOrNndZZbhp59pihb0ygTH2uzfh96y9VGVOacqMKNuY6YJ0pERMrY80QbL3kG/iHBht/HevEy9g98mXmiRETke3h1rgs680R10J0nqkNlyFqUzZg0Rj5XU892X+WrmbnG/HGsfauOX+XVPFHJ5c3APHmimmnJWtTOnHmihqjI1dSY8eirmbmGOHVwfKiOe5mv9ER5iwsREZFBPCdKRETq+ch4LnuiRESk3v+Gc42+YHA4d86cOWjQoAGCg4PRrl07fPPNN+XOf+bMGYwcORK1a9dGUFAQbr31Vqxevdrt9bEnSkREVUJ2djYyMjIwd+5ctGvXDllZWUhKSsKPP/6IqKioUvMXFRXhrrvuQlRUFFatWoW6devi8OHDuOGGG9xeJxtRIiJSTscTi2bOnIlhw4YhLS0NADB37lx88sknWLBgAZ555plS8y9YsACnT5/Gli1bEBAQAABo0KCBR+vkcC4RESknM5Rb8sres2fPOr0KC11fFV1UVITc3FwkJiY6pvn5+SExMRFbt251ucxHH32EDh06YOTIkYiOjkbz5s0xZcoUWK1Wt8vJRpSIiCqt2NhYhIeHO15Tp051Od+pU6dgtVoRHR3tND06Ohp5eXkulzl48CBWrVoFq9WK1atX4/nnn8eMGTPw4osvur19HM4lIiL1JC4OciwP4OjRo06P/QsKUvfMYpvNhqioKMyfPx/+/v6Ij4/H8ePHMX36dGRmZrr1HmxEiYhIOVXnRMPCwtx6dm6tWrXg7++P/Px8p+n5+fmIiYlxuUzt2rUREBAAf39/x7SmTZsiLy8PRUVFCAwMrHC9HM4lIiL1hIKXBwIDAxEfH4+cnBzHNJvNhpycHHTo0MHlMnfeeSd++ukn2Gw2x7T//ve/qF27tlsNKGCmKDQNtEehaVApYqK0xDwpiATTGE/lq3F/xlR3/rfP1PGqLyMjA4MGDUKbNm3Qtm1bZGVl4cKFC46rdVNTU1G3bl3HedXHH38cs2fPxpgxYzBq1Cjs378fU6ZMwejRo91ep0eN6Mfnl3gye5Vif0i3L7E/pNuIj88r3BAvM+OxVrG/zVhuWb5ax71Bx7NzU1JSUFBQgAkTJiAvLw+tWrXCmjVrHBcbHTlyBH5+fwzAxsbGYu3atRg7dixatGiBunXrYsyYMXj66afdXifzRImISBl7nuif5k+AX3XjeaK2S5dx5NEXqlaeqC3fePSPLD2xQ74Xj6UqCk0nXeWWqaOyzFrHdewz3fFvgG9EofkKz67ONWvUk9Ht9vl4LOMxUTrpKrfWqDyT1nHd8YLaIg5N+LnylK9EofEWFyIiUo8pLkRERFQe9kSJiOg6sMDw/UOO5Ss/NqJERKQeh3OJiIioPOyJEhGRej7SE2UjSkRE6ilKcans2IgSEZFyqlJcKjueEyUiIjKIPVEiIlKP50RdMGHUEwCJ7fa9eCxVUWg66Sq31qhAk9ZxHftMe/wb4BtRaD5yTpQpLkREpIw9xaXeP1+QTnE5NnpC1UpxISIicodFXH3JLG8GXotCk4080hMdxJgoozFRvhgBZ/YoNB3lNuvnGtBbx2V4LQqN50Rd0Bl5pCM6iDFRxrfBByPgzB6FpqXcZv1cAz4RZ0YV43AuERGp5yMXFrERJSIi9XxkOJcPWyAiIjKIPVEiIlLPR3qibESJiEg9NqJEREQG+ciFRTwnSkREZBB7okREpByfWERERGSUj5wT5XAuERGRQWxEiYiIDPJanqh0bqCW/D1mLRrmgzmqZs8T1VJus36uAd/IBJVggeQ5UWVbcn0xT5SIiJSx54nWf+VF+AVL5IlevozDT49nnigREfkgH7lP1Gt5ouakN2tRlrG8QzVZizrJltvosZalOz+Wddx7ZLNIZfb5x+eXSK3bbT5yda7X8kRNSXfWoiwjx0tV1qJOkuU2eqxl6d/XrONeI1m/9NcVsuNwLhERqceeKBERkTF8YhEREZFRPtIT5cMWiIiIDGJPlIiI1PORnigbUSIiUs5XzolyOJeIiMgg9kSJiEg9PrGIiIjIIB85J8rhXCIiIoO8FoVmTppjomQZOl6KYqJ0kiy30WMtS3f0Heu4F0l+l2rd527ylQuLGIVGRETK2KPQbp4wRToK7eALz1X6KDQO5xIRERnktSg0nXFJRqmIiZIhu890xkTJRD35aiSYbDyWMb4d9yfzuTZfua/yi/ZSPZMczjXLhUVei0Izf3SP9+OxpPeZzpgoU9cVTZFgOqIGfTzuT+Zzbbpye5uPXJ3LW1yIiEg9H2lEeU6UiIjIIPZEiYhIOV+5xYU9USIiIoPYiBIRERnE4VwiIlLPRy4sYiNKRETK8ZwoERERlYs9USIiuj5M0puUwUaUiIjU85FzohzOJSIiMshreaJmyL+7loqsRRnS+0xn1qLJ6kqlyNXUktfr25m5Mp9r85Xbu3zlwiLmiRIRkTL2PNHGT06Bf5DxPFFr4WXsn17580R5TpSIiJTzlZ6oR41ocujA67UdlVLJjEmZbE2jzJovKctX80R1UFHH9dRTGebPzDVFnqiP8KgR1Z/zqI9UtqZRJs2XlKW/npkwV1MBw3XcDNmWJVWFzFwz7HNNV+fOmTMH06dPR15eHlq2bIlZs2ahbdu2FS63fPlyDBgwAH/729/wwQcfuL0+Xp1LRETqCQUvD2VnZyMjIwOZmZnYtm0bWrZsiaSkJJw8ebLc5X7++WeMGzcOCQkJHq+TjSgREVUJM2fOxLBhw5CWloZmzZph7ty5CAkJwYIFC8pcxmq14qGHHsKkSZNw8803e7xONqJERKSc/cIimRdw9Wrfkq/CQtdD4UVFRcjNzUViYqJjmp+fHxITE7F169Yyt/OFF15AVFQUhg4daqicbESJiEg9RcO5sbGxCA8Pd7ymTp3qcnWnTp2C1WpFdHS00/To6Gjk5eW5XOarr77C22+/jTfffNNwMXmLCxERVVpHjx51uk80KEjNQy7OnTuHhx9+GG+++SZq1apl+H3YiBIRkXqKrs4NCwtz62ELtWrVgr+/P/Lz852m5+fnIyYmptT8Bw4cwM8//4zk5GTHNJvNBgCoVq0afvzxRzRq1KjC9XI4l4iIlFN1TtRdgYGBiI+PR05OjmOazWZDTk4OOnToUGr+Jk2aYNeuXdixY4fj1atXL3Tt2hU7duxAbGysW+tlT5SIiKqEjIwMDBo0CG3atEHbtm2RlZWFCxcuIC0tDQCQmpqKunXrYurUqQgODkbz5s2dlr/hhhsAoNT08rARJSIi9TQ8bCElJQUFBQWYMGEC8vLy0KpVK6xZs8ZxsdGRI0fg56d2AJaNKBERKafr2bnp6elIT093+beNGzeWu+yiRYs8Xp9HjagZ48xklIzHkooFM8qk0ViyfDYKTQMlddwEsVzOqkDcnxn2uY+EcjMKjYiIlLFHoTUdKR+F9sMcRqEREZEv8pGeKKPQyqEiHst89MdEyTISM1UZjrVMPJbx/a03+s6ssXe6yey3j88vUbglZbNA7myQhjNJhjAKzW16YsG8rhLERMmSr6cmjIAzur2ao+/0f6eY83Otf7+RHYdziYhIPQ7nEhERGaPrFhdv42P/iIiIDGJPlIiI1ONwLhERkQSTNIQyOJxLRERkEHuiRESknK9cWMRGlIiI1OM5USIiImN8pSfKc6JEREQGMQqtHCriscxHf0yULCP1tDIca6nPl+H9rTf6zqyxd7qZ4rvYR4ZzGYVGRETK2KPQWgyZAv9AiSi0osv4fkHlj0LjcC4REZFBHg3n2vK9H1ukgtHYIBWRSTqinmSoionSE+slQz4Czmzxb4Dz8ZaJvjMTXyxzSd6KQvOV4VzPrs41YWQQoCo2yFhkkrkji4zHRGmJ9ZKhIgLO1PFvktF3JuWLZfYaH2lEOZxLRERkEO8TJSIi5XzlPlE2okREpB6Hc4mIiKg87IkSEZFyFiFgkXgMgcyy3sRGlIiI1POR4Vw2okREpJyvXFjEc6JEREQGsSdKRETqcTiXiIjIGA7nEhERUbk864lqzIiUYTR7T0XuoCly/0pQlbWoJxtThoIcVZNlqALOZZXKjzURXyyzFj4ynMs8USIiUsaeJxqf8pJ0nmhu9v9jnigREVFVxTzRcqjIHdSZMWnMH7maMnmi5sNy68jW1J0dqyszVyfmiarFPFE3mTFj0hCnc6DG80RNh+XWk62pOTtWW2aujzDLFbYyOJxLRERkEO8TJSIi9YS4+pJZ3gTYiBIRkXK+8rAFNqJERKSej1xYxHOiREREBrEnSkREyllsV18yy5sBG1EiIlKPw7lERERUHvZEiYhIOV6dS0REZBTvE3XBh6PQzBiPZUx1538bjEIzH5ZbSyyY5tg7bXF/VGUwCo2IiJSxR6G1S56MagHGo9CKr1zGfz5+vtJHoXE4l4iI1PORq3O9FoUmGxtk1sgks8UlqYh/A+SOl566oj8KTabcxve3fBSazmNthO/W8av8os0Wz1i5eS0KTTo2yKSRSWaOS5KKxjJbXakEUWhS5Ta6vSqi0HQea0k+Vce9jFfnEhERGcWrc4mIiIzxlZ4on1hERERkEHuiRESkHq/OJSIiMobDuURERFQu9kSJiEg9m7j6klneBNiIEhGRej5yTpTDuURERAaxJ0pERMpZIHlhkbItub7YiBIRkXp8YpELEtl/0tl7Js0dNFvmoJIMVcCEdUV/nqhUuQ3vbwV5ojqPtQG+W8fpemCeKBERKWPPE+3414moVk0iT7T4Mr7aMNHjPNE5c+Zg+vTpyMvLQ8uWLTFr1iy0bdvW5bxvvvkm3nnnHezevRsAEB8fjylTppQ5vyu8sIiIiNQTCl4eys7ORkZGBjIzM7Ft2za0bNkSSUlJOHnypMv5N27ciAEDBuDzzz/H1q1bERsbi+7du+P48eNur9OjnqhMnqg56c+Y9D5fLDOgIldTlo58ShXZmnqyfmWYv47L5JF+fH6Jwi0pzd4TTeiSKd0T3bRxkkc90Xbt2uH222/H7NmzAQA2mw2xsbEYNWoUnnnmmQqXt1qtiIiIwOzZs5GamurWOr2WJ2pKlSBj0ut8scyAmlxNWZpzOXXkiWpRBeq47hxWbzp79qzT/4OCghAUVPqcclFREXJzc/Hss886pvn5+SExMRFbt251a10XL17ElStXcOONN7q9fRzOJSIi9WwKXgBiY2MRHh7ueE2dOtXl6k6dOgWr1Yro6Gin6dHR0cjLy3Nrk59++mnUqVMHiYmJbheTt7gQEZFyFiFgkbhu1b7s0aNHnYZzXfVCVXj55ZexfPlybNy4EcHB7g9DsxElIqJKKywszK1zorVq1YK/vz/y8/Odpufn5yMmJqbcZV999VW8/PLLWL9+PVq08Ox8M4dziYhIPS9fnRsYGIj4+Hjk5OQ4ptlsNuTk5KBDhw5lLjdt2jRMnjwZa9asQZs2bTxbKdgTJSKi60HDE4syMjIwaNAgtGnTBm3btkVWVhYuXLiAtLQ0AEBqairq1q3rOK/6yiuvYMKECVi2bBkaNGjgOHcaGhqK0NBQt9bJRpSIiKqElJQUFBQUYMKECcjLy0OrVq2wZs0ax8VGR44cgZ/fHwOwb7zxBoqKitC3b1+n98nMzMTEiRPdWicbUSIiUs4iJB9Ab3DZ9PR0pKenu/zbxo0bnf7/888/G1tJCWxEiYhIPR95AD0vLCIiIjKIPVEiIlLOYrv6klneDLwWhWZO+uOxvM8XywwoiQSTpSFaS0ksmOm+F8xfx00Rsegjw7mMQiMiImXsD6Dvcvv/k34A/cZvX/I4Cs3beE6UiIjIII+Gc3VGoclE/xiPatIbj6UnYkpNmXXEeskoGQmmKx6Lddxb9NdxnfyivbPdqp6dW9mZJgpNKvrH6HbrjsfSsb9VlVlzrJccPfFYrONeUgnquE/wkXOiHM4lIiIyiLe4EBGRegKOTFDDy5sAG1EiIlLOV86JcjiXiIjIIPZEiYhIPQHJC4uUbcl1xUaUiIjU49W5REREVB72RImISD0b5J5LXCUfQE9EROQGX7k6l40oERGp5yPnRE0ThSYV/WN4uzXHY2nZ34rKrCHWS4ZzWfXEY7GOe4v+Ok5VB6PQiIhIGXsUWrdm41DN3/iPlGJrIXL2vlrpo9A4nEtEROpxOLe05NCB12s7rivGRHnijzLLRILpiDOzM7bf5I+17PEyawScngg34+suWWYdn2tAb4yat6LQfIVHjaj+iCqDGBPlPqdzgcYjwbTWFSPbrOJYSx4v/Z8vY8dbS4SbinVD0+ca8I0YNd7iQkREZIyv3OLCJxYREREZxJ4oERGpxwuLiIiIDLIJwCLRENrM0YhyOJeIiMgg9kSJiEg9DucSEREZJdmImiSVm40oERGp5yM9UZ4TJSIiMog9USIiUs8mIDUka5Krc9mIEhGResJ29SWzvAl41IjqyHlUglmLHqju/G+Dz77UWlcM7TcFx1ryeJk1R1VPDqrxdZcss5bPNcAs0iqEeaJERKSMPU80MfZxVPOTyBO1FWL90TeYJ0pERD6I50RLs+Ubzw00X74koCJb03zlVpOhKpOXqDtXU1fGpA6680R1UFFmwHzltvv4/BLdm1CleNYTlcjAM12+JKAkW9N05VaVoWrWugKNGZPambCOSzNpZq4Z+Mh9ohzOJSIi9QQkG1FlW3Jd8WELREREBrEnSkRE6nE4l4iIyCCbDYDEAxNsVfBhC0RERG7xkZ4oz4kSEREZxJ4oERGp5yM9UTaiRESkno88sYjDuURERAaxJ0pERMoJYYOQiDOTWdabPGtEJeJ7zBeNBaiIBTNfuRXFv5msrlSKeCwNtEehaaCizID5yu11QsgNyZrknCij0IiISBl7FFq3G1JRzRJo+H2KRRFyzrzDKDQiIvJBQvLCIpP07zxqRJNDBxpekUw0FiAXO6QiCk1HPJaOfVYZIsHMWm7Z7dbD3HF/RvhqHbfzWhSazQZYJM5rVsVzolLRPxLRWNrWrSoWzCid+wwaI8HMWm7J7dbC7HF/kny1jpM6HM4lIiL1OJxLRERkjLDZICSGc6vmLS5ERETu8JGeKJ9YREREZBB7okREpJ5NAJaq3xNlI0pEROoJAalQbpM0ohzOJSIiMog9USIiUk7YBITEcK5ZnkjLnigREaknbPIvA+bMmYMGDRogODgY7dq1wzfffFPu/CtXrkSTJk0QHByMuLg4rF692qP1sRElIqIqITs7GxkZGcjMzMS2bdvQsmVLJCUl4eTJky7n37JlCwYMGIChQ4di+/bt6N27N3r37o3du3e7vU42okREpJywCemXp2bOnIlhw4YhLS0NzZo1w9y5cxESEoIFCxa4nP+1117D3XffjSeffBJNmzbF5MmT8Ze//AWzZ892e51sRImISD0vD+cWFRUhNzcXiYmJjml+fn5ITEzE1q1bXS6zdetWp/kBICkpqcz5XfHowqJ1tpWezK7UOs1PgPJa8oFCsvvMjGUGfLfcsvyijSWL6P5syjDrsTbDPi/GFakHFhXjCoCr+aQlBQUFISiodCD6qVOnYLVaER0d7TQ9Ojoa+/btc7mOvLw8l/Pn5eW5vZ28OpeIiJQJDAxETEwMvsrz7AIdV0JDQxEbG+s0LTMzExMnTpR+b1XYiBIRkTLBwcE4dOgQioqKpN9LCAGLxSmvz2UvFABq1aoFf39/5OfnO03Pz89HTEyMy2ViYmI8mt8VNqJERKRUcHAwgoODvbrOwMBAxMfHIycnB7179wYA2Gw25OTkID093eUyHTp0QE5ODp544gnHtHXr1qFDhw5ur5eNKBERVQkZGRkYNGgQ2rRpg7Zt2yIrKwsXLlxAWloaACA1NRV169bF1KlTAQBjxoxB586dMWPGDNxzzz1Yvnw5vvvuO8yfP9/tdbIRJSKiKiElJQUFBQWYMGEC8vLy0KpVK6xZs8Zx8dCRI0fg5/fHTSl33HEHli1bhvHjx+O5555D48aN8cEHH6B58+Zur9MizPJsJSIiokqG94kSEREZxEaUiIjIIDaiREREBrERJSIiMoiNKBERkUFsRImIiAxiI0pERGQQG1EiIiKD2IgSEREZxEaUiIjIIDaiREREBrERJSIiMuj/A5nM1Ur3VAgmAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "weights_to_display = tf.reshape(tensor_data, [tensor_data.shape[0],tf.reduce_prod(tensor_data.shape[1:])])\n", "weights_to_display = weights_to_display[0:width, 0:height]\n", "\n", "val_ones = np.ones([height, width])\n", "val_zeros = np.zeros([height, width])\n", "subset_values_to_display = np.where(abs(weights_to_display) > 0, val_ones, val_zeros)\n", "\n", "plot_separation_lines(height, width)\n", "\n", "plt.axis('off')\n", "plt.imshow(subset_values_to_display)\n", "plt.colorbar()\n", "plt.title(\"Unstructed pruned weights for Conv2D layer\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "vqsfZdMpWr_N" }, "source": [ "The TensorFlow Model Optimization Toolkit includes a python script that can be used to check whether which layers in the model from the given tflite file have the structurally pruned weights: [`check_sparsity_m_by_n.py`](https://github.com/tensorflow/model-optimization/blob/master/tensorflow_model_optimization/python/core/sparsity/keras/tools/check_sparsity_m_by_n.py). The following command demonstrates how to use this tool to check for 2 by 4 sparsity in a specific model." ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:20:04.065726Z", "iopub.status.busy": "2024-03-09T12:20:04.065068Z", "iopub.status.idle": "2024-03-09T12:20:04.285390Z", "shell.execute_reply": "2024-03-09T12:20:04.284321Z" }, "id": "7HDYffebWr_N" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "python3: can't open file '/tmpfs/src/temp/tensorflow_model_optimization/g3doc/guide/pruning/./tensorflow_model_optimization/python/core/sparsity/keras/tools/check_sparsity_m_by_n.py': [Errno 2] No such file or directory\r\n" ] } ], "source": [ "! python3 ./tensorflow_model_optimization/python/core/sparsity/keras/tools/check_sparsity_m_by_n.py --model_tflite=pruned_model.tflite --m_by_n=2,4\n" ] } ], "metadata": { "colab": { "collapsed_sections": [ "Tce3stUlHN0L" ], "name": "pruning_with_sparsity_2_by_4.ipynb", "provenance": [], "toc_visible": true }, "interpreter": { "hash": "5be03e09ac1816611305450014280c0b9eb46a3a95e12dcae8d73de01e2da776" }, "kernelspec": { "display_name": "Python 3.6.9 64-bit ('mo': venv)", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" } }, "nbformat": 4, "nbformat_minor": 0 }