{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "30155835fc9f" }, "source": [ "##### Copyright 2022 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2023-10-17T12:26:59.077356Z", "iopub.status.busy": "2023-10-17T12:26:59.076891Z", "iopub.status.idle": "2023-10-17T12:26:59.080622Z", "shell.execute_reply": "2023-10-17T12:26:59.080010Z" }, "id": "906e07f6e562" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "5hrbPTziJK15" }, "source": [ "# Load LM Checkpoints using Model Garden" ] }, { "cell_type": "markdown", "metadata": { "id": "-PYqCW1II75I" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "yyyk1KMlJdWd" }, "source": [ "This tutorial demonstrates how to load BERT, ALBERT and ELECTRA pretrained checkpoints and use them for downstream tasks.\n", "\n", "[Model Garden](https://www.tensorflow.org/tfmodels) contains a collection of state-of-the-art models, implemented with TensorFlow's high-level APIs. The implementations demonstrate the best practices for modeling, letting users to take full advantage of TensorFlow for their research and product development." ] }, { "cell_type": "markdown", "metadata": { "id": "uEG4RYHolQij" }, "source": [ "## Install TF Model Garden package" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:26:59.084350Z", "iopub.status.busy": "2023-10-17T12:26:59.083920Z", "iopub.status.idle": "2023-10-17T12:27:09.445433Z", "shell.execute_reply": "2023-10-17T12:27:09.444441Z" }, "id": "kPfC1NJZnJq1" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\r\n", "\u001b[0m" ] } ], "source": [ "!pip install -U -q \"tf-models-official\"" ] }, { "cell_type": "markdown", "metadata": { "id": "Op9R3zy3lUk8" }, "source": [ "## Import necessary libraries" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:09.449901Z", "iopub.status.busy": "2023-10-17T12:27:09.449639Z", "iopub.status.idle": "2023-10-17T12:27:11.927098Z", "shell.execute_reply": "2023-10-17T12:27:11.926326Z" }, "id": "6_y4Rfq23wK-" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-10-17 12:27:09.738068: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", "2023-10-17 12:27:09.738115: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", "2023-10-17 12:27:09.738155: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n" ] } ], "source": [ "import os\n", "import yaml\n", "import json\n", "\n", "import tensorflow as tf" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:11.931599Z", "iopub.status.busy": "2023-10-17T12:27:11.930811Z", "iopub.status.idle": "2023-10-17T12:27:13.968982Z", "shell.execute_reply": "2023-10-17T12:27:13.967896Z" }, "id": "xjgv3gllzbYQ" }, "outputs": [], "source": [ "import tensorflow_models as tfm\n", "\n", "from official.core import exp_factory" ] }, { "cell_type": "markdown", "metadata": { "id": "J-t2mo6VQNfY" }, "source": [ "## Load BERT model pretrained checkpoints" ] }, { "cell_type": "markdown", "metadata": { "id": "hdBsFnI20LDE" }, "source": [ "### Select required BERT model" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:13.973810Z", "iopub.status.busy": "2023-10-17T12:27:13.973290Z", "iopub.status.idle": "2023-10-17T12:27:23.597185Z", "shell.execute_reply": "2023-10-17T12:27:23.596119Z" }, "id": "apn3VgxUlr5G" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2023-10-17 12:27:14-- https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/cased_L-12_H-768_A-12.tar.gz\r\n", "Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.219.207, 209.85.146.207, 209.85.147.207, ...\r\n", "Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.219.207|:443... connected.\r\n", "HTTP request sent, awaiting response... " ] }, { "name": "stdout", "output_type": "stream", "text": [ "200 OK\r\n", "Length: 401886728 (383M) [application/octet-stream]\r\n", "Saving to: ‘cased_L-12_H-768_A-12.tar.gz’\r\n", "\r\n", "\r", " cased_L-1 0%[ ] 0 --.-KB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " cased_L-12 2%[ ] 8.01M 39.0MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " cased_L-12_ 6%[> ] 24.01M 44.1MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " cased_L-12_H 11%[=> ] 43.82M 58.9MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " cased_L-12_H- 15%[==> ] 59.02M 62.5MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " cased_L-12_H-7 19%[==> ] 73.48M 64.2MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " cased_L-12_H-76 25%[====> ] 96.01M 69.1MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " cased_L-12_H-768 28%[====> ] 108.73M 68.4MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " cased_L-12_H-768_ 31%[=====> ] 120.01M 61.9MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " cased_L-12_H-768_A 35%[======> ] 136.01M 61.9MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "cased_L-12_H-768_A- 39%[======> ] 152.01M 63.2MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "ased_L-12_H-768_A-1 45%[========> ] 176.01M 64.9MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "sed_L-12_H-768_A-12 52%[=========> ] 200.01M 68.6MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "ed_L-12_H-768_A-12. 55%[==========> ] 214.15M 68.7MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "d_L-12_H-768_A-12.t 61%[===========> ] 235.60M 71.0MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "_L-12_H-768_A-12.ta 65%[============> ] 251.98M 71.6MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "L-12_H-768_A-12.tar 66%[============> ] 256.01M 70.4MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "-12_H-768_A-12.tar. 73%[=============> ] 280.01M 73.8MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "12_H-768_A-12.tar.g 75%[==============> ] 288.01M 69.8MB/s eta 1s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "2_H-768_A-12.tar.gz 78%[==============> ] 301.40M 69.0MB/s eta 1s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "_H-768_A-12.tar.gz 83%[===============> ] 321.52M 70.8MB/s eta 1s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "H-768_A-12.tar.gz 89%[================> ] 344.01M 70.1MB/s eta 1s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "-768_A-12.tar.gz 95%[==================> ] 367.90M 73.5MB/s eta 1s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "cased_L-12_H-768_A- 100%[===================>] 383.27M 79.4MB/s in 5.3s \r\n", "\r\n", "2023-10-17 12:27:19 (72.9 MB/s) - ‘cased_L-12_H-768_A-12.tar.gz’ saved [401886728/401886728]\r\n", "\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "cased_L-12_H-768_A-12/\r\n", "cased_L-12_H-768_A-12/vocab.txt\r\n", "cased_L-12_H-768_A-12/bert_model.ckpt.index\r\n", "cased_L-12_H-768_A-12/bert_model.ckpt.data-00000-of-00001\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "cased_L-12_H-768_A-12/params.yaml\r\n", "cased_L-12_H-768_A-12/bert_config.json\r\n" ] } ], "source": [ "# @title Download Checkpoint of the Selected Model { display-mode: \"form\", run: \"auto\" }\n", "model_display_name = 'BERT-base cased English' # @param ['BERT-base uncased English','BERT-base cased English','BERT-large uncased English', 'BERT-large cased English', 'BERT-large, Uncased (Whole Word Masking)', 'BERT-large, Cased (Whole Word Masking)', 'BERT-base MultiLingual','BERT-base Chinese']\n", "\n", "if model_display_name == 'BERT-base uncased English':\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/uncased_L-12_H-768_A-12.tar.gz\"\n", " !tar -xvf \"uncased_L-12_H-768_A-12.tar.gz\"\n", "elif model_display_name == 'BERT-base cased English':\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/cased_L-12_H-768_A-12.tar.gz\"\n", " !tar -xvf \"cased_L-12_H-768_A-12.tar.gz\"\n", "elif model_display_name == \"BERT-large uncased English\":\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/uncased_L-24_H-1024_A-16.tar.gz\"\n", " !tar -xvf \"uncased_L-24_H-1024_A-16.tar.gz\"\n", "elif model_display_name == \"BERT-large cased English\":\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/cased_L-24_H-1024_A-16.tar.gz\"\n", " !tar -xvf \"cased_L-24_H-1024_A-16.tar.gz\"\n", "elif model_display_name == \"BERT-large, Uncased (Whole Word Masking)\":\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/wwm_uncased_L-24_H-1024_A-16.tar.gz\"\n", " !tar -xvf \"wwm_uncased_L-24_H-1024_A-16.tar.gz\"\n", "elif model_display_name == \"BERT-large, Cased (Whole Word Masking)\":\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/wwm_cased_L-24_H-1024_A-16.tar.gz\"\n", " !tar -xvf \"wwm_cased_L-24_H-1024_A-16.tar.gz\"\n", "elif model_display_name == \"BERT-base MultiLingual\":\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/multi_cased_L-12_H-768_A-12.tar.gz\"\n", " !tar -xvf \"multi_cased_L-12_H-768_A-12.tar.gz\"\n", "elif model_display_name == \"BERT-base Chinese\":\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/chinese_L-12_H-768_A-12.tar.gz\"\n", " !tar -xvf \"chinese_L-12_H-768_A-12.tar.gz\"" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:23.601469Z", "iopub.status.busy": "2023-10-17T12:27:23.600918Z", "iopub.status.idle": "2023-10-17T12:27:23.609425Z", "shell.execute_reply": "2023-10-17T12:27:23.608830Z" }, "id": "jzxyziRuaC95" }, "outputs": [ { "data": { "text/plain": [ "'cased_L-12_H-768_A-12'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Lookup table of the directory name corresponding to each model checkpoint\n", "folder_bert_dict = {\n", " 'BERT-base uncased English': 'uncased_L-12_H-768_A-12',\n", " 'BERT-base cased English': 'cased_L-12_H-768_A-12',\n", " 'BERT-large uncased English': 'uncased_L-24_H-1024_A-16',\n", " 'BERT-large cased English': 'cased_L-24_H-1024_A-16',\n", " 'BERT-large, Uncased (Whole Word Masking)': 'wwm_uncased_L-24_H-1024_A-16',\n", " 'BERT-large, Cased (Whole Word Masking)': 'wwm_cased_L-24_H-1024_A-16',\n", " 'BERT-base MultiLingual': 'multi_cased_L-12_H-768_A-1',\n", " 'BERT-base Chinese': 'chinese_L-12_H-768_A-12'\n", "}\n", "\n", "folder_bert = folder_bert_dict.get(model_display_name)\n", "folder_bert" ] }, { "cell_type": "markdown", "metadata": { "id": "q1WrYswpZPlc" }, "source": [ "### Construct BERT Model Using the New `params.yaml`\n", "\n", "params.yaml can be used for training with the bundled trainer in addition to constructing the BERT encoder here." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:23.612467Z", "iopub.status.busy": "2023-10-17T12:27:23.612154Z", "iopub.status.idle": "2023-10-17T12:27:23.619764Z", "shell.execute_reply": "2023-10-17T12:27:23.619198Z" }, "id": "quu1s8Hi2szo" }, "outputs": [ { "data": { "text/plain": [ "{'task': {'model': {'encoder': {'bert': {'attention_dropout_rate': 0.1,\n", " 'dropout_rate': 0.1,\n", " 'hidden_activation': 'gelu',\n", " 'hidden_size': 768,\n", " 'initializer_range': 0.02,\n", " 'intermediate_size': 3072,\n", " 'max_position_embeddings': 512,\n", " 'num_attention_heads': 12,\n", " 'num_layers': 12,\n", " 'type_vocab_size': 2,\n", " 'vocab_size': 28996},\n", " 'type': 'bert'}}}}" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "config_file = os.path.join(folder_bert, \"params.yaml\")\n", "config_dict = yaml.safe_load(tf.io.gfile.GFile(config_file).read())\n", "config_dict" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:23.622726Z", "iopub.status.busy": "2023-10-17T12:27:23.622491Z", "iopub.status.idle": "2023-10-17T12:27:23.636255Z", "shell.execute_reply": "2023-10-17T12:27:23.635705Z" }, "id": "3t8o0iG9v8ac" }, "outputs": [ { "data": { "text/plain": [ "{'vocab_size': 28996,\n", " 'hidden_size': 768,\n", " 'num_layers': 12,\n", " 'num_attention_heads': 12,\n", " 'hidden_activation': 'gelu',\n", " 'intermediate_size': 3072,\n", " 'dropout_rate': 0.1,\n", " 'attention_dropout_rate': 0.1,\n", " 'max_position_embeddings': 512,\n", " 'type_vocab_size': 2,\n", " 'initializer_range': 0.02,\n", " 'embedding_size': None,\n", " 'output_range': None,\n", " 'return_all_encoder_outputs': False,\n", " 'return_attention_scores': False,\n", " 'norm_first': False}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Method 1: pass encoder config dict into EncoderConfig\n", "encoder_config = tfm.nlp.encoders.EncoderConfig(config_dict[\"task\"][\"model\"][\"encoder\"])\n", "encoder_config.get().as_dict()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:23.639341Z", "iopub.status.busy": "2023-10-17T12:27:23.638854Z", "iopub.status.idle": "2023-10-17T12:27:23.652252Z", "shell.execute_reply": "2023-10-17T12:27:23.651682Z" }, "id": "2I5PetB6wPvb" }, "outputs": [ { "data": { "text/plain": [ "{'vocab_size': 28996,\n", " 'hidden_size': 768,\n", " 'num_layers': 12,\n", " 'num_attention_heads': 12,\n", " 'hidden_activation': 'gelu',\n", " 'intermediate_size': 3072,\n", " 'dropout_rate': 0.1,\n", " 'attention_dropout_rate': 0.1,\n", " 'max_position_embeddings': 512,\n", " 'type_vocab_size': 2,\n", " 'initializer_range': 0.02,\n", " 'embedding_size': None,\n", " 'output_range': None,\n", " 'return_all_encoder_outputs': False,\n", " 'return_attention_scores': False,\n", " 'norm_first': False}" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Method 2: use override_params_dict function to override default Encoder params\n", "encoder_config = tfm.nlp.encoders.EncoderConfig()\n", "tfm.hyperparams.override_params_dict(encoder_config, config_dict[\"task\"][\"model\"][\"encoder\"], is_strict=True)\n", "encoder_config.get().as_dict()" ] }, { "cell_type": "markdown", "metadata": { "id": "5yHiG_9oS3Uw" }, "source": [ "### Construct BERT Model Using the Old `bert_config.json`" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:23.655525Z", "iopub.status.busy": "2023-10-17T12:27:23.655104Z", "iopub.status.idle": "2023-10-17T12:27:23.659968Z", "shell.execute_reply": "2023-10-17T12:27:23.659407Z" }, "id": "WEyaqLcW3nne" }, "outputs": [ { "data": { "text/plain": [ "{'hidden_size': 768,\n", " 'initializer_range': 0.02,\n", " 'intermediate_size': 3072,\n", " 'max_position_embeddings': 512,\n", " 'num_attention_heads': 12,\n", " 'num_layers': 12,\n", " 'type_vocab_size': 2,\n", " 'vocab_size': 28996,\n", " 'hidden_activation': 'gelu',\n", " 'dropout_rate': 0.1,\n", " 'attention_dropout_rate': 0.1}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bert_config_file = os.path.join(folder_bert, \"bert_config.json\")\n", "config_dict = json.loads(tf.io.gfile.GFile(bert_config_file).read())\n", "config_dict" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:23.663102Z", "iopub.status.busy": "2023-10-17T12:27:23.662622Z", "iopub.status.idle": "2023-10-17T12:27:23.675255Z", "shell.execute_reply": "2023-10-17T12:27:23.674666Z" }, "id": "xSIcaW9tdrl4" }, "outputs": [ { "data": { "text/plain": [ "{'vocab_size': 28996,\n", " 'hidden_size': 768,\n", " 'num_layers': 12,\n", " 'num_attention_heads': 12,\n", " 'hidden_activation': 'gelu',\n", " 'intermediate_size': 3072,\n", " 'dropout_rate': 0.1,\n", " 'attention_dropout_rate': 0.1,\n", " 'max_position_embeddings': 512,\n", " 'type_vocab_size': 2,\n", " 'initializer_range': 0.02,\n", " 'embedding_size': None,\n", " 'output_range': None,\n", " 'return_all_encoder_outputs': False,\n", " 'return_attention_scores': False,\n", " 'norm_first': False}" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "encoder_config = tfm.nlp.encoders.EncoderConfig({\n", " 'type':'bert',\n", " 'bert': config_dict\n", "})\n", "\n", "encoder_config.get().as_dict()" ] }, { "cell_type": "markdown", "metadata": { "id": "yZznAP--TDLe" }, "source": [ "### Construct a classifier with `encoder_config`\n", "\n", "Here, we construct a new BERT Classifier with 2 classes and plot its model architecture. A BERT Classifier consists of a BERT encoder using the selected encoder config, a Dropout layer and a MLP classification head." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:23.678489Z", "iopub.status.busy": "2023-10-17T12:27:23.677896Z", "iopub.status.idle": "2023-10-17T12:27:27.415328Z", "shell.execute_reply": "2023-10-17T12:27:27.414369Z" }, "id": "Ny962I8nqs4n" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-10-17 12:27:24.243086: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2211] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.\n", "Skipping registering GPU devices...\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4EAAAFgCAYAAAAFLXzCAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzde3hU1b3/8c/kNrmSAEJIAkoEgcqBaPGGQhGjQCQxkEMCCCqtt9a2ihRrW23lVI+2SkV7pFXoxdpaSYIChpsiYn9ya5EiWJCAIOWSBJIAISEkhGT9/vBkDsPkMhMymczs9+t58vBkzZq9v3vvNZv5ZPasbTPGGAEAAAAArCA/yNcVAAAAAAA6DiEQAAAAACyEEAgAAAAAFkIIBAAAAAALCbmwYdOmTXrxxRd9UQsANGvWrFkaPny4V5bNeQ+A5N3zTHZ2tleWCwCtyc/Pd2lz+STw0KFDWrx4cYcUBHhi8+bN2rx5s6/LgA8sXrxYhw4d8tryOe+hEecZ6/L2eWbx4sU6fPiw15YPtITxZ02HDx9u9v2NyyeBjZpKjIAvNf4VlbFpPTabrUPWw9gC5xnr6ojzzKOPPqqcnByvrwe4kM1mY/xZUF5eniZPntzkY3wnEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQFo0aJFstlsstlsCg8PD5h1tUV0dLSjvsafuXPn+rqsNgmkbQHaw9y5cx2vhd69e/usjkB6bQbStgCdXWd+DxVI54JA2pb21C4hsKqqSldccYXS09PbY3G4SFOmTJExRqmpqW16vifH82LX5W1VVVXatm2bJCkzM1PGGM2ePdvHVbVNIG1LIOC853uzZ8+WMUYpKSk+rSOQXpuBtC3+jnOMfwqU91CBdC4IpG1pT+0SAo0xamhoUENDQ3sszquio6M1YsQIX5fRqfnT8Qw0jE//4U+vE8YVzsd48A9WO8cEyrj0p+MWSAJl/HSkkPZYSExMjPbt29cei0InwPEEWsfrBIA3cY7xTxw3+Au+EwgAAAAAFnLRIXDp0qVOX7Ssqalpsv3AgQOaPHmy4uLi1L17d6Wnpzv9peTCL/hv2bJFqampiomJUWRkpEaPHq0NGzY4+j/zzDOO/ud//Lt69WpH+yWXXOKy/NOnT2vDhg2OPiEh7n8YOmHCBKdtOn+9a9eulc1mU0FBgaNt5syZTv3PnTsnSSovL9esWbPUr18/hYWFqWvXrkpLS9O6deua3a+FhYXKyclR9+7dHW1lZWWSpN27d2vChAmKjY1VVFSURo4cqfXr17u9Xedr7ng28mRdtbW1+tnPfqZBgwYpMjJS3bp1U0ZGht59913V19e3qb72Eojj80Lnzp1Tbm6ubrvtNvXq1UsREREaMmSIXn75ZcdlKidPnnT5svQzzzzjeP757ZMmTXIsu7S0VA8//LD69u2rsLAw9ejRQ1lZWfr000+b3cctjWF/Y6Xz3oXb9O9//1uTJ09WTEyMunfvrrvuuksnTpzQgQMHlJGRoZiYGCUkJOj+++9XZWWl07LcGZONLub88Ze//MVlXJeUlLi9ze0lEMfDhTjPeIeVzjEtLcPTsePp9jZyZ6y5wwrvoQJxDF7IMuc1c4Hc3FzTRHOrMjMzjSRz5syZJtszMzPNxo0bTVVVlVmzZo2JiIgw1157rctyUlJSTFRUlBk+fLij/5YtW8zQoUNNWFiY+eijj5z6R0VFmZtuusllOcOGDTPdu3d3aW+uv7vmz59vJJk333zTqX3GjBlGkpk8ebJT+5IlS0xqaqrj9+LiYpOcnGzi4+NNQUGBqaioMIWFhSYrK8vYbDazcOFCp+c37r9Ro0aZdevWmdOnT5vNmzeb4OBgU1paavbu3Wvi4uJMUlKSef/9901lZaXZsWOHGTNmjOnbt6+x2+1t2s6mjqen67rvvvtMbGysef/99011dbUpKSkxs2fPNpLMunXrPK5p0qRJZtKkSR4/b9u2bY4x2Nx2+sv4bGlbLlRQUGAkmWeffdYcP37clJaWml//+tcmKCjIzJ4926nv2LFjTVBQkPniiy9cljN8+HCn8V5UVGQuu+wyEx8fb1asWGEqKyvNv/71LzNq1CgTHh5uNm7c6PT81sawOySZ3Nxct/q2Bec997c1KyvLfPLJJ6aqqsq88cYbRpJJS0szmZmZZtu2baaystK8+uqrRpJ59NFHnZbhyZj05PyRkpJikpKSHL+fO3fOzJo1y9x2223m+PHjHm0n5xnOM97SluVb6RzT0jI8GTvGeLa9no41d3TG91BtGX+c1/z/vNbC+5u8DguBBQUFTu2TJk0yklw2IiUlxUgy27Ztc2rfsWOHkWRSUlKc2jv6RFVeXm7CwsLMuHHjHG3V1dWma9eupn///iYiIsKcOnXK8djEiRPNn/70J8fvjWHxrbfeclpuTU2NSUxMNBEREaakpMTR3rj/Vq5c2WQ92dnZRpJZvHixU/uRI0eM3W5v1xDo6bqSk5PNjTfe6LLsAQMGdLoQ6C/j09OT2M033+zSPn36dBMaGmoqKiocbe+9956RZB566CGnvuvXrzdJSUnm7NmzjrZ77rmnyT+EFBcXG7vdboYNG+bU3toYdoe/hkB/GVfuaNymFStWOLUPHjzYSDJ/+9vfnNqTk5PNwIEDndo8GZOenD/OD4EnTpwwY8eONY888og5d+6cR9toDOeZ1rblQpxnvLt8K51jWlqGJ2PHGM+219Ox5o7O+B7KWyHQX8agVc9rLYXADvtO4LXXXuv0e58+fSRJRUVFLn2joqJ01VVXObUNGTJEiYmJ2r59u4qLi71XaCu6deum22+/XWvWrHFcXrRs2TJdf/31+u53v6szZ87onXfekSQdP35cH330kbKyshzPX7JkiSRp/PjxTsu12+1KTU3VmTNn9N5777ms97rrrmuyntWrV0uSxo4d69SemJioAQMGtHErm+bpusaNG6eNGzfqgQce0ObNmx2XLxQWFurmm29u19ouVqCMz/Olp6c7XWLcKCUlRXV1ddq5c6ejbcyYMRoyZIhef/11lZeXO9pfeOEFff/731doaKijbenSpQoKCnKZ/rpXr14aPHiwtm7dqsOHD7ust7kxHMgCcVxdc801Tr8nJiY22Z6UlOSynZ6MybacPwoLC3X99dcrKChIL730koKDgz3ePm8KxPHAeca3AnFMNcWTsdPI3e1t61jzVKC+hwrEMWiV81qHhcDY2Fin38PCwiSpySl04+LimlxGz549JUnHjh1r5+o8c/fdd6u+vl5//etfJUl//vOfdffdd2vq1KkKDg7Wm2++KUl66623lJ6erujoaElfXd9dUVGh8PBwxcTEuCw3Pj5ekpr87kpUVJRLW21trSorKxUeHu5Yx/ka91d7aMu65s+frzfeeEP79+9XamqqunTponHjxjmCcGcSSOOzUUVFhX72s59pyJAh6tq1q+M68scee0ySVF1d7dR/5syZqq6u1m9+8xtJ0p49e/Thhx/qgQcecPRpHMMNDQ2KjY11uR7+n//8pyRp7969LvU0NYYDXSCOqy5dujj9HhQUpODgYEVGRjq1BwcHu2ynJ2PS0/PHiRMnNGHCBPXu3VurVq3SX/7yl/bY3HYViOOB84xvBeKYao47Y+d87mzvxYw1TwTye6hAHINWOa91ytlBy8vLZYxxaW8cHOe/WIKCgnT27FmXvidPnmxy2Tab7aLrGz9+vLp166Y///nPKi0t1ebNmzVhwgTFx8drzJgx+vDDD1VcXKw//elPuvvuux3Ps9vtio2NVU1NjcuECZJ09OhRSV/9RcAddrtdMTExqqmpUVVVlcvjx48fb+MWts+6bDab7rrrLn3wwQc6efKkli5dKmOMsrKy9OKLL7ZbbR2ts4/PRhkZGXr66ad1//33a8+ePWpoaJAxRvPmzZMkl22YNm2a4uPj9corr6i2tla/+tWvdM8996hr166OPna7XXFxcQoJCVFdXZ2MMU3+jB49ut22wyr8ZVxdDE/GpKfnj5CQEH3wwQdatmyZhgwZovvvv19btmzpsG1rb/4yHjjP+I/OPqZaW4Y7Y+d87mxvR4013kN9pbOPwUZWOa91yhBYU1Pj8p/3Z599pqKiIqWkpCghIcHRnpCQoCNHjjj1LSkp0cGDB5tcdmRkpNOgGjhwoBYsWOBRfWFhYZo8ebI+/fRTPfHEE8rMzFRERIQk6a677lJ9fb2eeuopFRcX65ZbbnF67sSJEyVJK1ascGqvra3V2rVrFRER4XKpQEvS0tIk/d9lBo3KyspUWFjo0Xa197ri4uK0e/duSVJoaKhuu+02x4xHF26/P+ns4zMkJEQ7d+7Uhg0b1KtXLz388MPq0aOH4wR55syZJp9nt9v10EMP6dixY/rVr36lN998U4888ohLv6ysLJ07d67JGdZ++ctf6tJLL3XMhAv3dfZxdbHq6+s9GpOenj9iYmKUlJSk6Ohovfvuu4qOjtaECRM6zeVFnurs44HzjP/p7GOqtWW4O3Yaubu9HTXWeA/V+ceg1c5rnTIExsbG6ic/+Yk2bdqk06dP65NPPtH06dMVFhaml19+2anvmDFjVFRUpFdeeUVVVVXat2+fHnnkkWYvhfz617+uPXv26NChQ9q0aZP279+vkSNHelzjXXfdJUlauHCh06d9EyZMUExMjBYuXKhp06YpKMh5Fz/33HNKTk7WzJkztXz5clVWVmrPnj268847VVxcrJdfftlxWag7nn32WXXr1k0zZ87UmjVrVFVVpV27dmn69OlNXnJwMdqyrm9/+9vasWOHamtrdezYMT3//PMyxriEY3/iD+MzODhYN998s0pKSvTCCy+orKxMZ86c0bp16/Tqq682+7yHHnpIERERevLJJ3Xrrbeqf//+Ln2ee+459evXT9/61re0atUqVVRU6Pjx43rttdf085//XHPnzr2oqZmtyh/G1cVoy5hs6/mjb9++Wrx4sUpLS5WVlaXa2lpvbJJX+cN44DzjXzr7mHJnGe6MHU+3t6PGGu+hOv8YlCx2XvNgFpkmLVmyxEhy+pk2bZrZtGmTS/sTTzxhzFefoTr9jB8/3rG8xlnedu3aZcaOHWtiYmJMRESEGTVqlFm/fr3L+k+ePGnuu+8+k5CQYCIiIsyIESPMli1bzLBhwxzLf/zxxx39d+/ebUaOHGmioqJMnz59zPz5893e1gtdccUV5tJLLzUNDQ1O7Y0zgO7cubPJ55WVlZmZM2ea5ORkExoaamJjY83YsWPN2rVrHX2a2n/NHZfCwkIzYcIE06VLF8fUvMuXLzepqamO5917771ubVNzx7Mt6/r000/Ngw8+aL72ta+ZyMhI061bN3PDDTeYhQsXuuwzd7Rl1r6oqCiX7XnhhRf8cnw2tS3N/Xz++eemtLTUPPjgg6ZPnz4mNDTUxMfHmxkzZpgf/ehHjn5NzXp2//33Nznb4/nKy8vNrFmzzOWXX25CQ0NNjx49zJgxY8yaNWscfTwZw61RJ5sd1Ernvea2acuWLS7tzz33nPn4449d2p966iljjPFoTLpz/njrrbdc1jVv3rwmaz7/PNYSzjOcZ7zFk+Vb6Rzj6TLcGTuebq87Y80dnfk9lKfjm/NaYJzXOuQWEe3lwvs9AY3aOnV7e7LC+PzDH/7QpimxvamzhcD2ZoVx5S84z3QMK55nvL38lgTSmHJn7ATS9rYXX44/Y6xxTDrjea1T3CICgH949dVXNWvWLF+XASCAcZ5BWzF20Fn529gkBAIW97vf/U4TJ05UVVWVXn31VZ04cUI5OTm+LgtAAOE8g7Zi7KCz8vex2WlC4Ny5c2Wz2bR9+3YdOXJENptNTz75ZIet/8L7dTT1M2fOnA6rxxussI3e4uvx6W1Lly5V165d9dvf/laLFi1iwoUO4utxxTmhc/H1ePA2zjMdz9djqr3OMe6OHW9uL+fLtvH1GPQ2fz6v2YxxvtlFXl6eJk+e3OR9PABfys7OliTl5+f7uBJ0NJvNptzcXK/9hY3zHhpxnrEub59nvL18oCWMP2tq4f1Nfqf5JBAAAAAA4H2EQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWEtLcA9nZ2R1ZB9CqzZs3SwqcsVlfX689e/YoKSlJXbp08XU5UOCMLbRdoJ1n0LnMmzdP+fn5vi6j06moqFBRUZEGDRokm83m63ICFuPPeg4fPtzsYy4hsE+fPpo0aZJXCwLa4oYbbvB1Ce3q9OnT2rdvn3bu3Kno6GglJSUpMTFR3bp14z/BC0yaNEl9+vTx2vI57+Hzzz+XFHjnGbjP2+cZzjH/xxij48ePq6ioSEeOHFFVVZUiIiLUp08fRUdH+7q8gMT4s6bevXs3e+xtxhjTwfUA+F8NDQ3atm2bCgoKlJeXp88//1yXXHKJ0tLSlJ2drTFjxshut/u6TCDg5eTkSJLy8vJ8XAkQmOrr67Vp0ybl5+fr7bff1pEjR9S3b1/dcccdys7O1o033qigIL6lBHSQfEIg0Ins379fBQUFys/P18aNGxUREaFbbrlF2dnZmjBhApeNAl5CCATaX01NjdavX6+CggLl5ubq6NGjuvLKK5WRkaH09HTddNNNXPkC+AYhEOisDh48qNWrV6ugoEDvvfeegoODNWLECKWnpysnJ0cJCQm+LhEIGIRAoH1UV1dr7dq1ys/P17Jly3Tq1CldeeWVys7O1pQpUzRo0CBflwiAEAj4h+PHj2v58uVavny5Vq1aperqag0fPlwZGRmaOHGiBgwY4OsSAb9GCATa7vz/o1auXKmamhrdcMMNys7O1qRJk5SUlOTrEgE4IwQC/ubMmTP64IMPlJ+fr4KCAp08eZLLa4CLRAgEPFNaWqpVq1YpPz/f5WqVKVOmKD4+3tclAmgeIRDwZy190T4jI0M333yzQkKavRMMgP9FCARa9+WXX+rdd9/le+uA/yMEAoFk586dys/PZ6ZRwEOEQKBpO3fu1PLly1VQUKANGzaoW7duGj9+vDIyMnT77bcrKirK1yUC8BwhEAhULc00mpmZqdjYWF+XCHQahEDgK+ffumjRokUqLCxUjx49NG7cOGVnZ2vcuHEKDQ31dZkALg4hELCCQ4cOadWqVY6ZRoOCgjRy5EhmGgX+FyEQVtbaPfz4rjkQcAiBgNU0NdPo1VdfrfT0dE2dOlUDBw70dYlAhyMEwmpqamq0Zs0aLV++XEuXLtWxY8ecJhkbMWKEr0sE4D2EQMDKGmcabe5NAH/9hVUQAmEFLd3Djz8CApZCCATwlfMvB3rnnXd0+PBhXXbZZcrMzGSmUQQ8QiACVePVH/n5+VqzZo3OnTvHPfwAEAIBNK1xptH8/Hzt2rVL3bt31+23385MowhIhEAEkgu/B849/ABcgBAIoHXMNIpARwiEv+MefgA8QAgE4JkL/8JsjNH111+v7OxsZWdnKzEx0dclAh4jBMIfNV6xsXz5cm3dupV7+AFwFyEQQNudOHFCH3zwgQoKCrRkyRJmGoXfIgTCH5x/D7+33npLe/bsUe/evXX77bcrPT2de/gBcBchEED7YKZR+DNCIDqr8yftWrx4sYqKipScnKyMjAzu4QegrQiBANofM43C3xAC0Zk0dw+/7OxsZWRkaNiwYb4uEYB/IwQC8L7mZhrleyvoLAiB8LXz7+G3dOlSnT59msvrAXgLIRBAx2KmUXRGhED4Qnl5uVasWME9/AB0NEIgAN8pLS3VqlWrlJ+fr/fff1/19fWON0DMNIqORAhER2nqHn633nqrMjIyNGHCBPXs2dPXJQIIfIRAAJ3D+TONXngp1JQpUzRo0CBfl4gARgiEN3EPPwCdDCEQQOdz/kyjy5Yt09GjR5lpFF5FCER7a+4eftnZ2RozZozsdruvSwRgXYRAAJ3b+TONLlmyRIcOHdJll12msWPHcl8stBtCIC5WU/fw69Onj9LS0jhXAehsCIEA/MuFf11nplG0B0Ig2oJ7+AHwU4RAAP7rwplGw8PDlZqaquzsbN1xxx2Ki4vzdYnwE4RAuIt7+AEIAIRAAIGBmUZxMQiBaElL9/C78847NWDAAF+XCACeIAQCCDzMNApPEQJxoZbu4ccflgD4OUIggMB2/qVbjTONXn755UpPT+c7O3AgBEKSDh48qNWrV6ugoECrV69WSEgI9/ADEIgIgQCso3ESh+XLl+vtt9/WF198oUsvvVTjxo1j9j6LIwRa14XfLT7/Hn4TJ05UTEyMr0sEgPZGCARgXc3dxysjI0NpaWmKjo72dYnoIIRAa2lulmHu4QfAIgiBACBJX375pd59913l5+dr06ZNstvtzDRqIYTAwHb+Pfz++te/au/evU738EtLS1NISIivywSAjkIIBIALtTTT6KRJk5SUlOTrEtHOCIGB5/x7+OXn56u4uJjvAwPAVwiBANCSkydPas2aNY6ZRisrKx33BJs8ebK+9rWv+bpEtANCYGA4c+aMPvjgA+Xn56ugoEAnT57kHn4A4IoQCADuqqmp0fr161VQUKC8vDyVlJTwyYIfevPNN/X73/9eDQ0NjrbCwkJJ0sCBAx1tQUFBuvfeezVt2rQOrxHuu/APNdzDDwBaRQgEgLY4f6bRd955R3v37mWmUT+xfft2XXXVVW71/fTTT5WSkuLliuCp8+/hd+El29zDDwBaRQgEgPbATKP+ZdCgQY5P/5rTv39/7d27t4MqQmsOHjyoJUuWaPny5froo4+c7uE3ceJE9ejRw9clAoC/IAQCQHtrbqZRbjjdefz3f/+3/uu//kt1dXVNPh4aGqo5c+boJz/5SQdXhvNdeA+/yMhIjR49mnv4AcDFIQQCgDeVlZVp5cqV7T7TaH19vUpKSpiptI3279+v/v37q6X/Avfu3av+/ft3YFWB4cCBA0pKSmrz5dCNn6rn5+dr165dTvfwGzt2rMLCwtq5YgCwHEIgAHSU06dP68MPP1R+fr6WLVumU6dOtXmm0Y8++kjZ2dn685//rHHjxnmx6sA1bNgwbdu2zSUI2mw2ff3rX9cnn3zio8r817JlyzRjxgzl5eXptttuc+s5DQ0N2rhxY7Pfr+UefgDQ7vKDfF0BAFhFVFSUMjIy9MYbb+jo0aNas2aNbr31Vr322mu68sor1a9fPz3yyCNav359i59QSdLSpUtVXl6utLQ0/fjHP9a5c+c6aCsCx913363g4GCX9uDgYN19990+qMh/1dXV6Qc/+IEmTpyoiooKvfPOOy32r6+v1/r16/XII4+od+/eGjlypPLz85WWlqaPP/5YBw4c0GuvvaaMjAwCIAB4AZ8EAoCPNfVJSJ8+fZSWltbsTKO9e/fWkSNHJH0VWlJSUvT222+rb9++PtgC/3Ts2DElJCQ43SpC+urWEEeOHFGvXr18VJl/OXTokLKzs/XJJ5+ovr5ektStWzeVlpYqKOj//tbc0j38cnJydOWVV/pqEwDAargcFAA6m6ZmGk1NTVV6erqysrJUWFioa665xuk5oaGhioyM1F//+lfdfvvtPqrc/4wePVoff/yxI7wEBwfrG9/4hj788EMfV+YfVqxYoWnTpqm6utplkp3169dr8ODBjnv4LVmyRNXV1Ro+fLgyMjKUlZWlK664wkeVA4ClEQIBoDP74osvtGTJEi1dulSbN29WeHi4UlJStHXrVp09e9apb1BQkBoaGvT9739fv/rVr7hPoRv+8Ic/6IEHHnAKgQsXLtQ3v/lNH1fWuZ07d05PP/20nn76adlsNpdPU8PCwpScnKz9+/fLZrPplltu0cSJE5kdFwA6B0IgAPiLkpISLVu2THPmzFFJSUmz/RovD128eLGSk5M7sEL/c+rUKV1yySWOT7FCQ0N17NgxxcXF+biyzuvIkSOaNGmStmzZ4gjPTYmNjdX8+fM1fvx49icAdC5MDAMA/qJXr15KTU1tMQBKX0268dlnn+nqq6/WihUrOqg6/9SlSxfH7JMhISG6/fbbCSwt+PDDDx2fRLcUACWpoqJCgwcPZn8CQCdECAQAP/LOO++4NVtiXV2dKisrlZ6erocffrjZm6JDmj59uurr61VfX69p06b5upxOqb6+XnPmzNGtt96qEydOuDWewsLCtGTJkg6oDgDgKeZdBuD3Dh8+rI0bN/q6jA6xYMGCVj+BadT4Pa3/+Z//0cqVK/WDH/xA3bt392Z5fqmurk5hYWEyxqi2tlZ5eXm+LqlTOXHihF566SXt3r1bklq9fUmjs2fPauHChRo8eLA3y+tUbrzxRvXu3dvXZQBAq/hOIAC/l5eXp8mTJ/u6DAAWl5ubq5ycHF+XAQCtyeeTQAABI9D/plVUVKTPPvtMwcHBioqKUlhYmMLDwxUREaGwsDBFRUUpODhYXbp08XWpfmf16tWy2Wz63e9+J0nKz8/3cUX+48SJE5Kk6upq1dbWqqGhQRUVFZKk06dPa8CAAZa456LNZvN1CQDgNkIgAPiJxMREJSYm+rqMgHTrrbdKkiMEwn1du3Z1+hcA0PkRAgEAlufOZDsAAAQKZgcFAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEALGfu3Lmy2Wyy2Wzq3bu3r8uxtEWLFjmORXh4uK/LaZPo6GjHNlz4ExkZqZSUFL344ouqr6/vtHVe+PPJJ5/4tFZ3BMLYAQBfIQQCsJzZs2fLGKOUlBRfl2J5U6ZMkTFGqampvi6lzaqqqrRt2zZJUmZmpowxMsbo1KlTWr16tSTpBz/4gR577DFfltlsnRf+xMbG+rROdwXC2AEAXyEEAoCPREdHa8SIEb4uA14SExOjb3zjG3r11VclSa+99prq6uq8uk7GFFVkyxgAACAASURBVADAHSG+LgAAgEA2cOBASVJ1dbUqKip0ySWX+Liilp08edLXJQAAvIxPAgEA8KLCwkJJUo8ePTp1ABwxYoRef/11X5cBAOgAhEAAlrd7926NHz9esbGxioyM1OjRo7VhwwaXfqWlpXr44YfVt29fhYWFqUePHsrKytKnn37q6LN06VKnCTYKCwuVk5Oj7t27O9p+9KMfyWaz6fTp09qwYYOjPSTE84sz2lLTgQMHNHnyZMXFxal79+5KT0/Xvn37XJZdXl6uWbNmqV+/frLb7erdu7duvfVWvf766zpz5kyT/cLCwtS1a1elpaVp3bp1Te7rCRMmKDY2VlFRURo5cqTWr1/frtvX1D4vKyvzeN9erKqqKn388cf69re/rcjISMdloefrjGPqfIwd34wdAPA6AwB+Ljc317TldJaSkmJiY2PN6NGjzfr1601lZaXZsmWLGTp0qAkLCzMfffSRo29RUZG57LLLTHx8vFmxYoWprKw0//rXv8yoUaNMeHi42bhxo9OyMzMzjSQzatQos27dOnP69GmzefNmExwcbEpLS40xxkRFRZmbbrqpzdvd1poyMzPNxo0bTVVVlVmzZo2JiIgw1157rVPf4uJik5ycbHr16mUKCgrMqVOnTElJiXn66aeNJDNv3jynfvHx8aagoMBUVFSYwsJCk5WVZWw2m1m4cKFjmXv37jVxcXEmKSnJvP/++6aystLs2LHDjBkzxvTt29fY7fZ22b6W9nlrJk2aZCZNmuT2MWi0bds2I6nJn4EDB5q3337b5Tm+GFMt1SnJ/PGPf2zyeYyd1kkyubm5bvUFAB/LIwQC8HsXEwIlmU2bNjm179ixw0gyKSkpjrZ77rnHSDJvvvmmU9/i4mJjt9vNsGHDnNob31SuXLmy2fVfbAhsa00FBQVO7ZMmTTKSnN7szpgxo9k3tePGjXO8kW/s99Zbbzn1qampMYmJiSYiIsKUlJQYY4zJzs42kszixYud+h45csTY7XaXN/Le2OetudgQmJmZ6Wirq6sz+/fvN0899ZSx2WwmKyvLnD171vG4L8ZUU3U2uummm1oNgYyd5hECAfiRPC4HBWBp4eHhuv76653ahgwZosTERG3fvl3FxcWSvrpsLCgoSOnp6U59e/XqpcGDB2vr1q06fPiwy/Kvu+46r9Xe1pquvfZap9/79OkjSSoqKnK0LVmyRJKUlpbm8vxVq1Zp5syZTv3Gjx/v1Mdutys1NVVnzpzRe++9J0mO2yWMHTvWqW9iYqIGDBjQbtvnzX3uiZCQECUnJ2vOnDm688479c477+jXv/6143F/3D7GDgAEBkIgAEtr/O7PhXr27ClJOnbsmGpra1VRUaGGhgbFxsa63Fj7n//8pyRp7969LsuJiorySt0XU9OF94ELCwuTJDU0NDgtOzw8XDExMa3W0Fy/+Ph4SVJJSYlqa2tVWVmp8PBwRUdHu/Rt3N/tsX3e2ucX4xvf+IYkae3atZI65/atX79eM2bMaLEPYwcAAgO3iABgaRUVFU22Hzt2TNJXbzDtdrvi4uJUVVWlM2fOXPRkG42aCp/u8lZNjcuOjY1VRUWFKisrm30z31q/o0ePSvrq0xe73a6YmBhVVlaqqqrK5c388ePHO2z7fMEYI+mr20RI3tu+ixlT7YGxAwD+gU8CAVhaVVWVtm/f7tT22WefqaioSCkpKUpISJAkZWVl6dy5c03OGvrLX/5Sl156qc6dO+fRuiMjI3X27FnH7wMHDtSCBQvcfr43amo0ceJESdLKlStdHrv66qv16KOPOvVbsWKFU5/a2lqtXbtWERERjkv4Gi8PbLy0r1FZWZnjNgrn8+b2dbSPP/5YkvPllJ1xTEnSNddco0WLFnn0nPMxdgDAD/j6W4kAcLEuZmKYqKgoM2LECLN582ZTVVXV7OygR48eNf369TOXX365WblypTl58qQpLy83r776qomMjHSZEKJxookzZ840u/5x48aZ2NhYc/DgQbNx40YTEhJidu3a5Xb97VXT448/biSZbdu2OdoaZ25MSEgwy5cvN6dOnTKHDh0y3/nOd0x8fLz597//7dSvcYbHU6dOOc3wuGDBAscyv/jiC9OtWzenGR537txpxo4da3r27OkyuYc39nlr2ntimC+//NIxMUxSUpIpKiry6va1NqZamhim0bBhw1wma2HstE5MDAPAfzA7KAD/52kIfOGFFxxT4iclJZl//OMfZvTo0SY6OtpERESYUaNGmfXr17s8r7y83MyaNctcfvnlJjQ01PTo0cOMGTPGrFmzxtFn06ZNTU6935Tdu3ebkSNHmqioKNOnTx8zf/58j7e9rTU98cQTxhjj0j5+/HjH88rKyszMmTNNcnKyCQ0NNQkJCWbKlClmz549TjVc2C82NtaMHTvWrF271qXewsJCM2HCBNOlSxfH7QWWL19uUlNTHTXce++9XtvnrWlLCIyKimpy/TabzcTExJiUlBTzwx/+0Bw9etTluR05ppqrs6mfxhDI2HEfIRCAH8mzGfO/X1QAAD+Vl5enyZMni9MZLlZ2drYkKT8/38eVwN/YbDbl5uYqJyfH16UAQGvy+U4gAAAAAFgIIRAAAAAALIQQCACdzIX3NWvqZ86cOb4uEwAA+ClungMAnQzfbQQAAN7EJ4EAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWEuLrAgCgveTl5fm6BPi5w4cPS2IsAQACGyEQQMCYPHmyr0tAgGAsAQACmc0YY3xdBAAAvpSTkyOJTwABAJaQz3cCAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCQnxdAAAAHenvf/+7tm/f7tS2f/9+SdKCBQuc2ocOHaobbrihw2oDAKAjEAIBAJZy7NgxPfjggwoODlZQ0FcXxBhjJEnf+973JEkNDQ2qr6/Xu+++67M6AQDwFptp/J8PAAALqKur0yWXXKJTp0612C8mJkZlZWUKCwvroMoAAOgQ+XwnEABgKaGhoZoyZUqL4S40NFRTp04lAAIAAhIhEABgOVOnTtXZs2ebfbyurk533nlnB1YEAEDH4XJQAIDlNDQ0KDExUUePHm3y8R49eqikpMTxnUEAAAIIl4MCAKwnKChI06dPb/Jyz7CwMN1zzz0EQABAwOJ/OACAJTV3SejZs2c1depUH1QEAEDH4HJQAIBl9e/fX/v27XNqu+yyy3TgwAHfFAQAgPdxOSgAwLqmT5+u0NBQx+9hYWH65je/6cOKAADwPj4JBABY1hdffKErrrjCqa2wsFADBgzwUUUAAHgdnwQCAKyrf//+Gjp0qGw2m2w2m4YOHUoABAAEPEIgAMDS7r77bgUHBys4OFh33323r8sBAMDruBwUAGBpRUVF6tOnj4wxOnjwoHr37u3rkgAA8Kb8EF9XAACBxmaz+boEtFGfPn18XQI8xN+yAcBzhEAA8IKZM2dq+PDhvi4Dbvrggw9ks9mUmprabJ9NmzbppZdeUm5ubgdWhuY0Hg8AgOcIgQDgBcOHD1dOTo6vy4CbGsNf9+7dW+z30ksvcVw7EUIgALQNIRAAYHmthT8AAAIJs4MCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAB0QosWLZLNZpPNZlN4eLivy/FbK1eu1IABAxQSEuKT9UdHRzuOY+NPUFCQunbtqpSUFD300EPaunWrT2oDAFgXIRAAOqEpU6bIGKPU1FRfl+KX9u3bpzvuuEM//vGPdfToUZ/VUVVVpW3btkmSMjMzZYxRXV2ddu/erZ///OfavXu3rrnmGn3zm99UdXW1z+oEAFgLIRAA0KlER0drxIgRF7WMn/70p7rxxhu1detWxcTEtFNl7SM4OFjx8fHKzMzUhx9+qB/+8Id6/fXXNXXqVBljfF1eh2mP4wwAaBvfXB8DAIAX/f73v1dERISvy3DLL37xC/3tb3/Tu+++q0WLFmnq1Km+LgkAEOD4JBAAEHD8JQBKks1m0/e+9z1J0m9+8xsfVwMAsAJCIAB0Art379aECRMUGxurqKgojRw5UuvXr3fpt3TpUqdJRgoLC5WTk6Pu3bs72srKyiRJ5eXlmjVrlvr166ewsDB17dpVaWlpWrdunWN5c+fOdTyvd+/e2rJli1JTUxUTE6PIyEiNHj1aGzZscKnDnWU/88wzjmWff9nf6tWrHe2XXHKJSy2nT5/Whg0bHH18NalLR2rcP5s3b1ZdXR3HGQDgXQYA0K4kmdzcXLf7792718TFxZmkpCTz/vvvm8rKSrNjxw4zZswY07dvX2O3212ek5mZaSSZUaNGmXXr1pnTp0+bzZs3m+DgYFNaWmqKi4tNcnKyiY+PNwUFBaaiosIUFhaarKwsY7PZzMKFC52Wl5KSYqKioszw4cPNxo0bTVVVldmyZYsZOnSoCQsLMx999JGjr6fLjoqKMjfddJPLNgwbNsx0797dpb25/m2VlJRkgoODL3o5ubm5pi3/bW7bts1IMpmZmc32OXPmjJFkJJmioiJHO8e5eW09HgAAk8fZEwDamachMDs720gyixcvdmo/cuSIsdvtLYbAlStXNrnMGTNmGEnmrbfecmqvqakxiYmJJiIiwpSUlDjaU1JSjCSzbds2p/47duwwkkxKSkqbl00IbD0EVldXtxgCOc6uCIEA0GZ5XA4KAD62evVqSdLYsWOd2hMTEzVgwIAWn3vdddc12b5kyRJJ0vjx453a7Xa7UlNTdebMGb333ntOj0VFRemqq65yahsyZIgSExO1fft2FRcXt3nZaFnjvg0NDXW6dLIRxxkA0J4IgQDgQ7W1taqsrFR4eLiio6NdHu/Zs2eLz4+KimpymRUVFQoPD2/y9gjx8fGSpJKSEqf2uLi4JtfRWMOxY8favGy0rPH7n8OHD1doaKjL4xxnAEB7IgQCgA/Z7XbFxMSopqZGVVVVLo8fP368TcuMjY1VTU2NKisrXR5vvHl6r169nNrLy8ubvE/dsWPHJH0VEtqy7KCgIJ09e9al78mTJ5us32azNbdpAamhoUHz58+XJH33u991+3kcZwBAWxECAcDH0tLSJP3fZaGNysrKVFhY2KZlTpw4UZK0YsUKp/ba2lqtXbtWERERLpef1tTUaMuWLU5tn332mYqKipSSkqKEhIQ2LTshIUFHjhxx6ltSUqKDBw82WXtkZKRTmBg4cKAWLFjQ6jb7qx//+Mf6xz/+oYkTJyo7O9uj53KcAQBtQQgEAB979tln1a1bN82cOVNr1qxRVVWVdu3apenTpzd5iag7nnvuOSUnJ2vmzJlavny5KisrtWfPHt15550qLi7Wyy+/7Likr1FsbKx+8pOfaNOmTTp9+rQ++eQTTZ8+XWFhYXr55ZfbvOwxY8aoqKhIr7zyiqqqqrRv3z498sgjzV7q+vWvf1179uzRoUOHtGnTJu3fv18jR45s037ojBoaGnTs2DEtW7ZMqampev755/Wtb31Lb775psefjnGcAQBt4uupaQAg0MjD2UGNMaawsNBMmDDBdOnSxURERJhrr73WLF++3KSmpjpmjbz33nvNpk2bHL+f/9OUsrIyM3PmTJOcnGxCQ0NNbGysGTt2rFm7dq1L35SUFJOUlGR27dplxo4da2JiYkxERIQZNWqUWb9+/UUt++TJk+a+++4zCQkJJiIiwowYMcJs2bLFDBs2zFH/448/7ui/e/duM3LkSBMVFWX69Olj5s+f79G+NMaYgoKCJveTJJdbG7irLbNRRkVFuazfZrOZ2NhYM2TIEPOd73zHbN261eV5HOfWMTsoALRZns2YJr4YAABoM5vNptzcXOXk5Pi6FLddddVVKisr0+HDh31dSqeVl5enyZMnN/l9On8RSMc5EI4HAPhIPpeDAgAAAICFEAIBAAAAwEIIgQBgYXPnzpXNZtP27dt15MgR2Ww2Pfnkk74uq0k2m63Vnzlz5vi6zE7Jn44zAMD7QnxdAADAd2bPnq3Zs2f7ugy38N2vtvOn4wwA8D4+CQQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQmzHG+LoIAAgkNpvN1yUAlsHbGADwWH6IrysAgECTm5vr6xLgoXnz5kmSHn30UR9XAgCA9/FJIADA8nJyciRJeXl5Pq4EAACvy+c7gQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEJCfF0AAAAdqbq6WrW1tU5tZ8+elSSdOHHCqd1utysyMrLDagMAoCMQAgEAlvLHP/5R3/ve95p8rFu3bk6/v/LKK/rud7/bEWUBANBhbMYY4+siAADoKKWlpUpISFB9fX2L/YKDg1VcXKwePXp0UGUAAHSIfL4TCACwlB49euiWW25RcHBws32Cg4OVmppKAAQABCRCIADAcqZPn66WLoQxxmj69OkdWBEAAB2Hy0EBAJZTWVmpHj16uEwQ0ygsLEylpaXq0qVLB1cGAIDXcTkoAMB6YmJilJ6ertDQUJfHQkJCdMcddxAAAQABixAIALCkadOm6dy5cy7t9fX1mjZtmg8qAgCgY3A5KADAks6ePatLLrlElZWVTu3R0dEqKyuT3W73UWUAAHgVl4MCAKwpLCxMkyZNUlhYmKMtNDRUOTk5BEAAQEAjBAIALOvOO+/U2bNnHb/X1dXpzjvv9GFFAAB4H5eDAgAsq6GhQfHx8SorK5Mkde/eXUePHm3xHoIAAPg5LgcFAFhXUFCQpk2bprCwMIWGhmr69OkEQABAwCMEAgAsberUqTp79iyXggIALCPE1wUAaF8vvviiNm3a5OsyAL8SGRkpSXrhhRd8XAngX4YPH65Zs2b5ugwAHuKTQCDAbNq0SZs3b/Z1GYBfueyyy3TZZZc1+djhw4e1ePHiDq4I6Pw2b97MHx0BP8UngUAAuuGGG5Sfn+/rMgC/sXPnTknS4MGDXR7Ly8vT5MmTeU0BF8jOzvZ1CQDaiBAIALC8psIfAACBistBAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAA4LBo0SLZbDbZbDaFh4f7upyAsGXLFs2YMUPJycmKiIhQt27d9B//8R/6z//8T/32t7/Vvn37JElz58517PvevXv7uGr3ff7555o8ebJ69eqlkJAQxzbExcX5ujSfiI6OduyDxp+5c+c69endu7dLnyeffLLDa+X1DlgXIRAA4DBlyhQZY5SamuryWFVVla644gqlp6d7Zd3eXn5Ha2ho0GOPPaYbb7xRPXv21KpVq3Ty5El9/vnnmjdvnk6dOqWHHnpI/fv317lz5zR79mwZY5SSkuLr0p20dFwOHDig4cOH6/PPP9c777yjU6dO6dSpU8rLy1NQkO/fYvhiTFVVVWnbtm2SpMzMTBljNHv2bKc+hw8f1pdffilJuummm2SM0TPPPNNhNTZq6fUOILD5/gwNAP8rOjpaI0aM8HUZaIYxRg0NDWpoaGjzMlo6xu2x/M7kpz/9qebOnavf/OY3ev755zVo0CDZ7XbFx8frtttu0+rVq5WWlubrMlvV0nFZsGCBKioqNH/+fN14442KjIxUTEyMsrOzdfz48Q6pz0pjCgDaS4ivCwAA+IeYmBjHpYv+uPyOtHv3bv3iF7/QsGHDdP/99zfZJzg4WD/96U+1atWqDq7OMy0dl71790qShg4d2pEluS2QxhQAtCdCIAAA7WzBggVqaGhQdnZ2i/2GDx8uY0wHVdX+6urqJEl2u93HlQAAPMHloABUW1urn/3sZxo0aJAiIyPVrVs3ZWRk6N1331V9fb1T39LSUj388MPq27evwsLC1KNHD2VlZenTTz919Fm6dKnThAcHDhzQ5MmTFRcXp+7duys9Pd3pr/ONE2KcPn1aGzZscDwvJCTE6+tuVF5erlmzZqlfv36y2+3q3bu3br31Vr3++us6c+aMRzW448JJQLZs2aLU1FTFxMQoMjJSo0eP1oYNG5rdrsLCQuXk5Kh79+6OtrKyMo9r3L17tyZMmKDY2FhFRUVp5MiRWr9+vUu/C9dfU1Pj0f5r7Rh7svywsDB17dpVaWlpWrduXbM1unvsveH//b//J6n9PiE7d+6ccnNzddttt6lXr16KiIjQkCFD9PLLL7tc6uju69mdfs0dl8b2ZcuWSZIiIiJcJjqx2WyaMWOGU23uvM7c3dZAHlPuvoY9GReS+693ABZgAASUSZMmmUmTJnn0nPvuu8/Exsaa999/31RXV5uSkhIze/ZsI8msW7fO0a+oqMhcdtllJj4+3qxYscJUVlaaf/3rX2bUqFEmPDzcbNy40Wm5mZmZRpLJzMw0GzduNFVVVWbNmjUmIiLCXHvttS51REVFmZtuuqnJGr257uLiYpOcnGx69eplCgoKzKlTp0xJSYl5+umnjSQzb968NtXgjpSUFBMVFWWGDx/uqHPLli1m6NChJiwszHz00UdNbteoUaPMunXrzOnTp83mzZtNcHCwKS0t9ajGvXv3mri4OJOUlGTef/99U1lZaXbs2GHGjBlj+vbta+x2u0u9jes/c+aMx/vPmJaPcWvLj4+PNwUFBaaiosIUFhaarKwsY7PZzMKFC5tchrvjrjW5ubnG0/8uExISjCTz97//3eP1paSkmKSkJKe2goICI8k8++yz5vjx46a0tNT8+te/NkFBQWb27NlOfd19Pbvbz5imj0tL7aWlpUaSueeeexxt7o4TT7bVmM45prZt22YkufVzYe2evIY92Vdteb23pi3/3wDoFPIIgUCAact/ysnJyebGG290aR8wYIDTm8F77rnHSDJvvvmmU7/i4mJjt9vNsGHDnNob3zgVFBS41CjJlJaWOrW39GbOm+ueMWOGkWRyc3Nd1jtu3DjHm1NPa3BHSkqKkWS2bdvm1L5jxw4jyaSkpDS5XStXrmxyeZ7UmJ2dbSSZxYsXO/U9cuSIsdvtbodAd/efMW17w964/Lfeesupb01NjUlMTDQRERGmpKTEZRnujrvWXEwI/Mc//uHR84xpPgTefPPNLn2nT59uQkNDTUVFhaPN3dezu/2MaZ8Q6O448WRbjemcY6oxBGZmZjZb15dfftlkCPTkNezJvmrL6701hEDAb+VxOSgAjRs3Ths3btQDDzygzZs3Oy4FKyws1M033+zot3TpUgUFBblMt96rVy8NHjxYW7du1eHDh12Wf+211zr93qdPH0lSUVGR2zV6c91LliyRpCZnaly1apVmzpx5UTW0JioqSldddZVT25AhQ5SYmKjt27eruLjY5TnXXXddk8vypMbVq1dLksaOHevUNzExUQMGDHC7fnf3X1s1Ln/8+PFO7Xa7XampqTpz5ozee+89l+e1x7hrq8TERElyXKJ7sdLT050uU2yUkpKiuro67dy509Hm7uvZ3X7txd1x4sm2XmwtnXFMefIa9mRftdfrHUBgIAQC0Pz58/XGG29o//79Sk1NVZcuXTRu3DjHGyXpq+8PVVRUqKGhQbGxsS7f/fnnP/8p6f9mCzxfbGys0+9hYWGS5Pa07d5cd+Oyw8PDFRMT45UaWtPcTbV79uwpSTp27JjLY1FRURdVY21trSorKxUeHq7o6Ohm190ad/dfW7W2/Pj4eElSSUmJy2MXO+4uxqhRoyRJO3bsaJflVVRU6Gc/+5mGDBmirl27Oo7pY489Jkmqrq529HXn9exJv/bgyTjxZFu9UYsvx5Sn5xl391V7vd4BBA5CIADZbDbddddd+uCDD3Ty5EktXbpUxhhlZWXpxRdflPTVX8jj4uIUEhKiuro6GWOa/Bk9evRF1dEUb67bbrcrNjZWNTU1qqysbLGft2ooLy9vcobIxvDn7hs0T2q02+2KiYlRTU2NqqqqXJbl7j3e3N1/jZo7xm1d/tGjRyV99SlJZ/Lggw8qJCREixcvbrHfD3/4QwUFBWn37t0t9svIyNDTTz+t+++/X3v27FFDQ4OMMZo3b54kOY0fd17PnvRrD56Mg3u7uQAABUJJREFUE0+2tXE72rMWX44pT88z7u6r9nq9AwgchEAAiouLc7wJDQ0N1W233eaYFW/FihWOfllZWTp37pzTrJWNfvnLX+rSSy/VuXPn2lxHZGSkzp496/h94MCBWrBggdfXPXHiREnSypUrXR67+uqr9eijj3q1hpqaGm3ZssWp7bPPPlNRUZFSUlKUkJDg9rI8qbHxsrzGy8QalZWVqbCw0O11urv/pJaPcWvLP38sSl99urF27VpFRES4XOLmawMGDNBTTz2lTz75RH/4wx+a7FNYWKjXXntNOTk5GjRoULPLqq+v14YNG9SrVy89/PDD6tGjhyP4nD9zbSN3X8/u9msv7owTT7dVCrwx5e5r2NN91V6vdwABwnvfNwTgC235on5sbKwZNWqU2b59u6mpqTFHjx41c+bMMZLMM8884+h39OhR069fP3P55ZeblStXmpMnT5ry8nLz6quvmsjISJcJH5qbNOLxxx9vcjKUcePGmdjYWHPw4EGzceNGExISYnbt2uX1dTfOFJiQkGCWL19uTp06ZQ4dOmS+853vmPj4ePPvf/+7TTW4IyUlxcTGxprU1FSPZge9cLsaeVLjF198Ybp16+Y0W+DOnTvN2LFjTc+ePT2eHbS1/WdMy8e4teU3zuR46tQpp5kcFyxY4NY+am7ctaYtE8M0+tGPfmRCQ0PN448/bgoLC01tba05fPiw+d3vfmcSEhLMiBEjTFVVldNzmpoY5pZbbjGSzPPPP29KS0tNdXW1+fDDD82ll15qJJk1a9Y4+rr7ena3nzHtOztoa+PEk201pnOOqYuZGMaT17An+6otr/fWMDEM4LeYHRQING35T/nTTz81Dz74oPna175mIiMjTbdu3cwNN9xgFi5caBoaGpz6lpeXm1mzZpnLL7/chIaGmh49epgxY8Y4vdnYtGmTyzToTzzxhDHGuLSPHz/e8bzdu3ebkSNHmqioKNOnTx8zf/78Dlt3WVmZmTlzpklOTjahoaEmISHBTJkyxfz/9u5Qp5UgCgPwbAKCQEEigHdALgqBWIugQRASEgSvgOANqniNVmCwKBANDkMCGhS8AOpcRZMLlLvLJSztfF9StZvZszsz2f5pZnp/f9+4hiZev/Df3t5GVVXR6XRibm4uNjc34+rq6tP7GhdMmtR4d3cX29vbsbi4ONru/vz8PLa2tkbXODw8jLOzs3fX3tvba/z8xvVx0/aXlpaiqqq4uLj49BnV6ft/+Z8QGBFxfX0d+/v7sba2FrOzs9HpdKIsyzg9PY2Xl5fReb1eb2z9T09PcXR0NGpjeXk5Dg4O4vj4eHTu666RdedznfPG9ctn/VVV1btjl5eXEVFvnDS514jfN6bm5+ffHev1en+NiZWVlbHtRtSfw02fVd35XpcQCBOrX0R8sBAFmFjdbjellNJgMGi5EupYX19Pz8/PX9pVlJ/R7/fT7u7uh+s2IWfeNzCxBtYEAgAAZEQIBAAAyIgQCPDN3v6310efhYWFVBRFurm5SY+Pj6koinRyctJ26QBABmbaLgBg2lg7BgD8Zn4JBAAAyIgQCAAAkBEhEAAAICNCIAAAQEaEQAAAgIwIgQAAABkRAgEAADIiBAIAAGRECAQAAMiIEAgAAJARIRAAACAjQiAAAEBGhEAAAICMzLRdAPD9hsNh6na7bZcBU+Hh4SGllMwpeGM4HKayLNsuA/gCIRCmzMbGRtslwFRZXV1NOzs7bZcBv05Zlt45MKGKiIi2iwAAAOBHDKwJBAAAyIgQCAAAkBEhEAAAICNCIAAAQEb+AJXjLbFFoDEfAAAAAElFTkSuQmCC", "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bert_encoder = tfm.nlp.encoders.build_encoder(encoder_config)\n", "bert_classifier = tfm.nlp.models.BertClassifier(network=bert_encoder, num_classes=2)\n", "\n", "tf.keras.utils.plot_model(bert_classifier)" ] }, { "cell_type": "markdown", "metadata": { "id": "IStKfxXkTJMu" }, "source": [ "### Load Pretrained Weights into the BERT Classifier\n", "\n", "The provided pretrained checkpoint only contains weights for the BERT Encoder within the BERT Classifier. Weights for the Classification Head is still randomly initialized." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:27.419655Z", "iopub.status.busy": "2023-10-17T12:27:27.419007Z", "iopub.status.idle": "2023-10-17T12:27:27.843139Z", "shell.execute_reply": "2023-10-17T12:27:27.842304Z" }, "id": "G9_XCBpOEo4y" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "checkpoint = tf.train.Checkpoint(encoder=bert_encoder)\n", "checkpoint.read(\n", " os.path.join(folder_bert, 'bert_model.ckpt')).expect_partial().assert_existing_objects_matched()" ] }, { "cell_type": "markdown", "metadata": { "id": "E6Hu1FFgQWUU" }, "source": [ "## Load ALBERT model pretrained checkpoints" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:27.847077Z", "iopub.status.busy": "2023-10-17T12:27:27.846567Z", "iopub.status.idle": "2023-10-17T12:27:42.948033Z", "shell.execute_reply": "2023-10-17T12:27:42.946986Z" }, "id": "TWUtFeWxQn0V" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2023-10-17 12:27:27-- https://storage.googleapis.com/tf_model_garden/nlp/albert/albert_xxlarge.tar.gz\r\n", "Resolving storage.googleapis.com (storage.googleapis.com)... 172.253.114.207, 172.217.214.207, 142.251.6.207, ...\r\n", "Connecting to storage.googleapis.com (storage.googleapis.com)|172.253.114.207|:443... connected.\r\n", "HTTP request sent, awaiting response... " ] }, { "name": "stdout", "output_type": "stream", "text": [ "200 OK\r\n", "Length: 826059238 (788M) [application/octet-stream]\r\n", "Saving to: ‘albert_xxlarge.tar.gz’\r\n", "\r\n", "\r", "albert_xxlarge.tar. 0%[ ] 0 --.-KB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 1%[ ] 8.01M 36.0MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 5%[> ] 40.01M 87.1MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 9%[> ] 72.01M 102MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 13%[=> ] 104.01M 113MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 17%[==> ] 136.01M 122MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 20%[===> ] 160.01M 120MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 23%[===> ] 189.05M 123MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 26%[====> ] 208.35M 120MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 29%[====> ] 235.73M 122MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 33%[=====> ] 267.62M 125MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 36%[======> ] 291.15M 125MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 40%[=======> ] 315.29M 124MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 41%[=======> ] 329.50M 121MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 46%[========> ] 363.59M 124MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 49%[========> ] 392.01M 123MB/s eta 3s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 53%[=========> ] 424.01M 124MB/s eta 3s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 57%[==========> ] 456.01M 123MB/s eta 3s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 61%[===========> ] 486.80M 130MB/s eta 3s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 64%[===========> ] 509.65M 129MB/s eta 3s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 65%[============> ] 517.79M 124MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 70%[=============> ] 552.01M 121MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 74%[=============> ] 584.01M 119MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 77%[==============> ] 610.95M 121MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 80%[===============> ] 632.01M 119MB/s eta 2s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 84%[===============> ] 664.01M 121MB/s eta 1s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 88%[================> ] 696.01M 118MB/s eta 1s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 93%[=================> ] 736.01M 122MB/s eta 1s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 97%[==================> ] 768.01M 118MB/s eta 1s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "albert_xxlarge.tar. 100%[===================>] 787.79M 117MB/s in 6.5s \r\n", "\r\n", "2023-10-17 12:27:34 (122 MB/s) - ‘albert_xxlarge.tar.gz’ saved [826059238/826059238]\r\n", "\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "albert_xxlarge/\r\n", "albert_xxlarge/bert_model.ckpt.index\r\n", "albert_xxlarge/30k-clean.model\r\n", "albert_xxlarge/30k-clean.vocab\r\n", "albert_xxlarge/bert_model.ckpt.data-00000-of-00001\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "albert_xxlarge/params.yaml\r\n", "albert_xxlarge/albert_config.json\r\n" ] } ], "source": [ "# @title Download Checkpoint of the Selected Model { display-mode: \"form\", run: \"auto\" }\n", "albert_model_display_name = 'ALBERT-xxlarge English' # @param ['ALBERT-base English', 'ALBERT-large English', 'ALBERT-xlarge English', 'ALBERT-xxlarge English']\n", "\n", "if albert_model_display_name == 'ALBERT-base English':\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/albert/albert_base.tar.gz\"\n", " !tar -xvf \"albert_base.tar.gz\"\n", "elif albert_model_display_name == 'ALBERT-large English':\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/albert/albert_large.tar.gz\"\n", " !tar -xvf \"albert_large.tar.gz\"\n", "elif albert_model_display_name == \"ALBERT-xlarge English\":\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/albert/albert_xlarge.tar.gz\"\n", " !tar -xvf \"albert_xlarge.tar.gz\"\n", "elif albert_model_display_name == \"ALBERT-xxlarge English\":\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/albert/albert_xxlarge.tar.gz\"\n", " !tar -xvf \"albert_xxlarge.tar.gz\"" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:42.952880Z", "iopub.status.busy": "2023-10-17T12:27:42.952150Z", "iopub.status.idle": "2023-10-17T12:27:42.958377Z", "shell.execute_reply": "2023-10-17T12:27:42.957763Z" }, "id": "5lZDWD7zUAAO" }, "outputs": [ { "data": { "text/plain": [ "'albert_xxlarge'" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Lookup table of the directory name corresponding to each model checkpoint\n", "folder_albert_dict = {\n", " 'ALBERT-base English': 'albert_base',\n", " 'ALBERT-large English': 'albert_large',\n", " 'ALBERT-xlarge English': 'albert_xlarge',\n", " 'ALBERT-xxlarge English': 'albert_xxlarge'\n", "}\n", "\n", "folder_albert = folder_albert_dict.get(albert_model_display_name)\n", "folder_albert" ] }, { "cell_type": "markdown", "metadata": { "id": "ftXwmObdU2fS" }, "source": [ "### Construct ALBERT Model Using the New `params.yaml`\n", "\n", "params.yaml can be used for training with the bundled trainer in addition to constructing the BERT encoder here." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:42.962052Z", "iopub.status.busy": "2023-10-17T12:27:42.961488Z", "iopub.status.idle": "2023-10-17T12:27:42.968925Z", "shell.execute_reply": "2023-10-17T12:27:42.968354Z" }, "id": "VXn20q2oU1UJ" }, "outputs": [ { "data": { "text/plain": [ "{'task': {'model': {'encoder': {'albert': {'attention_dropout_rate': 0.0,\n", " 'dropout_rate': 0.0,\n", " 'embedding_width': 128,\n", " 'hidden_activation': 'gelu',\n", " 'hidden_size': 4096,\n", " 'initializer_range': 0.02,\n", " 'intermediate_size': 16384,\n", " 'max_position_embeddings': 512,\n", " 'num_attention_heads': 64,\n", " 'num_layers': 12,\n", " 'type_vocab_size': 2,\n", " 'vocab_size': 30000},\n", " 'type': 'albert'}}}}" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "config_file = os.path.join(folder_albert, \"params.yaml\")\n", "config_dict = yaml.safe_load(tf.io.gfile.GFile(config_file).read())\n", "config_dict" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:42.972118Z", "iopub.status.busy": "2023-10-17T12:27:42.971752Z", "iopub.status.idle": "2023-10-17T12:27:42.984904Z", "shell.execute_reply": "2023-10-17T12:27:42.984323Z" }, "id": "Uo_TSMSvWOX_" }, "outputs": [ { "data": { "text/plain": [ "{'vocab_size': 30000,\n", " 'embedding_width': 128,\n", " 'hidden_size': 4096,\n", " 'num_layers': 12,\n", " 'num_attention_heads': 64,\n", " 'hidden_activation': 'gelu',\n", " 'intermediate_size': 16384,\n", " 'dropout_rate': 0.0,\n", " 'attention_dropout_rate': 0.0,\n", " 'max_position_embeddings': 512,\n", " 'type_vocab_size': 2,\n", " 'initializer_range': 0.02}" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Method 1: pass encoder config dict into EncoderConfig\n", "encoder_config = tfm.nlp.encoders.EncoderConfig(config_dict[\"task\"][\"model\"][\"encoder\"])\n", "encoder_config.get().as_dict()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:42.988191Z", "iopub.status.busy": "2023-10-17T12:27:42.987558Z", "iopub.status.idle": "2023-10-17T12:27:43.001705Z", "shell.execute_reply": "2023-10-17T12:27:43.001007Z" }, "id": "u7oJe93uWcy0" }, "outputs": [ { "data": { "text/plain": [ "{'vocab_size': 30000,\n", " 'embedding_width': 128,\n", " 'hidden_size': 4096,\n", " 'num_layers': 12,\n", " 'num_attention_heads': 64,\n", " 'hidden_activation': 'gelu',\n", " 'intermediate_size': 16384,\n", " 'dropout_rate': 0.0,\n", " 'attention_dropout_rate': 0.0,\n", " 'max_position_embeddings': 512,\n", " 'type_vocab_size': 2,\n", " 'initializer_range': 0.02}" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Method 2: use override_params_dict function to override default Encoder params\n", "encoder_config = tfm.nlp.encoders.EncoderConfig()\n", "tfm.hyperparams.override_params_dict(encoder_config, config_dict[\"task\"][\"model\"][\"encoder\"], is_strict=True)\n", "encoder_config.get().as_dict()" ] }, { "cell_type": "markdown", "metadata": { "id": "abpQFw80Wx6c" }, "source": [ "### Construct ALBERT Model Using the Old `albert_config.json`" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:43.004660Z", "iopub.status.busy": "2023-10-17T12:27:43.004433Z", "iopub.status.idle": "2023-10-17T12:27:43.009490Z", "shell.execute_reply": "2023-10-17T12:27:43.008876Z" }, "id": "Xb99qms6WuPa" }, "outputs": [ { "data": { "text/plain": [ "{'hidden_size': 4096,\n", " 'initializer_range': 0.02,\n", " 'intermediate_size': 16384,\n", " 'max_position_embeddings': 512,\n", " 'num_attention_heads': 64,\n", " 'type_vocab_size': 2,\n", " 'vocab_size': 30000,\n", " 'embedding_width': 128,\n", " 'attention_dropout_rate': 0.0,\n", " 'dropout_rate': 0.0,\n", " 'num_layers': 12,\n", " 'hidden_activation': 'gelu'}" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "albert_config_file = os.path.join(folder_albert, \"albert_config.json\")\n", "config_dict = json.loads(tf.io.gfile.GFile(albert_config_file).read())\n", "config_dict" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:43.012497Z", "iopub.status.busy": "2023-10-17T12:27:43.012171Z", "iopub.status.idle": "2023-10-17T12:27:43.024585Z", "shell.execute_reply": "2023-10-17T12:27:43.024014Z" }, "id": "mCW0RJHcEtVV" }, "outputs": [ { "data": { "text/plain": [ "{'vocab_size': 30000,\n", " 'embedding_width': 128,\n", " 'hidden_size': 4096,\n", " 'num_layers': 12,\n", " 'num_attention_heads': 64,\n", " 'hidden_activation': 'gelu',\n", " 'intermediate_size': 16384,\n", " 'dropout_rate': 0.0,\n", " 'attention_dropout_rate': 0.0,\n", " 'max_position_embeddings': 512,\n", " 'type_vocab_size': 2,\n", " 'initializer_range': 0.02}" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "encoder_config = tfm.nlp.encoders.EncoderConfig({\n", " 'type':'albert',\n", " 'albert': config_dict\n", "})\n", "\n", "encoder_config.get().as_dict()" ] }, { "cell_type": "markdown", "metadata": { "id": "EIAMaOxdZw5u" }, "source": [ "### Construct a Classifier with `encoder_config`\n", "\n", "Here, we construct a new BERT Classifier with 2 classes and plot its model architecture. A BERT Classifier consists of a BERT encoder using the selected encoder config, a Dropout layer and a MLP classification head." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:43.027646Z", "iopub.status.busy": "2023-10-17T12:27:43.027237Z", "iopub.status.idle": "2023-10-17T12:27:45.257212Z", "shell.execute_reply": "2023-10-17T12:27:45.256122Z" }, "id": "xTkUisEEFEey" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4EAAAFgCAYAAAAFLXzCAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzde3hU1b3/8c/kNrmSAEIgIUIUkcqBqHhD4QcYIUQSA2kCKKhYRU9tq0ix7anWeopHW0XRVlqFHmq1lEuogOEmiHhOuaSiRbAgAUXKJQkkXEJCSMhl/f7wZMowucyEJJOZ/X49Tx6erFmz9nfPrNnMJ7NnbZsxxggAAAAAYAU5Ad6uAAAAAADQfgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYSNDFDdu2bdMrr7zijVoAoFEzZszQkCFD2mRsjnsApLY9zmRnZ7fJuADQnJycHJc2l08CDx8+rGXLlrVLQYAn8vLylJeX5+0y4AXLli3T4cOH22x8jnuox3HGutr6OLNs2TIdOXKkzcYHmsL8s6YjR440+v7G5ZPAeg0lRsCb6v+Kyty0HpvN1i7bYW6B44x1tcdx5oknntCECRPafDvAxWw2G/PPgpYuXaqJEyc2eBvfCQQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQqAfWrx4sWw2m2w2m0JDQ/1mWy0RGRnpqK/+Z/bs2d4uq0X8aV+A1jB79mzHa6FXr15eq8OfXpv+tC9AR9eR30P507HAn/alNbVKCCwvL9dVV12ltLS01hgOl2jSpEkyxig5OblF9/fk+bzUbbW18vJy7dixQ5KUkZEhY4xmzpzp5apaxp/2xR9w3PO+mTNnyhijpKQkr9bhT69Nf9oXX8cxxjf5y3sofzoW+NO+tKZWCYHGGNXV1amurq41hmtTkZGRGjp0qLfL6NB86fn0N8xP3+FLrxPmFS7EfPANVjvG+Mu89KXnzZ/4y/xpT0GtMUhUVJS++uqr1hgKHQDPJ9A8XicA2hLHGN/E8wZfwXcCAQAAAMBCLjkErlixwumLlpWVlQ22Hzx4UBMnTlRMTIy6du2qtLQ0p7+UXPwF/+3btys5OVlRUVEKDw/XyJEjtWXLFkf/5557ztH/wo9/161b52i/7LLLXMY/e/astmzZ4ugTFOT+h6Hjxo1z2qcLt7tx40bZbDbl5uY62qZPn+7Uv6amRpJ04sQJzZgxQ1deeaVCQkLUuXNnpaamatOmTY0+rvn5+ZowYYK6du3qaCspKZEk7d27V+PGjVN0dLQiIiI0bNgwbd682e39ulBjz2c9T7ZVVVWlZ555Rv3791d4eLi6dOmi9PR0vffee6qtrW1Rfa3FH+fnxWpqarRkyRKNGjVKPXr0UFhYmAYOHKjXXnvNcZrK6dOnXb4s/dxzzznuf2F7VlaWY+zi4mI99thj6tOnj0JCQtStWzdlZmbqs88+a/QxbmoO+xorHfcu3qd//vOfmjhxoqKiotS1a1fde++9OnXqlA4ePKj09HRFRUWpZ8+emjZtmsrKypzGcmdO1ruU48ef/vQnl3ldVFTk9j63Fn+cDxfjONM2rHSMaWoMT+eOp/tbz5255g4rvIfyxzl4Mcsc18xFlixZYhpoblZGRoaRZM6dO9dge0ZGhtm6daspLy83GzZsMGFhYebGG290GScpKclERESYIUOGOPpv377dDBo0yISEhJiPPvrIqX9ERIS57bbbXMYZPHiw6dq1q0t7Y/3dNXfuXCPJLFy40Kl96tSpRpKZOHGiU/vy5ctNcnKy4/fCwkKTmJhoYmNjTW5uriktLTX5+fkmMzPT2Gw2M3/+fKf71z9+w4cPN5s2bTJnz541eXl5JjAw0BQXF5v9+/ebmJgYEx8fb9avX2/KysrMrl27zOjRo02fPn2M3W5v0X429Hx6uq2HHnrIREdHm/Xr15uKigpTVFRkZs6caSSZTZs2eVxTVlaWycrK8vh+O3bscMzBxvbTV+ZnU/tysdzcXCPJPP/88+bkyZOmuLjY/PrXvzYBAQFm5syZTn1TUlJMQECA+fLLL13GGTJkiNN8LygoML179zaxsbFm9erVpqyszPzjH/8ww4cPN6GhoWbr1q1O929uDrtDklmyZIlbfVuC4577+5qZmWk++eQTU15ebt5++20jyaSmppqMjAyzY8cOU1ZWZt544w0jyTzxxBNOY3gyJz05fiQlJZn4+HjH7zU1NWbGjBlm1KhR5uTJkx7tJ8cZjjNtpSXjW+kY09QYnswdYzzbX0/nmjs64nuolsw/jmu+f1xr4v3N0nYLgbm5uU7tWVlZRpLLTiQlJRlJZseOHU7tu3btMpJMUlKSU3t7H6hOnDhhQkJCzJgxYxxtFRUVpnPnzqZv374mLCzMnDlzxnHb+PHjzR//+EfH7/VhcdGiRU7jVlZWmri4OBMWFmaKiooc7fWP35o1axqsJzs720gyy5Ytc2o/evSosdvtrRoCPd1WYmKiufXWW13G7tevX4cLgb4yPz09iI0YMcKlfcqUKSY4ONiUlpY62t5//30jyTz66KNOfTdv3mzi4+PN+fPnHW33339/g38IKSwsNHa73QwePNipvbk57A5fDYG+Mq/cUb9Pq1evdmofMGCAkWT+53/+x6k9MTHRXH311U5tnsxJT44fF4bAU6dOmZSUFPP444+bmpoaj/bRGI4zze3LxTjOtO34VjrGNDWGJ3PHGM/219O55o6O+B6qrUKgr8xBqx7XmgqB7fadwBtvvNHp94SEBElSQUGBS9+IiAhde+21Tm0DBw5UXFycdu7cqcLCwrYrtBldunTRnXfeqQ0bNjhOL1q5cqVuvvlmfe9739O5c+f07rvvSpJOnjypjz76SJmZmY77L1++XJI0duxYp3HtdruSk5N17tw5vf/++y7bvemmmxqsZ926dZKklJQUp/a4uDj169evhXvZME+3NWbMGG3dulUPP/yw8vLyHKcv5Ofna8SIEa1a26Xyl/l5obS0NKdTjOslJSWpurpau3fvdrSNHj1aAwcO1FtvvaUTJ0442l966SX94Ac/UHBwsKNtxYoVCggIcFn+ukePHhowYIA+/fRTHTlyxGW7jc1hf+aP8+qGG25w+j0uLq7B9vj4eJf99GROtuT4kZ+fr5tvvlkBAQF69dVXFRgY6PH+tSV/nA8cZ7zLH+dUQzyZO/Xc3d+WzjVP+et7KH+cg1Y5rrVbCIyOjnb6PSQkRJIaXEI3JiamwTG6d+8uSTp+/HgrV+eZ++67T7W1tfrzn/8sSXrnnXd033336e6771ZgYKAWLlwoSVq0aJHS0tIUGRkp6Zvzu0tLSxUaGqqoqCiXcWNjYyWpwe+uREREuLRVVVWprKxMoaGhjm1cqP7xag0t2dbcuXP19ttv68CBA0pOTlanTp00ZswYRxDuSPxpftYrLS3VM888o4EDB6pz586O88iffPJJSVJFRYVT/+nTp6uiokK//e1vJUn79u3Thx9+qIcfftjRp34O19XVKTo62uV8+L///e+SpP3797vU09Ac9nf+OK86derk9HtAQIACAwMVHh7u1B4YGOiyn57MSU+PH6dOndK4cePUq1cvrV27Vn/6059aY3dblT/OB44z3uWPc6ox7sydC7mzv5cy1zzhz++h/HEOWuW41iFXBz1x4oSMMS7t9ZPjwhdLQECAzp8/79L39OnTDY5ts9kuub6xY8eqS5cueuedd1RcXKy8vDyNGzdOsbGxGj16tD788EMVFhbqj3/8o+677z7H/ex2u6Kjo1VZWemyYIIkHTt2TNI3fxFwh91uV1RUlCorK1VeXu5y+8mTJ1u4h62zLZvNpnvvvVcffPCBTp8+rRUrVsgYo8zMTL3yyiutVlt76+jzs156erpmzZqladOmad++faqrq5MxRnPmzJEkl32YPHmyYmNj9frrr6uqqkovv/yy7r//fnXu3NnRx263KyYmRkFBQaqurpYxpsGfkSNHttp+WIWvzKtL4cmc9PT4ERQUpA8++EArV67UwIEDNW3aNG3fvr3d9q21+cp84DjjOzr6nGpuDHfmzoXc2d/2mmu8h/pGR5+D9axyXOuQIbCystLlP+/PP/9cBQUFSkpKUs+ePR3tPXv21NGjR536FhUV6dChQw2OHR4e7jSprr76as2bN8+j+kJCQjRx4kR99tlneuqpp5SRkaGwsDBJ0r333qva2lr9/Oc/V2FhoW6//Xan+44fP16StHr1aqf2qqoqbdy4UWFhYS6nCjQlNTVV0r9OM6hXUlKi/Px8j/artbcVExOjvXv3SpKCg4M1atQox4pHF++/L+no8zMoKEi7d+/Wli1b1KNHDz322GPq1q2b4wB57ty5Bu9nt9v16KOP6vjx43r55Ze1cOFCPf744y79MjMzVVNT0+AKa7/61a90+eWXO1bChfs6+ry6VLW1tR7NSU+PH1FRUYqPj1dkZKTee+89RUZGaty4cR3m9CJPdfT5wHHG93T0OdXcGO7OnXru7m97zTXeQ3X8OWi141qHDIHR0dH66U9/qm3btuns2bP65JNPNGXKFIWEhOi1115z6jt69GgVFBTo9ddfV3l5ub766is9/vjjjZ4Kef3112vfvn06fPiwtm3bpgMHDmjYsGEe13jvvfdKkubPn+/0ad+4ceMUFRWl+fPna/LkyQoIcH6IX3jhBSUmJmr69OlatWqVysrKtG/fPt1zzz0qLCzUa6+95jgt1B3PP/+8unTpounTp2vDhg0qLy/Xnj17NGXKlAZPObgULdnWv//7v2vXrl2qqqrS8ePH9eKLL8oY4xKOfYkvzM/AwECNGDFCRUVFeumll1RSUqJz585p06ZNeuONNxq936OPPqqwsDA9/fTTuuOOO9S3b1+XPi+88IKuvPJKfec739HatWtVWlqqkydP6s0339QvfvELzZ49+5KWZrYqX5hXl6Ilc7Klx48+ffpo2bJlKi4uVmZmpqqqqtpil9qUL8wHjjO+paPPKXfGcGfueLq/7TXXeA/V8eegZLHjmgeryDRo+fLlRpLTz+TJk822bdtc2p966iljvvkM1eln7NixjvHqV3nbs2ePSUlJMVFRUSYsLMwMHz7cbN682WX7p0+fNg899JDp2bOnCQsLM0OHDjXbt283gwcPdoz/4x//2NF/7969ZtiwYSYiIsIkJCSYuXPnur2vF7vqqqvM5Zdfburq6pza61cA3b17d4P3KykpMdOnTzeJiYkmODjYREdHm5SUFLNx40ZHn4Yev8ael/z8fDNu3DjTqVMnx9K8q1atMsnJyY77Pfjgg27tU2PPZ0u29dlnn5lHHnnEfOtb3zLh4eGmS5cu5pZbbjHz5893eczc0ZJV+yIiIlz256WXXvLJ+dnQvjT288UXX5ji4mLzyCOPmISEBBMcHGxiY2PN1KlTzU9+8hNHv4ZWPZs2bVqDqz1e6MSJE2bGjBnmiiuuMMHBwaZbt25m9OjRZsOGDY4+nszh5qiDrQ5qpeNeY/u0fft2l/YXXnjB/PWvf3Vp//nPf26MMR7NSXeOH4sWLXLZ1pw5cxqs+cLjWFM4znCcaSuejG+lY4ynY7gzdzzdX3fmmjs68nsoT+c3xzX/OK61yyUiWsvF13sC6rV06fbWZIX5uWDBghYtid2WOloIbG1WmFe+guNM+7Dicaatx2+KP80pd+aOP+1va/Hm/DPGGs9JRzyudYhLRADwDW+88YZmzJjh7TIA+DGOM2gp5g46Kl+bm4RAwOJ+//vfa/z48SovL9cbb7yhU6dOacKECd4uC4Af4TiDlmLuoKPy9bnZYULg7NmzZbPZtHPnTh09elQ2m01PP/10u23/4ut1NPTz7LPPtls9bcEK+9hWvD0/29qKFSvUuXNn/e53v9PixYtZcKGdeHtecUzoWLw9H9oax5n25+051VrHGHfnTlvuL8fLlvH2HGxrvnxcsxnjfLGLpUuXauLEiQ1exwPwpuzsbElSTk6OlytBe7PZbFqyZEmb/YWN4x7qcZyxrrY+zrT1+EBTmH/W1MT7m5wO80kgAAAAAKDtEQIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWEhQYzdkZ2e3Zx1As/Ly8iT5z9ysra3Vvn37FB8fr06dOnm7HMh/5hZazt+OM+hY5syZo5ycHG+X0eGUlpaqoKBA/fv3l81m83Y5fov5Zz1Hjhxp9DaXEJiQkKCsrKw2LQhoiVtuucXbJbSqs2fP6quvvtLu3bsVGRmp+Ph4xcXFqUuXLvwneJGsrCwlJCS02fgc9/DFF19I8r/jDNzX1scZjjH/YozRyZMnVVBQoKNHj6q8vFxhYWFKSEhQZGSkt8vzS8w/a+rVq1ejz73NGGPauR4A/6eurk47duxQbm6uli5dqi+++EKXXXaZUlNTlZ2drdGjR8tut3u7TMDvTZgwQZK0dOlSL1cC+Kfa2lpt27ZNOTk5+stf/qKjR4+qT58+uuuuu5Sdna1bb71VAQF8SwloJzmEQKADOXDggHJzc5WTk6OtW7cqLCxMt99+u7KzszVu3DhOGwXaCCEQaH2VlZXavHmzcnNztWTJEh07dkzXXHON0tPTlZaWpttuu40zXwDvIAQCHdWhQ4e0bt065ebm6v3331dgYKCGDh2qtLQ0TZgwQT179vR2iYDfIAQCraOiokIbN25UTk6OVq5cqTNnzuiaa65Rdna2Jk2apP79+3u7RACEQMA3nDx5UqtWrdKqVau0du1aVVRUaMiQIUpPT9f48ePVr18/b5cI+DRCINByF/4ftWbNGlVWVuqWW25Rdna2srKyFB8f7+0SATgjBAK+5ty5c/rggw+Uk5Oj3NxcnT59mtNrgEtECAQ8U1xcrLVr1yonJ8flbJVJkyYpNjbW2yUCaBwhEPBlTX3RPj09XSNGjFBQUKNXggHwfwiBQPO+/vprvffee3xvHfB9hEDAn+zevVs5OTmsNAp4iBAINGz37t1atWqVcnNztWXLFnXp0kVjx45Venq67rzzTkVERHi7RACeIwQC/qqplUYzMjIUHR3t7RKBDoMQCHzjwksXLV68WPn5+erWrZvGjBmj7OxsjRkzRsHBwd4uE8ClIQQCVnD48GGtXbvWsdJoQECAhg0bxkqjwP8hBMLKmruGH981B/wOIRCwmoZWGr3uuuuUlpamu+++W1dffbW3SwTaHSEQVlNZWakNGzZo1apVWrFihY4fP+60yNjQoUO9XSKAtkMIBKysfqXRxt4E8NdfWAUhEFbQ1DX8+CMgYCmEQADfuPB0oHfffVdHjhxR7969lZGRwUqj8HuEQPir+rM/cnJytGHDBtXU1HANPwCEQAANq19pNCcnR3v27FHXrl115513stIo/BIhEP7k4u+Bcw0/ABchBAJoHiuNwt8RAuHruIYfAA8QAgF45uK/MBtjdPPNNys7O1vZ2dmKi4vzdomAxwiB8EX1Z2ysWrVKn376KdfwA+AuQiCAljt16pQ++OAD5ebmavny5aw0Cp9FCIQvuPAafosWLdK+ffvUq1cv3XnnnUpLS+MafgDcRQgE0DpYaRS+jBCIjurCRbuWLVumgoICJSYmKj09nWv4AWgpQiCA1sdKo/A1hEB0JI1dwy87O1vp6ekaPHiwt0sE4NsIgQDaXmMrjfK9FXQUhEB424XX8FuxYoXOnj3L6fUA2gohEED7YqVRdESEQHjDiRMntHr1aq7hB6C9EQIBeE9xcbHWrl2rnJwcrV+/XrW1tY43QKw0ivZECER7aegafnfccYfS09M1btw4de/e3dslAvB/hEAAHcOFK41efCrUpEmT1L9/f2+XCD9GCERb4hp+ADoYQiCAjufClUZXrlypY8eOsdIo2hQhEK2tsWv4ZWdna/To0bLb7d4uEYB1EQIBdGwXrjS6fPlyHT58WL1791ZKSgrXxUKrIQTiUjV0Db+EhASlpqZyrALQ0RACAfiWi/+6zkqjaA2EQLQE1/AD4KMIgQB818UrjYaGhio5OVnZ2dm66667FBMT4+0S4SMIgXAX1/AD4AcIgQD8AyuN4lIQAtGUpq7hd88996hfv37eLhEAPEEIBOB/WGkUniIE4mJNXcOPPywB8HGEQAD+7cJTt+pXGr3iiiuUlpbGd3bgQAiEJB06dEjr1q1Tbm6u1q1bp6CgIK7hB8AfEQIBWEf9Ig6rVq3SX/7yF3355Ze6/PLLNWbMGFbvszhCoHVd/N3iC6/hN378eEVFRXm7RABobYRAANbV2HW80tPTlZqaqsjISG+XiHZCCLSWxlYZ5hp+ACyCEAgAkvT111/rvffeU05OjrZt2ya73c5KoxZCCPRvF17D789//rP279/vdA2/1NRUBQUFebtMAGgvhEAAuFhTK41mZWUpPj7e2yWilREC/c+F1/DLyclRYWEh3wcGgG8QAgGgKadPn9aGDRscK42WlZU5rgk2ceJEfetb3/J2iWgFhED/cO7cOX3wwQfKyclRbm6uTp8+zTX8AMAVIRAA3FVZWanNmzcrNzdXS5cuVVFREZ8s+KCFCxfqv//7v1VXV+doy8/PlyRdffXVjraAgAA9+OCDmjx5crvXCPdd/IcaruEHAM0iBAJAS1y40ui7776r/fv3s9Koj9i5c6euvfZat/p+9tlnSkpKauOK4KkLr+F38SnbXMMPAJpFCASA1sBKo76lf//+jk//GtO3b1/t37+/nSpCcw4dOqTly5dr1apV+uijj5yu4Td+/Hh169bN2yUCgK8gBAJAa2tspVEuON1x/Nd//Zf+8z//U9XV1Q3eHhwcrGeffVY//elP27kyXOjia/iFh4dr5MiRXMMPAC4NIRAA2lJJSYnWrFnT6iuN1tbWqqioiJVKW+jAgQPq27evmvovcP/+/erbt287VuUfDh48qPj4+BafDl3/qXpOTo727NnjdA2/lJQUhYSEtHLFAGA5hEAAaC9nz57Vhx9+qJycHK1cuVJnzpxp8UqjH330kbKzs/XOO+9ozJgxbVi1/xo8eLB27NjhEgRtNpuuv/56ffLJJ16qzHetXLlSU6dO1dKlSzVq1Ci37lNXV6etW7c2+v1aruEHAK0uJ8DbFQCAVURERCg9PV1vv/22jh07pg0bNuiOO+7Qm2++qWuuuUZXXnmlHn/8cW3evLnJT6gkacWKFTpx4oRSU1P1H//xH6qpqWmnvfAf9913nwIDA13aAwMDdd9993mhIt9VXV2tH/7whxo/frxKS0v17rvvNtm/trZWmzdv1uOPP65evXpp2LBhysnJUWpqqv7617/q4MGDevPNN5Wenk4ABIA2wCeBAOBlDX0SkpCQoNTU1EZXGu3Vq5eOHj0q6ZvQkpSUpL/85S/q06ePF/bANx0/flw9e/Z0ulSE9M2lIY4ePaoePXp4qTLfcvjwYWVnZ+uTTz5RbW2tJKlLly4qLi5WQMC//tbc1DX8JkyYoGuuucZbuwAAVsPpoADQ0TS00mhycrLS0tKUmZmp/Px83XDDDU73CQ4OVnh4uP785z/rzjvv9FLlvmfkyJH661//6ggvgYGB+n//7//pww8/9HJlvmH16tWaPHmyKioqXBbZ2bx5swYMGOC4ht/y5ctVUVGhIUOGKD09XZmZmbrqqqu8VDkAWBohEAA6si+//FLLly/XihUrlJeXp9DQUCUlJenTTz/V+fPnnfoGBASorq5OP/jBD/Tyyy9znUI3LFiwQA8//LBTCJw/f74eeOABL1fWsdXU1GjWrFmaNWuWbDaby6epISEhSkxM1IEDB2Sz2XT77bdr/PjxrI4LAB0DIRAAfEVRUZFWrlypZ599VkVFRY32qz89dNmyZUpMTGzHCn3PmTNndNlllzk+xQoODtbx48cVExPj5co6rqNHjyorK0vbt293hOeGREdHa+7cuRo7diyPJwB0LCwMAwC+okePHkpOTm4yAErfLLrx+eef67rrrtPq1avbqTrf1KlTJ8fqk0FBQbrzzjsJLE348MMPHZ9ENxUAJam0tFQDBgzg8QSADogQCAA+5N1333VrtcTq6mqVlZUpLS1Njz32WKMXRYc0ZcoU1dbWqra2VpMnT/Z2OR1SbW2tnn32Wd1xxx06deqUW/MpJCREy5cvb4fqAACeYt1lAF61bds2HT582Ntl+Ix58+Y1+wlMvfrvaf3mN7/RmjVr9MMf/lBdu3Zty/J8UnV1tUJCQmSMUVVVlZYuXertkjqUU6dO6dVXX9XevXslqdnLl9Q7f/685s+frwEDBrRleX4nISFBQ4YM8XYZAPwc3wkE4FXZ2dlatmyZt8sAgA4hKytLOTk53i4DgH/L4ZNAAF7Hmx73FBQU6PPPP1dgYKAiIiIUEhKi0NBQhYWFKSQkRBEREQoMDFSnTp28XarPWbdunWw2m37/+99LEvPRA6dOnZIkVVRUqKqqSnV1dSotLZUknT17Vv369eOai27Kzs72dgkALIIQCAA+Ii4uTnFxcd4uwy/dcccdkuQIgXBf586dnf4FAHR8hEAAgOW5s9gOAAD+gtVBAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAL81e/Zs2Ww22Ww29erVq9l2tL/Fixc7novQ0FBvl9Mq9u/fL5vNpltuuaXB2315XkZGRjpqbO7nk08+8Xa5zfLH+QcA7iAEAvBbM2fOlDFGSUlJbrWj/U2aNEnGGCUnJ3u7lFbzhz/8QZL0t7/9TXv27HG53ZfnZXl5uXbs2CFJysjIkDGmwZ/o6GgvV+oef5x/AOAOQiAAeElkZKSGDh3q7TLQiurq6vT222/ruuuuk/SvQOhLmJcA4P8IgQAAtJL169crKChI8+bNkyS989XryNYAACAASURBVM47qqmp8XJV7e/06dO64YYbvF0GAKARhEAAAFrJggULNHXqVN1www0aNGiQjh07pjVr1ni7rHYzdOhQvfXWW94uAwDQDEIgAJ9TU1OjJUuWaNSoUerRo4fCwsI0cOBAvfbaa6qrq/N4vL1792rs2LGKjo5WeHi4Ro4cqS1btrj0Ky4u1mOPPaY+ffooJCRE3bp1U2Zmpj777DNHnxUrVjgtjpGfn68JEyaoa9eujraf/OQnstlsOnv2rLZs2eJoDwoK8rj2ltR08OBBTZw4UTExMeratavS0tL01VdfuYx94sQJzZgxQ1deeaXsdrt69eqlO+64Q2+99ZbOnTvXYL+QkBB17txZqamp2rRpU4OP9bhx4xQdHa2IiAgNGzZMmzdvbtX9a+gxLykp8fix9dTJkyeVm5ur+++/X5L0wAMPSPomGLaEL8/LCzH/2mf+AYBHDAB4UVZWlsnKyvLoPrm5uUaSef75583JkydNcXGx+fWvf20CAgLMzJkzXfonJSWZ+Pj4Btujo6PNyJEjzebNm01ZWZnZvn27GTRokAkJCTEfffSRo29BQYHp3bu3iY2NNatXrzZlZWXmH//4hxk+fLgJDQ01W7dudRo7IyPDSDLDhw83mzZtMmfPnjV5eXkmMDDQFBcXG2OMiYiIMLfddptH+36hltaUkZFhtm7dasrLy82GDRtMWFiYufHGG536FhYWmsTERNOjRw+Tm5trzpw5Y4qKisysWbOMJDNnzhynfrGxsSY3N9eUlpaa/Px8k5mZaWw2m5k/f75jzP3795uYmBgTHx9v1q9fb8rKysyuXbvM6NGjTZ8+fYzdbm+V/WvqMW9OS+Zjvd/85jdm5MiRjt+Li4tNcHCwCQoKMseOHXPp76vzcseOHUZSoz9/+MMfGrwf8695lzL/AMADSwmBALyqpSFwxIgRLu1TpkwxwcHBprS01Km9qTfbksy2bduc2nft2mUkmaSkJEfb/fffbySZhQsXOvUtLCw0drvdDB482Km9/g3hmjVrGt2PSw2BLa0pNzfXqT0rK8tIcnqjOnXqVCPJLFmyxGW7Y8aMcbwJr++3aNEipz6VlZUmLi7OhIWFmaKiImOMMdnZ2UaSWbZsmVPfo0ePGrvd7vImvC0e8+Zcypvw66+/3rz99ttObePHjzeSzOzZs136++q8rA+BGRkZLrfddtttzYZA5l/jCIEA2slSTgcF4HPS0tIaPNUrKSlJ1dXV2r17t9tjhYaG6uabb3ZqGzhwoOLi4rRz504VFhZK+uaUr4CAAKWlpTn17dGjhwYMGKBPP/1UR44ccRn/pptucrsWT7W0phtvvNHp94SEBElSQUGBo2358uWSpNTUVJf7r127VtOnT3fqN3bsWKc+drtdycnJOnfunN5//31J0rp16yRJKSkpTn3j4uLUr1+/Vtu/tnzMG7Nr1y7t379f3/72t53a608J9XSVUF+el81h/gGA913aif4A4AWlpaV6+eWXtXz5ch05ckSnT592ur2iosLtseq/t3Ox7t27q6CgQMePH1eXLl1UWloqSU1e/2z//v0uF/mOiIhwuxZPVFVVtbimi/uHhIRIkuP7lPVjh4aGKioqqtkaGusXGxsrSSoqKlJVVZXKysoUGhqqyMhIl77du3fXvn37WmX/2uoxb8qCBQtUVlbW6LZ3796tjz/+2O2A4Kvzsqnv19Vj/gGA9/FJIACfk56erlmzZmnatGnat2+f6urqZIzRnDlzJEnGGLfHqn+jd7Hjx49L+ubNod1uV0xMjIKCglRdXd3oBbJHjhzp0X409CbfXW1VU/3Y0dHRqqysVFlZWYv7HTt2TNI3n5zY7XZFRUWpsrJS5eXlLn1PnjzZbvvX2qqrq7Vw4UJt2bKlwRrrP7Xy5NNAX52XrYH5BwBtjxAIwKfU1tZqy5Yt6tGjhx577DF169bN8ab1whUD3VVeXq6dO3c6tX3++ecqKChQUlKSevbsKUnKzMxUTU1Ng6sz/upXv9Lll1/u8fXgwsPDdf78ecfvV199teP6cu5oi5rqjR8/XpIavLzBddddpyeeeMKp3+rVq536VFVVaePGjQoLC3Ocfld/al/9aXn1SkpKlJ+f77Kdtty/1pSbm6vLLrtMt956a4O3P/jgg5KkRYsWuT1HfXleStINN9ygxYsXe3SfCzH/AKCNtelXDgGgGS1ZCOH22283ksyLL75oiouLTUVFhfnwww/N5ZdfbiSZDRs2OPVvagGOiIgIM3ToUJOXl2fKy8sbXYXx2LFj5sorrzRXXHGFWbNmjTl9+rQ5ceKEeeONN0x4eLjLAhb1i0ScO3eu0f0YM2aMiY6ONocOHTJbt241QUFBZs+ePW4/Dq1V049//GMjyezYscPRVr/qYs+ePc2qVavMmTNnzOHDh813v/tdExsba/75z3869atfnfHMmTNOqzPOmzfPMeaXX35punTp4rQ64+7du01KSorp3r27y8IcbfGYN6cl8zEtLc28+OKLTfa56aabjCTzpz/9ydHmq/OyqYVh6g0ePNhlsRbmX/NYGAZAO2F1UADe1ZI3PcXFxeaRRx4xCQkJJjg42MTGxpqpU6ean/zkJ45l6gcPHmxeeukll+Xrn3rqKaf2+Ph48/HHH5uRI0eayMhIExYWZoYPH242b97sst0TJ06YGTNmmCuuuMIEBwebbt26mdGjRzuFzm3btjW4bH5D9u7da4YNG2YiIiJMQkKCmTt3rmcP3iXU9NRTTxljjEv72LFjHfcrKSkx06dPN4mJiSY4ONj07NnTTJo0yezbt8+phov7RUdHm5SUFLNx40aXevPz8824ceNMp06dHJcGWLVqlUlOTnbU8OCDD7bZY94cT+bj4cOHnbZ38803u/T5+uuvm7ycgq/Ny4iICLf2Rxes2Mn8cx8hEEA7WWozxoMvzwBAK8vOzpYk5eTkeLkSgPkI72L+AWgnOXwnEAAAAAAshBAIAAAAABZCCASADsZmszX78+yzz3q7TAAA4KO4WDwAdDB8VRsAALQlPgkEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwkCBvFwAAR44c0dKlS71dBqAjR45IEvMRXnHkyBH16tXL22UAsABCIACvy8vL08SJE71dBuDAfIS3ZGVlebsEABZgM8YYbxcBAIA3TZgwQRKfAAIALCGH7wQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUEebsAAADa09/+9jft3LnTqe3AgQOSpHnz5jm1Dxo0SLfccku71QYAQHsgBAIALOX48eN65JFHFBgYqICAb06IMcZIkr7//e9Lkurq6lRbW6v33nvPa3UCANBWbKb+fz4AACygurpal112mc6cOdNkv6ioKJWUlCgkJKSdKgMAoF3k8J1AAIClBAcHa9KkSU2Gu+DgYN19990EQACAXyIEAgAs5+6779b58+cbvb26ulr33HNPO1YEAED74XRQAIDl1NXVKS4uTseOHWvw9m7duqmoqMjxnUEAAPwIp4MCAKwnICBAU6ZMafB0z5CQEN1///0EQACA3+J/OACAJTV2Suj58+d19913e6EiAADaB6eDAgAsq2/fvvrqq6+c2nr37q2DBw96pyAAANoep4MCAKxrypQpCg4OdvweEhKiBx54wIsVAQDQ9vgkEABgWV9++aWuuuoqp7b8/Hz169fPSxUBANDm+CQQAGBdffv21aBBg2Sz2WSz2TRo0CACIADA7xECAQCWdt999ykwMFCBgYG67777vF0OAABtjtNBAQCWVlBQoISEBBljdOjQIfXq1cvbJQEA0JZygrxdAQD4G5vN5u0S0EIJCQneLgEe4m/ZAOA5QiAAtIHp06dryJAh3i4Dbvrggw9ks9mUnJzcaJ9t27bp1Vdf1ZIlS9qxMjSm/vkAAHiOEAgAbWDIkCGaMGGCt8uAm+rDX9euXZvs9+qrr/K8diCEQABoGUIgAMDymgt/AAD4E1YHBQAAAAALIQQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAA6IAWL14sm80mm82m0NBQb5fj8+666y7ZbDY999xz7brdyMhIx/NY/xMQEKDOnTsrKSlJjz76qD799NN2rQkAAEIgAHRAkyZNkjFGycnJ3i7F57399tvKzc31yrbLy8u1Y8cOSVJGRoaMMaqurtbevXv1i1/8Qnv37tUNN9ygBx54QBUVFV6pEQBgPYRAAECHEhkZqaFDh7bKWAUFBZo+fbruvffeVhmvNQQGBio2NlYZGRn68MMP9aMf/UhvvfWW7r77bhljvF1eu2nN5xkA4BlCIADAb02bNk3Z2dkaPXq0t0tp1C9/+UvdfPPNeu+997R48WJvlwMAsABCIADALy1YsEC7d+/W7NmzvV1Kk2w2m77//e9Lkn772996uRoAgBUQAgGgA9i7d6/GjRun6OhoRUREaNiwYdq8ebNLvxUrVjgtMpKfn68JEyaoa9eujraSkhJJ0okTJzRjxgxdeeWVCgkJUefOnZWamqpNmzY5xps9e7bjfr169dL27duVnJysqKgohYeHa+TIkdqyZYtLHe6M/dxzzznGvvC0v3Xr1jnaL7vsMpdazp49qy1btjj6BAUFefx4HjlyRD/84Q+1YMECRUVFeXz/9lb/+OTl5am6uprnGQDQtgwAoFVJMkuWLHG7//79+01MTIyJj48369evN2VlZWbXrl1m9OjRpk+fPsZut7vcJyMjw0gyw4cPN5s2bTJnz541eXl5JjAw0BQXF5vCwkKTmJhoYmNjTW5uriktLTX5+fkmMzPT2Gw2M3/+fKfxkpKSTEREhBkyZIjZunWrKS8vN9u3bzeDBg0yISEh5qOPPnL09XTsiIgIc9ttt7nsw+DBg03Xrl1d2hvr74mUlBTz6KOPOn5/5513jCQza9asFo+5ZMkS05L/Nnfs2GEkmYyMjEb7nDt3zkgykkxBQYGjnee5cS19PgAAZilHTwBoZZ6GwOzsbCPJLFu2zKn96NGjxm63NxkC16xZ0+CYU6dONZLMokWLnNorKytNXFycCQsLM0VFRY72pKQkI8ns2LHDqf+uXbuMJJOUlNTisds7HMybN89cccUVpry83NHW0UNgRUVFkyGQ59kVIRAAWmwpp4MCgJetW7dOkpSSkuLUHhcXp379+jV535tuuqnB9uXLl0uSxo4d69Rut9uVnJysc+fO6f3333e6LSIiQtdee61T28CBAxUXF6edO3eqsLCwxWO3l0OHDunJJ5/UggULFBER4ZUaWqL+sQ0ODnY6dbIezzMAoDURAgHAi6qqqlRWVqbQ0FBFRka63N69e/cm799Q0KmqqlJpaalCQ0Mb/D5cbGysJKmoqMipPSYmpsFt1Ndw/PjxFo/dXnJzc1VaWqoRI0Y4faeu/hIRP/vZzxxtX375pVdqbEj99z+HDBmi4OBgl9t5ngEArYkQCABeZLfbFRUVpcrKSpWXl7vcfvLkyRaNGR0drcrKSpWVlbncfuzYMUlSjx49nNpPnDjR4HXqjh8/LumbkNCSsQMCAnT+/HmXvqdPn26wfpvN1tiuNet73/uejDEuP++8844kadasWY62vn37tng7ramurk5z586V9E397rLy8wwAuDSEQADwstTUVEn/Oi20XklJifLz81s05vjx4yVJq1evdmqvqqrSxo0bFRYW5nL6aWVlpbZv3+7U9vnnn6ugoEBJSUnq2bNni8bu2bOnjh496tS3qKhIhw4darD28PBwpzBx9dVXa968ec3us6/6j//4D3388ccaP368srOzPbovzzMAoCUIgQDgZc8//7y6dOmi6dOna8OGDSovL9eePXs0ZcqUBk8RdccLL7ygxMRETZ8+XatWrVJZWZn27dune+65R4WFhXrttdccp/TVi46O1k9/+lNt27ZNZ8+e1SeffKIpU6YoJCREr732WovHHj16tAoKCvT666+rvLxcX331lR5//PFGT3W9/vrrtW/fPh0+fFjbtm3TgQMHNGzYsBY9Dh1RXV2djh8/rpUrVyo5OVkvvviivvOd72jhwoUefzrG8wwAaBGvrUkDAH5KHq4Oaowx+fn5Zty4caZTp04mLCzM3HjjjWbVqlUmOTnZsWrkgw8+aLZt2+b4/cKfhpSUlJjp06ebxMREExwcbKKjo01KSorZuHGjS9+kpCQTHx9v9uzZY1JSUkxUVJQJCwszw4cPN5s3b76ksU+fPm0eeugh07NnTxMWFmaGDh1qtm/fbgYPHuyo/8c//rGj/969e82wYcNMRESESUhIMHPnzvXosbzQI4880uDjlZKS4vFYLVmNMiIiwmXbNpvNREdHm4EDB5rvfve75tNPP3W5H89z81gdFABabKnNmAa+GAAAaDGbzaYlS5ZowoQJ3i7Fbddee61KSkp05MgRb5fSYS1dulQTJ05s8Pt0vsKfnmd/eD4AwEtyOB0UAAAAACyEEAgAAAAAFkIIBAALmz17tmw2m3bu3KmjR4/KZrPp6aef9nZZDbrwun+N/Tz77LPeLrND8qXnGQDQ9oK8XQAAwHtmzpypmTNnersMt/Ddr5bzpecZAND2+CQQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUQAgEAAADAQmzGGOPtIgDAn9hsNm+XAFgGb2MAwGM5Qd6uAAD8zZIlS7xdAjw0Z84cSdITTzzh5UoAAGh7fBIIALC8CRMmSJKWLl3q5UoAAGhzOXwnEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWEiQtwsAAKA9VVRUqKqqyqnt/PnzkqRTp045tdvtdoWHh7dbbQAAtAdCIADAUv7whz/o+9//foO3denSxen3119/Xd/73vfaoywAANqNzRhjvF0EAADtpbi4WD179lRtbW2T/QIDA1VYWKhu3bq1U2UAALSLHL4TCACwlG7duun2229XYGBgo30CAwOVnJxMAAQA+CVCIADAcqZMmaKmToQxxmjKlCntWBEAAO2H00EBAJZTVlambt26uSwQUy8kJETFxcXq1KlTO1cGAECb43RQAID1REVFKS0tTcHBwS63BQUF6a677iIAAgD8FiEQAGBJkydPVk1NjUt7bW2tJk+e7IWKAABoH5wOCgCwpPPnz+uyyy5TWVmZU3tkZKRKSkpkt9u9VBkAAG2K00EBANYUEhKirKwshYSEONqCg4M1YcIEAiAAwK8RAgEAlnXPPffo/Pnzjt+rq6t1zz33eLEiAADaHqeDAgAsq66uTrGxsSopKZEkde3aVceOHWvyGoIAAPg4TgcFAFhXQECAJk+erJCQEAUHB2vKlCkEQACA3yMEAgAs7e6779b58+c5FRQAYBlB3i4AQOt65ZVXtG3bNm+XAfiU8PBwSdJLL73k5UoA3zJkyBDNmDHD22UA8BCfBAJ+Ztu2bcrLy/N2GYBP6d27t3r37t3gbUeOHNGyZcvauSKg48vLy+OPjoCP4pNAwA/dcsstysnJ8XYZgM/YvXu3JGnAgAEuty1dulQTJ07kNQVcJDs729slAGghQiAAwPIaCn8AAPgrTgcFAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQACAw+LFi2Wz2WSz2RQaGurtcvzC9u3bNXXqVCUmJiosLExdunTRv/3bv+nb3/62fve73+mrr76SJM2ePdvx2Pfq1cvLVbvviy++0MSJE9WjRw8FBQU59iEmJsbbpXlFZGSk4zGo/5k9e7ZTn169ern0efrpp9u9Vl7vgHURAgEADpMmTZIxRsnJyS63lZeX66qrrlJaWlqbbLutx29vdXV1evLJJ3Xrrbeqe/fuWrt2rU6fPq0vvvhCc+bM0ZkzZ/Too4+qb9++qqmp0cyZM2WMUVJSkrdLd9LU83Lw4EENGTJEX3zxhd59912dOXNGZ86c0dKlSxUQ4P23GN6YU+Xl5dqxY4ckKSMjQ8YYzZw506nPkSNH9PXXX0uSbrvtNhlj9Nxzz7VbjfWaer0D8G/eP0IDwP+JjIzU0KFDvV0GGmGMUV1dnerq6lo8RlPPcWuM35H87Gc/0+zZs/Xb3/5WL774ovr37y+73a7Y2FiNGjVK69atU2pqqrfLbFZTz8u8efNUWlqquXPn6tZbb1V4eLiioqKUnZ2tkydPtkt9VppTANBagrxdAADAN0RFRTlOXfTF8dvT3r179ctf/lKDBw/WtGnTGuwTGBion/3sZ1q7dm07V+eZpp6X/fv3S5IGDRrUniW5zZ/mFAC0JkIgAACtbN68eaqrq1N2dnaT/YYMGSJjTDtV1fqqq6slSXa73cuVAAA8wemgAFRVVaVnnnlG/fv3V3h4uLp06aL09HS99957qq2tdepbXFysxx57TH369FFISIi6deumzMxMffbZZ44+K1ascFrw4ODBg5o4caJiYmLUtWtXpaWlOf11vn5BjLNnz2rLli2O+wUFBbX5tuudOHFCM2bM0JVXXim73a5evXrpjjvu0FtvvaVz5855VIM7Ll4EZPv27UpOTlZUVJTCw8M1cuRIbdmypdH9ys/P14QJE9S1a1dHW0lJicc17t27V+PGjVN0dLQiIiI0bNgwbd682aXfxduvrKz06PFr7jn2ZPyQkBB17txZqamp2rRpU6M1uvvct4X//d//ldR6n5DV1NRoyZIlGjVqlHr06KGwsDANHDhQr732msupju6+nt3p19jzUt++cuVKSVJYWJjLQic2m01Tp051qs2d15m7++rPc8rd17An80Jy//UOwAIMAL+SlZVlsrKyPLrPQw89ZKKjo8369etNRUWFKSoqMjNnzjSSzKZNmxz9CgoKTO/evU1sbKxZvXq1KSsrM//4xz/M8OHDTWhoqNm6davTuBkZGUaSycjIMFu3bjXl5eVmw4YNJiwszNx4440udURERJjbbrutwRrbctuFhYUmMTHR9OjRw+Tm5pozZ86YoqIiM2vWLCPJzJkzp0U1uCMpKclERESYIUOGOOrcvn27GTRokAkJCTEfffRRg/s1fPhws2nTJnP27FmTl5dnAgMDTXFxsUc17t+/38TExJj4+Hizfv16U1ZWZnbt2mVGjx5t+vTpY+x2u0u99ds/d+6cx4+fMU0/x82NHxsba3Jzc01paanJz883mZmZxmazmfnz5zc4hrvzrjlLliwxnv532bNnTyPJ/O1vf/N4e0lJSSY+Pt6pLTc310gyzz//vDl58qQpLi42v/71r01AQICZOXOmU193X8/u9jOm4eelqfbi4mIjydx///2ONnfniSf7akzHnFM7duwwktz6ubh2T17DnjxWLXm9N6cl/98A6BCWEgIBP9OS/5QTExPNrbfe6tLer18/pzeD999/v5FkFi5c6NSvsLDQ2O12M3jwYKf2+jdOubm5LjVKMsXFxU7tTb2Za8ttT5061UgyS5YscdnumDFjHG9OPa3BHUlJSUaS2bFjh1P7rl27jCSTlJTU4H6tWbOmwfE8qTE7O9tIMsuWLXPqe/ToUWO3290Oge4+fsa07A17/fiLFi1y6ltZWWni4uJMWFiYKSoqchnD3XnXnEsJgR9//LFH9zOm8RA4YsQIl75TpkwxwcHBprS01NHm7uvZ3X7GtE4IdHeeeLKvxnTMOVUfAjMyMhqt6+uvv24wBHryGvbksWrJ6705hEDAZy3ldFAAGjNmjLZu3aqHH35YeXl5jlPB8vPzNWLECEe/FStWKCAgwGW59R49emjAgAH69NNPdeTIEZfxb7zxRqffExISJEkFBQVu19iW216+fLkkNbhS49q1azV9+vRLqqE5ERERuvbaa53aBg4cqLi4OO3cuVOFhYUu97npppsaHMuTGtetWydJSklJceobFxenfv36uV2/u49fS9WPP3bsWKd2u92u5ORknTt3Tu+//77L/Vpj3rVUXFycJDlO0b1UaWlpTqcp1ktKSlJ1dbV2797taHP39exuv9bi7jzxZF8vtZaOOKc8ZusTKQAABuhJREFUeQ178li11usdgH8gBALQ3Llz9fbbb+vAgQNKTk5Wp06dNGbMGMcbJemb7w+Vlpaqrq5O0dHRLt/9+fvf/y7pX6sFXig6Otrp95CQEElye9n2ttx2/dihoaGKiopqkxqa09hFtbt37y5JOn78uMttERERl1RjVVWVysrKFBoaqsjIyEa33Rx3H7+Wam782NhYSVJRUZHLbZc67y7F8OHDJUm7du1qlfFKS0v1zDPPaODAgercubPjOX3yySclSRUVFY6+7ryePenXGjyZJ57sa1vU4s055elxxt3HqrVe7wD8ByEQgGw2m+6991598MEHOn36tFasWCFjjDIzM/XKK69I+uYv5DExMQoKClJ1dbWMMQ3+jBw58pLqaEhbbttutys6OlqVlZUqKytrsl9b1XDixIkGV4isD3/uvkHzpEa73a6oqChVVlaqvLzcZSx3r/Hm7uNXr7HnuKXjHzt2TNI3n5J0JI888oiCgoK0bNmyJvv96Ec/UkBAgPbu3dtkv/T0dM2aNUvTpk3Tvn37VFdXJ2OM5syZI0lO88ed17Mn/VqDJ/PEk32t34/WrMWbc8rT44y7j1Vrvd4B+A9CIADFxMQ43oQGBwdr1KhRjlXxVq9e7eiXmZmpmpoap1Ur6/3qV7/S5ZdfrpqamhbXER4ervPnzzt+v/rqqzVv3rw23/b48eMlSWvWrHG57brrrtMTTzzRpjVUVlZq+/btTm2ff/65CgoKlJSUpJ49e7o9lic11p+WV3+aWL2SkhLl5+e7vU13Hz+p6ee4ufEvnIvSN59ubNy4UWFhYS6nuHlbv3799POf/1yffPKJFixY0GCf/Px8vfnmm5owYYL69+/f6Fi1tbXasmWLevTooccee0zdunVzBJ8LV66t5+7r2d1+rcWdeeLpvkr+N6fcfQ17+li11usdgJ9ou+8bAvCGlnxRPzo62gwfPtzs3LnTVFZWmmPHjplnn33WSDL/v737e2X3jeM4fq3cB36uVmj5UThyxOEUKStTThwQJaUcLGdyshWHjjhyoIS/gAMny4k4IZYjDgg5wRzIVlpSTry+R1sNs937+Bru56Oc2LX7ft/XfV1tr3S/zc3Npcfd39+rpaVFzc3N2tra0uPjoxKJhJaXl1VWVvau4UO2phGhUOjDZih9fX1yu926ubnRwcGBSkpKdHZ29r+fO9Up0Ov1KhKJKJlM6vb2VpOTk6qtrdX19XVBNeSjra1Nbrdbfr/fVnfQt9eVYqfGq6sreTyejG6Bp6enCgQCqqmpsd0dNNf8SZ/f41zHT3VyTCaTGZ0cV1ZW8pqjbOsul0Iaw6SEw2FZlqVQKKSLiwu9vLwoFotpbW1NXq9XnZ2denp6ynjPR41henp6ZIzR/Py8Hh4e9Pz8rN3dXTU2NsoYo+3t7fTYfPdzvuOkr+0Ommud2LlW6WeuqX9pDGNnD9uZq0L2ey40hgF+LbqDAn9NIR/Kx8fHCgaDam1tVVlZmTwej3w+n1ZXV/X6+poxNpFIaHp6Ws3NzbIsS9XV1ert7c34snF4ePiuDfrMzIwkvft9f39/+n3n5+fq6upSeXm5GhoatLS09G3njsfjmpqaUlNTkyzLktfr1cjIiC4vL23XYEfqC//Z2ZkCgYAqKytVWlqq7u5u7e/vf3pd2YKJnRovLi40MDCgqqqqdLv7SCQiv9+fPsfExIQ2NzffnXt0dNT2/GW7x3aP73a7FQgEtLOz8+kc5XPvc/mXEChJR0dHGhsbU0NDgyzLUmVlpXw+nxYXF/Xy8pIet7CwkLX+h4cHBYPB9DFqa2s1Pj6ucDicHpvqGpnvfs5nXLb78tn9CgQC717b29uTlN86sXOt0s9bU+Xl5e9eW1hYyFgTdXV1WY8r5b+H7c5Vvvs9X4RA4Ndad0kfPIgC4NcaGhoyxhizsbFR5EqQj/b2dhOPxwvqKorvsb6+boaHhz98bhNwMj5vgF9rg2cCAQAAAMBBCIEAAAAA4CCEQAD4Ym//t9dHPxUVFcblcpmTkxNzd3dnXC6XmZ2dLXbpAADAAUqKXQAA/DU8OwYAAH4y/hIIAAAAAA5CCAQAAAAAByEEAgAAAICDEAIBAAAAwEEIgQAAAADgIIRAAAAAAHAQQiAAAAAAOAghEAAAAAAchBAIAAAAAA5CCAQAAAAAByEEAgAAAICDEAIBAAAAwEEIgQAAAADgICXFLgDA14tGo2ZoaKjYZQB/QiwWM8YY9hTwRjQaNT6fr9hlACgAIRD4Yzo6OopdAvCn1NfXm8HBwWKXAfw4Pp+Pzxzgl3JJUrGLAAAAAAB8iw2eCQQAAAAAByEEAgAAAICDEAIBAAAAwEEIgQAAAADgIP8BOgyMM03BCsgAAAAASUVORK5CYII=", "text/plain": [ "" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "albert_encoder = tfm.nlp.encoders.build_encoder(encoder_config)\n", "albert_classifier = tfm.nlp.models.BertClassifier(network=albert_encoder, num_classes=2)\n", "\n", "tf.keras.utils.plot_model(albert_classifier)" ] }, { "cell_type": "markdown", "metadata": { "id": "m6EG_7CaZ2rI" }, "source": [ "### Load Pretrained Weights into the Classifier\n", "\n", "The provided pretrained checkpoint only contains weights for the ALBERT Encoder within the ALBERT Classifier. Weights for the Classification Head is still randomly initialized." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:45.261529Z", "iopub.status.busy": "2023-10-17T12:27:45.260851Z", "iopub.status.idle": "2023-10-17T12:27:45.624803Z", "shell.execute_reply": "2023-10-17T12:27:45.624073Z" }, "id": "7dOG3agXZ9Dx" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "checkpoint = tf.train.Checkpoint(encoder=albert_encoder)\n", "checkpoint.read(\n", " os.path.join(folder_albert, 'bert_model.ckpt')).expect_partial().assert_existing_objects_matched()" ] }, { "cell_type": "markdown", "metadata": { "id": "6xsbeS-EcCqu" }, "source": [ "## Load ELECTRA model pretrained checkpoints" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:45.628328Z", "iopub.status.busy": "2023-10-17T12:27:45.628068Z", "iopub.status.idle": "2023-10-17T12:27:48.685718Z", "shell.execute_reply": "2023-10-17T12:27:48.684634Z" }, "id": "VpwIrAR4cIBF" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2023-10-17 12:27:45-- https://storage.googleapis.com/tf_model_garden/nlp/electra/small.tar.gz\r\n", "Resolving storage.googleapis.com (storage.googleapis.com)... 172.253.114.207, 172.217.214.207, 142.251.6.207, ...\r\n", "Connecting to storage.googleapis.com (storage.googleapis.com)|172.253.114.207|:443... connected.\r\n", "HTTP request sent, awaiting response... " ] }, { "name": "stdout", "output_type": "stream", "text": [ "200 OK\r\n", "Length: 157951922 (151M) [application/octet-stream]\r\n", "Saving to: ‘small.tar.gz’\r\n", "\r\n", "\r", "small.tar.gz 0%[ ] 0 --.-KB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "small.tar.gz 10%[=> ] 16.01M 79.9MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "small.tar.gz 39%[======> ] 59.93M 150MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "small.tar.gz 63%[===========> ] 96.01M 153MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "small.tar.gz 92%[=================> ] 138.75M 167MB/s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "small.tar.gz 100%[===================>] 150.63M 173MB/s in 0.9s \r\n", "\r\n", "2023-10-17 12:27:46 (173 MB/s) - ‘small.tar.gz’ saved [157951922/157951922]\r\n", "\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "small/\r\n", "small/ckpt-1000000.data-00000-of-00001\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "small/params.yaml\r\n", "small/checkpoint\r\n", "small/ckpt-1000000.index\r\n" ] } ], "source": [ "# @title Download Checkpoint of the Selected Model { display-mode: \"form\", run: \"auto\" }\n", "electra_model_display_name = 'ELECTRA-small English' # @param ['ELECTRA-small English', 'ELECTRA-base English']\n", "\n", "if electra_model_display_name == 'ELECTRA-small English':\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/electra/small.tar.gz\"\n", " !tar -xvf \"small.tar.gz\"\n", "elif electra_model_display_name == 'ELECTRA-base English':\n", " !wget \"https://storage.googleapis.com/tf_model_garden/nlp/electra/base.tar.gz\"\n", " !tar -xvf \"base.tar.gz\"" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:48.689731Z", "iopub.status.busy": "2023-10-17T12:27:48.689450Z", "iopub.status.idle": "2023-10-17T12:27:48.695251Z", "shell.execute_reply": "2023-10-17T12:27:48.694629Z" }, "id": "fy4FmsNOhlNa" }, "outputs": [ { "data": { "text/plain": [ "'small'" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Lookup table of the directory name corresponding to each model checkpoint\n", "folder_electra_dict = {\n", " 'ELECTRA-small English': 'small',\n", " 'ELECTRA-base English': 'base'\n", "}\n", "\n", "folder_electra = folder_electra_dict.get(electra_model_display_name)\n", "folder_electra" ] }, { "cell_type": "markdown", "metadata": { "id": "rgAcf-Fl3RTG" }, "source": [ "### Construct BERT Model Using the `params.yaml`\n", "\n", "params.yaml can be used for training with the bundled trainer in addition to constructing the BERT encoder here." ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:48.698883Z", "iopub.status.busy": "2023-10-17T12:27:48.698324Z", "iopub.status.idle": "2023-10-17T12:27:48.708951Z", "shell.execute_reply": "2023-10-17T12:27:48.708368Z" }, "id": "ZNBg5xzqh0Gr" }, "outputs": [ { "data": { "text/plain": [ "{'model': {'cls_heads': [{'activation': 'tanh',\n", " 'cls_token_idx': 0,\n", " 'dropout_rate': 0.1,\n", " 'inner_dim': 64,\n", " 'name': 'next_sentence',\n", " 'num_classes': 2}],\n", " 'disallow_correct': False,\n", " 'discriminator_encoder': {'type': 'bert',\n", " 'bert': {'attention_dropout_rate': 0.1,\n", " 'dropout_rate': 0.1,\n", " 'embedding_size': 128,\n", " 'hidden_activation': 'gelu',\n", " 'hidden_size': 256,\n", " 'initializer_range': 0.02,\n", " 'intermediate_size': 1024,\n", " 'max_position_embeddings': 512,\n", " 'num_attention_heads': 4,\n", " 'num_layers': 12,\n", " 'type_vocab_size': 2,\n", " 'vocab_size': 30522}},\n", " 'discriminator_loss_weight': 50.0,\n", " 'generator_encoder': {'type': 'bert',\n", " 'bert': {'attention_dropout_rate': 0.1,\n", " 'dropout_rate': 0.1,\n", " 'embedding_size': 128,\n", " 'hidden_activation': 'gelu',\n", " 'hidden_size': 64,\n", " 'initializer_range': 0.02,\n", " 'intermediate_size': 256,\n", " 'max_position_embeddings': 512,\n", " 'num_attention_heads': 1,\n", " 'num_layers': 12,\n", " 'type_vocab_size': 2,\n", " 'vocab_size': 30522}},\n", " 'num_classes': 2,\n", " 'num_masked_tokens': 76,\n", " 'sequence_length': 512,\n", " 'tie_embeddings': True}}" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "config_file = os.path.join(folder_electra, \"params.yaml\")\n", "config_dict = yaml.safe_load(tf.io.gfile.GFile(config_file).read())\n", "config_dict" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:48.712338Z", "iopub.status.busy": "2023-10-17T12:27:48.711773Z", "iopub.status.idle": "2023-10-17T12:27:48.724602Z", "shell.execute_reply": "2023-10-17T12:27:48.724016Z" }, "id": "i-yX-KgJyduv" }, "outputs": [ { "data": { "text/plain": [ "{'vocab_size': 30522,\n", " 'hidden_size': 256,\n", " 'num_layers': 12,\n", " 'num_attention_heads': 4,\n", " 'hidden_activation': 'gelu',\n", " 'intermediate_size': 1024,\n", " 'dropout_rate': 0.1,\n", " 'attention_dropout_rate': 0.1,\n", " 'max_position_embeddings': 512,\n", " 'type_vocab_size': 2,\n", " 'initializer_range': 0.02,\n", " 'embedding_size': 128,\n", " 'output_range': None,\n", " 'return_all_encoder_outputs': False,\n", " 'return_attention_scores': False,\n", " 'norm_first': False}" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "disc_encoder_config = tfm.nlp.encoders.EncoderConfig(\n", " config_dict['model']['discriminator_encoder']\n", ")\n", "\n", "disc_encoder_config.get().as_dict()" ] }, { "cell_type": "markdown", "metadata": { "id": "1AdrMkH73VYz" }, "source": [ "### Construct a Classifier with `encoder_config`\n", "\n", "Here, we construct a Classifier with 2 classes and plot its model architecture. A Classifier consists of a ELECTRA discriminator encoder using the selected encoder config, a Dropout layer and a MLP classification head.\n", "\n", "**Note**: The generator is discarded and the discriminator is used for downstream tasks" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:48.727907Z", "iopub.status.busy": "2023-10-17T12:27:48.727540Z", "iopub.status.idle": "2023-10-17T12:27:51.168411Z", "shell.execute_reply": "2023-10-17T12:27:51.167342Z" }, "id": "98Pt-SxszAvN" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4EAAAFgCAYAAAAFLXzCAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzde3hU1b3/8c/kNrmSAEJIAkIEgcqBaPGGgohBQiQxkEMCCiKtt9b2KCLWXpVTPdoqFbViEVprba0kQQHDTRDx/OTWIkWwIAFByiUJJFxyISRAsn5/2MxhmCTMhCQzmf1+PU8enqxZs+a796zZzCezZ22bMcYIAAAAAGAFeQHergAAAAAA0HYIgQAAAABgIYRAAAAAALAQQiAAAAAAWEjQhQ0bN27USy+95I1aAKBR06dP15AhQ1plbI57AKTWPc5kZWW1yrgAcDF5eXkubS6fBB48eFALFy5sk4IAT2zatEmbNm3ydhnwgoULF+rgwYOtNj7HPdTjOGNdrX2cWbhwoQ4dOtRq4wNNYf5Z06FDhxp9f+PySWC9hhIj4E31f0VlblqPzWZrk8dhboHjjHW1xXHmscceU3Z2dqs/DnAhm83G/LOg3NxcTZgwocHb+E4gAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRAC/dCCBQtks9lks9kUGhrqN4/VHJGRkY766n9mzZrl7bKaxZ+2BWgJs2bNcrwWunfv7rU6/Om16U/bAvg6X34P5U/HAn/alpbUIiGwsrJSV155pdLS0lpiOFyiiRMnyhij5OTkZt3fk+fzUh+rtVVWVmrr1q2SpIyMDBljNGPGDC9X1Tz+tC3+gOOe982YMUPGGCUlJXm1Dn96bfrTtrR3HGPaJ395D+VPxwJ/2paW1CIh0Bijuro61dXVtcRwrSoyMlJDhw71dhk+rT09n/6G+dl+tKfXCfMK52M+tA9WO8b4y7xsT8+bP/GX+dOWglpikKioKO3du7clhoIP4PkELo7XCYDWxDGmfeJ5Q3vBdwIBAAAAwEIuOQQuXrzY6YuW1dXVDbbv379fEyZMUExMjDp37qy0tDSnv5Rc+AX/zZs3Kzk5WVFRUQoPD9eIESO0fv16R/9nn33W0f/8j39XrlzpaL/ssstcxj916pTWr1/v6BMU5P6HoWPHjnXapvMfd82aNbLZbMrPz3e0TZs2zan/uXPnJEnHjh3T9OnT1bt3b4WEhKhjx45KTU3V2rVrG92vBQUFys7OVufOnR1tpaWlkqRdu3Zp7Nixio6OVkREhIYNG6Z169a5vV3na+z5rOfJY9XU1Oipp55S//79FR4erk6dOik9PV0ffPCBamtrm1VfS/HH+Xmhc+fOKScnR7fffru6deumsLAwDRw4UK+88orjNJWTJ0+6fFn62Wefddz//Pbx48c7xi4pKdEjjzyiXr16KSQkRF26dFFmZqY+//zzRvdxU3O4vbHSce/CbfrXv/6lCRMmKCoqSp07d9Y999yjEydOaP/+/UpPT1dUVJTi4uL0wAMPqKKiwmksd+ZkvUs5fvzlL39xmdfFxcVub3NL8cf5cCGOM63DSseYpsbwdO54ur313Jlr7rDCeyh/nIMXssxxzVwgJyfHNNB8URkZGUaSOX36dIPtGRkZZsOGDaaystKsXr3ahIWFmeuuu85lnKSkJBMREWGGDBni6L9582YzaNAgExISYj755BOn/hEREebmm292GWfw4MGmc+fOLu2N9XfXnDlzjCTzzjvvOLVPnTrVSDITJkxwal+0aJFJTk52/F5UVGQSExNNbGysyc/PN2VlZaagoMBkZmYam81m5s+f73T/+v03fPhws3btWnPq1CmzadMmExgYaEpKSsyePXtMTEyMSUhIMKtWrTIVFRVm+/btZtSoUaZXr17Gbrc3azsbej49faz777/fREdHm1WrVpmqqipTXFxsZsyYYSSZtWvXelzT+PHjzfjx4z2+39atWx1zsLHtbC/zs6ltuVB+fr6RZJ577jlz/PhxU1JSYl599VUTEBBgZsyY4dQ3JSXFBAQEmK+++splnCFDhjjN98LCQtOzZ08TGxtrli1bZioqKsw///lPM3z4cBMaGmo2bNjgdP+LzWF3SDI5OTlu9W0Ojnvub2tmZqb57LPPTGVlpXn77beNJJOammoyMjLM1q1bTUVFhZk7d66RZB577DGnMTyZk54cP5KSkkxCQoLj93Pnzpnp06eb22+/3Rw/ftyj7eQ4w3GmtTRnfCsdY5oaw5O5Y4xn2+vpXHOHL76Has7847jW/o9rTby/yW2zEJifn+/UPn78eCPJZSOSkpKMJLN161an9u3btxtJJikpyam9rQ9Ux44dMyEhIWb06NGOtqqqKtOxY0fTp08fExYWZsrLyx23jRs3zvzpT39y/F4fFt99912ncaurq018fLwJCwszxcXFjvb6/bd8+fIG68nKyjKSzMKFC53aDx8+bOx2e4uGQE8fKzEx0dx0000uY/ft29fnQmB7mZ+eHsRuvfVWl/bJkyeb4OBgU1ZW5mj78MMPjSTz8MMPO/Vdt26dSUhIMGfOnHG03XvvvQ3+IaSoqMjY7XYzePBgp/aLzWF3tNcQ2F7mlTvqt2nZsmVO7QMGDDCSzP/+7/86tScmJpp+/fo5tXkyJz05fpwfAk+cOGFSUlLMo48+as6dO+fRNhrDceZi23IhjjOtO76VjjFNjeHJ3DHGs+31dK65wxffQ7VWCGwvc9Cqx7WmQmCbfSfwuuuuc/q9R48ekqTCwkKXvhEREbr66qud2gYOHKj4+Hht27ZNRUVFrVfoRXTq1El33HGHVq9e7Ti9aMmSJbrhhhv0gx/8QKdPn9b7778vSTp+/Lg++eQTZWZmOu6/aNEiSdKYMWOcxrXb7UpOTtbp06f14Ycfujzu9ddf32A9K1eulCSlpKQ4tcfHx6tv377N3MqGefpYo0eP1oYNG/Tggw9q06ZNjtMXCgoKdOutt7ZobZfKX+bn+dLS0pxOMa6XlJSks2fPaseOHY62UaNGaeDAgXrrrbd07NgxR/uLL76o//qv/1JwcLCjbfHixQoICHBZ/rpbt24aMGCAtmzZokOHDrk8bmNz2J/547y69tprnX6Pj49vsD0hIcFlOz2Zk805fhQUFOiGG25QQECAXn75ZQUGBnq8fa3JH+cDxxnv8sc51RBP5k49d7e3uXPNU/76Hsof56BVjmttFgKjo6Odfg8JCZGkBpfQjYmJaXCMrl27SpKOHj3awtV5ZsqUKaqtrdVf//pXSdKf//xnTZkyRXfddZcCAwP1zjvvSJLeffddpaWlKTIyUtI353eXlZUpNDRUUVFRLuPGxsZKUoPfXYmIiHBpq6mpUUVFhUJDQx2Pcb76/dUSmvNYc+bM0dtvv619+/YpOTlZHTp00OjRox1B2Jf40/ysV1ZWpqeeekoDBw5Ux44dHeeRP/HEE5Kkqqoqp/7Tpk1TVVWVXn/9dUnS7t279fHHH+vBBx909Kmfw3V1dYqOjnY5H/4f//iHJGnPnj0u9TQ0h/2dP86rDh06OP0eEBCgwMBAhYeHO7UHBga6bKcnc9LT48eJEyc0duxYde/eXStWrNBf/vKXltjcFuWP84HjjHf545xqjDtz53zubO+lzDVP+PN7KH+cg1Y5rvnk6qDHjh2TMcalvX5ynP9iCQgI0JkzZ1z6njx5ssGxbTbbJdc3ZswYderUSX/+859VUlKiTZs2aezYsYqNjdWoUaP08ccfq6ioSH/60580ZcoUx/3sdruio6NVXV3tsmCCJB05ckTSN38RcIfdbldUVJSqq6tVWVnpcvvx48ebuYUt81g2m0333HOPPvroI508eVKLFy+WMUaZmZl66aWXWqy2tubr87Neenq6nnnmGT3wwAPavXu36urqZIzR7NmzJcllGyZNmqTY2Fi99tprqqmp0W9+8xvde++96tixo6OP3W5XTEyMgoKCdPbsWRljGvwZMWJEi22HVbSXeXUpPJmTnh4/goKC9NFHH2nJkiUaOHCgHnjgAW3evLnNtq2ltZf5wHGm/fD1OXWxMdyZO+dzZ3vbaq7xHuobvj4H61nluOaTIbC6utrlP+8vvvhChYWFSkpKUlxcnKM9Li5Ohw8fdupbXFysAwcONDh2eHi406Tq16+f5s2b51F9ISEhmjBhgj7//HP97Gc/U0ZGhsLCwiRJ99xzj2pra/X000+rqKhIt912m9N9x40bJ0latmyZU3tNTY3WrFmjsLAwl1MFmpKamirp/04zqFdaWqqCggKPtqulHysmJka7du2SJAUHB+v22293rHh04fa3J74+P4OCgrRjxw6tX79e3bp10yOPPKIuXbo4DpCnT59u8H52u10PP/ywjh49qt/85jd655139Oijj7r0y8zM1Llz5xpcYe3Xv/61Lr/8csdKuHCfr8+rS1VbW+vRnPT0+BEVFaWEhARFRkbqgw8+UGRkpMaOHeszpxd5ytfnA8eZ9sfX59TFxnB37tRzd3vbaq7xHsr356DVjms+GQKjo6P105/+VBs3btSpU6f02WefafLkyQoJCdErr7zi1HfUqFEqLCzUa6+9psrKSu3du1ePPvpoo6dCfvvb39bu3bt18OBBbdy4Ufv27dOwYcM8rvGee+6RJM2fP9/p076xY8cqKipK8+fP16RJkxQQ4LyLn3/+eSUmJmratGlaunSpKioqtHv3bt19990qKirSK6+84jgt1B3PPfecOnXqpGnTpmn16tWqrKzUzp07NXny5AZPObgUzXms733ve9q+fbtqamp09OhRvfDCCzLGuITj9qQ9zM/AwEDdeuutKi4u1osvvqjS0lKdPn1aa9eu1dy5cxu938MPP6ywsDD9/Oc/18iRI9WnTx+XPs8//7x69+6t7373u1qxYoXKysp0/PhxvfHGG/rlL3+pWbNmXdLSzFbVHubVpWjOnGzu8aNXr15auHChSkpKlJmZqZqamtbYpFbVHuYDx5n2xdfnlDtjuDN3PN3etpprvIfy/TkoWey45sEqMg1atGiRkeT0M2nSJLNx40aX9p/97GfGfPMZqtPPmDFjHOPVr/K2c+dOk5KSYqKiokxYWJgZPny4Wbduncvjnzx50tx///0mLi7OhIWFmaFDh5rNmzebwYMHO8Z/8sknHf137dplhg0bZiIiIkyPHj3MnDlz3N7WC1155ZXm8ssvN3V1dU7t9SuA7tixo8H7lZaWmmnTppnExEQTHBxsoqOjTUpKilmzZo2jT0P7r7HnpaCgwIwdO9Z06NDBsTTv0qVLTXJysuN+9913n1vb1Njz2ZzH+vzzz81DDz1kvvWtb5nw8HDTqVMnc+ONN5r58+e77DN3NGfVvoiICJftefHFF9vl/GxoWxr7+fLLL01JSYl56KGHTI8ePUxwcLCJjY01U6dONT/+8Y8d/Rpa9eyBBx5ocLXH8x07dsxMnz7dXHHFFSY4ONh06dLFjBo1yqxevdrRx5M5fDHysdVBrXTca2ybNm/e7NL+/PPPm08//dSl/emnnzbGGI/mpDvHj3fffdflsWbPnt1gzecfx5rCcYbjTGvxZHwrHWM8HcOduePp9roz19zhy++hPJ3fHNf847jWJpeIaCkXXu8JqNfcpdtbkhXm55tvvtmsJbFbk6+FwJZmhXnVXnCcaRtWPM609vhN8ac55c7c8aftbSnenH/GWOM58cXjmk9cIgJA+zB37lxNnz7d22UA8GMcZ9BczB34qvY2NwmBgMX9/ve/17hx41RZWam5c+fqxIkTys7O9nZZAPwIxxk0F3MHvqq9z02fCYGzZs2SzWbTtm3bdPjwYdlsNv385z9vs8e/8HodDf3MnDmzzeppDVbYxtbi7fnZ2hYvXqyOHTvqd7/7nRYsWMCCC23E2/OKY4Jv8fZ8aG0cZ9qet+dUSx1j3J07rbm9HC+bx9tzsLW15+OazRjni13k5uZqwoQJDV7HA/CmrKwsSVJeXp6XK0Fbs9lsysnJabW/sHHcQz2OM9bV2seZ1h4faArzz5qaeH+T5zOfBAIAAAAAWh8hEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFBDV2Q1ZWVlvWAVzUpk2bJPnP3KytrdXu3buVkJCgDh06eLscyH/mFprP344z8C2zZ89WXl6et8vwOWVlZSosLFT//v1ls9m8XY7fYv5Zz6FDhxq9zSUE9ujRQ+PHj2/VgoDmuPHGG71dQos6deqU9u7dqx07digyMlIJCQmKj49Xp06d+E/wAuPHj1ePHj1abXyOe/jyyy8l+d9xBu5r7eMMx5j/Y4zR8ePHVVhYqMOHD6uyslJhYWHq0aOHIiMjvV2eX2L+WVP37t0bfe5txhjTxvUA+Le6ujpt3bpV+fn5ys3N1ZdffqnLLrtMqampysrK0qhRo2S3271dJuD3srOzJUm5ublergTwT7W1tdq4caPy8vL03nvv6fDhw+rVq5fuvPNOZWVl6aabblJAAN9SAtpIHiEQ8CH79u1Tfn6+8vLytGHDBoWFhem2225TVlaWxo4dy2mjQCshBAItr7q6WuvWrVN+fr5ycnJ05MgRXXXVVUpPT1daWppuvvlmznwBvIMQCPiqAwcOaOXKlcrPz9eHH36owMBADR06VGlpacrOzlZcXJy3SwT8BiEQaBlVVVVas2aN8vLytGTJEpWXl+uqq65SVlaWJk6cqP79+3u7RACEQKB9OH78uJYuXaqlS5dqxYoVqqqq0pAhQ5Senq5x48apb9++3i4RaNcIgUDznf9/1PLly1VdXa0bb7xRWVlZGj9+vBISErxdIgBnhECgvTl9+rQ++ugj5eXlKT8/XydPnuT0GuASEQIBz5SUlGjFihXKy8tzOVtl4sSJio2N9XaJABpHCATas6a+aJ+enq5bb71VQUGNXgkGwL8RAoGL+/rrr/XBBx/wvXWg/SMEAv5kx44dysvLY6VRwEOEQKBhO3bs0NKlS5Wfn6/169erU6dOGjNmjNLT03XHHXcoIiLC2yUC8BwhEPBXTa00mpGRoejoaG+XCPgMQiDwjfMvXbRgwQIVFBSoS5cuGj16tLKysjR69GgFBwd7u0wAl4YQCFjBwYMHtWLFCsdKowEBARo2bBgrjQL/RgiElV3sGn581xzwO4RAwGoaWmn0mmuuUVpamu666y7169fP2yUCbY4QCKuprq7W6tWrtXTpUi1evFhHjx51WmRs6NCh3i4RQOshBAJWVr/SaGNvAvjrL6yCEAgraOoafvwRELAUQiCAb5x/OtD777+vQ4cOqWfPnsrIyGClUfg9QiD8Vf3ZH3l5eVq9erXOnTvHNfwAEAIBNKx+pdG8vDzt3LlTnTt31h133MFKo/BLhED4kwu/B841/ABcgBAI4OJYaRT+jhCI9o5r+AHwACEQgGcu/AuzMUY33HCDsrKylJWVpfj4eG+XCHiMEIj2qP6MjaVLl2rLli1cww+AuwiBAJrvxIkT+uijj5Sfn69Fixax0ijaLUIg2oPzr+H37rvvavfu3erevbvuuOMOpaWlcQ0/AO4iBAJoGaw0ivaMEAhfdf6iXQsXLlRhYaESExOVnp7ONfwANBchEEDLY6VRtDeEQPiSxq7hl5WVpfT0dA0ePNjbJQJo3wiBAFpfYyuN8r0V+ApCILzt/Gv4LV68WKdOneL0egCthRAIoG2x0ih8ESEQ3nDs2DEtW7aMa/gBaGuEQADeU1JSohUrVigvL0+rVq1SbW2t4w0QK42iLREC0VYauobfyJEjlZ6errFjx6pr167eLhGA/yMEAvAN5680euGpUBMnTlT//v29XSL8GCEQrYlr+AHwMYRAAL7n/JVGlyxZoiNHjrDSKFoVIRAtrbFr+GVlZWnUqFGy2+3eLhGAdRECAfi281caXbRokQ4ePKiePXsqJSWF62KhxRACcakauoZfjx49lJqayrEKgK8hBAJoXy786zorjaIlEALRHFzDD0A7RQgE0H5duNJoaGiokpOTlZWVpTvvvFMxMTHeLhHtBCEQ7uIafgD8ACEQgH9gpVFcCkIgmtLUNfzuvvtu9e3b19slAoAnCIEA/A8rjcJThEBcqKlr+PGHJQDtHCEQgH87/9St+pVGr7jiCqWlpfGdHTgQAiFJBw4c0MqVK5Wfn6+VK1cqKCiIa/gB8EeEQADWUb+Iw9KlS/Xee+/pq6++0uWXX67Ro0ezep/FEQKt68LvFp9/Db9x48YpKirK2yUCQEsjBAKwrsau45Wenq7U1FRFRkZ6u0S0EUKgtTS2yjDX8ANgEYRAAJCkr7/+Wh988IHy8vK0ceNG2e12Vhq1EEKgfzv/Gn5//etftWfPHqdr+KWmpiooKMjbZQJAWyEEAsCFmlppdPz48UpISPB2iWhhhED/c/41/PLy8lRUVMT3gQHgG4RAAGjKyZMntXr1asdKoxUVFY5rgk2YMEHf+ta3vF0iWgAh0D+cPn1aH330kfLy8pSfn6+TJ09yDT8AcEUIBAB3VVdXa926dcrPz1dubq6Ki4v5ZKEdeuedd/SHP/xBdXV1jraCggJJUr9+/RxtAQEBuu+++zRp0qQ2rxHuu/APNVzDDwAuihAIAM1x/kqj77//vvbs2cNKo+3Etm3bdPXVV7vV9/PPP1dSUlIrVwRPnX8NvwtP2eYafgBwUYRAAGgJrDTavvTv39/x6V9j+vTpoz179rRRRbiYAwcOaNGiRVq6dKk++eQTp2v4jRs3Tl26dPF2iQDQXhACAaClNbbSKBec9h3/8z//o//+7//W2bNnG7w9ODhYM2fO1E9/+tM2rgznu/AafuHh4RoxYgTX8AOAS0MIBIDWVFpaquXLl7f4SqO1tbUqLi5mpdJm2rdvn/r06aOm/gvcs2eP+vTp04ZV+Yf9+/crISGh2adD13+qnpeXp507dzpdwy8lJUUhISEtXDEAWA4hEADayqlTp/Txxx8rLy9PS5YsUXl5ebNXGv3kk0+UlZWlP//5zxo9enQrVu2/Bg8erK1bt7oEQZvNpm9/+9v67LPPvFRZ+7VkyRJNnTpVubm5uv322926T11dnTZs2NDo92u5hh8AtLi8AG9XAABWERERofT0dL399ts6cuSIVq9erZEjR+qNN97QVVddpd69e+vRRx/VunXrmvyESpIWL16sY8eOKTU1VT/5yU907ty5NtoK/zFlyhQFBga6tAcGBmrKlCleqKj9Onv2rB5//HGNGzdOZWVlev/995vsX1tbq3Xr1unRRx9V9+7dNWzYMOXl5Sk1NVWffvqp9u/frzfeeEPp6ekEQABoBXwSCABe1tAnIT169FBqamqjK412795dhw8flvRNaElKStJ7772nXr16eWEL2qejR48qLi7O6VIR0jeXhjh8+LC6devmpcral4MHDyorK0ufffaZamtrJUmdOnVSSUmJAgL+72/NTV3DLzs7W1dddZW3NgEArIbTQQHA1zS00mhycrLS0tKUmZmpgoICXXvttU73CQ4OVnh4uP7617/qjjvu8FLl7c+IESP06aefOsJLYGCgbrnlFn388cderqx9WLZsmSZNmqSqqiqXRXbWrVunAQMGOK7ht2jRIlVVVWnIkCFKT09XZmamrrzySi9VDgCWRggEAF/21VdfadGiRVq8eLE2bdqk0NBQJSUlacuWLTpz5oxT34CAANXV1em//uu/9Jvf/IbrFLrhzTff1IMPPugUAufPn6/vfOc7Xq7Mt507d07PPPOMnnnmGdlsNpdPU0NCQpSYmKh9+/bJZrPptttu07hx41gdFwB8AyEQANqL4uJiLVmyRDNnzlRxcXGj/epPD124cKESExPbsML2p7y8XJdddpnjU6zg4GAdPXpUMTExXq7Mdx0+fFjjx4/X5s2bHeG5IdHR0ZozZ47GjBnD/gQA38LCMADQXnTr1k3JyclNBkDpm0U3vvjiC11zzTVatmxZG1XXPnXo0MGx+mRQUJDuuOMOAksTPv74Y8cn0U0FQEkqKyvTgAED2J8A4IMIgQDQjrz//vturZZ49uxZVVRUKC0tTY888kijF0WHNHnyZNXW1qq2tlaTJk3ydjk+qba2VjNnztTIkSN14sQJt+ZTSEiIFi1a1AbVAQA8xbrLAHxWbm6ut0vwOfPmzbvoJzD16r+n9dvf/lbLly/X448/rs6dO7dmee3S2bNnFRISImOMampqmHcXOHHihF5++WXt2rVLki56+ZJ6Z86c0fz58zVgwIDWLK/dys7O9nYJACyM7wQC8Fk2m83bJQBAq+DtFwAvyuOTQAA+LScnh7+Y/1thYaG++OILBQYGKiIiQiEhIQoNDVVYWJhCQkIUERGhwMBAdejQwdultjsrV66UzWZTSkqK2/ex2WzMT33zSaEkVVVVqaamRnV1dSorK5MknTp1Sn379uWai+fJzc3VhAkTvF0GAIsjBAJAOxEfH6/4+Hhvl+GXRo4c6e0S2q2OHTs6/QsA8H2EQACA5bmz2A4AAP6C1UEBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAvzBr1izZbDbZbDZ1797d2+VY2oIFCxzPRWhoqLfLcbJ8+XL17dtXQUFBbf7YkZGRjv1y4U94eLiSkpL00ksvqba2ts1rc7fOC38+++wzr9bqDl+ejwDgLYRAAH5hxowZMsYoKSnJ26VY3sSJE2WMUXJysrdLcdi7d6/uvPNO/eQnP9GRI0e8UkNlZaW2bt0qScrIyJAxRsYYlZeXa+XKlZKkxx9/XE888YRX6qvXWJ0X/kRHR3u1Tnf54nwEAG8jBAJAC4qMjNTQoUO9XQYu8Itf/EI33XSTtmzZovO6RfcAACAASURBVKioKG+X4yQqKkq33HKL5s6dK0l64403dPbs2VZ9TOYpAFhb258PAwBAG/vDH/6gsLAwb5fRpH79+kmSqqqqVFZWpssuu8zLFTXt5MmT3i4BANBMfBIIAPB7vh4AJamgoECS1KVLF58OgEOHDtVbb73l7TIAAJeAEAjAL+3atUtjxoxRdHS0wsPDNWLECK1fv96lX0lJiR555BH16tVLISEh6tKlizIzM/X55587+ixevNhpMYyCggJlZ2erc+fOjrYf//jHstlsOnXqlNavX+9ob84CJM2paf/+/ZowYYJiYmLUuXNnpaWlae/evS5jHzt2TNOnT1fv3r1lt9vVvXt3jRw5Um+99ZZOnz7dYL+QkBB17NhRqampWrt2bYP7euzYsYqOjlZERISGDRumdevWtej2NbTPS0tLPd63vqiyslKffvqpvve97yk8PNxxWuj5fHGeno/56D/zEYBFGADwUZJMTk6OR/dJSkoy0dHRZsSIEWbdunWmoqLCbN682QwaNMiEhISYTz75xNG3sLDQ9OzZ08TGxpply5aZiooK889//tMMHz7chIaGmg0bNjiNnZGRYSSZ4cOHm7Vr15pTp06ZTZs2mcDAQFNSUmKMMSYiIsLcfPPNzd7m5taUkZFhNmzYYCorK83q1atNWFiYue6665z6FhUVmcTERNOtWzeTn59vysvLTXFxsXnmmWeMJDN79mynfrGxsSY/P9+UlZWZgoICk5mZaWw2m5k/f75jzD179piYmBiTkJBgVq1aZSoqKsz27dvNqFGjTK9evYzdbm+R7Wtqn3sqISHBBAYGNuu+52vO/Ny6dauR1OBPv379zHvvvedyH2/M06bqlGT++Mc/Nng/5uPF5eTkGN5+AfCyXI5CAHxWc0OgJLNx40an9u3btxtJJikpydF27733GknmnXfecepbVFRk7Ha7GTx4sFN7/RvA5cuXN/r4lxoCm1tTfn6+U/v48eONJKc3plOnTm10n44ePdrxpru+37vvvuvUp7q62sTHx5uwsDBTXFxsjDEmKyvLSDILFy506nv48GFjt9td3nS3xj73lC+EwIyMDEfb2bNnzb59+8zTTz9tbDabyczMNGfOnHHc7o152lCd9W6++eaLhkDmY+MIgQB8QC6ngwLwO6Ghobrhhhuc2gYOHKj4+Hht27ZNRUVFkr45xSsgIEBpaWlOfbt166YBAwZoy5YtOnTokMv4119/favV3tyarrvuOqffe/ToIUkqLCx0tC1atEiSlJqa6nL/FStWaNq0aU79xowZ49THbrcrOTlZp0+f1ocffihJjksbpKSkOPWNj49X3759W2z7WnOfe1tQUJASExM1c+ZM3X333Xr//ff16quvOm5vj/uM+QgAvo0QCMDv1H9P50Jdu3aVJB09elQ1NTUqKytTXV2doqOjXS6C/Y9//EOStGfPHpdxIiIiWqXuS6npwmu2hYSESJLq6uqcxg4NDW3yEgkX6xcbGytJKi4uVk1NjSoqKhQaGqrIyEiXvvX7uyW2r7X2ua+55ZZbJElr1qyR5Jv7bN26dZo6dWqTfZiPAODbuEQEAL9TVlbWYPvRo0clffNm0G63KyYmRpWVlTp9+vQlL4xRr6Hw6a7Wqql+7OjoaJWVlamioqLRN94X61d/ofVu3brJbrcrKipKFRUVqqysdHnjffz48TbbPn9hjJH0zWUipNbbZ5cyT1sC8xEAvItPAgH4ncrKSm3bts2p7YsvvlBhYaGSkpIUFxcnScrMzNS5c+caXDX017/+tS6//HKdO3fOo8cODw/XmTNnHL/369dP8+bNc/v+rVFTvXHjxkmSli9f7nLbNddco8cee8yp37Jly5z61NTUaM2aNQoLC3Ocbld/Kl/9aXj1SktLHZc8OF9rbp8/+PTTTyU5n07pi/NUkq699lotWLDAo/ucj/kIAF7k7W8lAkBj1MyFYSIiIszQoUPNpk2bTGVlZaOrgx45csT07t3bXHHFFWb58uXm5MmT5tixY2bu3LkmPDzc5bHrF4U4ffp0o48/evRoEx0dbQ4cOGA2bNhggoKCzM6dO92uv6VqevLJJ40ks3XrVkdb/SqLcXFxZunSpaa8vNwcPHjQfP/73zexsbHmX//6l1O/+tUYy8vLnVZjnDdvnmPMr776ynTq1MlpNcYdO3aYlJQU07VrV5eFOFpjn3vKFxeG+frrrx0LwyQkJJjCwkLH7d6Yp00tDFNv8ODBLou1MB8vjoVhAPgAVgcF4Ls8eZP94osvOpavT0hIMH//+9/NiBEjTGRkpAkLCzPDhw8369atc7nfsWPHzPTp080VV1xhgoODTZcuXcyoUaPM6tWrHX02btzY4DL5Ddm1a5cZNmyYiYiIMD169DBz5szxeLubW9PPfvYzY4xxaR8zZozjfqWlpWbatGkmMTHRBAcHm7i4ODNx4kSze/dupxou7BcdHW1SUlLMmjVrXOotKCgwY8eONR06dHBcCmDp0qUmOTnZUcN9993XavvcHfn5+Y1e7uD8Swx4wtMQGBER0eDj22w2ExUVZZKSksyPfvQjc+TIEZf7tuU8bazOhn7qQyDz0X2EQAA+INdmzL+/gAAAPsZmsyknJ0fZ2dneLgVwwfxEc+Tm5mrChAni7RcAL8rjO4EAAAAAYCGEQAAAAACwEEIgALSBC69B1tDPzJkzvV1mu8I+BQCgebgoDgC0Ab7/0/LYpwAANA+fBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFhIkLcLAICmbNy40dslAI1ifsJTzBkAvsBmjDHeLgIAGmKz2bxdAgC0Ct5+AfCiPD4JBOCzeJOEtpKdnS1Jys3N9XIlAAC0Pr4TCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWEuTtAgAAaEt/+9vftG3bNqe2ffv2SZLmzZvn1D5o0CDdeOONbVYbAABtgRAIALCUo0eP6qGHHlJgYKACAr45IcYYI0n64Q9/KEmqq6tTbW2tPvjgA6/VCQBAa7GZ+v/5AACwgLNnz+qyyy5TeXl5k/2ioqJUWlqqkJCQNqoMAIA2kcd3AgEAlhIcHKyJEyc2Ge6Cg4N11113EQABAH6JEAgAsJy77rpLZ86cafT2s2fP6u67727DigAAaDucDgoAsJy6ujrFx8fryJEjDd7epUsXFRcXO74zCACAH+F0UACA9QQEBGjy5MkNnu4ZEhKie++9lwAIAPBb/A8HALCkxk4JPXPmjO666y4vVAQAQNvgdFAAgGX16dNHe/fudWrr2bOn9u/f752CAABofZwOCgCwrsmTJys4ONjxe0hIiL7zne94sSIAAFofnwQCACzrq6++0pVXXunUVlBQoL59+3qpIgAAWh2fBAIArKtPnz4aNGiQbDabbDabBg0aRAAEAPg9QiAAwNKmTJmiwMBABQYGasqUKd4uBwCAVsfpoAAASyssLFSPHj1kjNGBAwfUvXt3b5cEAEBrygvydgUA4G9sNpu3S0Az9ejRw9slwEP8LRsAPEcIBIBWMG3aNA0ZMsTbZcBNH330kWw2m5KTkxvts3HjRr388svKyclpw8rQmPrnAwDgOUIgALSCIUOGKDs729tlwE314a9z585N9nv55Zd5Xn0IIRAAmocQCACwvIuFPwAA/AmrgwIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAAAAAFgIIRAAAAAALIQQCAAAAAAWQggEAAAAAAshBAIAAACAhRACAQAAAMBCCIEAAAAAYCGEQAAAAACwEEIgAPigBQsWyGazyWazKTQ01NvltCtz58517LvGflJTU9uklsjISJfHDggIUMeOHZWUlKSHH35YW7ZsaZNaAACoRwgEAB80ceJEGWOUnJzs7VL80k033dQmj1NZWamtW7dKkjIyMmSM0dmzZ7Vr1y798pe/1K5du3TttdfqO9/5jqqqqtqkJgAACIEAAJ8SGRmpoUOHXtIY9YHrwp/du3fLbrfrgQceaKFqPRcYGKjY2FhlZGTo448/1o9+9CO99dZbuuuuu2SM8Vpdba0lnmcAQPMQAgEAfqVPnz4aNmxYg7f99re/1dixY9WtW7c2rqpxv/rVr3TDDTfogw8+0IIFC7xdDgDAAgiBAAC/MnLkSD3++OMu7RUVFfrTn/6khx9+2AtVNc5ms+mHP/yhJOn111/3cjUAACsgBAKAD9i1a5fGjh2r6OhoRUREaNiwYVq3bp1Lv8WLFzstMlJQUKDs7Gx17tzZ0VZaWipJOnbsmKZPn67evXsrJCREHTt2VGpqqtauXesYb9asWY77de/eXZs3b1ZycrKioqIUHh6uESNGaP369S51uDP2s88+6xj7/NP+Vq5c6Wi/7LLLXGo5deqU1q9f7+gTFBTUIvv4j3/8oy6//HLdcsstLTJeS6rfP5s2bdLZs2d5ngEArcsAAFqUJJOTk+N2/z179piYmBiTkJBgVq1aZSoqKsz27dvNqFGjTK9evYzdbne5T0ZGhpFkhg8fbtauXWtOnTplNm3aZAIDA01JSYkpKioyiYmJJjY21uTn55uysjJTUFBgMjMzjc1mM/Pnz3caLykpyURERJghQ4aYDRs2mMrKSrN582YzaNAgExISYj755BNHX0/HjoiIMDfffLPLNgwePNh07tzZpb2x/peirq7O9O3b17z++uvNHiMnJ8c057/NrVu3GkkmIyOj0T6nT582kowkU1hY6GjneW5cc58PAIDJ5egJAC3M0xCYlZVlJJmFCxc6tR8+fNjY7fYmQ+Dy5csbHHPq1KlGknn33Xed2qurq018fLwJCwszxcXFjvakpCQjyWzdutWp//bt240kk5SU1OyxfSEELlu2zERFRZmKiopmj9GaIbCqqqrJEMjz7IoQCADNlsvpoADgZStXrpQkpaSkOLXHx8erb9++Td73+uuvb7B90aJFkqQxY8Y4tdvtdiUnJ+v06dP68MMPnW6LiIjQ1Vdf7dQ2cOBAxcfHa9u2bSoqKmr22N726quvasqUKYqMjPR2KQ2q37fBwcFOp07W43kGALQkQiAAeFFNTY0qKioUGhraYEDp2rVrk/ePiIhocMyysjKFhoYqKirK5fbY2FhJUnFxsVN7TExMg49RX8PRo0ebPbY37d69W6tWrfK5BWHOV//9zyFDhig4ONjldp5nAEBLIgQCgBfZ7XZFRUWpurpalZWVLrcfP368WWNGR0erurpaFRUVLrcfOXJEklwuk3Ds2LEGr1N39OhRSd+EhOaMHRAQoDNnzrj0PXnyZIP122y2xjatWV599VXdcsstuuqqq1p03JZSV1enOXPmSJJ+8IMfuH0/nmcAQHMRAgHAy1JTUyX932mh9UpLS1VQUNCsMceNGydJWrZsmVN7TU2N1qxZo7CwMJfTT6urq7V582anti+++EKFhYVKSkpSXFxcs8aOi4vT4cOHnfoWFxfrwIEDDdYeHh7uFCb69eunefPmXXSbG1JeXq63337bo3DV1n7yk5/o73//u8aNG6esrCyP7svzDABoDkIgAHjZc889p06dOmnatGlavXq1KisrtXPnTk2ePLnZ32F7/vnnlZiYqGnTpmnp0qWqqKjQ7t27dffdd6uoqEivvPKK45S+etHR0frpT3+qjRs36tSpU/rss880efJkhYSE6JVXXmn22KNGjVJhYaFee+01VVZWau/evXr00UcbPdX129/+tnbv3q2DBw9q48aN2rdvX6MXf7+YN998U5GRkY5A4wvq6up09OhRLVmyRMnJyXrhhRf03e9+V++8847Hn47xPAMAmsXbS9MAgL+Rh6uDGmNMQUGBGTt2rOnQoYMJCwsz1113nVm6dKlJTk52rBp53333mY0bNzp+P/+nIaWlpWbatGkmMTHRBAcHm+joaJOSkmLWrFnj0jcpKckkJCSYnTt3mpSUFBMVFWXCwsLM8OHDzbp16y5p7JMnT5r777/fxMXFmbCwMDN06FCzefNmM3jwYEf9Tz75pKP/rl27zLBhw0xERITp0aOHmTNnjkf7sl5dXZ3p06ePeeqpp5p1/ws1ZzXKiIgIl+fKZrOZ6OhoM3DgQPP973/fbNmyxeV+PM8Xx+qgANBsuTZjGvhiAACg2Ww2m3JycpSdne3tUtx29dVXq7S0VIcOHfJ2KT4rNzdXEyZMaPD7dO2FPz3P/vB8AICX5HE6KAAAAABYCCEQAAAAACyEEAgAFjZr1izZbDZt27ZNhw8fls1m089//nNvl9Ugm8120Z+ZM2d6u0yf1J6eZwBA6wvydgEAAO+ZMWOGZsyY4e0y3MJ3v5qvPT3PAIDWxyeBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFmIzxhhvFwEA/sRms3m7BMAyeBsDAB7LC/J2BQDgb3JycrxdAjw0e/ZsSdJjjz3m5UoAAGh9fBIIALC87OxsSVJubq6XKwEAoNXl8Z1AAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIYRAAAAAALAQQiAAAAAAWAghEAAAAAAshBAIAAAAABZCCAQAAAAACyEEAgAAAICFEAIBAAAAwEIIgQAAAABgIUHeLgAAgLZUVVWlmpoap7YzZ85Ikk6cOOHUbrfbFR4e3ma1AQDQFgiBAABL+eMf/6gf/vCHDd7WqVMnp99fe+01/eAHP2iLsgAAaDM2Y4zxdhEAALSVkpISxcXFqba2tsl+gYGBKioqUpcuXdqoMgAA2kQe3wkEAFhKly5ddNtttykwMLDRPoGBgUpOTiYAAgD8EiEQAGA5kydPVlMnwhhjNHny5DasCACAtsPpoAAAy6moqFCXLl1cFoipFxISopKSEnXo0KGNKwMAoNVxOigAwHqioqKUlpam4OBgl9uCgoJ05513EgABAH6LEAgAsKRJkybp3LlzLu21tbWaNGmSFyoCAKBtcDooAMCSzpw5o8suu0wVFRVO7ZGRkSotLZXdbvdSZQAAtCpOBwUAWFNISIjGjx+vkJAQR1twcLCys7MJgAAAv0YIBABY1t13360zZ844fj979qzuvvtuL1YEAEDr43RQAIBl1dXVKTY2VqWlpZKkzp0768iRI01eQxAAgHaO00EBANYVEBCgSZMmKSQkRMHBwZo8eTIBEADg9wiBAABLu+uuu3TmzBlOBQUAWEaQtwsA0LJeeuklbdy40dtlAO1KeHi4JOnFF1/0ciVA+zJkyBBNnz7d22UA8BCfBAJ+ZuPGjdq0aZO3ywDalZ49e6pnz54N3nbo0CEtXLiwjSsCfN+mTZv4oyPQTvFJIOCHbrzxRuXl5Xm7DKDd2LFjhyRpwIABLrfl5uZqwoQJvKaAC2RlZXm7BADNRAgEAFheQ+EPAAB/xemgAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAAAAAFkIIBAAAAAALIQQCAAAAgIUQAgEAAADAQgiBAAAAAGAhhEAAAAAAsBBCIAAAAABYCCEQAAAAACyEEAgAcFiwYIFsNptsNptCQ0O9XY5f2Lx5s6ZOnarExESFhYWpU6dO+o//+A/953/+p373u99p7969kqRZs2Y59n337t29XLX7vvzyS02YMEHdunVTUFCQYxtiYmK8XZpXREZGOvZB/c+sWbOc+nTv3t2lz89//vM2r5XXO2BdhEAAgMPEiRNljFFycrLLbZWVlbryyiuVlpbWKo/d2uO3tbq6Oj3xxBO66aab1LVrV61YsUInT57Ul19+qdmzZ6u8vFwPP/yw+vTpo3PnzmnGjBkyxigpKcnbpTtp6nnZv3+/hgwZoi+//FLvv/++ysvLVV5ertzcXAUEeP8thjfmVGVlpbZu3SpJysjIkDFGM2bMcOpz6NAhff3115Kkm2++WcYYPfvss21WY72mXu8A/Jv3j9AA8G+RkZEaOnSot8tAI4wxqqurU11dXbPHaOo5bonxfckvfvELzZo1S6+//rpeeOEF9e/fX3a7XbGxsbr99tu1cuVKpaamervMi2rqeZk3b57Kyso0Z84c3XTTTQoPD1dUVJSysrJ0/PjxNqnPSnMKAFpKkLcLAAC0D1FRUY5TF9vj+G1p165d+tWvfqXBgwfrgQceaLBPYGCgfvGLX2jFihVtXJ1nmnpe9uzZI0kaNGhQW5bkNn+aUwDQkgiBAAC0sHnz5qmurk5ZWVlN9hsyZIiMMW1UVcs7e/asJMlut3u5EgCAJzgdFIBqamr01FNPqX///goPD1enTp2Unp6uDz74QLW1tU59S0pK9Mgjj6hXr14KCQlRly5dlJmZqc8//9zRZ/HixU4LHuzfv18TJkxQTEyMOnfurLS0NKe/ztcviHHq1CmtX7/ecb+goKBWf+x6x44d0/Tp09W7d2/Z7XZ1795dI0eO1FtvvaXTp097VIM7LlwEZPPmzUpOTlZUVJTCw8M1YsQIrV+/vtHtKigoUHZ2tjp37uxoKy0t9bjGXbt2aezYsYqOjlZERISGDRumdevWufS78PGrq6s92n8Xe449GT8kJEQdO3ZUamqq1q5d22iN7j73reH//b//J6nlPiE7d+6ccnJydPvtt6tbt24KCwvTwIED9corr7ic6uju69mdfo09L/XtS5YskSSFhYW5LHRis9k0depUp9rceZ25u63+PKfcfQ17Mi8k91/vACzAAPAr48ePN+PHj/foPvfff7+Jjo42q1atMlVVVaa4uNjMmDHDSDJr16519CssLDQ9e/Y0sbGxZtmyZaaiosL885//NMOHDzehoaFmw4YNTuNmZGQYSSYjI8Ns2LDBVFZWmtWrV5uwsDBz3XXXudQRERFhbr755gZrbM3HLioqMomJiaZbt24mPz/flJeXm+LiYvPMM88YSWb27NnNqsEdSUlJJiIiwgwZMsRR5+bNm82gQYNMSEiI+eSTTxrcruHDh5u1a9eaU6dOmU2bNpnAwEBTUlLiUY179uwxMTExJiEhwaxatcpUVFSY7du3m1GjRplevXoZu93uUm/9458+fdrj/WdM08/xxcaPjY01+fn5pqyszBQUFJjMzExjs9nM/PnzGxzD3Xl3MTk5OcbT/y7j4uKMJPO3v/3N48dLSkoyCQkJTm35+flGknnuuefM8ePHTUlJiXn11VdNQECAmTFjhlNfd1/P7vYzpuHnpan2kpISI8nce++9jjZ354kn22qMb86prVu3Gklu/VxYuyevYU/2VXNe7xfTnP9vAPiEXEIg4Gea859yYmKiuemmm1za+/bt6/Rm8N577zWSzDvvvOPUr6ioyNjtdjN48GCn9vo3Tvn5+S41SjIlJSVO7U29mWvNx546daqRZHJyclwed/To0Y43p57W4I6kpCQjyWzdutWpffv27UaSSUpKanC7li9f3uB4ntSYlZVlJJmFCxc69T18+LCx2+1uh0B3958xzXvDXj/+u+++69S3urraxMfHm7CwMFNcXOwyhrvz7mIuJQT+/e9/9+h+xjQeAm+99VaXvpMnTzbBwcGmrKzM0ebu69ndfsa0TAh0d554sq3G+Oacqg+BGRkZjdb19ddfNxgCPXkNe7KvmvN6vxhCINBu5XI6KACNHj1aGzZs0IMPPqhNmzY5TgUrKCjQrbfe6ui3ePFiBQQEuCy33q1bNw0YMEBbtmzRoUOHXMa/7rrrnH7v0aOHJKmwsNDtGlvzsRctWiRJDa7UuGLFCk2bNu2SariYiIgIXX311U5tAwcOVHx8vLZt26aioiKX+1x//fUNjuVJjStXrpQkpaSkOPWNj49X37593a7f3f3XXPXjjxkzxqndbrcrOTlZp0+f1ocffuhyv5aYd80VHx8vSY5TdC9VWlqa02mK9ZKSknT27Fnt2LHD0ebu69ndfi3F3XniybZeai2+OKc8eQ17sq9a6vUOwD8QAgFozpw5evvtt7Vv3z4lJyerQ4cOGj16tOONkvTN94fKyspUV1en6Ohol+/+/OMf/5D0f6sFni86Otrp95CQEElye9n21nzs+rFDQ0MVFRXVKjVcTGMX1e7atask6ejRoy63RUREXFKNNTU1qqioUGhoqCIjIxt97Itxd/8118XGj42NlSQVFxe73Hap8+5SDB8+XJK0ffv2FhmvrKxMTz31lAYOHKiOHTs6ntMnnnhCklRVVeXo687r2ZN+LcGTeeLJtrZGLd6cU54eZ9zdVy31egfgPwiBAGSz2XTPPffoo48+0smTJ7V48WIZ3fU8vQAABfdJREFUY5SZmamXXnpJ0jd/IY+JiVFQUJDOnj0rY0yDPyNGjLikOhrSmo9tt9sVHR2t6upqVVRUNNmvtWo4duxYgytE1oc/d9+geVKj3W5XVFSUqqurVVlZ6TKWu9d4c3f/1WvsOW7u+EeOHJH0zackvuShhx5SUFCQFi5c2GS/H/3oRwoICNCuXbua7Jeenq5nnnlGDzzwgHbv3q26ujoZYzR79mxJcpo/7ryePenXEjyZJ55sa/12tGQt3pxTnh5n3N1XLfV6B+A/CIEAFBMT43gTGhwcrNtvv92xKt6yZcsc/TIzM3Xu3DmnVSvr/frXv9bll1+uc+fONbuO8PBwnTlzxvF7v379NG/evFZ/7HHjxkmSli9f7nLbNddco8cee6xVa6iurtbmzZud2r744gsVFhYqKSlJcXFxbo/lSY31p+XVnyZWr7S0VAUFBW4/prv7T2r6Ob7Y+OfPRembTzfWrFmjsLAwl1PcvK1v3756+umn9dlnn+nNN99ssE9BQYHeeOMNZWdnq3///o2OVVtbq/Xr16tbt2565JFH1KVLF0fwOX/l2nruvp7d7ddS3Jknnm6r5H9zyt3XsKf7qqVe7wD8ROt93xCANzTni/rR0dFm+PDhZtu2baa6utocOXLEzJw500gyzz77rKPfkSNHTO/evc0VV1xhli9fbk6ePGmOHTtm5s6da8LDw10WfGhs0Ygnn3yywcVQRo8ebaKjo82BAwfMhg0bTFBQkNm5c2erP3b9SoFxcXFm6dKlpry83Bw8eNB8//vfN7GxseZf//pXs2pwR1JSkomOjjbJyckerQ564XbV86TGr776ynTq1MlptcAdO3aYlJQU07VrV49XB73Y/jOm6ef4YuPXr+RYXl7utJLjvHnz3NpHjc27i2nOwjD1fvzjH5vg4GDz5JNPmoKCAlNTU2MOHTpkfv/735u4uDgzdOhQU1lZ6XSfhhaGue2224wk88ILL5iSkhJTVVVlPv74Y3P55ZcbSWb16tWOvu6+nt3tZ0zLrg56sXniybYa45tz6lIWhvHkNezJvmrO6/1iWBgGaLdYHRTwN835T/nzzz83Dz300P9v745VGomiMABPYFNojIIgIppCOzvLsRALwRQ2FgYFEQQLW7ES9A1SWVj5BlrY2FopGOxsRK21Mj6AjWerDbhRN9ldjDrfB2kyNzNn5s4l8xPuTYyPj0d3d3f09/dHmqaxv78fz8/PL9o+Pj7G5uZmjI2NRT6fj4GBgZidnX3xsHF+ft60DPr29nZERNP7c3Nzjc9dX1/H1NRUFAqFKJVKsbe392HHrtfrsbGxEaOjo5HP52NoaCiWlpbi9va27Rra8euB/+rqKsrlchSLxejq6orp6ek4Ozt797zeCibt1HhzcxPz8/PR29vbWO7++Pg4ZmZmGsdYW1uLo6OjpmMvLy+3ff3e6uN299/X1xflcjlOTk7evUat9P2f/EsIjIi4uLiIlZWVKJVKkc/no1gsRpqmsbu7G09PT4121Wr1zfofHh5ifX29sY/BwcFYXV2Nra2tRttfq0a2Op5bafdWv7zXX+VyuWnb6elpRLR2n7RzrhGf754qFApN26rV6ot7Ynh4+M39RrQ+htu9Vq2O91YJgfBlHeQiXpmIAnxZlUolSZIkOTw87HAltGJiYiKp1+t/taooH+Pg4CBZXFx8dd4mZJnvG/iyDs0JBAAAyBAhEAAAIEOEQID/7Pf/9nrt1dPTk+RyueTy8jK5v79PcrlcsrOz0+nSAYAM+NHpAgC+G3PHAIDPzC+BAAAAGSIEAgAAZIgQCAAAkCFCIAAAQIYIgQAAABkiBAIAAGSIEAgAAJAhQiAAAECGCIEAAAAZIgQCAABkiBAIAACQIUIgAABAhgiBAAAAGfKj0wUA/1+tVksqlUqny4Bv4e7uLkmSxJiC39RqtSRN006XAfwFIRC+mcnJyU6XAN/KyMhIsrCw0Oky4NNJ09R3DnxRuYiIThcBAADAhzg0JxAAACBDhEAAAIAMEQIBAAAyRAgEAADIkJ8vxFjPHciukgAAAABJRU5ErkJggg==", "text/plain": [ "" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "disc_encoder = tfm.nlp.encoders.build_encoder(disc_encoder_config)\n", "elctra_dic_classifier = tfm.nlp.models.BertClassifier(network=disc_encoder, num_classes=2)\n", "tf.keras.utils.plot_model(elctra_dic_classifier)" ] }, { "cell_type": "markdown", "metadata": { "id": "aWQ2FKj64X5U" }, "source": [ "### Load Pretrained Weights into the Classifier\n", "\n", "The provided pretrained checkpoint contains weights for the entire ELECTRA model. We are only loading its discriminator (conveninently named as `encoder`) wights within the Classifier. Weights for the Classification Head is still randomly initialized." ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2023-10-17T12:27:51.172362Z", "iopub.status.busy": "2023-10-17T12:27:51.172065Z", "iopub.status.idle": "2023-10-17T12:27:51.379102Z", "shell.execute_reply": "2023-10-17T12:27:51.378428Z" }, "id": "99pznFJszQfV" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "checkpoint = tf.train.Checkpoint(encoder=disc_encoder)\n", "checkpoint.read(\n", " tf.train.latest_checkpoint(os.path.join(folder_electra))\n", " ).expect_partial().assert_existing_objects_matched()" ] } ], "metadata": { "colab": { "name": "load_lm_ckpts.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" } }, "nbformat": 4, "nbformat_minor": 0 }