{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Tce3stUlHN0L" }, "source": [ "##### Copyright 2022 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2024-04-20T11:08:54.061808Z", "iopub.status.busy": "2024-04-20T11:08:54.061554Z", "iopub.status.idle": "2024-04-20T11:08:54.065665Z", "shell.execute_reply": "2024-04-20T11:08:54.065098Z" }, "id": "tuOe1ymfHZPu" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "36EdAGhThQov" }, "source": [ "# Learning to Rank with Decision Forests\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View on GitHub\n", " \n", " Download notebook\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "id": "kvvDY0LVhuaW" }, "source": [ "Welcome to the **Learning to Rank Colab** for **TensorFlow Decision Forests** (**TF-DF**).\n", "In this colab, you will learn how to use **TF-DF** for ranking.\n", "\n", "This colab assumes you are familiar with the concepts presented the [Beginner colab](beginner_colab.ipynb), notably about the installation about TF-DF.\n", "\n", "In this colab, you will:\n", "\n", "1. Learn what a ranking model is.\n", "1. Train a Gradient Boosted Trees models on the LETOR3 dataset.\n", "1. Evaluate the quality of this model." ] }, { "cell_type": "markdown", "metadata": { "id": "jK9tCTcwqq4k" }, "source": [ "## Installing TensorFlow Decision Forests\n", "\n", "Install TF-DF by running the following cell." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:08:54.069169Z", "iopub.status.busy": "2024-04-20T11:08:54.068765Z", "iopub.status.idle": "2024-04-20T11:08:57.618544Z", "shell.execute_reply": "2024-04-20T11:08:57.617595Z" }, "id": "Pa1Pf37RhEYN" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting tensorflow_decision_forests\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Downloading tensorflow_decision_forests-1.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.0 kB)\r\n", "Requirement already satisfied: numpy in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (1.26.4)\r\n", "Requirement already satisfied: pandas in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (2.2.2)\r\n", "Requirement already satisfied: tensorflow~=2.16.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (2.16.1)\r\n", "Requirement already satisfied: six in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (1.16.0)\r\n", "Requirement already satisfied: absl-py in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (1.4.0)\r\n", "Requirement already satisfied: wheel in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (0.41.2)\r\n", "Collecting wurlitzer (from tensorflow_decision_forests)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Downloading wurlitzer-3.0.3-py3-none-any.whl.metadata (1.9 kB)\r\n", "Requirement already satisfied: tf-keras~=2.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (2.16.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: astunparse>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (1.6.3)\r\n", "Requirement already satisfied: flatbuffers>=23.5.26 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (24.3.25)\r\n", "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (0.5.4)\r\n", "Requirement already satisfied: google-pasta>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (0.2.0)\r\n", "Requirement already satisfied: h5py>=3.10.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (3.11.0)\r\n", "Requirement already satisfied: libclang>=13.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (18.1.1)\r\n", "Requirement already satisfied: ml-dtypes~=0.3.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (0.3.2)\r\n", "Requirement already satisfied: opt-einsum>=2.3.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (3.3.0)\r\n", "Requirement already satisfied: packaging in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (24.0)\r\n", "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (3.20.3)\r\n", "Requirement already satisfied: requests<3,>=2.21.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (2.31.0)\r\n", "Requirement already satisfied: setuptools in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (69.5.1)\r\n", "Requirement already satisfied: termcolor>=1.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (2.4.0)\r\n", "Requirement already satisfied: typing-extensions>=3.6.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (4.11.0)\r\n", "Requirement already satisfied: wrapt>=1.11.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (1.16.0)\r\n", "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (1.63.0rc2)\r\n", "Requirement already satisfied: tensorboard<2.17,>=2.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (2.16.2)\r\n", "Requirement already satisfied: keras>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (3.2.1)\r\n", "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (0.36.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: python-dateutil>=2.8.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pandas->tensorflow_decision_forests) (2.9.0.post0)\r\n", "Requirement already satisfied: pytz>=2020.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pandas->tensorflow_decision_forests) (2024.1)\r\n", "Requirement already satisfied: tzdata>=2022.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pandas->tensorflow_decision_forests) (2024.1)\r\n", "Requirement already satisfied: rich in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (13.7.1)\r\n", "Requirement already satisfied: namex in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (0.0.8)\r\n", "Requirement already satisfied: optree in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (0.11.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: charset-normalizer<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow~=2.16.1->tensorflow_decision_forests) (3.3.2)\r\n", "Requirement already satisfied: idna<4,>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow~=2.16.1->tensorflow_decision_forests) (3.7)\r\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow~=2.16.1->tensorflow_decision_forests) (2.2.1)\r\n", "Requirement already satisfied: certifi>=2017.4.17 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow~=2.16.1->tensorflow_decision_forests) (2024.2.2)\r\n", "Requirement already satisfied: markdown>=2.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (3.6)\r\n", "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (0.7.2)\r\n", "Requirement already satisfied: werkzeug>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (3.0.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: importlib-metadata>=4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from markdown>=2.6.8->tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (7.1.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: MarkupSafe>=2.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (2.1.5)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: markdown-it-py>=2.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from rich->keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (3.0.0)\r\n", "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from rich->keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (2.17.2)\r\n", "Requirement already satisfied: zipp>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (3.18.1)\r\n", "Requirement already satisfied: mdurl~=0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (0.1.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Downloading tensorflow_decision_forests-1.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.5 MB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Downloading wurlitzer-3.0.3-py3-none-any.whl (7.3 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: wurlitzer, tensorflow_decision_forests\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed tensorflow_decision_forests-1.9.0 wurlitzer-3.0.3\r\n" ] } ], "source": [ "!pip install tensorflow_decision_forests\n" ] }, { "cell_type": "markdown", "metadata": { "id": "vZGda2dOe-hH" }, "source": [ "[Wurlitzer](https://pypi.org/project/wurlitzer/) is needed to display the detailed training logs in Colabs (when using `verbose=2` in the model constructor)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:08:57.623108Z", "iopub.status.busy": "2024-04-20T11:08:57.622824Z", "iopub.status.idle": "2024-04-20T11:08:59.602638Z", "shell.execute_reply": "2024-04-20T11:08:59.601816Z" }, "id": "lk26uBSCe8Du" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: wurlitzer in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (3.0.3)\r\n" ] } ], "source": [ "!pip install wurlitzer" ] }, { "cell_type": "markdown", "metadata": { "id": "3oinwbhXlggd" }, "source": [ "## Importing libraries" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:08:59.607082Z", "iopub.status.busy": "2024-04-20T11:08:59.606785Z", "iopub.status.idle": "2024-04-20T11:09:02.035993Z", "shell.execute_reply": "2024-04-20T11:09:02.035306Z" }, "id": "52W45tmDjD64" }, "outputs": [], "source": [ "import os\n", "# Keep using Keras 2\n", "os.environ['TF_USE_LEGACY_KERAS'] = '1'\n", "\n", "import tensorflow_decision_forests as tfdf\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import tensorflow as tf\n", "import tf_keras\n", "import math" ] }, { "cell_type": "markdown", "metadata": { "id": "0LPPwWxYxtDM" }, "source": [ "The hidden code cell limits the output height in colab.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2024-04-20T11:09:02.040722Z", "iopub.status.busy": "2024-04-20T11:09:02.039932Z", "iopub.status.idle": "2024-04-20T11:09:02.044296Z", "shell.execute_reply": "2024-04-20T11:09:02.043692Z" }, "id": "2AhqJz3VmQM-" }, "outputs": [], "source": [ "#@title\n", "\n", "from IPython.core.magic import register_line_magic\n", "from IPython.display import Javascript\n", "from IPython.display import display as ipy_display\n", "\n", "# Some of the model training logs can cover the full\n", "# screen if not compressed to a smaller viewport.\n", "# This magic allows setting a max height for a cell.\n", "@register_line_magic\n", "def set_cell_height(size):\n", " ipy_display(\n", " Javascript(\"google.colab.output.setIframeHeight(0, true, {maxHeight: \" +\n", " str(size) + \"})\"))" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:02.047413Z", "iopub.status.busy": "2024-04-20T11:09:02.046987Z", "iopub.status.idle": "2024-04-20T11:09:02.050488Z", "shell.execute_reply": "2024-04-20T11:09:02.049863Z" }, "id": "8gVQ-txtjFU4" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found TensorFlow Decision Forests v1.9.0\n" ] } ], "source": [ "# Check the version of TensorFlow Decision Forests\n", "print(\"Found TensorFlow Decision Forests v\" + tfdf.__version__)" ] }, { "cell_type": "markdown", "metadata": { "id": "S54mR6i9jkhp" }, "source": [ "## What is a ranking model?\n", "\n", "The goal of a [ranking](https://en.wikipedia.org/wiki/Learning_to_rank) model is to **correctly order** items. For example, ranking can be used to select the best *documents* to retrieve following a user *query*.\n", "\n", "A common way to represent a Ranking dataset is with a \"relevance\" score: The order of the elements is defined by their relevance: Items of greater relevance should be before lower relevance items. The cost of a mistake is defined by the difference between the relevance of the predicted item with the relevance of the correct item. For example, misordering two items with respective relevance 3 and 4 is not as bad as misordering two items with respective relevance 1 and 5.\n", "\n", "TF-DF expects ranking datasets to be presented in a \"flat\" format.\n", "A dataset of queries and corresponding documents might look like this:\n", "\n", "query | document_id | feature_1 | feature_2 | relevance\n", "----- | ----------- | --------- | --------- | ---------------\n", "cat | 1 | 0.1 | blue | 4\n", "cat | 2 | 0.5 | green | 1\n", "cat | 3 | 0.2 | red | 2\n", "dog | 4 | NA | red | 0\n", "dog | 5 | 0.2 | red | 0\n", "dog | 6 | 0.6 | green | 1\n", "\n", "\n", "The *relevance/label* is a floating point numerical value between 0 and 5\n", "(generally between 0 and 4) where 0 means \"completely unrelated\", 4 means \"very\n", "relevant\" and 5 means \"same as the query\".\n", "\n", "In this example, Document 1 is very relevant to the query \"cat\", while document 2 is only \"related\" to cats. There are no documents is really talking about \"dog\" (the highest relevance is 1 for the document 6). However, the dog query is still expecting to return document 6 (since this is the document that talks the \"most\" about dogs).\n", "\n", "Interestingly, decision forests are often good rankers, and many\n", "state-of-the-art ranking models are decision forests." ] }, { "cell_type": "markdown", "metadata": { "id": "QvcRIhFF0BoN" }, "source": [ "## Let's train a Ranking model\n", "\n", "In this example, use a sample of the\n", "[LETOR3](https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval/#!letor-3-0)\n", "dataset. More precisely, we want to download the `OHSUMED.zip` from [the LETOR3 repo](https://onedrive.live.com/?authkey=%21ACnoZZSZVfHPJd0&id=8FEADC23D838BDA8%21107&cid=8FEADC23D838BDA8). This dataset is stored in the\n", "libsvm format, so we will need to convert it to csv." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:02.053782Z", "iopub.status.busy": "2024-04-20T11:09:02.053297Z", "iopub.status.idle": "2024-04-20T11:09:11.825968Z", "shell.execute_reply": "2024-04-20T11:09:11.825197Z" }, "id": "axD6x1ZivHCS" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading data from https://download.microsoft.com/download/E/7/E/E7EABEF1-4C7B-4E31-ACE5-73927950ED5E/Letor.zip\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 8192/61824018 [..............................] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 98304/61824018 [..............................] - ETA: 44s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 278528/61824018 [..............................] - ETA: 29s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 729088/61824018 [..............................] - ETA: 15s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 1753088/61824018 [..............................] - ETA: 8s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 2105344/61824018 [>.............................] - ETA: 12s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 4202496/61824018 [=>............................] - ETA: 8s " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 6299648/61824018 [==>...........................] - ETA: 7s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 8396800/61824018 [===>..........................] - ETA: 7s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "10493952/61824018 [====>.........................] - ETA: 6s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "12591104/61824018 [=====>........................] - ETA: 6s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "14688256/61824018 [======>.......................] - ETA: 5s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "16785408/61824018 [=======>......................] - ETA: 5s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "18882560/61824018 [========>.....................] - ETA: 5s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20979712/61824018 [=========>....................] - ETA: 4s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "23076864/61824018 [==========>...................] - ETA: 4s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "25174016/61824018 [===========>..................] - ETA: 4s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "27271168/61824018 [============>.................] - ETA: 4s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "29368320/61824018 [=============>................] - ETA: 3s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "31465472/61824018 [==============>...............] - ETA: 3s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "32702464/61824018 [==============>...............] - ETA: 3s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "33562624/61824018 [===============>..............] - ETA: 3s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "35659776/61824018 [================>.............] - ETA: 3s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "37756928/61824018 [=================>............] - ETA: 2s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39854080/61824018 [==================>...........] - ETA: 2s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "41418752/61824018 [===================>..........] - ETA: 2s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "41951232/61824018 [===================>..........] - ETA: 2s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "44048384/61824018 [====================>.........] - ETA: 2s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "45809664/61824018 [=====================>........] - ETA: 1s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "46145536/61824018 [=====================>........] - ETA: 1s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "48242688/61824018 [======================>.......] - ETA: 1s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "49905664/61824018 [=======================>......] - ETA: 1s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "50339840/61824018 [=======================>......] - ETA: 1s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "52436992/61824018 [========================>.....] - ETA: 1s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "54050816/61824018 [=========================>....] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "54534144/61824018 [=========================>....] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "56631296/61824018 [==========================>...] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58236928/61824018 [===========================>..] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58728448/61824018 [===========================>..] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "60825600/61824018 [============================>.] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "61824018/61824018 [==============================] - 7s 0us/step\n" ] } ], "source": [ "archive_path = tf_keras.utils.get_file(\"letor.zip\",\n", " \"https://download.microsoft.com/download/E/7/E/E7EABEF1-4C7B-4E31-ACE5-73927950ED5E/Letor.zip\",\n", " extract=True)\n", "\n", "# Path to a ranking ataset using libsvm format.\n", "raw_dataset_path = os.path.join(os.path.dirname(archive_path),\"OHSUMED/Data/Fold1/trainingset.txt\")" ] }, { "cell_type": "markdown", "metadata": { "id": "f5UB-Kwn6JKK" }, "source": [ "Here are the first lines of the dataset:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:11.830056Z", "iopub.status.busy": "2024-04-20T11:09:11.829797Z", "iopub.status.idle": "2024-04-20T11:09:11.955416Z", "shell.execute_reply": "2024-04-20T11:09:11.954656Z" }, "id": "pDjamyHv6K7B" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2 qid:1 1:3.00000000 2:2.07944154 3:0.27272727 4:0.26103413 5:37.33056511 6:11.43124125 7:37.29975005 8:1.13865735 9:15.52428944 10:8.83129655 11:12.00000000 12:5.37527841 13:0.08759124 14:0.08649364 15:28.30306459 16:9.34002375 17:24.80878473 18:0.39309068 19:57.41651698 20:3.29489291 21:25.02310000 22:3.21979940 23:-3.87098000 24:-3.90273000 25:-3.87512000 #docid = 40626\r", "\r\n", "0 qid:1 1:3.00000000 2:2.07944154 3:0.42857143 4:0.40059418 5:37.33056511 6:11.43124125 7:37.29975005 8:1.81447983 9:17.45499227 10:11.61793065 11:10.00000000 12:5.19295685 13:0.08547009 14:0.08453711 15:28.30306459 16:9.34002375 17:24.80878473 18:0.34920457 19:43.24062605 20:2.65472417 21:23.49030000 22:3.15658757 23:-3.96838000 24:-4.00865000 25:-3.98670000 #docid = 11852\r", "\r\n", "2 qid:1 1:0.00000000 2:0.00000000 3:0.00000000 4:0.00000000 5:37.33056511 6:11.43124125 7:37.29975005 8:0.00000000 9:0.00000000 10:0.00000000 11:8.00000000 12:4.38202663 13:0.07692308 14:0.07601813 15:28.30306459 16:9.34002375 17:24.80878473 18:0.24031887 19:25.81698944 20:1.55134225 21:15.86500000 22:2.76411543 23:-4.28166000 24:-4.33313000 25:-4.44161000 #docid = 12693\r", "\r\n", "2 qid:1 1:4.00000000 2:2.77258872 3:0.33333333 4:0.32017083 5:37.33056511 6:11.43124125 7:37.29975005 8:1.26080803 9:17.97524177 10:8.86378153 11:3.00000000 12:1.79175947 13:0.03409091 14:0.03377241 15:28.30306459 16:9.34002375 17:24.80878473 18:0.11149640 19:10.09242586 20:0.64975836 21:14.27780000 22:2.65870588 23:-4.77772000 24:-4.73563000 25:-4.86759000 #docid = 12694\r", "\r\n", "0 qid:1 1:0.00000000 2:0.00000000 3:0.00000000 4:0.00000000 5:37.33056511 6:11.43124125 7:37.29975005 8:0.00000000 9:0.00000000 10:0.00000000 11:6.00000000 12:3.87120101 13:0.04761905 14:0.04736907 15:28.30306459 16:9.34002375 17:24.80878473 18:0.18210403 19:23.54629629 20:1.62139253 21:15.27640000 22:2.72630915 23:-4.43073000 24:-4.45985000 25:-4.57053000 #docid = 15450\r", "\r\n", "1 qid:1 1:1.00000000 2:0.69314718 3:0.14285714 4:0.13353139 5:37.33056511 6:11.43124125 7:37.29975005 8:0.62835774 9:6.12170704 10:4.15689134 11:10.00000000 12:4.43081680 13:0.08333333 14:0.08191707 15:28.30306459 16:9.34002375 17:24.80878473 18:0.32796715 19:43.13226482 20:2.12249256 21:16.33990000 22:2.79360997 23:-4.75652000 24:-4.66814000 25:-4.82965000 #docid = 17665\r", "\r\n", "0 qid:1 1:0.00000000 2:0.00000000 3:0.00000000 4:0.00000000 5:37.33056511 6:11.43124125 7:37.29975005 8:0.00000000 9:0.00000000 10:0.00000000 11:3.00000000 12:2.07944154 13:0.05357143 14:0.05309873 15:28.30306459 16:9.34002375 17:24.80878473 18:0.25524876 19:14.92698785 20:2.59607100 21:16.00510000 22:2.77290742 23:-4.54349000 24:-4.52334000 25:-4.69865000 #docid = 18432\r", "\r\n", "0 qid:1 1:1.00000000 2:0.69314718 3:0.50000000 4:0.40546511 5:37.33056511 6:11.43124125 7:37.29975005 8:1.07964603 9:3.88727484 10:3.22375250 11:14.00000000 12:5.19295685 13:0.10000000 14:0.09779062 15:28.30306459 16:9.34002375 17:24.80878473 18:0.41939944 19:65.74099134 20:2.81316484 21:20.37810000 22:3.01446079 23:-4.25087000 24:-4.18235000 25:-4.21824000 #docid = 18540\r", "\r\n", "0 qid:1 1:3.00000000 2:2.07944154 3:0.60000000 4:0.54696467 5:37.33056511 6:11.43124125 7:37.29975005 8:2.13084866 9:15.65986863 10:10.90521468 11:9.00000000 12:4.68213123 13:0.13043478 14:0.12827973 15:28.30306459 16:9.34002375 17:24.80878473 18:0.44756051 19:33.23097043 20:2.69791902 21:21.26510000 22:3.05706723 23:-4.18472000 24:-4.18399000 25:-4.03491000 #docid = 44695\r", "\r\n", "0 qid:1 1:0.00000000 2:0.00000000 3:0.00000000 4:0.00000000 5:37.33056511 6:11.43124125 7:37.29975005 8:0.00000000 9:0.00000000 10:0.00000000 11:16.00000000 12:6.10479323 13:0.11510791 14:0.11289797 15:28.30306459 16:9.34002375 17:24.80878473 18:0.37130060 19:55.42216381 20:2.33077909 21:19.21490000 22:2.95568602 23:-4.09988000 24:-4.15679000 25:-4.17349000 #docid = 18541\r", "\r\n" ] } ], "source": [ "!head {raw_dataset_path}" ] }, { "cell_type": "markdown", "metadata": { "id": "rcManr98ZGID" }, "source": [ "The first step is to convert this dataset to the \"flat\" format mentioned above." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:11.959628Z", "iopub.status.busy": "2024-04-20T11:09:11.958952Z", "iopub.status.idle": "2024-04-20T11:09:12.096376Z", "shell.execute_reply": "2024-04-20T11:09:12.095777Z" }, "id": "mkiM9HJox-e8" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
relevancegroupf_1f_2f_3f_4f_5f_6f_7f_8...f_16f_17f_18f_19f_20f_21f_22f_23f_24f_25
02g_13.02.0794420.2727270.26103437.33056511.43124137.299751.138657...9.34002424.8087850.39309157.4165173.29489325.02313.219799-3.87098-3.90273-3.87512
10g_13.02.0794420.4285710.40059437.33056511.43124137.299751.814480...9.34002424.8087850.34920543.2406262.65472423.49033.156588-3.96838-4.00865-3.98670
22g_10.00.0000000.0000000.00000037.33056511.43124137.299750.000000...9.34002424.8087850.24031925.8169891.55134215.86502.764115-4.28166-4.33313-4.44161
\n", "

3 rows × 27 columns

\n", "
" ], "text/plain": [ " relevance group f_1 f_2 f_3 f_4 f_5 f_6 \\\n", "0 2 g_1 3.0 2.079442 0.272727 0.261034 37.330565 11.431241 \n", "1 0 g_1 3.0 2.079442 0.428571 0.400594 37.330565 11.431241 \n", "2 2 g_1 0.0 0.000000 0.000000 0.000000 37.330565 11.431241 \n", "\n", " f_7 f_8 ... f_16 f_17 f_18 f_19 \\\n", "0 37.29975 1.138657 ... 9.340024 24.808785 0.393091 57.416517 \n", "1 37.29975 1.814480 ... 9.340024 24.808785 0.349205 43.240626 \n", "2 37.29975 0.000000 ... 9.340024 24.808785 0.240319 25.816989 \n", "\n", " f_20 f_21 f_22 f_23 f_24 f_25 \n", "0 3.294893 25.0231 3.219799 -3.87098 -3.90273 -3.87512 \n", "1 2.654724 23.4903 3.156588 -3.96838 -4.00865 -3.98670 \n", "2 1.551342 15.8650 2.764115 -4.28166 -4.33313 -4.44161 \n", "\n", "[3 rows x 27 columns]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def convert_libsvm_to_csv(src_path, dst_path):\n", " \"\"\"Converts a libsvm ranking dataset into a flat csv file.\n", " \n", " Note: This code is specific to the LETOR3 dataset.\n", " \"\"\"\n", " dst_handle = open(dst_path, \"w\")\n", " first_line = True\n", " for src_line in open(src_path,\"r\"):\n", " # Note: The last 3 items are comments.\n", " items = src_line.split(\" \")[:-3]\n", " relevance = items[0]\n", " group = items[1].split(\":\")[1]\n", " features = [ item.split(\":\") for item in items[2:]]\n", "\n", " if first_line:\n", " # Csv header\n", " dst_handle.write(\"relevance,group,\" + \",\".join([\"f_\" + feature[0] for feature in features]) + \"\\n\")\n", " first_line = False\n", " dst_handle.write(relevance + \",g_\" + group + \",\" + (\",\".join([feature[1] for feature in features])) + \"\\n\")\n", " dst_handle.close()\n", "\n", "# Convert the dataset.\n", "csv_dataset_path=\"/tmp/ohsumed.csv\"\n", "convert_libsvm_to_csv(raw_dataset_path, csv_dataset_path)\n", "\n", "# Load a dataset into a Pandas Dataframe.\n", "dataset_df = pd.read_csv(csv_dataset_path)\n", "\n", "# Display the first 3 examples.\n", "dataset_df.head(3)" ] }, { "cell_type": "markdown", "metadata": { "id": "jdelXEgw6bsq" }, "source": [ "In this dataset, each row represents a pair of query/document (called \"group\"). The \"relevance\" tells how much the query matches the document.\n", "\n", "The features of the query and the document are merged together in \"f1-25\". The exact definition of the features is not known, but it would be omething like:\n", "\n", "- Number of words in queries\n", "- Number of common words between the query and the document\n", "- Cosinus similarity between an embedding of the query and an embedding of the document.\n", "- ...\n", "\n", "Let's convert the Pandas Dataframe into a TensorFlow Dataset:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:12.099664Z", "iopub.status.busy": "2024-04-20T11:09:12.099433Z", "iopub.status.idle": "2024-04-20T11:09:14.480135Z", "shell.execute_reply": "2024-04-20T11:09:14.479406Z" }, "id": "5QMbBkCEXxu_" }, "outputs": [], "source": [ "dataset_ds = tfdf.keras.pd_dataframe_to_tf_dataset(dataset_df, label=\"relevance\", task=tfdf.keras.Task.RANKING)" ] }, { "cell_type": "markdown", "metadata": { "id": "sOPVcfs_7xxf" }, "source": [ "Let's configure and train our Ranking model." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:14.484063Z", "iopub.status.busy": "2024-04-20T11:09:14.483802Z", "iopub.status.idle": "2024-04-20T11:09:23.351465Z", "shell.execute_reply": "2024-04-20T11:09:23.350733Z" }, "id": "Ba1gb75SX1rr" }, "outputs": [ { "data": { "application/javascript": [ "google.colab.output.setIframeHeight(0, true, {maxHeight: 400})" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Use /tmpfs/tmp/tmpzqjjgty3 as temporary training directory\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Reading training dataset...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[WARNING 24-04-20 11:09:14.5069 UTC gradient_boosted_trees.cc:1840] \"goss_alpha\" set but \"sampling_method\" not equal to \"GOSS\".\n", "[WARNING 24-04-20 11:09:14.5069 UTC gradient_boosted_trees.cc:1851] \"goss_beta\" set but \"sampling_method\" not equal to \"GOSS\".\n", "[WARNING 24-04-20 11:09:14.5069 UTC gradient_boosted_trees.cc:1865] \"selective_gradient_boosting_ratio\" set but \"sampling_method\" not equal to \"SELGB\".\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training dataset read in 0:00:03.986733. Found 9219 examples.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training model...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model trained in 0:00:00.757738\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Compiling model...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO 24-04-20 11:09:19.2736 UTC kernel.cc:1233] Loading model from path /tmpfs/tmp/tmpzqjjgty3/model/ with prefix fa7585ffd7c24e56\n", "[INFO 24-04-20 11:09:19.2748 UTC quick_scorer_extended.cc:911] The binary was compiled without AVX2 support, but your CPU supports it. Enable it for faster model inference.\n", "[INFO 24-04-20 11:09:19.2749 UTC abstract_model.cc:1344] Engine \"GradientBoostedTreesQuickScorerExtended\" built\n", "[INFO 24-04-20 11:09:19.2749 UTC kernel.cc:1061] Use fast generic engine\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model compiled.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%set_cell_height 400\n", "\n", "model = tfdf.keras.GradientBoostedTreesModel(\n", " task=tfdf.keras.Task.RANKING,\n", " ranking_group=\"group\",\n", " num_trees=50)\n", "\n", "model.fit(dataset_ds)" ] }, { "cell_type": "markdown", "metadata": { "id": "kz9kdege8T_y" }, "source": [ "We can now look at the quality of the model on the validation dataset. By default, TF-DF trains ranking models to optimize the [NDCG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain). The NDCG is a value between 0 and 1, where 1 is the perfect score. For this reason, -NDCG is the model loss." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:23.354714Z", "iopub.status.busy": "2024-04-20T11:09:23.354431Z", "iopub.status.idle": "2024-04-20T11:09:24.273912Z", "shell.execute_reply": "2024-04-20T11:09:24.273200Z" }, "id": "lt5qysPs8zex" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/oAAAFzCAYAAACOxnKYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAACuj0lEQVR4nOzdd3xT9foH8E9G26Qj3ZPVBbRlFVqpRWQWinAVBC7ohYtWBBcO4IpwRUTEX3GxFMUrSxQueBUUFMFSQAUqCKUIdAgdtHQP2rRpm2ac3x/pOSV0JSWjSZ7365XXpcnJyTdcafKcZ3x5DMMwIIQQQgghhBBCiFXgm3sBhBBCCCGEEEIIMRwK9AkhhBBCCCGEECtCgT4hhBBCCCGEEGJFKNAnhBBCCCGEEEKsCAX6hBBCCCGEEEKIFaFAnxBCCCGEEEIIsSIU6BNCCCGEEEIIIVaEAn1CCCGEEEIIIcSKCM29AEulVqtRVFQEFxcX8Hg8cy+HEEKIjWMYBrW1tQgICACfT9fxDYE+6wkhhHQ3un7eU6DfRUVFRejVq5e5l0EIIYRoKSgoQM+ePc29DKtAn/WEEEK6q84+7ynQ7yIXFxcAmr9giURi5tUQQgixdVKpFL169eI+n8i9o896Qggh3Y2un/cU6HcRW8InkUjow58QQki3QSXmhkOf9YQQQrqrzj7vqYmPEEIIIYQQQgixIhToE0IIIYQQQgghVoQCfUIIIYQQQgghxIpQoE8IIYQQQgghhFgRCvQJIYQQQgghhBArQoE+IYQQQgghhBBiRSjQJ4QQQgghhBBCrAgF+oQQQgghhBBCiBWhQJ8QQgghhBBCCLEiFOgT0oGCqnoUVTeYexmEEEIs3OnrFfjuUiFkcqW5l0IIIcQGUKBPSDuq65swZfNvmLblDBQqtbmXQwghxIIt+m8qXtmfhkK6eEwIIcQEKNAnpB3HM8ogbVSirFaO7PI6cy+HEEKIBfNwsgcAVMmazLwSQgghtoACfULacfRqCffnjGKpGVdCCCHE0nlSoE8IIcSEKNAnpA0yuRK/XS/nfk4vokCfEEJI17EZ/UoK9AkhhJgABfqEtOGXv8ohV7b05WcU15pxNYQQQiydh5MDAKCqjgJ9QgghxkeBPiFtYMv2Y4M9AQDpxVIwDGPOJRFCCLFgHk52AIAqmdzMKyGEEGILKNAn5C5ypQonM8sAAC+ODwWfp+mpLKulL2eEEEK6hs3oU+k+IYQQU6BAn5C7nM2uRK1cCR8XB9wf5IkQb2cA1KdPCCGk69hhfLfrKdAnhBBifBToE3KXY81l+/ED/MDn8xDuLwGgKd8nhBBCuoIbxkc9+oQQQkyAAn1C7qBSM0hKLwUATBroBwCICKBAnxBCyL3xoO31CCGEmBAF+oTc4UJeFSplTXAV22F4kAcAcBn9DAr0CSGEdJHHHaX7NNyVEEKIsVGgT8gdjl7TlO3HhfvCTqD55xHRHOjnVshQ36Q029oIIYRYLjbQV6gYSBvps4QQQohxUaBPSDOGYfDzNe2yfQDwdnGAl7MDGAbIKqk11/IIIYRYMJGdAE72AgDAbSrfJ4QQYmQU6BPS7GqhFIXVDXC0F+DBvl5aj1GfPiGEkHvl4dw8kI8CfUIIIUZGgT4hzY5eKwYAjOnvDZGdQOuxcH8XANSnTwghpOs8nBwA0EA+QgghxkeBPiHNjt6xrd7d2D799CIK9AkhhHSNh6MdAKBKJjfzSgghhFg7CvQJAXCjrBbZ5TLYCXgYG+bT6nE20M8sqYVaTdOSCSFEV1VVVZgzZw4kEgnc3Nwwf/581NXVdXj8iy++iP79+0MsFqN379546aWXUFNTwx2za9cu8Hi8Nm9lZWUAgFOnTrX5eElJidHfc3vYjD6V7hNCCDE2obkXQEh3cKx5CN8DoV6QiOxaPR7k5QQHIR/1TSrcrKpHkJeTqZdICCEWac6cOSguLkZSUhIUCgUSEhKwcOFC7N27t83ji4qKUFRUhA8++AARERG4efMmnn32WRQVFeGbb74BAMyePRuTJk3Set6TTz6JxsZG+PhoX6zNysqCRCLhfr77cVPybO7Rp2F8hBBCjM3sGf0tW7YgMDAQIpEIMTExOH/+vE7P27dvH3g8HqZNm8bdp1Ao8Nprr2HQoEFwcnJCQEAA5s2bh6KiIq3nBgYGtrrCv27dOkO+LWJh2LL9SW2U7QOAUMBHfz/q0yeEEH1kZGTg6NGj2LZtG2JiYjBy5Eh89NFH2LdvX6vPZtbAgQPx7bff4uGHH0ZISAjGjRuHd955B4cPH4ZSqdmWTiwWw8/Pj7sJBAKcOHEC8+fPb3U+Hx8frWP5fPN99WG32KOMPiGEEGMza6C/f/9+LFmyBG+++SZSU1MxZMgQxMfHc2V37cnLy8O//vUvPPjgg1r319fXIzU1FW+88QZSU1Nx4MABZGVl4ZFHHml1jjVr1qC4uJi7vfjiiwZ9b8RyFFY34EphDfg8IC7Ct93jqE+fEEL0k5KSAjc3N0RHR3P3xcXFgc/n49y5czqfp6amBhKJBEJh24WIu3fvhqOjI2bOnNnqscjISPj7+2PChAk4c+ZMh68jl8shlUq1bobEBvo0jI8QQoixmTXQX79+PRYsWICEhARERERg69atcHR0xI4dO9p9jkqlwpw5c/DWW28hODhY6zFXV1ckJSVh1qxZ6N+/P+6//358/PHHuHjxIvLz87WOdXFx0brC7+REpdi26lhzNj860ANezg7tHhfeHOhTRp8QQnRTUlLSqlReKBTCw8ND5175iooKvP3221i4cGG7x2zfvh3/+Mc/IBaLufv8/f2xdetWfPvtt/j222/Rq1cvjBkzBqmpqe2eJzExEa6urtytV69eOq1RV54U6BNCCDERswX6TU1NuHjxIuLi4loWw+cjLi4OKSkp7T5vzZo18PHxabM8ry01NTXg8Xhwc3PTun/dunXw9PTE0KFD8f7773PlgMT2HL3Wcdk+KyKgOaNPgT4hxMYtX7683WF47C0zM/OeX0cqlWLKlCmIiIjA6tWr2zwmJSUFGRkZrb4X9O/fH8888wyioqIwYsQI7NixAyNGjMCGDRvafb0VK1agpqaGuxUUFNzze7iTO1u6X0eBPiGEEOMy2zC+iooKqFQq+Ppql0r7+vq2++Xg9OnT2L59O9LS0nR6jcbGRrz22mt4/PHHtQbxvPTSSxg2bBg8PDxw9uxZrFixAsXFxVi/fn2755LL5ZDLW7bDMXQ5HzGPijo5/sirAgBMHNB+2T4AhDX36BfXNOK2rIn7wkYIIbZm6dKlePLJJzs8Jjg4GH5+fq3a8ZRKJaqqquDn1/HF1draWkyaNAkuLi44ePAg7OxaD0oFgG3btiEyMhJRUVGdrnv48OE4ffp0u487ODjAwaH9yq57RRl9QgghpmIxU/dra2vxz3/+E59//jm8vLw6PV6hUGDWrFlgGAaffvqp1mNLlizh/jx48GDY29vjmWeeQWJiYrsf8ImJiXjrrbfu7U2Qbud4eikYBhjUwxU93R07PNZFZIfeHo7Ir6pHRrEUI0I7/++QEEKskbe3N7y9vTs9LjY2FtXV1bh48SIXiJ84cQJqtRoxMTHtPk8qlSI+Ph4ODg44dOgQRCJRm8fV1dXh66+/RmJiok7rTktLg7+/v07HGgPbo9+gUKGhSQWxvcBsayGEEGLdzFa67+XlBYFAgNLSUq37S0tL27zKn52djby8PDz88MMQCoUQCoXYvXs3Dh06BKFQiOzsbO5YNsi/efMmkpKStLL5bYmJiYFSqUReXl67xxi7nI+YB1e2P7DjzBIr3F+T1afyfUII6Vx4eDgmTZqEBQsW4Pz58zhz5gwWLVqExx57DAEBAQCAwsJChIWFcbvuSKVSTJw4ETKZDNu3b4dUKkVJSQlKSkqgUqm0zr9//34olUrMnTu31Wtv3LgR33//PW7cuIGrV6/ilVdewYkTJ/DCCy8Y/423w9lBCHuB5qtXVT1l9QkhhBiP2TL69vb2iIqKQnJyMrdFnlqtRnJyMhYtWtTq+LCwMFy5ckXrvpUrV6K2thabNm3iBuawQf7169dx8uRJeHp6drqWtLQ08Pn8DvfWNXY5HzE9aaMCZ25UAADiOynbZ0X4u+LYtVIK9AkhREd79uzBokWLMH78ePD5fMyYMQObN2/mHlcoFMjKykJ9fT0AIDU1lZvIHxoaqnWu3NxcBAYGcj9v374d06dPbzWHB9DMAlq6dCkKCwvh6OiIwYMH4/jx4xg7dqzh36SOeDwePJzsUSJtRFVdE3q4iTt/EiGEENIFZi3dX7JkCZ544glER0dj+PDh2LhxI2QyGRISEgAA8+bNQ48ePZCYmAiRSISBAwdqPZ/9YGfvVygUmDlzJlJTU/HDDz9ApVJxU309PDxgb2+PlJQUnDt3DmPHjoWLiwtSUlKwePFizJ07F+7u7qZ788TsTmaWQaFiEOLthFAfF52ew2X0aYs9QgjRiYeHB/bu3dvu44GBgWAYhvt5zJgxWj935OzZs+0+tmzZMixbtkz3hZqIe3OgXymTd34wIYQQ0kVmDfRnz56N8vJyrFq1CiUlJYiMjMTRo0e5AX35+fng83XvLigsLMShQ4cAaPbNvdPJkycxZswYODg4YN++fVi9ejXkcjmCgoKwePFirb59YhuO6Vm2D7RM3s8ur0OTUg17oVl3qCSEEGJhaCAfIYQQUzD7ML5Fixa1WaoPAKdOnerwubt27dL6+e6sQFuGDRuG33//XZ8lEivUqFDhZGY5ACC+k2317tTDTQyJSAhpoxLXy2oxIMDVWEskhBBihTwo0CeEEGICZg/0CQGA3Sl5+PjEDTQoVJ0fbABqNYMGhQoBriIM6qF7sM7j8RDuL8G53CpkFFOgTwghRD8U6BNCCDEFCvSJWanUDN7+IR27zuaZ5fVn3dcLPB5Pr+ewgX56kRTofNtmQgghhEOl+4QQQkyBAn1iNjK5Ei/+9xJOZJYBAJZN6o9JepTR3yt7Ib9LE4/ZPv0MmrxPCCFET+7NgX4lBfqEEEKMiAJ9YhbFNQ2Yv+sC0oulcBDysXF2JB4a5G/uZekkwl8T6KcXS8EwjN4VAYQQQmwXZfQJIYSYAgX6xOSuFtZg/hd/oFQqh5ezPT6fF42hvS1na8O+vs4Q8nmoaVCguKYRAbQPMiGEEB2xPfq3KdAnhBBiRLQ3GDGp5IxSzPosBaVSOfr6OOPg8w9YVJAPAA5CAUJ9nAFA06dPCCGE6MjTmUr3CSGEGB8F+sRkdp3JxYLdF1DfpMLIUC9889wI9PJwNPeyuiTcn/r0CSGE6M/DyQEAUNOggEKlNvNqCCGEWCsK9InRqdQMVh+6htWH06FmgNnRvbAz4T64iu3MvbQuu7NPnxBCCNGVq9gO/ObRLrfrKatPCCHEOKhHn3RZmbQRT33xB8pr5R0ep1Ax3NCh1yaF4dnRwRY/wI4y+oQQQrpCwOfBzdEeVbImVMma4OMiMveSCCGEWCEK9EmXHbxUiKuFugW6DkI+NsyOxGQLmazfmXB/FwBAXmU96uRKODvQPyVCCCG68XBqCfQJIYQQY6DohHTZqaxyAMCisaF4aJBfh8f2cBPDzdHeFMsyCU9nB/hKHFAqlSOrRIqoPh7mXhIhhBAL4UFb7BFCCDEyCvRJl9Q2KvBHXhUAYGZUTwR6OZl5RaYX4S9BqbQc6UUU6BNCCNGdJwX6hBBCjIyG8ZEuOXOjEko1g0BPR5sM8oGWPv304lozr4QQQoglYTP6lXUU6BNCCDEOCvRJl/zyVxkAYEx/HzOvxHwiAmjyPiGEEP1R6T4hhBBjo0Cf6I1hGJzM1PTnj+nvbebVmA+b0c8qkUKlZsy8GkIIIZaCAn1CCCHGRoE+0VtWaS1KpI1wEPJxf7CnuZdjNoGeThDbCdCoUCO3Qmbu5RBCCLEQFOgTQggxNgr0id7YafuxIZ4Q2QnMvBrzEfB56O+n2WYvg8r3CSGE6MjTyQEABfqEEEKMhwJ9oreTmZr+/LE23J/Poj59Qggh+uKG8VGgTwghxEgo0Cd6qW1U4OLN2wBsuz+fxfbpU0afEEKIrthA/3Z9E9Q044UQQogRUKBP9HLmRgWUagbBXk7o42mb2+rdKYLdYq+IAn1CCCG6cXeyAwCo1AykjQozr4YQQog1okCf6IWdtj+asvkAgDA/F/B4QFmtHBV1cnMvhxBCiAVwEArg4iAEQH36hBBCjENo7gUQy8EwDH75i91Wj/rzAcDJQYhATyfkVsgw89OzHQ4nFPB5WDgqGFMje5hwhYQQQrojD2d71MqVqJI1IZiunRNCCDEwCvSJzjJLNNvqiez4iAnyMPdyuo37gz2RWyFDXmV9p8duOn6dAn1CCCHwcLLHzcp6GshHCCHEKCjQJzo7maWZtj8ixMumt9W72+pHIjA1MgBKVfsDleRKFeZ/cQE5FTJUyZq4QUyEEEJsk2fz5wCV7hNCCDEGCvSJzk5lacr2x1J/vhYHoQD3B3t2elyojzNulNXhUv5tjA/3NcHKCCGEdFfujhToE0IIMR4axkd0ItXaVo/687tiWG83AOD+HgkhhNguD2cK9AkhhBgPBfpEJ6evV0ClZhDs7YReHo7mXo5FiurjDoACfUIIIVS6TwghxLgo0Cc6OdXcnz+Wsvldxgb6f96qgUKlNvNqDEulZrDk6zS8/UO6uZdCCOlmqqqqMGfOHEgkEri5uWH+/Pmoq6vr8DnPPPMMQkJCIBaL4e3tjalTpyIzM1PrmPz8fEyZMgWOjo7w8fHBq6++CqVSqXXMqVOnMGzYMDg4OCA0NBS7du0y9NvrMg8nBwCgYXyEEEKMggJ90imGYbj+/DHUn99lwV7OkIiEaFCokFlca+7lGNSVwhocSC3E9tO5kDYqzL0cQkg3MmfOHFy7dg1JSUn44Ycf8Ouvv2LhwoUdPicqKgo7d+5ERkYGjh07BoZhMHHiRKhUKgCASqXClClT0NTUhLNnz+KLL77Arl27sGrVKu4cubm5mDJlCsaOHYu0tDS88sorePrpp3Hs2DGjvl9dtWT05WZeCSGEEGtk9kB/y5YtCAwMhEgkQkxMDM6fP6/T8/bt2wcej4dp06Zp3c8wDFatWgV/f3+IxWLExcXh+vXrWsd0Jbtgy9KLpSirlUNsJ8Bw2lavy/h8HoZx5ftVZl6NYaVkV3J/ztdhm0FCiG3IyMjA0aNHsW3bNsTExGDkyJH46KOPsG/fPhQVFbX7vIULF2LUqFEIDAzEsGHDsHbtWhQUFCAvLw8A8PPPPyM9PR1fffUVIiMj8dBDD+Htt9/Gli1b0NSkyZBv3boVQUFB+PDDDxEeHo5FixZh5syZ2LBhgyneeqfc2UC/jjL6hBBCDM+sgf7+/fuxZMkSvPnmm0hNTcWQIUMQHx+PsrKyDp+Xl5eHf/3rX3jwwQdbPfbee+9h8+bN2Lp1K86dOwcnJyfEx8ejsbGRO6Yr2QVbxmbzHwj1hIOQttW7F1G9mwP9/GrzLsTAUnJaAv28SpkZV0II6U5SUlLg5uaG6Oho7r64uDjw+XycO3dOp3PIZDLs3LkTQUFB6NWrF3feQYMGwde3ZQeT+Ph4SKVSXLt2jTsmLi5O61zx8fFISUlp97XkcjmkUqnWzVi4jH49BfqEEEIMz6yB/vr167FgwQIkJCQgIiICW7duhaOjI3bs2NHuc1QqFebMmYO33noLwcHBWo8xDIONGzdi5cqVmDp1KgYPHozdu3ejqKgI3333HYCuZxds2S/Ngf5o6s+/Z2yffqoVDeRTqNS4kNdSoZBXQYE+IUSjpKQEPj7anx1CoRAeHh4oKSnp8LmffPIJnJ2d4ezsjJ9++glJSUmwt7fnzntnkA+A+5k9b3vHSKVSNDQ0tPmaiYmJcHV15W7shQVj8GgO9BsVatQ3KTs5mhBCCNGP2QL9pqYmXLx4UetqO5/PR1xcXIdX29esWQMfHx/Mnz+/1WO5ubkoKSnROqerqytiYmK4c3Y1u2DKq/zdSU2DAhfzm7fV60f9+fdqSC838HlAYXUDSmoaO3+CBfjzVjXqm1Tcz3lUuk+I1Vu+fDl4PF6Ht7uH5+lrzpw5uHTpEn755Rf069cPs2bN0qrOM4YVK1agpqaGuxUUFBjttRztBXAQar6GVRqhfL9RoaJWKkIIsWFCc71wRUUFVCpVm1fb2/tycPr0aWzfvh1paWltPs5exW/rnHde4e9KdiExMRFvvfVWh+/JGrHb6oX6ONO2egbg5CBEmJ8E6cVSpObfxuRB/uZe0j1j+/NFdnw0KtS4SaX7hFi9pUuX4sknn+zwmODgYPj5+bVqx1MqlaiqqoKfn1+Hz2ez6n379sX9998Pd3d3HDx4EI8//jj8/PxazfQpLS0FAO68fn5+3H13HiORSCAWi9t8TQcHBzg4OHS4LkPh8XjwdLJHUU0jqmRNBv+MXfJ1Go5cKcGRlx5ERIDEoOcmhBDS/Zl9GJ+uamtr8c9//hOff/45vLy8TP76przK352cbN5Wj7L5hhPFDeSzjvJ9tj9/yqAAAJTR10V9kxJKK9tikdgWb29vhIWFdXizt7dHbGwsqqurcfHiRe65J06cgFqtRkxMjM6vxzAMGIaBXK6ZUB8bG4srV65oXURISkqCRCJBREQEd0xycrLWeZKSkhAbG3svb92guIF8Rthi70Ke5jPmepl17fJCCCFEN2YL9L28vCAQCNq82t7WVf7s7Gzk5eXh4YcfhlAohFAoxO7du3Ho0CEIhUJkZ2dzz+vonF3NLjg4OEAikWjdrJ1azeCXvzT9+WPDqD/fUIb1cQMApOZbfqAvV6q4L5OPD9f0spbXyiGTU79peyrr5Bj3wS945OMzYBjG3MshxKjCw8MxadIkLFiwAOfPn8eZM2ewaNEiPPbYYwgI0FwcLCwsRFhYGJehz8nJQWJiIi5evIj8/HycPXsWf//73yEWizF58mQAwMSJExEREYF//vOfuHz5Mo4dO4aVK1fihRde4DLyzz77LHJycrBs2TJkZmbik08+wddff43Fixeb5y+jDWyffqWBA/3aRgXKajUXRaSN9PuYEEJskdkCfXt7e0RFRWldbVer1UhOTm7zantYWBiuXLmCtLQ07vbII49w++P26tULQUFB8PPz0zqnVCrFuXPnuHMaKrtgC9KLpSivlcPRXoDoQHdzL8dqRPXWbFF4tbAGjQpVJ0d3b2n51ZAr1fBydkBUH3e4O9oBAG5SVr9dn/+WixJpI9KLpaigbbWIDdizZw/CwsIwfvx4TJ48GSNHjsR//vMf7nGFQoGsrCzU12t+b4hEIvz222+YPHkyQkNDMXv2bLi4uODs2bNc651AIMAPP/wAgUCA2NhYzJ07F/PmzcOaNWu48wYFBeHHH39EUlIShgwZgg8//BDbtm1DfHy8af8COsBO3r9t4EA/r6Lld7C0QWHQcxNCCLEMZuvRB4AlS5bgiSeeQHR0NIYPH46NGzdCJpMhISEBADBv3jz06NEDiYmJEIlEGDhwoNbz3dzcAEDr/ldeeQVr165F3759ERQUhDfeeAMBAQGYNm0aAO3swtatW6FQKFplF4jGqeay/QdCvWhbPQPq5SGGl7MDKurkuFpYg+hAD3MvqcvYsv37gz3A4/HQx9MJt+urcbNSRj2hbaisk2N3Sh73c055HbxdTNMPTIi5eHh4YO/eve0+HhgYqFXdEhAQgCNHjnR63j59+nR63JgxY3Dp0iXdF2tiHk6af/+GzujnVNRxf6ZAnxBCbJNZA/3Zs2ejvLwcq1atQklJCSIjI3H06FFumF5+fj74fP2KDpYtWwaZTIaFCxeiuroaI0eOxNGjRyESibhj9uzZg0WLFmH8+PHg8/mYMWMGNm/ebND3Zg1ONW+rN6Y/9ecbEo/HQ1QfNxy7VoqLN29bdqDfPIgvNsQTABDo6Yi0gmrq02/Hf37L0dqhIKdChphgTzOuiBBiTp7ObI++3KDnzb1jm1NpIwX6hBBii8wa6APAokWLsGjRojYfO3XqVIfP3bVrV6v7eDwe1qxZo1W+d7fOsgsEqKlXcD3kY/pTf76hDevtjmPXSi26T79RocKl/GoAQGxzsNrH0wkAaPJ+Gyrr5Nh99iYAoK+PM66X1SG7rK6TZxFCrJmHkYbx5ZTfEeg3UI8+IYTYIouZuk9M69fr5VAzQD9fZ/Rwa3sbItJ1LZP3qy12IFvqzdtoUqnhK3FAkJcmwGf/N48C/Vb+81sOGhQqDO7pinkjAgFoMvqEENvl7micYXx3ZvRrqHSfEEJsEgX6pE2/XWfL9imbbwwDe7jCTsBDRZ0cBVUN5l5Ol7D9+bHBnuDxeACAPp6afaBpGJ+2O7P5r8T1RUjzBZGccsroE2LL2NJ9Qw7jYxiGSvcJIYRQoE/aVlzTCAAI83Mx80qsk8hOgAEBrgAsd5u9u/vzASCwuXS/uKbR4ncUMKT//KrJ5g/p6Yqx/X0Q7O0MACi43YAmpdrMqyOEmIsxttcrr5Oj7o4tTmkYHyGE2CYK9Emb2C8GrmI7M6/EerWU71teoF/fpMTlW9UAgNhgL+5+N0c7SESa0R+U1deoqJNjdwqbze8HHo8HX4kDnOwFUKkZ5FdR+T4htordXq+2UWmwi35sfz5fU2gFaSP16BNCiC2iQJ+0ie3pk1CgbzSWHOhfyLsNhYpBDzcxenm0zHDg8XgIpD59LZ/fkc1nd7Dg8XgI8tb8PWWX098TIbZKIrKDoDkiv11vmKw+W7bfz1dTkSdtUFjsLBhCCCFdR4E+aRObAaCMvvGwgX5miVSrzNISsP3599/Rn8+iyfst2srms0Kay/dzKNAnxGbx+Ty4O2o+ZyvrDBvoD+npBgBQqhmtbT0JIYTYBgr0SSsMw3Cl+xIRBfrG4isRoYebGGoG+LOg2tzL0Utb/fmswOaBfHlUut9mNp8V7KUJ9LNpIB8hNo3t0zdURp+9eDiwhwTC5moBGshHCCG2hwJ90kqDQgWlWlPmJxELzbwa6zbMAsv3axsVuFJYA6DtQJ8y+hodZfMBINibJu8TQgw/kC+nQvM7JdjbmWu/kzZYVtUYIYSQe0eBPmmF7c8X8nkQ2wnMvBrrFtXbDQBw0YIm7/+RVwWVmkFvD0f0cBO3epzL6FfYdkafm7Tfy61VNh+4I9CvsO0LIoTYOk8nBwBAVZ38ns+lVKmR31xNFeTlxA1HpYw+IYTYHgr0SSvslX9XsV2rLCQxLDajn3rzNtRqyxiWxJXtB7fO5gMtGf2imgbIlbbZF6rJ5ucBAF6J69vmv6Og5qGF1fUKVBlway1CiGVhM/qG+D1w63YDlGoGIjs+/CQiLqNfU0+BPiGE2BoK9Ekr7JV/mrhvfOH+Eojs+JA2Krlyy+6OHcTXVtk+AHg528PJXgCGAQqqGky5tG7jP7/moFGh1mTz+7XO5gOAo70QAa4iAFS+T4gtczdg6T47iC/Q0wl8Po8bqEsZfUIIsT16B/pyuRy//vorvvzyS3z22Wc4cOAAcnNzjbE2YiYtg/ioP9/Y7AR8bjKyOfr0v/r9Jp798qLOmaSaegWuFUkBtB/o83g8m+7TL6/tPJvPCqbJ+4TYPE8DDuNjh3uyrUHsQF32c50QQojt0DmSO3PmDDZt2oTDhw9DoVDA1dUVYrEYVVVVkMvlCA4OxsKFC/Hss8/CxcXFmGsmRsb26FNG3zSi+rjjXG4VLt68jdn39TbZ69Y3KfF/RzJQ36SCQqXGtieiO23VOJdbCYYBgr2c4CsRtXtcoJcj0oulNjl5/z+/ZneazWeFeDvh9I0KZFtINQchxPC4YXwG2F6Pzeizu3qwA3XZLXMJIYTYDp0y+o888ghmz56NwMBA/Pzzz6itrUVlZSVu3bqF+vp6XL9+HStXrkRycjL69euHpKQkY6+bGJGUAn2TGtbbPJP3j2eUcXsrJ2eWYdfZvE6fw5bt399ONp9lqxn98lo5vvydnbTfcTYfaMnoZ5fZ1t8TIaSFpwF79NlAn50Bwmb0ayijTwghNkenjP6UKVPw7bffws6u7cAvODgYwcHBeOKJJ5Ceno7i4mKDLpKYFnvln/2CQIyLHciXXS5DdX0T3BztTfK6h9IKAQB9fZxxvawOiUcycV+gBwb2cG33OZ0N4mNxk/dtLKP/3/P5OmfzgTsn71NGnxBb5eFshECfLd0XU+k+IYTYKp0y+s8880y7Qf7dIiIiMH78+HtaFDGvlow+9eibgoeTPYKbsy+X8qtN8prV9U345a9yAMCWOcMwMcIXTSo1XvrvJcjkbZd4VsmakFlSCwC4v9NAX/N+8mxs67ik9FIAwJzhvXXasYLN6OdX1kOhUht1bYSQ7snjjh79e9l9pb5JieKaRgDgPlMkNIyPEEJsVpen7jc1NeHWrVvIz8/XuhHLx03dp4y+yQw1cfn+T1dLoFAxCPeXoJ+vC96bORj+riLkVMjw5qFrbT7nXHPZfl8fZ3i7OHR4/sDmL5m3btejSWkbAWxxTQOuFNaAxwPGhfvo9Bx/iQgiOz6UagYFVbZV/UDMSy6/9z3biWG4N1dxqRmg+h4y72w238PJnqsMY4fqstvmEkIIsR16B/rXr1/Hgw8+CLFYjD59+iAoKAhBQUEIDAxEUFCQMdZITIyG8ZleVB/TBvrfN5ftPzIkAADg5miPjbMjwecB31y8xT1+p8621buTj4sDRHZ8qBmgsNo2ttg7nlEGQDNzwcu54wshLD6fhyAvmrxPjO+nn37CE088geDgYNjZ2cHR0RESiQSjR4/GO++8g6KiInMv0WbZCfhcQH4v5ft39+cDlNEnhBBbpneg/+STT4LP5+OHH37AxYsXkZqaitTUVFy6dAmpqanGWCMxMfbKvysF+ibDBvqXb1VDaeQS7uKaBpzLrQIAPDzEn7s/JtgTL47rCwB4/eDVVoP0dO3PBzRb7HHl+zYykI8t258Q4avX86hPnxjTwYMH0a9fPzz11FMQCoV47bXXcODAARw7dgzbtm3D6NGjcfz4cQQHB+PZZ59FeXm5uZdskzybLw7eU6Bf3jrQZz/HaRgfIYTYHr2bsNPS0nDx4kWEhYUZYz2kG2gp3acefVPp6+MMFwchauVKZJbUdjgQ7179cLkYDAPcF+iOnu6OWo+9OC4UKdmVOJ9XhRf/ewnfPDsC9kI+ymvluF6mCURjdAj0AaCPpyMyS2pxs0IG9Df42+hWahsVSMmuAADEhesX6Id4U0afGM97772HDRs24KGHHgKf3/ra/qxZswAAhYWF+Oijj/DVV19h8eLFpl6mzfNwskduhQxVsq63VLSZ0RfRMD5CCLFVemf0IyIiUFFRYYy1kG6CC/Qpo28yfD4Pkb3dAACp+cYt3z90WVOi+0hkj1aPCQV8bHwsEq5iO/x5qwYf/JwFAPi9uWw/zM+FGxzVmZaMvvX3nv/6VwUUKgbBXk4I9XHW67khbEafAn1iBCkpKZgyZUqbQf6devTogXXr1lGQbybs79XKe8joZzcH+sFapfuaC/a1cuU9DfojhBBiefQO9N99910sW7YMp06dQmVlJaRSqdaNWL6aehrGZw5s+X6qEfv0s8vrcKWwBgI+D5MH+rV5TICbGO/PHAwA+M+vOTiVVaZXfz6rT3Ogf3cLgDVKSi8BoH/ZPgAEN/foZ5dT6T4htsqjeXheVV3XAn2GYZDb/DuE3c0DaPkcZxigrokG8hFCiC3RuzY7Li4OAFptoccwDHg8HlQqlWFWRsxCrWZQK6cefXNgA/3zuVVcP3x7RHZ8DOnpBj6/8y3c7nQoTZPNf7CvF9cT2paJA/wwL7YPdqfcxL/+dxkOQgEA3frzWYGemraAm1ae0Veo1DiRqRnEF9eFQJ/d77pS1oSaegVcHenfHTEOlUqFXbt2ITk5GWVlZVCrteeBnDhxwkwrIx7OzYF+fdcC/SpZE6SNSvB4mrYplshOAHshH01KNWrqFXQBnxBCbIjegf7JkyeNsQ7STdQ1KcE0V/e5UI++SUX2cgOPBxTVNOLxz3/v9PiXx/fF4gn9dD4/wzA43Fy2PzUyoNPj/z05HOdzq5BZUgsA4PGAmCA9MvrN5aMFt+uhVKkhFHR5N89u7Y+8KkgblfBwssew5m0S9eHsIISvxAGlUjmyK+q6dA5CdPHyyy9j165dmDJlCgYOHAgeT78LhcR4PJtL97s6jI/tzw9wFUNkJ9B6zFVsh/JaOU3eJ4QQG6N3JDd69GhjrIN0E+zAHgchv9WXBWJcLiI7vDg2FD9dLenwOBXDIKdchk9O3cDDQwJ07gm/WihFToUMIjs+JkS0XbZ/J5GdAB//Yyge/ugMGhQqDAiQ6JVt9peIuExScU0jenk4dv4kC3Q8XZPNHxfmA4GeFRasYC9nlErlyCmXUaBPjGbfvn34+uuvMXnyZHMvhdzF4x4DfXbGB7uLx50kIqEm0G+g0n1CCLElXUrZVldXY/v27cjIyAAADBgwAE899RRcXY03KZyYBrsFDw3iM48lE/tjycSOR9QzDIOndv2Bk1nlWPndFfx3wf06Zea+TysEoJkK7+yg2z/9UB8XvPPoQLz27Z+YMaynTs9h8fk89PZwxI2yOuRVyqwy0GcYBkkZXe/PZwV7OyElpxI51KdPjMje3h6hoaHmXgZpAzeMr4s9+jltDOJjsZ/nlNEnhBDbonct7YULFxASEoINGzagqqoKVVVVWL9+PUJCQpCammqMNRITYq/4U39+98Xj8bBm6kCI7Pj4PacKBy8VdvoclZrB4T+bp+0P6bxs/07Th/XEtbcmIeGBIL3XyvbpW+vk/azSWhRUNcBByMeDfb26fJ5g2mKPmMDSpUuxadMmMIxpp69XVVVhzpw5kEgkcHNzw/z581FX1/FFrWeeeQYhISEQi8Xw9vbG1KlTkZmZyT1++fJlPP744+jVqxfEYjHCw8OxadMmrXOcOnUKPB6v1a2kpOOqKXO414x+boXm7zOorUC/uS+/hrbYI4QQm6J3Rn/x4sV45JFH8Pnnn0Mo1DxdqVTi6aefxiuvvIJff/3V4IskpsNtrUf9+d1aLw9HvDS+L947moV3fszAuDAfuDm2v+3d+dwqlErlkIiEGN3fW+/Xsxd2rb+em7xfYZ0B7PH0UgDAyFAvONp3/d8Mt8VeBWX0ifGcPn0aJ0+exE8//YQBAwbAzk77gu6BAweM8rpz5sxBcXExkpKSoFAokJCQgIULF2Lv3r3tPicqKgpz5sxB7969UVVVhdWrV2PixInIzc2FQCDAxYsX4ePjg6+++gq9evXC2bNnsXDhQggEAixatEjrXFlZWZBIJNzPPj4+Rnmf94IL9OubuOHG+mB79IO8W7dysRfupRToE0KITdH7m+mFCxe0gnwAEAqFWLZsGaKjow26OGJ6UirdtxhPjwzGwdRCXC+rw7tHM5E4fXC7xx66rMn6Tx7kz03QN4WWjL51BvpJzYH+vZTtA0BI85fzvIp6qNRMl3v9CemIm5sbHn30UZO+ZkZGBo4ePYo//viD+47w0UcfYfLkyfjggw8QENB2hdHChQu5PwcGBmLt2rUYMmQI8vLyEBISgqeeekrr+ODgYKSkpODAgQOtAn0fHx+4ubkZ9o0ZmKeTZheUJqUasiaVzu1VgKZii62aart0X3MuaSP16BNCiC3RO00nkUiQn5/f6v6CggK4uLjovYAtW7YgMDAQIpEIMTExOH/+fLvHHjhwANHR0XBzc4OTkxMiIyPx5Zdfah3TVpkej8fD+++/zx0TGBjY6vF169bpvXZrxPXo0xY83Z69kI93Hh0EAPjv+QJcvFnV5nFypQpHrmhKVR/RYdq+IbEZfWss3S+VNuLyrRrweMC48HvLEAa4iTWDC1Vq3LptfX9XpHvYuXNnhzdjSElJgZubm1YiIC4uDnw+H+fOndPpHDKZDDt37kRQUBB69erV7nE1NTXw8PBodX9kZCT8/f0xYcIEnDlzRv83YQJiewHEzQNwq/Ts0y+qbkCTUg17IR8BbuJWj7Of55TRJ4QQ26J3oD979mzMnz8f+/fvR0FBAQoKCrBv3z48/fTTePzxx/U61/79+7FkyRK8+eabSE1NxZAhQxAfH4+ysrI2j/fw8MDrr7+OlJQU/Pnnn0hISEBCQgKOHTvGHVNcXKx127FjB3g8HmbMmKF1rjVr1mgd9+KLL+r7V2GV2Cv+bAaAdG/DgzwwK1ozJO/1g1ehUKlbHfPrXxWoaVDAV+Kg1/Z4hsD2i+ZXajLV1uR4hiabH9nLDT4uons6l4DPQ1DzRRHq0yfGVl5ejtOnT+P06dMoLy836muVlJS0KpUXCoXw8PDotFf+k08+gbOzM5ydnfHTTz8hKSkJ9vZttyidPXsW+/fv16oE8Pf3x9atW/Htt9/i22+/Ra9evTBmzJgO5wnJ5XJIpVKtm6lwA/lkcr2exw7iC/R0bLMaiIbxEUKIbdI70P/ggw8wffp0zJs3D4GBgQgMDMSTTz6JmTNn4t1339XrXOvXr8eCBQuQkJCAiIgIbN26FY6OjtixY0ebx48ZMwaPPvoowsPDERISgpdffhmDBw/G6dOnuWP8/Py0bt9//z3Gjh2L4OBgrXO5uLhoHefk1LrczRaxV/xpGJ/lWP5QONwd7ZBZUoudZ3JbPX7osmYI398GB5i8JNzfVQQ7AQ9NKjVKpI0mfW1jY/vz48LvrWyfxW6LlU2T94mRyGQyPPXUU/D398eoUaMwatQoBAQEYP78+aiv16+SZPny5e1W0LG3O4fndcWcOXNw6dIl/PLLL+jXrx9mzZqFxsbWv0euXr2KqVOn4s0338TEiRO5+/v3749nnnkGUVFRGDFiBHbs2IERI0Zgw4YN7b5mYmIiXF1duVtHFQSG5unctYF8ueXtD+IDqEefEEJsld6Bvr29PTZt2oTbt28jLS0NaWlpqKqqwoYNG+Dg4KDzeZqamnDx4kXExcW1LIbPR1xcHFJSUjp9PsMwSE5ORlZWFkaNGtXmMaWlpfjxxx8xf/78Vo+tW7cOnp6eGDp0KN5//30olR33rpnzKr8ptQzjo0DfUng42WPF5HAAwIak61ql3zK5EknpmqzZVBOX7QOAUMBHL3dNn741DeSTyZU4k10JAJh4j/35rGBuIJ/1/D2R7mXJkiX45ZdfcPjwYVRXV6O6uhrff/89fvnlFyxdulSvcy1duhQZGRkd3oKDg+Hn59eqSk+pVKKqqgp+fn4dvoarqyv69u2LUaNG4ZtvvkFmZiYOHjyodUx6ejrGjx+PhQsXYuXKlZ2ue/jw4bhx40a7j69YsQI1NTXcraCgoNNzGoq7I5vR1y/QZ39nBHm1HsQH3Fm6Tz36hBBiS7pcn+3o6IhBgwZ1+YUrKiqgUqng66v9JdnX17fDLEBNTQ169OgBuVwOgUCATz75BBMmTGjz2C+++AIuLi6YPn261v0vvfQShg0bBg8PD5w9exYrVqxAcXEx1q9f3+7rJiYm4q233tLjHVomGsZnmf4e1RPfXLyF87lVWH0oHdue0PTDJqWXolGhRpCXEwb1cDXL2vp4OiKnQoa8ynqMsJItvH/9qxxNSjX6eDoi1KftL9f6CvZit9ijjD4xjm+//RbffPMNxowZw903efJkiMVizJo1C59++qnO5/L29oa3d+c7eMTGxqK6uhoXL15EVFQUAODEiRNQq9WIiYnR+fUYhgHDMJDLW8rar127hnHjxuGJJ57AO++8o9N50tLS4O/v3+7jDg4OeiUtDMmzuXT/tr4Z/eZAv61BfMCdw/goo08IIbZEp0B/+vTp2LVrFyQSSaug+W7G2p6H5eLigrS0NNTV1SE5ORlLlixBcHCw1hcX1o4dOzBnzhyIRNr9s0uWLOH+PHjwYNjb2+OZZ55BYmJiux/wK1as0HqeVCo1aUmfqbBX/Cmjb1l4PB7emTYQD236DcczSvHztRJMHOCH79M00/YfGRKg93ZNhqIZyFeOm1Y0eT+puT9/Qrivwf5eQ3zYQN96/p5I91JfX9/q4jqgmUqvb+m+rsLDwzFp0iQsWLAAW7duhUKhwKJFi/DYY49xE/cLCwsxfvx47N69G8OHD0dOTg7279+PiRMnwtvbG7du3cK6desgFosxefJkAJpy/XHjxiE+Ph5Llizh+v0FAgF3AWLjxo0ICgrCgAED0NjYiG3btuHEiRP4+eefjfJe7xW3xZ6+Gf3m3xlsVdDdaBgfIYTYJp0CfVdXV+7LrEQiMcgXWy8vLwgEApSWlmrdX1pa2mE5H5/PR2ioJi0YGRmJjIwMJCYmtgr0f/vtN2RlZWH//v2driUmJgZKpRJ5eXno379/m8eY8yq/KbFX/KlH3/L09XXBwlHB+ORUNlYfuoaIAAl+u14BwPTT9u9kbVvsKVVqnMjUlCLf67Z6d2K/pJfVylHbqIALXWwjBhYbG4s333wTu3fv5i6ANzQ04K233kJsbKzRXnfPnj1YtGgRxo8fDz6fjxkzZmDz5s3c4wqFAllZWdzFBpFIhN9++w0bN27E7du34evri1GjRuHs2bPcYL9vvvkG5eXl+Oqrr/DVV19x5+rTpw/y8vIAaFoEly5disLCQjg6OmLw4ME4fvw4xo4da7T3ei88nPUv3W9UqFBU0wCg8x79Ggr0CSHEpugU6N+57c6uXbsM8sL29vaIiopCcnIypk2bBgBQq9VITk5utQduR9RqtVYpH2v79u2IiorCkCFDOj1HWloa+Hx+q8nAtqildJ+m7luiF8f1xeE/i1BQ1YC5285BqWYwsIeE26fdHPo0f/m8aSVb7F28eRvV9Qq4Odohqo+7wc4rEdnBy9kBFXVy5JTLMKSXm8HOTQgAbNq0CfHx8ejZsyf32Xj58mWIRCKt3WsMzcPDA3v37m338cDAQDBMy64cAQEBOHLkSIfnXL16NVavXt3hMcuWLcOyZcv0Wqs5eXYho59XKQPDABKRkKsIuBvbiidrUkGpUkMo0Hs8EyGEEAuk92/7cePGobq6utX9UqkU48aN0+tcS5Ysweeff44vvvgCGRkZeO655yCTyZCQkAAAmDdvHlasWMEdn5iYiKSkJOTk5CAjIwMffvghvvzyS8ydO7fVWv73v//h6aefbvWaKSkp2LhxIy5fvoycnBzs2bMHixcvxty5c+Hubrgv7ZaKveJPpfuWSWwvwJpHBgJo2bt+6pAe5lwSApu3jdN8IbX8LfaSmqftjwvzMfgX5paBfNSnTwxv4MCBuH79OhITExEZGYnIyEisW7cO169fx4ABA8y9PJvXlWF8uc1l+0Hezu1WW7qIWi7c1zbSQD5CCLEVeqdtT506haam1h9CjY2N+O233/Q61+zZs1FeXo5Vq1ahpKQEkZGROHr0KNdDmJ+fDz6/5Yu0TCbD888/j1u3bkEsFiMsLAxfffUVZs+erXXeffv2gWEYPP74461e08HBAfv27cPq1ashl8sRFBSExYsXa/Xf2yqlSg1ZkwoADeOzZGPDfDB5kB+OXCkBjwf8bUj7g6dMoYebGAI+D40KNcpq5fCV3Nue8+bEMIxWf76hhXg74XxuFfXpE6NxdHTEggULzL0M0gZ2ez19hvHldDKIDwDsBHw42gtQ36SCtFEB93Yy/4QQQqyLzoH+n3/+yf05PT2dG3wDACqVCkePHkWPHvpnDhctWtRuqf6pU6e0fl67di3Wrl3b6TkXLlyIhQsXtvnYsGHD8Pvvv+u9Tltw55V+iYhK9y3Zqr8NwI2yOkQHesDfVWzWtdgL+ejhJkZ+VT3yKmQWHejfKKvDzcp62Av4GNWv84nj+mqZvE+BPjGMQ4cO4aGHHoKdnR0OHTrU4bGPPPKIiVZF2uLhpJkDpE/pfmcT91muYjvUN6moT58QQmyIztFcZGQkeDweeDxemyX6YrEYH330kUEXR0yLHcTnZC+gHj4L5+cqws+LR5t7GZw+no6aQL9ShphgT3Mvp8t+bi7bHxHqCScHw18MY0v3s2mLPWIg06ZNQ0lJCXx8fLh5OG3h8XhQqVSmWxhphe2xr5MrIVeq4CAUdPocdjvOoHYm7rMkIjsU1zRyO+sQQgixfjp/U83NzQXDMAgODsb58+e19s+1t7eHj48PBILOP5RI98VtrUdl+8TAAj2d8Nv1Cm5ugKU6zpbtG3Da/p3YoYl5lTKo1Qz4fPNsiUish1qtbvPPpPuRiIQQ8nlQqhlUyZp0qsZiM/rtTdznzt08YJe9oE8IIcT66Rzo9+nTBwB9UbBmNIiPGEuf5i32buqwxZ5KzeCnq8VwEAoQ6uOM3h6OEHSDgLesthFpBdUAgDgj9OcDQE93MewEmnkGRTUN6OnuaJTXIbZp9+7dmD17dqutYpuamrBv3z7MmzfPTCsjgKaqwt3JHuW1clTWdR7o35Y14Xa95nObHXraHvZzXUql+4QQYjO6XHuanp6O/Pz8VoP5qMfPcrFX+mlrPWJo3OT9is4z+mt/TMfOM3ncz/ZCPoK9nBDq44y+Pi7o6+uMUB9nBHo6wV5ouhaTExllYBhgSE9Xo80ZEAr46OPphBtldcgul1GgTwwqISEBkyZNarWVbG1tLRISEijQ7wY8mwP92/Wd9+nnNl849ZOIOm0lYiv1KKNPCCG2Q++ILicnB48++iiuXLkCHo/HbZfFbutCPX6Wi73S70ql+8TAApvLSm82b7HX3jZQX6bkcUF+uL8EuRV1aFSokVlSi8ySWgDF3LECPg+eTvZo51QGx7a2GKtsnxXspQn0c8rrMNoIA/+I7Wrv396tW7fg6upqhhWRu7F9+roM5GO31gvupD8faPlcp2F8hBBiO/QO9F9++WUEBQUhOTkZQUFBOH/+PCorK7F06VJ88MEHxlgjMREuo0+l+8TAenmIweMBsiYVKuqa4O3i0OqYX/4qx+rD6QCAV+P744WxoVCrGRRWN+B6WS1ulNXhemkdbpTX4UZpHWrlSpTVyk36PuwFfEwZHGDU1wj2dgZQSpP3icEMHTqUG6Y7fvx4CIUtH/0qlQq5ubmYNGmSGVdIWGygX1nXeaCfU9E8iK+T/nygZScdGsZHCCG2Q+9APyUlBSdOnICXlxf4fD74fD5GjhyJxMREvPTSS7h06ZIx1klMgOvRp4w+MTAHoQABrmIUVjfgZqWsVaD/V2ktFu1JhUrNYMawnnh+TAgAgM/noZeHI3p5OGJcWEsmnWEYlErlqJSZNtD3dnGAj4txtwdks3Psl3hC7hU7bT8tLQ3x8fFwdnbmHrO3t0dgYCBmzJhhptWRO3k2B/p/ldZ2eqyug/gAKt0nhBBbpHegr1Kp4OLiAgDw8vJCUVER+vfvjz59+iArK8vgCySmw03dF1GPPjG8QC9HFFY3IK+yHtGBHtz9FXVyPLXrD9TKlRge5IHE6YPaLe1n8Xg8+LmK4Odq3KDbHELYQJ8y+sRA3nzzTQBAYGAgZs+eDZHI+v7dWIvYEE98kXIT+/4oQLC3ExaOCmn32Bw9SvdpGB8hhNgevSdZDRw4EJcvXwYAxMTE4L333sOZM2ewZs0aBAcHG3yBxHRahvFRRp8YXh/Plj59VqNChQW7L+DW7QYEejris7lRJh2w1x0Fe2myrcU1jahvojJbYjhPPPEEBfnd3KSB/lg6oR8A4P+OZGL76dw2j1OrGeQ1/y5lf2d0REI9+oQQYnP0/ka9cuVKbou9NWvWIDc3Fw8++CCOHDmCzZs3G3yBxHSkVLpPjCiweYu9vErN5H2GYfDqN3/iUn41JCIhtj95H9yby1ZtmbuTPdenS1l9YkgqlQoffPABhg8fDj8/P3h4eGjdSPfw4vi+eGl8XwDA2z+kY3dKXqtjiqWNaFSoIeTz0NO94234gJbddKSNdPGQEEJshd6Bfnx8PKZPnw4ACA0NRWZmJioqKlBWVoZx48YZfIHEdLgefRrGR4zg7oz+huPXcfhyEYR8Hrb+Mwoh3p1npWxFcHPPbXY59ekTw3nrrbewfv16zJ49GzU1NViyZAmmT58OPp+P1atXm3t55A6L4/pys0pWfX8Ne87d1Hqcnbjf29MRQkHnX+WodJ8QQmyPQWpkPTw8Ou2pJd0fe6WfvfJPiCEFNgf6uRUyfHepEJuTrwMA3nl0IEaEeJlzad1OMPXpEyPYs2cPPv/8cyxduhRCoRCPP/44tm3bhlWrVuH333839/LIHXg8Hl6N74+FozQtka8fvIr9f+Rzj+c2D+sM1mEQH9CyvR4N4yOEENuhU0THZvB1ceDAgS4vhpgXe6XflUr3iRH09tCU7tc2KvHqN5o5H8+MDsbs+3qbc1ndUnBzdUNOBQX6xHBKSkowaNAgAICzszNqamoAAH/729/wxhtvmHNppA08Hg8rHgqDUsVgx5lcLD9wBQI+HzOjenK/G3SZuA+0tOQ1KtSQK1VwEAqMtm5CCCHdg04ZfVdXV+4mkUiQnJyMCxcucI9fvHgRycnJcHV1NdpCifFxw/iodJ8YgdheAD+JZhCYQsUgfoAvXosPM/Oquic2S5dDpfvEgHr27Ini4mIAQEhICH7++WcAwB9//AEHB4eOnkrMhMfj4Y2/heOJ2D5gGODVby7ju0uFd0zc163lycVBCLbwkt1hhxBCiHXTKaO/c+dO7s+vvfYaZs2aha1bt0Ig0FwRVqlUeP755yGRSIyzSmJ0cqUKjQrNkEUaxkeMJcjLCSXSRgzsIcGG2ZHg86nlpy3sl/fcChkYhqHWKGIQjz76KJKTkxETE4MXX3wRc+fOxfbt25Gfn4/Fixebe3mkHTweD6sfGQCFmsHec/lY8nUaRHaa71+6ZvT5fB6cHYSobVRC2qiAtwtd2CHE2kgbFTh7owJKNWPupbTiKxEhzM8FLpRMNCm9m7F37NiB06dPc0E+AAgEAixZsgQjRozA+++/b9AFEtNgr/DzeJor/4QYw+IJ/RCUVohXxveFoz39d9ae3h6OEPB5qG9SoUTaCH/XzqdqE9KZdevWcX+ePXs2evfujZSUFPTt2xcPP/ywGVdGOsPj8bB26kCoVAz2XyhAfZMKgO49+oCmWq+2UUkD+QixUsu//RNHrpSYexkd6u3hiHB/F4T7SxDuL0GEvwQ93cWU0DASvb9pK5VKZGZmon///lr3Z2ZmctvuEcvDlu07Owgpy0qMZniQB4YH0TZenbEX8tHHwxE5FTLM/DQFDnbtd1kJ+Tw8NyYEjw7tacIVEmsQGxuL2NhYcy+D6IjP5yFx+iAo1Qy+Tb0FV7GdXpl5V7EdCqsbaIs9QqxQfZMSyRllAIDoPu4QCrrPd3k1AxRU1aO4phH5VfXIr6rHsWul3OMuIiHC/SQY3d8bC0cFw06HnUSIbvQO9BMSEjB//nxkZ2dj+PDhAIBz585h3bp1SEhIMPgCiWnQID5Cupf7Aj2QUyFDYXVDp8d++PNfmBbZg66Ik1YOHTqk87GPPPKIEVdCDIHP5+G9mYMxIECCIG8nvf7Nszvq1FBGnxCr89v1CsiVavRwE+N/z8Z2y+8Dt2VNyCiWIr1YioziWqQXS3GjrBa1jUqcz6vC+bwqCPg8PDs6xNxLtRp6B/offPAB/Pz88OGHH3JDffz9/fHqq69i6dKlBl8gMQ32g58G8RHSPbw9bSBm3dcLqg567VRqBk/uPI9btxuQVVqLMD+ak0K0TZs2TetnHo8HhmFa3Qdo5u2Q7k/A5+GpkUF6P4/9fKfSfUKsz/F0TYZ8QoRvtwzyAcDdyR4jQr0wIrRlS+UmpRrZ5XX46UoxNp+4gU3Hr+Nvg/3R093RjCu1HnrXRvD5fCxbtgyFhYWorq5GdXU1CgsLsWzZMq2+fWJZ2FI+9oo/IcS87IV8RPVx59od2rrFhnhiZPMHZtIdZXCEsNRqNXf7+eefERkZiZ9++on7/P7pp58wbNgwHD161NxLJUbGDtplW/UIIdZBpWZwIlNTtj8xwtfMq9GPvZCPcH8JFk/oh+FBHmhQqPDW4XRzL8tq3FMThEQioUn7VkJKGX1CLNKE5g/14xkU6JOOvfLKK9i0aRPi4+O5z+/4+HisX78eL730krmXR4yMbc2j7fUIsS6X8m+jUtYEiUiI+yx0DhKPx8PaaQMh5POQlF7KVSiQe6NT+nbYsGFITk6Gu7s7hg4d2mFJSGpqqsEWR0yHvcJPPfqEWJZx4T7g8YDLt2pQKm2Er0Rk7iWRbio7Oxtubm6t7nd1dUVeXp7J10NMi72QTz36hFiXpOageGyYj0UPsuvn64KnHwzG1l+y8eahaxgR6tntd2hiGAZL/3cZZ29U6nT8M6ODkfCA/q1XXaXT397UqVPh4KCZ7Hp3vx+xDuwVfgkF+oRYFB8XESJ7ueFSfjWS0ksx9/4+5l4S6abuu+8+LFmyBF9++SV8fTWVIKWlpXj11Ve54brEerGteVS6T4h1YQP9uHDLKttvy0vjQ3H4chEKqxuwOfkGlj8UZu4ldeibi7dwILVQ5+NlctNWVOkU6L/55ptt/plYDxrGR4jlmhDhi0v51TieQYE+ad+OHTvw6KOPonfv3ujVqxcAoKCgAH379sV3331n3sURo6NhfIRYn+zyOuRUyGAn4GF0f29zL+eeOdoLsfqRAViw+wK2/ZaD6cN6oJ+vi7mX1abbsib835EMAMCisaGYNNCv0+f4SHTfEtUQunc9BDEZ9go/DeMjxPJMCPfFe0ezcPZGJerkSjg70L/jrmIYBsoOdjpg8Xk8CPjdc7Jxe0JDQ/Hnn38iKSkJmZmZAIDw8HDExcV12ynNxHC4Hv1G6tEnxFqw2fz7gz2tJlk3IcIXceG+OJ5RipXfXcX+hfd3y8+od49m4na9Av19XfByXN9u2Tah07dBd3d3nf+Cq6qq7mlBxDxoGB8hlivUxxmBno7Iq6zHb3+V46FB/uZeksVRqtT4Pq0IW07eQE6FrNPjHYR8fDp3GMaFWVapJI/Hw8SJEzFx4kRzL4WYGDd1nzL6hFiNO7fVsyarH4nAmRsVOJ9bhW9TCzEzqqe5l6Tl4s0q7PujAACw9tGB3TLIB3QM9Ddu3GjkZRBzY6/w0zA+QiwPj8dDXLgvtp3ORVJ6KQX6elCq1Dh0uQgfnbiBXB0CfJZcqca/D1zF8aWe3bqCYvPmzVi4cCFEIhE2b97c4bHGmrxfVVWFF198EYcPHwafz8eMGTOwadMmODs7t/ucZ555BsePH0dRURGcnZ0xYsQIvPvuuwgLa+nXbCsB8d///hePPfYY9/OpU6ewZMkSXLt2Db169cLKlSvx5JNPGvT9WQquR58CfUKsQkWdHBfzbwOwjv78O/V0d8TLcX2x7qdM/N+RDMSF+8DN0d7cywKg+d7w+sGrAIC/R/XEfYHdd6cDnb6dPPHEE8ZeBzEzLqNPgT4hFmlChCbQP5FVBqVKDWE3vbrcXajUDA5fLsLm5OtcBt/DyR4LRwVjZlTPDq/OK1RqTP/kLPKr6rEx6S+s/FuEqZattw0bNmDOnDkQiUTYsGFDu8fxeDyjBfpz5sxBcXExkpKSoFAokJCQgIULF2Lv3r3tPicqKgpz5sxB7969UVVVhdWrV2PixInIzc2FQCDgjtu5cycmTZrE/XznrgK5ubmYMmUKnn32WezZswfJycl4+umn4e/vj/j4eKO81+6M69FvVIBhmG5ZCksI0d2JzDIwDDAgQIIAN7G5l2Nw80cG4UDqLfxVWod3j2Yhcfogcy8JALDrbB4yS2rh5miHFZPDzb2cDt1TGqKxsRFNTU1a90kkkntaEDGPlkC/+2amCCHti+rjDjdHO1TXK3Dh5m3cH+xp7iV1Syo1gx/+LMKm5OvIKdcE+O6OdlgwKhhPxAbCScfs/FtTByBh5x/YeTYPM6J6Ity/e3725ebmtvlnU8nIyMDRo0fxxx9/IDo6GgDw0UcfYfLkyfjggw8QEBDQ5vMWLlzI/TkwMBBr167FkCFDkJeXh5CQEO4xNzc3+Pm1PQBp69atCAoKwocffghAM4/g9OnT2LBhg20G+s0X8hUqBo0KNcT2gk6eQQjpzpKstGyfZSfgY+20QZj1WQr+ez4fM6N6IqqPu1nXVFzTgA1JfwEAlk8Kg4dT96gyaI/eKR+ZTIZFixbBx8cHTk5OcHd317rpa8uWLQgMDIRIJEJMTAzOnz/f7rEHDhxAdHQ03Nzc4OTkhMjISHz55Zdaxzz55JPg8Xhatzuv9gOaMsI5c+ZAIpHAzc0N8+fPR11dnd5rtxYMw7QM46MefUIsklDAx7gwHwAtPXu2QtqoQHJGKZLSO77tO5+P+I2/4uV9acgpl8HN0Q6vxvfHb6+Nw/NjQnUO8gFgbH8fPDTQDyo1g9cPXoFahwF+tiglJQVubm5ckA8AcXFx4PP5OHfunE7nkMlk2LlzJ4KCgrjdAlgvvPACvLy8MHz4cOzYsQMM0/L/Q0pKCuLi4rSOj4+PR0pKSruvJZfLIZVKtW7WwslewA2QpC32CLFsjQoVfrteDsD6yvbvNDzIg+vPX/ndVShVarOuZ83hdMiaVIjq445Z0b06f4KZ6Z2+XbZsGU6ePIlPP/0U//znP7FlyxYUFhbis88+w7p16/Q61/79+7FkyRJs3boVMTEx2LhxI+Lj45GVlQUfH59Wx3t4eOD1119HWFgY7O3t8cMPPyAhIQE+Pj5aV+cnTZqEnTt3cj87OGhvZdCVMkJr1qhQQ6HSfDmiHn1CLNfECF8cSC1EUkYpXp8SbhOluWo1g6d2/oELN2/r/BxXsR0WPBiEJ0YEwuUeLm6uejgCv/5VjtT8anx9oQCPDe/d5XMZy5IlS3Q+dv369QZ//ZKSklaf50KhEB4eHigpKenwuZ988gmWLVsGmUyG/v37IykpCfb2LdmTNWvWYNy4cXB0dMTPP/+M559/HnV1dVwLQklJCXx9tb8A+/r6QiqVoqGhAWJx61LXxMREvPXWW119u90aj8eDRCTE7XoFahoU8JWIzL0kQkgXnb5egUaFGj3cxBgQ0D0rygxlxUNhSEovRUaxFLvO5uHpB4PNso6TWWX46WoJBHwe1k4bCL4F7Lyjd6B/+PBh7N69G2PGjEFCQgIefPBBhIaGok+fPtizZw/mzJmj87nWr1+PBQsWICEhAYCmzO7HH3/Ejh07sHz58lbHjxkzRuvnl19+GV988QVOnz6tFeg7ODi0W8rX1TJCa1bTXLYv4PPgSKV8hFisB/t6w17Ix83Kelwvq+u2e88a0v4LBbhw8zZEdnyE+XX8ZUfA52FMP288+cC9Bfgsf1cxFk/oh7U/ZmDd0UxMHODX7cr4Ll26pNNx+l4UWr58Od59990Oj8nIyNDrnHebM2cOJkyYgOLiYnzwwQeYNWsWzpw5A5FIE6C+8cYb3LFDhw6FTCbD+++/f0+zBlasWKF1cUQqlbaqIrBkErEdbtcraCAfIRbueIamci8u3MfqL+p7Ojtg+UNhWHHgCjYk/QVPZ/tOp9z383Ux6HegRoUKq77XDOB76oHAbtuudze9A/2qqioEB2uupEgkEm47vZEjR+K5557T+TxNTU24ePEiVqxYwd3H5/MRFxfXYVkdi2EYnDhxAllZWa2+bJw6dQo+Pj5wd3fHuHHjsHbtWnh6avpVOysjfPTRR9t8PblcDrlczv1sTeV8LWX7Qqv/ZUGINXNyEOKBEE+czCpHUnqp1Qf6lXVyrPtJsx/8vyb2N8tV/idHBOKbi7eQWVKLxCMZeP/vQ0y+ho6cPHnSKOddunRpp9Prg4OD4efnh7KyMq37lUolqqqq2r0gz3J1dYWrqyv69u2L+++/H+7u7jh48CAef/zxNo+PiYnB22+/Dblczl3wLy3VbmMpLS2FRCJpM5sPaBIFd1cBWpM7B/IRQiyTWs3geIbm92qclfbn3212dC/870IBUvOrsXj/ZZ2eMzHCF6/E9UOEASoePj5xAwVVDfB3FeGVuH73fD5T0TvQDw4ORm5uLnr37o2wsDB8/fXXGD58OA4fPqw17bYzFRUVUKlUbZbVZWZmtvu8mpoa9OjRA3K5HAKBAJ988gkmTJjAPT5p0iRMnz4dQUFByM7Oxr///W889NBDSElJgUAg6HIZoTWX89HEfUKsR1yELxfovzA21NzLMarEnzJR06BAuL8ET44INMsahAI+3nl0IGZ8moL/XbyFv0f3wvCg7rvVjqF4e3vD29u70+NiY2NRXV2NixcvIioqCgBw4sQJqNVqxMTE6Px6DMOAYRitC+53S0tLg7u7Oxeox8bG4siRI1rHJCUlITY2VufXtTZse560QWnmlRBCuupSQTUq6uRwcRAiJsg2Bu/y+Tx88PchWPtjBuqbOv79pVAxSM2/jZ/TS/FzeikmDfDDy3F9u5yFv1FWh89+zQYAvPlwhF7zfMxN75UmJCTg8uXLGD16NJYvX46HH34YH3/8MRQKhVH6++7m4uKCtLQ01NXVITk5GUuWLEFwcDBX1n/n/rmDBg3C4MGDERISglOnTmH8+PFdfl1rLudjr+xTfz4hli8u3BevH7yKtIJqlNU2wsfFOvtwz+dW4ZuLtwAAa6cNNOt2glF9PPDYfb2w748CrPzuCn586cFOywrN5cKFC/j666+Rn5/fatecAwcOGPz1wsPDMWnSJCxYsABbt26FQqHAokWL8Nhjj3GtcoWFhRg/fjx2796N4cOHIycnB/v378fEiRPh7e2NW7duYd26dRCLxZg8eTIATRthaWkp7r//fohEIiQlJeH//u//8K9//Yt77WeffRYff/wxli1bhqeeegonTpzA119/jR9//NHg79NSsDvr1FDpPiEWiy3bH91f065nK4K9nbHjyft0OvZGWS02Jd/AD38W4ei1Ehy9VoLJg/zw0vi+nbb53YlhGLzx3VUoVAzGhfkgfkDHlWjdjd7/dSxevJjrf4uLi0NmZib27t2LS5cu4eWXX9b5PF5eXhAIBG2W1XVUzsfn8xEaGorIyEgsXboUM2fORGJiYrvHBwcHw8vLCzdu3ACALpcROjg4QCKRaN2sBfuBTxP3CbF8vhIRhvR0BQAkZ5R1crRlUqjUWPndFQDA48N7m327HQB4rXmbnb9K67D9tOm3sdPFvn37MGLECGRkZODgwYNQKBS4du0aTpw4AVdXV6O97p49exAWFobx48dj8uTJGDlyJP7zn/9wjysUCmRlZaG+vh4AIBKJ8Ntvv2Hy5MkIDQ3F7Nmz4eLigrNnz3IVeXZ2dtiyZQtiY2MRGRmJzz77DOvXr8ebb77JnTcoKAg//vgjkpKSMGTIEHz44YfYtm2bTW6tx+JK9ynQJ8RiWfu2eoYQ6uOCjx4fimOvjMKUwf7g8YAjV0owaeNveGFPKv4qrdXpPN+nFSElpxIiOz7eemSAxbU4653RLygo0Mpk9+nTB3369NH7he3t7REVFYXk5GRMmzYNAKBWq5GcnIxFixbpfB61Wt1hKd+tW7dQWVkJf39/AIYrI7QmbAkfe6WfEGLZJkT44vKtGhxPL8Xj3XAS/L3afjoXf5XWwdPJHq9N6m/u5QAA3J3ssfyhMCz75k9sOn4dfxvsj57ujuZelpb/+7//w4YNG/DCCy/AxcUFmzZtQlBQEJ555hnuM9IYPDw8OtzVJjAwUGtbvICAgFYl93ebNGlSq61z2zJmzBidBxLaArZFj3r0CbFMuRUy3Cirg5DPw5j+rXcoI9r6+bpgyz+GIaukFpuTr+PHK8X48Uoxjlwtxtj+Pp0O0D2RqUmYvDiuL3p5dK/PdF3oHdkFBgZi5MiRmDt3LmbOnAl3965nUpYsWYInnngC0dHRGD58ODZu3AiZTMZN4Z83bx569OjBZewTExMRHR2NkJAQyOVyHDlyBF9++SU+/fRTAEBdXR3eeustzJgxA35+fsjOzsayZcsQGhrKXcHXpYzQ1kgpo0+IVZkQ4YcPfv4Lp29UoL5JCUd767mId+t2PTYdvw4AWDE5HG6O3WfK/cxhPfG/CwX4I+823jqcjs/nRXf+JBPKzs7GlClTAGgutstkMvB4PCxevBjjxo2z2jk0pAX16BNi2Y43Z/Njgj2o5VYP/f1csGXOMLxYIsWm49fx09USLojvTIi3ExaYaUu/e6X3t78LFy5g7969WLNmDV588UVMmjQJc+fOxcMPP6z3pNrZs2ejvLwcq1atQklJCSIjI3H06FFuQF9+fj74/JbuAplMhueffx63bt2CWCxGWFgYvvrqK8yePRsAIBAI8Oeff+KLL75AdXU1AgICMHHiRLz99ttaa9uzZw8WLVqE8ePHg8/nY8aMGdi8ebO+fxVWg5u6T78wCLEK/Xyd0ctDjIKqBvz6VwUmDbSsnrKOrD6UjgaFCsODPDBjWA9zL0cLn8/D2mmDMGXzb0hKL8Xx9NJuNRHZ3d0dtbWacsUePXrg6tWrGDRoEKqrq7myeWLdJCLq0SfEkiU19+dPCO8+ny2WJMxPgk/nRiGjWIrfrpdDpe74eAEfeGigv8XOQtA70B86dCiGDh2K9957D6dOncLevXuxcOFCqNVqTJ8+HTt27NDrfIsWLWq3VP/UqVNaP69duxZr165t91xisRjHjh3r9DU7KyO0NewHPl0ZJMQ68Hg8TAj3w44zuTieUWo1gX5SeimOZ5RCyOdh7bSB3bJXrr+fC+Y/GITPfsnBm4euYUSoZ7epqBg1ahSSkpIwaNAg/P3vf8fLL7+MEydOICkp6Z6G1RLLQaX7hFiuKlkTLuRptjXvTheRLVG4v6TLU/gtSZe/ffB4PIwdOxZjx47Fc889h/nz5+OLL77QO9An5sf16Iu6x5dRQsi9i4vwwY4zuTiRWQaVmoGA3/2CYn3UNymx+tA1AMDTDwajn6+LmVfUvpfH98UPl4tRWN2AR7echZtj5xdR33l0EEJ9nI26ro8//hiNjY0AgNdffx12dnY4e/YsZsyYgZUrVxr1tUn3wA3jo0CfEItzMrMMakYTpHa3GTCke+pyZHfr1i3s3bsXe/fuxdWrVxEbG4stW7YYcm3ERKh0nxDrc1+gpn+vStaE1PzbuC/Qsvd235x8A4XVDejhJsZL40PNvZwOOdoLsfqRAViw+wKydJzsK5Mbv2faw6PlvwE+n4/ly5cb/TVJ9yKhHn1CLBY3bT+chvAR3egd6H/22WfYu3cvzpw5g7CwMMyZMwfff/99lybvk+6BC/RpGB8hVsNOwMfY/t74Lq0ISemlFh3o/1Vai22/5QAA3npkQLcphe/IhAhffPNsLEqkjTod38fT+NmZuLg4zJ07F9OnT7eqLWKJ7lybd9ehjD4hlqVRocKv18sBaAbuEqILvb8trV27Fo8//jg2b96MIUOGGGNNxMRattejQJ8QazIhwg/fpRXheHop/j053NzL6RKGYbDyu6tQqhlMiPC1qL7E6G52cWXAgAFYsWIFnn/+eUyZMgVz587F5MmTYWdHv/ttBVe636CAWs2Ab+EtPYTYipTsStQ3qeAnEWFgD7pQS3Sj9wjB/Px8vPfeexTkW5GWYXzdP0tGCNHd6P7esBfwkdO8764l+ja1EOdzqyC2E2D1IwPMvRyLtmnTJhQWFuK7776Dk5MT5s2bB19fXyxcuBC//PKLuZdHTIC9oK9mAFkTle8TYil+bi7bj4vw6ZaDaEn3pFNkl5+fj969ewOATv9xFRYWokeP7rXtka2oqVdAIhbq/EtArWZQS6X7hFglZwch7g/xxK9/lSMpvdTow97uVW2jApkltUgvkiKjmL1petxfjuuLHm5iM6/Q8vH5fEycOBETJ07E1q1bcfjwYbzzzjvYvn07VCqVuZdHjExkJ4C9kI8mpRrSRiVc6HOfELO6WSnDvw9e6XRuBnuxnsr2iT50CvTvu+8+TJs2DU8//TTuu+++No+pqanB119/jU2bNmHhwoV46aWXDLpQ0rlPT2XjvWOZeGV8P7wc11en58ialFAzmj9T6T4h1mdChC9+/aschy8XoW83C/TlSjWul9Uio1iK9GIpCqoa2jzu/mAPzB8ZZOLVWbeSkhLs27cPX331Ff78808MHz7c3EsiJiIR2aGiTg5pg4IunhFiZt+mFuLMjUqdjvVydsD9wd2rJYx0bzoF+unp6XjnnXcwYcIEiEQiREVFISAgACKRCLdv30Z6ejquXbuGYcOG4b333sPkyZONvW5yl//8mo13j2YCAM7cqNA50Jc2aq4g2gv5ENkJjLY+Qoh5xIX74I3vgPRiKZ7efcHcy+mUv6sI4f4SRDTvcRvu74JATyfqJTYAqVSKb7/9Fnv37sWpU6cQHByMOXPmYP/+/QgJCTH38oiJSMRCVNTJubY9QgyNYRhcL6tDiLezxW/tamzZ5ZpM/T9iemNCeMczaML8XeAgpO/qRHc6Bfqenp5Yv3493nnnHfz44484ffo0bt68iYaGBnh5eWHOnDmIj4/HwIEDjb1e0obtp3Pxf0cyuZ9zKmQ6P7emnsr2CbFm/q5iLI7rh5NZZeZeSisCPg+Bnk4I93dBRIAE4X4SuDvZm3tZVsvX1xfu7u6YPXs2EhMTER0dbe4lETO4cyAfIcbw09USPL8nFaP6eWPHE9EQCvQeCWYzcso139nHh/lgbBhtm0cMS6/pa2KxGDNnzsTMmTONtR6ip90peXj7h3QAwPyRQdh+OldTkteo0Cl457bWo0F8hFitl+P66lzlQ6zXoUOHMH78ePD59KXblrFtemxFHyGGdvHmbQDAr3+VY/Xha3h76kAaINcGtZpBboUmox/i3b1a64h1oE97C7bn3E2s+v4aAOD5MSFYOSUc3i4OAIDcct2y+uwVfcroE0KIdZswYQIF+QSuYsroE+O6WdnyHfSr3/Ox80ye+RbTjRXVNKBRoYadgIee7jQvgxgefeJbqP1/5OP1g1cBAAtHBePV+P7g8XgI8nICAOTqWL7PXtGnQXyEEGJ9Jk2ahN9//73T42pra/Huu+9iy5YtJlgVMSeJSFPBx1b0EWJoeZX1ADTl6ACw9sd0nMgsNeeSuiW2bL+PpxO1NxCjoHptC/TNxVtYfuAKACDhgUCseCiMK4kK9nLC+dwqnfv02WE8rhToE0KI1fn73/+OGTNmwNXVFQ8//DCio6NbDdM9ffo0jhw5gilTpuD9998395KJkbEX9mkYHzEGlZpBfnOgv/qRAfBydsD+CwV4ce8lfPPcCIT7S8y8wu4jp3kQX3Bzko4QQ6NA38J8d6kQr35zGQwDzIvtg1V/i9Dqe9I7o8+V7tN/CoQQYm3mz5+PuXPn4n//+x/279+P//znP6ipqQEA8Hg8REREID4+Hn/88QfCw8PNvFpiCi3D+KhHnxheibQRTSpNOXqAmxhvTxuI/Kp6pORUYv6uP/DdCw/ARyIy9zK7BTYpF0z9+cRIKLqzIIcvF2HJ12lgGM02HKsfHtBquAn7y4K9StiZlmF8lNEnhBBr5ODggLlz52Lu3LkAgJqaGjQ0NMDT0xN2dvS739ZwPfpUuk+M4GZz8NrLwxECPg8CPg9b50bh0U/OIKdChgW7L2DfwliI7WmbOLZ0P9ibMvrEOHQO9NVqNa5du4ZBgwYBALZu3YqmpibucYFAgOeee44G/RjJ8fRSvLI/DWoGmB3dC2unDmxzX+k7M/oMw3Q65ZS9ok/D+AghxDa4urrC1dXV3MsgZsLuskPD+IgxsP35gZ4twaurox12PHkfpn1yBpdv1WDp/9Lw8ePD2vwea0vYpFwIBfrESHQO9Pft24etW7fi119/BQC8+uqrcHNzg1CoOUVFRQVEIhHmz59vnJXauI9OXIdKzWD60B5InD6o3V+OvT0cwecB9U0qlNXK4dtJeRT16BNCCCG2g72wTz36xBjYift9PB217g/0csJnc6Mwd/s5HLlSgg88s7BsUpg5ltgt1DcpUVTTCAAI9qLSfWIcOgf6O3fuxAsvvKB13y+//ILg4GAAmgz/V199RYG+kZRINb8MEh4I6vAKqL2Qj14ejrhZWY+cclmngX5L6T51cRBCCCHWjm3Vq22kHn1ieHnNgf6dGX1WTLAn1k0fjKX/u4xPTmUjyMsJf4/upXWMWs3g1u0GXC+rxfWyOtwoq0NRdQMYxiTLBwA42gvw7ynhRt3bni3b93Cyh7uTvdFeh9g2naO7zMxMREdHt/v46NGj8e9//9sgiyLaGIZBlUzTJuHh3PkvgyAvJ9ysrEduhQyxIZ4dHtsyjI8y+oQQQoi143r0KaNPjCCvQlO6f3dGnzUjqidyKuqw5WQ2/n3wChoUKkgbFLheVofrpXXIqahDo0JtyiW3yd3JHh/8fYjRzs8N4qOJ+8SIdA70y8vLtX7OycmBp2dLEGlnZweZTLdJ70Q/dXIlFCrNpUwPx84D/WAvZ5zKKtdpIB97RZ+G8RFCCCHWj91lp1auhErNQGDjfdLEcNRqBjer2s/os5ZO6I/cChmOXCnBqu+vtXrcXshHsJcT+vq6INTbGX08HU3232l+VT3eP5aFE5llRv33wW2tR/35xIh0DvR9fX2RlZWFkJAQAIC3t7fW4xkZGfDz8zPs6ggAcNl8sZ1ApymlQd66b7FHPfqEEGIbCgoKwOPx0LNnTwDA+fPnsXfvXkRERGDhwoVmXh0xFZc7KvhqGxVw0yGBQIguymrlaFSoIeDz0MNd3O5xfD4P62dFgmHSUFjdgFAfZ/T1cWn+X2duYr85KFRqfPZLNqpkTUjNv437Aj2M8jotE/epP58Yj86B/vjx4/HOO+9g8uTJrR5jGAaJiYkYP368QRdHNCrZsn0de3iCvXQL9JUqNerk7NR96tEnhBBr9o9//AMLFy7EP//5T5SUlGDChAkYMGAA9uzZg5KSEqxatcrcSyQmYC/kQ2wnaC6ZVlKgTwyG7c/v6S6GnaDjXbhEdgJ8OjfKFMvSi52Aj3FhPvgurQjH00uNF+hXNGf0qXSfGJHOe+G9/vrruHr1KmJiYvC///0Ply9fxuXLl/H1118jJiYG165dox59I6mq0wT6njr05wMtW+zlV9VDoWq/z4kN8gHtK/yEEEKsz9WrVzF8+HAAwNdff42BAwfi7Nmz2LNnD3bt2mXexRGT4vr0G6lPnxhOy8R9yw5e4yJ8AQBJ6aVGOT/DMMiljD4xAZ3TuCEhIUhKSsKTTz6J2bNnc/uzMwyDsLAw/PzzzwgNDTXaQm1ZlZ4ZfT+JiLtaf+t2Axf4303aoAn0xXYC2At1vuZDCCHEAikUCjg4OAAAjh8/jkceeQQAEBYWhuLiYnMujZiYRCxEiZQG8lmDszcqUFYrx7ShPcy9FORVagbxBbYziM9SjO7nDTsBDzkVMmSX1xl8+n6pVA5ZkwpCPq/doYWEGIJe9drDhw9Heno60tLS8NdffwEA+vbti6FDhxplcUSjqr450NexvI7P5yHQywkZxVLklNe1H+jT1nqEEGIzBgwYgK1bt2LKlClISkrC22+/DQAoKirSGq5LrB+70w5l9C1bdnkdntz5B5pUavTxdMTQ3u5mXY+1ZPRdRHaIDfHCr3+VIym9FCGjDRvos4P4ens4dtriQMi90Ou/LqlUCrVajcjISMyaNQuzZs3C0KFDoVarIZVKjbVGm6dvRh/QrU+fBvERQojtePfdd/HZZ59hzJgxePzxxzFkiGbrqEOHDnEl/cQ2sDvt1FBG32IxDIM3vruKpuYWze/Tisy8opat9YK8LD9LPSHcB4BxyvezaeI+MRGdA/2DBw8iOjoajY2NrR5raGjAfffdh8OHDxt0cUSjsrlH30PHHn2gpU8/p4NAny3Zk1B/PiGEWL0xY8agoqICFRUV2LFjB3f/woULsXXrVqO9blVVFebMmQOJRAI3NzfMnz8fdXUdb//6zDPPICQkBGKxGN7e3pg6dSoyMzO5x3ft2gUej9fmraysDABw6tSpNh8vKSkx2nu1FFyPfoOykyNJd/VdWiHOZldyP//wZzGUHcxlMjaGYawmow+09Omn5t9GRZ3coOfOpv58YiI6B/qffvopli1bBkfH1lfpnJyc8Nprr+Hjjz826OKIRpVM8wvGU5+MPrvFXnkHgT5Xuk+BPiGEWLuGhgbI5XK4u2vKe2/evImNGzciKysLPj4+RnvdOXPm4Nq1a0hKSsIPP/yAX3/9tdPt/KKiorBz505kZGTg2LFjYBgGEydOhEqlAgDMnj0bxcXFWrf4+HiMHj261XvJysrSOs6Y79VSsDvtUOm+ZaqpV2DtDxkAgMVx/eDuaIeKOjlScio7eabxVNQ1QdakAp+nmbpv6fxdxRjYQwKGAU5klBn03GwSjibuE2PTOdC/evUqxowZ0+7jo0aNwpUrVwyxJnKXltJ9B52f05LRbz9rwl7Jp631CCHE+k2dOhW7d+8GAFRXVyMmJgYffvghpk2bhk8//dQor5mRkYGjR49i27ZtiImJwciRI/HRRx9h3759KCpqv9R44cKFGDVqFAIDAzFs2DCsXbsWBQUFyMvLAwCIxWL4+flxN4FAgBMnTmD+/PmtzuXj46N1LJ9PPbESLqNPgb4levdYJiplTejr44znxoRg8iB/AMAhM5bvs9n8ADcxHIQCs63DkCaE+wEAkjIMW76fw5XuU0afGJfOn3a3b9+GUtl+iZdCocDt27f1XsCWLVsQGBgIkUiEmJgYnD9/vt1jDxw4gOjoaLi5ucHJyQmRkZH48ssvtdbw2muvYdCgQXByckJAQADmzZvX6stEYGBgq1K+devW6b12U6nsQo8+G+iXSuWQydv+/4169AkhxHakpqbiwQcfBAB888038PX1xc2bN7F7925s3rzZKK+ZkpICNzc3REdHc/fFxcWBz+fj3LlzOp1DJpNh586dCAoKQq9evdo8Zvfu3XB0dMTMmTNbPRYZGQl/f39MmDABZ86c6fC15HI5pFKp1s0asS171KNveS7evI295/IBAGunDYS9kI9HhgQAAI5eLUGjQmWWdbVM3LeeLHVchKb657fr5WhoMszfa6NChcLqBgDUo0+MT+dAPzAwEBcuXGj38QsXLqBPnz56vfj+/fuxZMkSvPnmm0hNTcWQIUMQHx/P9dfdzcPDA6+//jpSUlLw559/IiEhAQkJCTh27BgAoL6+HqmpqXjjjTeQmpqKAwcOICsri9tC6E5r1qzRKuV78cUX9Vq7Kd3uQqDv5mjPHd/eQD4q3SeEENtRX18PFxcXAMDPP/+M6dOng8/n4/7778fNmzeN8polJSWtSuWFQiE8PDw67ZX/5JNP4OzsDGdnZ/z0009ISkqCvX3bn4Pbt2/HP/7xD4jFLSXD/v7+2Lp1K7799lt8++236NWrF8aMGYPU1NR2XzMxMRGurq7crb0LC5aO3W1H2kg9+pZEoVLj9YOa6tm/R/VETLBmt4z7Aj3g7ypCrVyJU1mGLTPXVUt/vuUP4mNF+EvQw02MRoUaZ25UGOSceZUyMIymmlafllxCukLnQH/69Ol4/fXXUVraunylpKQEK1euxIwZM/R68fXr12PBggVISEhAREQEtm7dCkdHR60hQXcaM2YMHn30UYSHhyMkJAQvv/wyBg8ejNOnTwMAXF1dkZSUhFmzZqF///64//778fHHH+PixYvIz8/XOpeLi4tWKZ+TU/e8qtaoUEHWfBVRn0Af6HzyPg3jI4QQ2xEaGorvvvsOBQUFOHbsGCZOnAgAKCsrg0Qi0etcy5cvb3cYHnu7c3heV8yZMweXLl3CL7/8gn79+mHWrFltDgROSUlBRkZGq7L9/v3745lnnkFUVBRGjBiBHTt2YMSIEdiwYUO7r7lixQrU1NRwt4KCgnt6D92VK5XuW6RdZ/KQWVILN0c7rJgczt3P5/O4rL65pu9bY0afx+MhzsDT93OaZ2eF+DiDx+MZ5JyEtEfnQH/58uVwcXFB37598fzzz2PTpk3YtGkTnnvuOfTr1w/Ozs5Yvny5zi/c1NSEixcvIi4urmUxfD7i4uKQkpLS6fMZhkFycjKysrIwatSodo+rqakBj8eDm5ub1v3r1q2Dp6cnhg4divfff7/DtgTAfOV8bH++nYCndy99UGeBfvOVfPbKPiGEEOu1atUq/Otf/0JgYCCGDx+O2NhYAJrs/tChQ/U619KlS5GRkdHhLTg4GH5+fq2q9JRKJaqqquDn59fha7i6uqJv374YNWoUvvnmG2RmZuLgwYOtjtu2bRsiIyMRFRXV6bqHDx+OGzdutPu4g4MDJBKJ1s0asRf4aRif5SisbsCG438BAP79UHir5M8jkZpAPzmzDLVm+P/VGjP6ADAhQvN7KjmzFCo1c8/nyy5r7s/3ov58Ynw6R3guLi44c+YMVqxYgf3793P9+G5ubpg7dy7eeecdriRQFxUVFVCpVPD19dW639fXt8MsQE1NDXr06AG5XA6BQIBPPvkEEyZMaPPYxsZGvPbaa3j88ce1PqxfeuklDBs2DB4eHjh79ixWrFiB4uJirF+/vt3XTUxMxFtvvaXz+zMUNtB3d7TX+8pfUHPvDzv0427Uo08IIbZj5syZGDlyJIqLizFkyBDu/vHjx+PRRx/V61ze3t7w9vbu9LjY2FhUV1fj4sWLXCB+4sQJqNVqxMTE6Px6DMOAYRjI5drbXNXV1eHrr79GYmKiTudJS0uDv7+/zq9rrdiWPerRtxyrD11DfZMK9wW6Y2ZUz1aPR/hLEOLthOxyGY5dK23zGGNhGIZLKgVa2ST54UEecHEQoqKuCWkF1Yjq435P5+Mm7lN/PjEBvVK5rq6u+OSTT7BlyxZUVFSAYRh4e3ubtPTExcUFaWlpqKurQ3JyMpYsWYLg4OBWOwIoFArMmjULDMO0mia8ZMkS7s+DBw+Gvb09nnnmGSQmJsLBoe3J9itWrNB6nlQqNUnvXlcG8bGodJ8QQsid2Ha1W7duAQB69uyJ4cOHG+31wsPDMWnSJCxYsABbt26FQqHAokWL8NhjjyEgQJOBLCwsxPjx47F7924MHz4cOTk52L9/PyZOnAhvb2/cunUL69atg1gsxuTJk7XOv3//fiiVSsydO7fVa2/cuBFBQUEYMGAAGhsbsW3bNpw4cQI///yz0d6vpeAy+g3Uo28Jfr5WgqT0Ugj5PLzz6CDw+a2/d/N4PEyN7IH1SX/h+7RCkwb6t+sVqG2uEu3tYV0ZfXshH2PCfHD4chGS0kvvPdBvTr6FUKBPTKBLe8xUVlbi5s2bKCgoQFVVVZde2MvLCwKBoFXPf2lpaYflfHw+H6GhoYiMjMTSpUsxc+bMVlfy2SD/5s2bSEpK6rT0LiYmBkqlktu2py3mKufryiA+FrttR06FDAzTutyIhvERQojtUKvVWLNmDVxdXdGnTx/06dMHbm5uePvtt6FWq432unv27EFYWBjGjx+PyZMnY+TIkfjPf/7DPa5QKJCVlYX6ek2Pr0gkwm+//YbJkycjNDQUs2fPhouLC86ePdtqsN/27dsxffr0Vu15gKZFcOnSpRg0aBBGjx6Ny5cv4/jx4xg/frzR3qulYCv5GhQqNCmN9/89uXcyuRKrD10DACwYFYx+vu1Xz7J9+mduVKC8Vt7ucYaW11y27+8qgsjOOrbWu9OECE318fF73GaPYRiuR5+21iOmoFdG/9q1a3juuedabU8zevRofPrpp+jfv7/O57K3t0dUVBSSk5Mxbdo0AJovIcnJyVi0aJHO51Gr1VqlfGyQf/36dZw8eRKenp6dniMtLQ18Pr/VF4ju4F4y+r09HMHjAbWNSlTKmuDlrF2twF7Jp4w+IYRYv9dffx3bt2/HunXr8MADDwAATp8+jdWrV6OxsRHvvPOOUV7Xw8MDe/fubffxwMBArYvRAQEBOHLkiE7nPnv2bLuPLVu2DMuWLdN9oTbE+Y6ZP7WNCng6t13NSMxvc/J1FNU0oqe7GC+N69vhsYFeThjSyw2XC6rx459FePKBIJOs0Vr781mj+3lDyOfhRlkdcitk3AwsfZXXyVErV4LPs96/K9K96Bzol5SUYPTo0fD29sb69esRFhYGhmGQnp6Ozz//HA8++CCuXr2qV7C8ZMkSPPHEE4iOjsbw4cOxceNGyGQyJCQkAADmzZuHHj16cBn7xMREREdHIyQkBHK5HEeOHMGXX37JleYrFArMnDkTqamp+OGHH6BSqbjtezw8PGBvb4+UlBScO3cOY8eOhYuLC1JSUrB48WLMnTsX7u73Vo5jDFUyzUWMrmzBIbIToIebGLduNyCnXKYV6Dcp1Who3muVhvERQoj1++KLL7Bt2zatLWcHDx6MHj164PnnnzdaoE+6HwGfBxcHIWrlSkgblRTod1OZJVJsO50LAFgzdQDE9p1ny6cOCcDlgmocuqxfoK9QqfHpqWz093NB/ICOB2XeLa/C+ibu38lVbIf7gz1x+kYFjqeXYsGo4C6dh83m93R3hIPQ+iofSPejc4S3YcMG9OnTB2fOnIFIJOLunzRpEp577jmMHDkSGzZs0HkgDgDMnj0b5eXlWLVqFUpKShAZGYmjR49yA/ry8/PB57d0F8hkMjz//PO4desWxGIxwsLC8NVXX2H27NkANH1+hw4dAgBERkZqvdbJkycxZswYODg4YN++fVi9ejXkcjmCgoKwePFirf777qSKy+h37UM4yMsJt243ILeiDsODPLj775y060IZfUIIsXpVVVUICwtrdX9YWFiX2/CI5ZKI7VArV9JAPjOorm9CckYZlJ20zOw9lw+VmsFDA/0wLsy3w2NZfxvsj7U/piM1vxr5lfXorWPmeOPxv7DlZDacHYQYv8oHQoHu3b0tGX3rDPQBIC7cB6dvVCDJAIE+9ecTU9E50E9KSsLy5cu1gnyWWCzGq6++ivfee0+vQB8AFi1a1G6p/qlTp7R+Xrt2LdauXdvuue4u/2vLsGHD8Pvvv+u1RnOqrGsO9J31z+gDQIi3M367XsFN+WSxg/hcHIQQtDHUhRBCiHUZMmQIPv74Y2zevFnr/o8//lhrCj+xDRKxHQqrG7jvA8R03jqcjoOXCnU61slegFUPR+h8bh+JCLEhnjhzoxKH/yzCC2NDO33O2RsV+ORUNgCgTq7ElcIaDO2te5VrXiWb0bfecvS4CF+sPpyOCzerUCVr6lJLbXbzID7qzyemonOgn5OTg2HDhrX7eHR0NHJycgyyKNLidr0m0O9K6T4Aro8ot/yuQL95OioN4iOEENvw3nvvYcqUKTh+/DhiY2MBACkpKSgoKNC5J55YD0lzn77UDHuu2zKGYXDmRgUA4P5gDzg7tP9VnMfjYVZ0L/i7ivV6jalDeuDMjUp8d6kQz48J6XB3rMo6OV7ZnwaGAYR8HpRqBik5lXoF+raQ0e/p7ohwfwkyiqU4kVnWpV0NcrhA33r/nkj3onOgX1tb2+GkeRcXF9TVtb1fO+k6dhifu+M9Bvp3ZfTZUj0XEfXnE0KILRg9ejT++usvbNmyBZmZmQCA6dOn4/nnn+e2uiO2g73QT1vsmVZhdQPKauUQ8nnYlTDcKFPq4wf6YeV3V3G9rA6ZJbUI92/7+7tazeBf/7uMslo5+vo4Y9rQHnj/WBZSsivx/JjOKwEAoKZegdv1mu+UgV7Wm9EHNNP3M4qlSEov6Vqg3/xdPNiLMvrENPSK8mpra9ss3Qc0+8p3VjZP9Mf26Ht2sXSfDfRvVtZDpWa4Mn22VM+VMvqEEGIzAgICWg3du3XrFhYuXKi15R2xfuyOO9Sjb1oXb94GAAzo4Wq0rehcxXYY098bP6eX4tDlonYD/Z1n83AyqxwOQj4++sdQMAzw/rEsXMi7jSalGvbCzvv0b1ZpglcfFwc42lt38mhCuC82J1/Hr39VoFGh0uv/P7lShYIqTYsD9egTU9F50gbDMOjXrx/c3d3bvOmztR7RjVKlRnXzVdKu9AIBQICbGPZCPppUahTebuDuZ0v1qHSfEEJsW2VlJbZv327uZRATYy/0U+m+aaU2B/rDersZ9XWmRvYAABxKK4Ja3ToRd7WwBut+ygAArPxbBML8JOjv6wJ3Rzs0KFT481a1Tq/T0p9v/cHrwB4S+ElEaFCocDa7Qq/n5lfWQ80Azg5CeLvQLhfENHS+9Hby5EljroO0gS2F4vG6Xrov4PMQ5OmErNJa5FTUcdNX2VI9CU3cJ4QQQmwOu7UuDeMzrYv5mkA/qo9xt3QeH+4DJ3sBCqsbkJp/G9GBLTsv1cmVePG/l6BQMYgf4Iu5Mb0BAHw+D/cHe+KnqyVIya7Uek57blaw/fnWXbYPaGYmxEX44Kvf85GUXqbzTggAkN08KyvY26nDmQmEGJLOgf7o0aONuQ7SBrZs301sd0+T8YO8NIF+boUMY5oLL1oy+tZdZkUIIYSQ1tgL/exwXmJ89U1KZBTXAjB+oC+yEyB+oB8OpBbi+7QiraB91fdXkVshQ4CrCO/OGKwVeMaGNAf6OZV4cXzfTl+Hy+h7WX9GHwAmRPjhq9/zcTyjFO+oB4Kv4/fznArNHLMQmrhPTEj3TTKJybGBvnsXy/ZZQd6tB/LVUI8+IYQQYrPY1j3q0TedywU1UKkZ+LuK9J6k3xVs+f6RK8VQqNQAgIOXbuFAaiH4PGDT40PhdlfFaGywJwDNLAG5UtXpa7RM3Lf+jD7QslNCea0cfxbW6Py87DJ2EJ9tXBAh3YPO6Vw+n99pqQmPx4NSSVeGDYUbxHevgX7zL5WcO7bYY0v1qHSfEEKs2/Tp0zt8vLq62jQLId0K16NPgb7JpDaX7Q8zcjaf9UCIJzyd7FEpa8KZGxXo4+mElQevAgBeieuH+9oozQ/1cYaXswMq6uS4lF+N+5sD//bYUo8+ADgIBRjdzxs/XilGUnoJInu56fQ8NqMfTBl9YkI6B/oHDx5s97GUlBRs3rwZarXaIIsiGlUyOYCuD+JjhbSR0WdL9WgYHyGEWDdXV9dOH583b56JVkO6C0nz9ro0jM902In7UXrsUX8vhAI+pgz2x+6Um/jm4i3crKyHrEmFmCAPvDC27e3zeDwe7g/2wA9/FiMlu7LDQL+2UYGKOs131d42ktEHNNvs/XilGEeulGDphP6dlu8zDMMl24Jp4j4xIZ0D/alTp7a6LysrC8uXL8fhw4cxZ84crFmzxqCLs3WVzRl9D6d7m84Z1LxfZ2F1A7cdSEtGn3r0CSHEmu3cudPcSyDdkITL6FMlpikwDGPyjD4ATI0MwO6Um/jhz2IAgJujHTY+Ftnh7KfYEE9NoJ9TicUdnPtmczbf08nepipEx4f7wEUkRG6FDEeuFuNvgwM6PL5K1oSaBgV4vJYqW0JMoUs9+kVFRViwYAEGDRoEpVKJtLQ0fPHFF+jTp4+h12fTDFW67+5ox5Xo5TX3UnGBPmX0CSGEEJsjoe31TCqnQobqegUchHxEtLOvvTEM6+2Onu4t8wDenzmk0/kAbJ9+Wn41GhXt9+mzgb6t9OezXER2mD8yCACw6fj1NrcvvFNOc0VtgKsYIjuB0ddHCEuvQL+mpgavvfYaQkNDce3aNSQnJ+Pw4cMYOHCgsdZn06q4jP69Bfo8Hq9Vnz77wU7D+AghhBDbw1b0NSnVHQZzxDDYsv0hPd1gLzTdLGwej4fH7usFAHjqgSBMiOh8S7ggLyf4ShzQpFJz624Lmzyylf78OyU8EAQXkRDXy+pw5Gpxh8fmlLP9+bb390TMS+ffNO+99x6Cg4Pxww8/4L///S/Onj2LBx980Jhrs3mGCvSBll8uuRUyMAzDlepRRp8QQgixPc4OQrDV2zSQz/hSb5q+bJ/13JhQHHtlFN74W7hOx/N4PC6rn5Jd2e5xLRP3bS+AdRXrntVnk2y0tR4xNZ0btJcvXw6xWIzQ0FB88cUX+OKLL9o87sCBAwZbnK0zaKB/R0ZfrlSjqXmbFerRJ4QQQmwPj8eDRGyH6noFpI0K+EhE5l6SVeP683u7mfy1BXwe+vu56PWc2BBPfJdWhJSc9gN9buK+l22V7rMSHgjCjtO5uF5Whx+vFOPhIW336mc3Z/RDKKNPTEznKG/evHmdbq9HDKvSgIE+O5Avt6KO2zOXzwOc7CnQJ4QQQmyRRKQJ9GtoIJ9R1TQo8FepJtgzR0a/K0aEeAEALhdUQyZXwsmh9fdFW87oA2xWPxgbjv+FzcnXMXmQf5tDDlsm7lNGn5iWzlHerl27jLgMcjeGYXCbHcbnbIhAv6V0/85BfJ1tCUIIIYQQ6yQRN2+xR6X7RnWpOZsf6OkIL+d720nJVHp5OKKHmxiF1Q24cPM2Rvfz1nq8vkmJUqlma71AGxvGd6cnHwjE9tM5ml79NrL6CpUa+VWaygfq0SemZrppIEQv0kYllM39Pu6Ohgv0b9cruCmptrQVCiGEEEK0udLkfZPg+vN7W0Y2nxUb0n6fPhu8ujnawc0A31MtFZvVB4DNydehuqtXP7+qHko1A0d7AfyoPYaYGAX63RTbn+9kLzDIVhxiewECXDW/YC7fqgbQciWfEEIIIbaHveBPGX3jSs2vBmA5ZfssbiBfG336eRXs1nqUpX7ygUBI2An8V7Qn8LNl+0FeTtQCTUyOAv1uqkqmKYfyMEDZPiuouWQoraAaAGX0CSGEEFvGBfqN1KNvLCo1w5XuR1laoN+c0b9aWIPau6o+bnJb69lu2T7rzqz+pruy+i1b61F/PjE9CvS7qco6dhCf4Xq52PL9y82BvittrUcIIYTYLLayr4Yy+kaTVVILWZMKzg5C9PPVb/K9uQW4idHH0xEqNYM/8qq0HmMn7lNGXyNhpCarf6N5Aj+LG8TnRX9PxPQo0O+m2NJ9TwNM3GcFN0/eZ6/cU0afEEIIsV1cjz4F+kZzsTmbH9nLrc2J7N0dV75/V58+ZfS1SUR2ePrB1r363NZ6PpTRJ6ZHgX43Zcit9VhBd037pB59QgghxHa5Ng9RK5U2mnkl1usSO4jPwsr2WdxAvpy7A33K6N+N7dW/M6ufU0EZfWI+FOh3U7eNktG/K9CnjD4hhBBis4b2cgMAnM+tglypMu9irNRFC+3PZ7EZ/WtFUtTUayo/GhUqFNU0AKCM/p3uzupXyZq4Cl3aWo+YAwX63RT7i8HdgIF+Dzcx7AQtZWMS6tEnhBBCbNaAAAm8XRwga1LhQt5tcy/H6lTUybnMd2TzRRVL4yMRIdjbCQwDnMvVZPVv3a4HwwAuDkKDVp5agzuz+puTrwMA/F1FcLSnKlpiehTod1PGKN0XCvhaJVY0jI8QQgixXTweD6P7eQMATmWVmXk11ie1uWy/n6+zRX/nYrP6Z5v79HPZrfW8HGnLuLvcmdX/IiUPAGXziflQoN9NGWMYH9AyeR+gHn1CCCHE1o3t7wMAOJlVbuaVWB9LL9tnsX36vzf36bOD+Kg/v21PPhAIV7EdmOZd9thh2ISYGgX63VSVETL6gHafPvXoE0IIMbaqqirMmTMHEokEbm5umD9/Purq6nR6LsMweOihh8Dj8fDdd99pPZafn48pU6bA0dERPj4+ePXVV6FUau8Hf+rUKQwbNgwODg4IDQ3Frl27DPSurMfIvl4Q8Hm4UVaHW7frzb0cq8Jm9If1tuxA//7mjH5mSS0q6+TIo4n7HZKI7PD0yCDuZ8roE3OhQL+basnoOxj0vNoZfQr0CSGEGNecOXNw7do1JCUl4YcffsCvv/6KhQsX6vTcjRs3tlkarFKpMGXKFDQ1NeHs2bP44osvsGvXLqxatYo7Jjc3F1OmTMHYsWORlpaGV155BU8//TSOHTtmsPdmDVzFdhjW2w0AcIqy+gbTpFTj8q0aAJY7cZ/l5eyAfr6arPS53CqauK+DJ5qz+gDQ39fFzKshtsrsgf6WLVsQGBgIkUiEmJgYnD9/vt1jDxw4gOjoaLi5ucHJyQmRkZH48ssvtY5hGAarVq2Cv78/xGIx4uLicP36da1j7iW7YAoNTSo0KDTTb92dDBuM3xnoW3K/GCGEkO4vIyMDR48exbZt2xATE4ORI0fio48+wr59+1BUVNThc9PS0vDhhx9ix44drR77+eefkZ6ejq+++gqRkZF46KGH8Pbbb2PLli1oatJcKN+6dSuCgoLw4YcfIjw8HIsWLcLMmTOxYcMGo7xXSzamuXyf+vQNJ71YiialGm6OdlaxtRrbp5+SXXlHRt/y35exSER22P5ENFZOCedaHwgxNbMG+vv378eSJUvw5ptvIjU1FUOGDEF8fDzKytr+oPHw8MDrr7+OlJQU/Pnnn0hISEBCQoLW1fn33nsPmzdvxtatW3Hu3Dk4OTkhPj4ejY0te8TeS3bBFCplcgCAvYAPZwfD9tGH+GiuyPJ5VLpPCCHEuFJSUuDm5obo6Gjuvri4OPD5fJw7d67d59XX1+Mf//gHtmzZAj8/vzbPO2jQIPj6+nL3xcfHQyqV4tq1a9wxcXFxWs+Lj49HSkpKu68rl8shlUq1brZgTH/NQL6z2ZW0zZ6BXGwu24/q7W4VA+vYYPW36+UovE1b6+kiOtADTz8YbBX//xPLZNZAf/369ViwYAESEhIQERGBrVu3wtHRsc2r9wAwZswYPProowgPD0dISAhefvllDB48GKdPnwagyeZv3LgRK1euxNSpUzF48GDs3r0bRUVFXG/fvWQXTOXO/nxD/3LwcnbAyinheONvERDbCwx6bkIIIeROJSUl8PHx0bpPKBTCw8MDJSUl7T5v8eLFGDFiBKZOndruee8M8gFwP7Pnbe8YqVSKhoaGNs+bmJgIV1dX7tarV6+O36CViPCXwMfFAfVNKvyRS9vsGQLXn2/hZfusmCBP8HhAXmU91AwgthPA28Ww7aWEEMMyW6Df1NSEixcval1t5/P5iIuL6/BqO4thGCQnJyMrKwujRo0CoOnHKykp0Tqnq6srYmJiuHN2NbtgSsbYWu9OTz8YjIQHgjo/kBBCCGnD8uXLwePxOrxlZmZ26dyHDh3CiRMnsHHjRsMuWgcrVqxATU0NdysoKDD5GsyBx+NxWf2TVL5vEBetZBAfy93JHmF+Eu7nPp60tR4h3Z3Z9lerqKiASqVq82p7R18Oampq0KNHD8jlcggEAnzyySeYMGECgJar+G2d884r/F3JLsjlcsjlcu5nY5bz3WYH8TkbJ9AnhBBC7sXSpUvx5JNPdnhMcHAw/Pz8WrXjKZVKVFVVtVmSDwAnTpxAdnY23NzctO6fMWMGHnzwQZw6dQp+fn6tZvqUlpYCAHdePz8/7r47j5FIJBCLxW2+toODAxwcbDNLOaa/D76+cAunssrwxt8izL0ci1ZU3YASaSMEfB6G9HI193IMJjbYExnFmu+/QVYwd4AQa2dxG6m7uLggLS0NdXV1SE5OxpIlSxAcHIwxY8YY9XUTExPx1ltvGfU1WMbaWo8QQggxBG9vb3h7e3d6XGxsLKqrq3Hx4kVERUUB0ATyarUaMTExbT5n+fLlePrpp7XuGzRoEDZs2ICHH36YO+8777yDsrIy7uJ9UlISJBIJIiIiuGOOHDmidZ6kpCTExsbq92ZtBLvNXna5DAVV9ejlQf3XXcVm8yP8JXC0t7iv2u2KDfHEjjO5AGjiPiGWwGyl+15eXhAIBG1ebW/vKj+gKe8PDQ1FZGQkli5dipkzZyIxMRFAy1X8js7ZlewCYNpyPrZ0392RAn1CCCGWKzw8HJMmTcKCBQtw/vx5nDlzBosWLcJjjz2GgIAAAEBhYSHCwsK4DL2fnx8GDhyodQOA3r17IyhI03Y2ceJERERE4J///CcuX76MY8eOYeXKlXjhhRe4jPyzzz6LnJwcLFu2DJmZmfjkk0/w9ddfY/HixWb4m+j+JCI7RDX3k9P0/XvTUrbvZt6FGNjwIA/wm6v1aRAfId2f2QJ9e3t7REVFITk5mbtPrVYjOTlZr6vtarWaK6kPCgqCn5+f1jmlUinOnTvHnfPO7AKrs+wCoCnnk0gkWjdjqaprLt2njD4hhBALt2fPHoSFhWH8+PGYPHkyRo4cif/85z/c4wqFAllZWaivr9f5nAKBAD/88AMEAgFiY2Mxd+5czJs3D2vWrOGOCQoKwo8//oikpCQMGTIEH374IbZt24b4+HiDvj9rwvbpn8oqN/NKLFtqvnUN4mO5iu0QG6IZyjekl5u5l0MI6YRZ64mWLFmCJ554AtHR0Rg+fDg2btwImUyGhIQEAMC8efPQo0cPLmOfmJiI6OhohISEQC6X48iRI/jyyy/x6aefAtAMk3nllVewdu1a9O3bF0FBQXjjjTcQEBCAadOmAdDOLmzduhUKhaJVdsHcuGF81KNPCCHEwnl4eGDv3r3tPh4YGAiGYTo8R1uP9+nTp1Vp/t3GjBmDS5cu6bZQgrH9ffDe0Sycza5Eo0IFkR3tzqOvhiYV0os0fexRVhboA8DHjw9DaW2j1mA+Qkj3ZNZAf/bs2SgvL8eqVatQUlKCyMhIHD16lBuml5+fDz6/pehAJpPh+eefx61btyAWixEWFoavvvoKs2fP5o5ZtmwZZDIZFi5ciOrqaowcORJHjx6FSCTijtmzZw8WLVqE8ePHg8/nY8aMGdi8ebPp3ngnqmSaCgXK6BNCCCHEVML8XOAnEaFE2ojzuVUY1a/zOQxE25+3qqFUM/BxcUAPt7aHPloydyd7uNP3U0IsAo/p7DI6aZNUKoWrqytqamoMXsY/9oNTyK2Q4etnYjE8yMOg5yaEEGKdjPm5ZKts8e/0tW/+xP4LBXjqgSCsepim7+vrk1M38N7RLDw00A+fzo0y93IIIVZI188ms/Xok/ZV1mky+jR1nxBCCCGmNDaM7dOngXxdkdo8iM8ay/YJIZbFevb8sBIKlRrSRiUACvQJIYQQYloPhHpByOchp0KG/Mp69Kbp6gCAgqp6LP36MmoaFB0el1shA2B9g/gIIZaHAv1u5nbzID4+D3AT25l5NYQQQgixJS7N2+ydy63Cqb/KMC820NxL6ha+Tb2F83lVOh3r5WyPAQG20epBCOm+KNDvZtiJ++6O9uCzm5USQgghhJjI2DAfnMutwslMCvRZ7CT9J0cEYkKEb4fH9vV1hoOQdiwghJgXBfrdDJvRp7J9QgghhJjDmP7eWPdTJlJyaJs9VkaJJtCfOMAXI0K8zLwaQgjpHA3j62YqKdAnhBBCiBn199Vss9eoUONcrm7l6tZM2qhAQVUDACDCn0ryCSGWgQL9bqaKAn1CCCGEmBGPx+Om75/MpOn7mcW1AIAAVxHcHOn7Gfn/9u49qqo6///46wACykUuIkig0OT9goppaOWNQJtx7PabGcdmvE02v6BUrEZ/3xLHpsDK0lxOzkxjP+dXpitNa6YV5ajQqIiGg6OGTJqOpqAoXhCT6/79gZw8AyoasM/ZPB9rnbU4e3/2Pu/9Oeib99mfz+cAroFC38lwRx8AAJhteLeOkqSsfxebHIn5vjxxXpLUiwX2ALgQCn0nU1JWLkkKptAHAAAmGXZHsDzcbDp8ukxHrnxlXGuVf+WOfk+G7QNwIRT6TuZsWe33s3JHHwAAmMXPu43ujAqSJGUWtO7h+18W1i7Ex/x8AK6EQt/JnLlyRz/I18vkSAAAQGs2onvtPP3MVjx8v6q6RgUnuaMPwPVQ6DuZusX4GLoPAADMNKJ77Tz97EO1X7PXGn19ukwVVTXy8XRX56B2ZocDAI1Goe9k6gr9QFZ1BQAAJuoW6qvw9t4qr6pR9tdnzA7HFPlXhu336OQvNzebydEAQONR6DuRmhpDZy/VztEP9qXQBwAA5rHZbBp+5a5+VkHrHL7/5Qnm5wNwTR5mB4DvnP+2UtU1hiTu6AMAAPON7B6i93Ye1brd36igqPS6bQPatdHM+G7qHubXQtE1v7qF+JifD8DVUOg7kZJLtcP2/bw95OnBYAsAAGCuYXd0kJ+Xh0ovVzVq+P7Wr05r2cSBurdbSAtE1/zqhu73CqfQB+BaKPSdCAvxAQAAZ+Lj5aENycPsQ9ivxZD0zo7/aOfhEk35v7v0wvg++vmQzi0TZDM5VXpZpy9WyM0mdQ+1zigFAK0Dhb4TOXPxykJ8FPoAAMBJ/CDEVz8I8b1hu8TeoZq7bq8++Odx/Z/1e/WfM2X6zZgeLruIXd2HG9EdfNTW093kaADg5jA+3IlwRx8AALgqLw93LfpJjGbFd5Mk/eHzr/XEu7v1bYVrfjVffmHtmgTMzwfgiij0nUhJWbkkKYhCHwAAuCCbzaYZ8V21+Kf95enupoz9RfrZH7N1qvSy2aHdtC+Znw/AhVHoO5GSstqv1gvy8TI5EgAAgFv3wIDb9M6vhiiwXRvt+ea8Hly2Xf8+ef1V+51NPivuA3BhFPpOpO6OPkP3AQCAqxscHaQPnhim6A4+On7uWz38++36x1fFZofVKJcrq/V18UVJUm8KfQAuiMX4nMiZK3P0GboPAACsILqDjz7430P1+Du52nm4RJPf3qW77+gg9xZcoG9snzD9r0GRN3VMQVGpaozamy8hfoy0BOB6KPSdSAmFPgAAsJhAH0/9v2mDNWfdXq3/53Fl/btl7+pvPXhaY/t2kq9X4//svXp+vs3mmt8aAKB1o9B3IhT6AADAirw83PXaT2L04IDbVHSh5RbmW/L3r3T83LfKKijWD/t1avRxzM8H4Ooo9J2EYRgU+gAAwLJsNpvu7RbSoq95qPii/pD1tTL2F91Uof/liSt39Cn0AbgoFuNzEpcqqlVeVSNJCval0AcAAPi+xvQOkyRtOXBK5VXVjTqmpsbgjj4Al0eh7yTq7uZ7t3FTO08GWgAArKGkpEQTJ06Uv7+/AgICNG3aNF28eLFRxxqGobFjx8pms2nDhg327Xv27NGECRMUGRmptm3bqmfPnlqyZInDsZmZmbLZbPUeRUVFTXl5cHIxEQEK9ffSxfIqbT94plHHHDt7SWUV1fL0cNPtIT7NHCEANA8qSidhX3G/HXfzAQDWMXHiRBUWFmrjxo2qrKzUlClTNH36dK1ateqGxy5evLjBhdByc3PVsWNHvfPOO4qMjNT27ds1ffp0ubu7Kzk52aFtQUGB/P2/uyvbsWPH739RcBlubjYl9g7TX7L/o4x9RRrZ48bvf92w/W6hvmrjzj0xAK7J9P+9li1bpqioKHl7e2vIkCHauXPnNdv+6U9/0j333KPAwEAFBgYqPj6+XvuGPr232Wx65ZVX7G2ioqLq7U9PT2+2a2yMkrJySVIQw/YBABaRn5+vjIwMvfXWWxoyZIjuvvtuLV26VKtXr9aJEyeue2xeXp4WLVqkFStW1Ns3depULVmyRMOHD9ftt9+uRx99VFOmTNEHH3xQr23Hjh0VFhZmf7i5mf6nD1pY3fD9jfknVVVdc8P2dcP2mZ8PwJWZmu3WrFmjlJQUpaamavfu3YqJiVFiYqJOnTrVYPvMzExNmDBBW7ZsUXZ2tiIjI5WQkKDjx4/b2xQWFjo8VqxYIZvNpocfftjhXAsWLHBo9+STTzbrtd7ImYt1C/HxXa0AAGvIzs5WQECABg0aZN8WHx8vNzc35eTkXPO4S5cu6ec//7mWLVumsLCwRr3W+fPnFRQUVG97//791alTJ913333atm3bdc9RXl6uCxcuODzg+gZHBymgXRuVlFXoi/+cvWH7L5mfD8ACTC30X3vtNT322GOaMmWKevXqpeXLl6tdu3YNfnovSe+++66eeOIJ9e/fXz169NBbb72lmpoabdq0yd7m6k/tw8LC9OGHH2rkyJG6/fbbHc7l5+fn0M7Hx9w5WGcv1Rb6way4DwCwiKKionpD5T08PBQUFHTdufKzZs3S0KFDNX78+Ea9zvbt27VmzRpNnz7dvq1Tp05avny51q1bp3Xr1ikyMlIjRozQ7t27r3metLQ0tW/f3v6IjIxs1OvDuXm4uym+Z6gkKWPfjddoyC8slcQdfQCuzbRCv6KiQrm5uYqPj/8uGDc3xcfHKzs7u1HnuHTpkiorKxv8BF+STp48qY8//ljTpk2rty89PV3BwcEaMGCAXnnlFVVVVd3ahTSRM3y1HgDARcyZM+eaU+XqHgcOHLilc3/00UfavHmzFi9e3Kj2+/bt0/jx45WamqqEhAT79u7du+vxxx9XbGyshg4dqhUrVmjo0KF6/fXXr3muuXPn6vz58/bHsWPHbuka4Hzqhu9/tr9IhmFcs925SxU6fu5bSVIPCn0ALsy0xfhOnz6t6upqhYaGOmwPDQ1t9B8Hv/nNbxQeHu7wYcHVVq5cKT8/Pz300EMO25966ikNHDhQQUFB2r59u+bOnavCwkK99tpr13yt8vJylZeX25839XC+kosU+gAA1zB79mxNnjz5um1uv/12hYWF1ZuOV1VVpZKSkmsOyd+8ebMOHTqkgIAAh+0PP/yw7rnnHmVmZtq3ffnllxo9erSmT5+u55577oZxDx48WFu3br3mfi8vL3l5MYXOiu7u2kHtPN114vxl7T1+Xv0iAhpsV3c3PyKwrdq3bdOCEQJA03LZVffT09O1evVqZWZmytvbu8E2K1as0MSJE+vtT0lJsf/cr18/eXp66vHHH1daWto1E3xaWpp++9vfNt0F/JcS7ugDAFxESEiIQkJCbtguLi5O586dU25urmJjYyXVFvI1NTUaMmRIg8fMmTNHv/rVrxy29e3bV6+//rrGjRtn37Z//36NGjVKkyZN0osvvtiouPPy8tSpU6dGtYW1eLdx18juHfXx3kJl7Cu6ZqHP/HwAVmHa0P0OHTrI3d1dJ0+edNh+8uTJGy688+qrryo9PV2fffaZ+vXr12Cbf/zjHyooKKj3x0JDhgwZoqqqKh05cuSabZp7OB9D9wEAVtOzZ0+NGTNGjz32mHbu3Klt27YpOTlZP/vZzxQeHi5JOn78uHr06GH/Fp2wsDD16dPH4SFJnTt3VnR0tKTa4fojR45UQkKCUlJSVFRUpKKiIhUXF9tfe/Hixfrwww918OBB7du3TzNnztTmzZuVlJTUwr0AZ5HYp/bvy4z9156nz4r7AKzCtELf09NTsbGxDgvp1S2sFxcXd83jXn75Zb3wwgvKyMhwWMX3v/35z39WbGysYmJibhhLXl6e3Nzcrvvdul5eXvL393d4NCUW4wMAWNG7776rHj16aPTo0br//vt19913649//KN9f2VlpQoKCnTp0qVGn3Pt2rUqLi7WO++8o06dOtkfd955p71NRUWFZs+erb59+2r48OHas2eP/v73v2v06NFNen1wHSO7h8jT3U1fF5fp4KnSBtt8eYI7+gCswdSh+ykpKZo0aZIGDRqkwYMHa/HixSorK9OUKVMkSb/85S912223KS0tTZK0cOFCzZs3T6tWrVJUVJR9xV5fX1/5+vraz3vhwgW9//77WrRoUb3XzM7OVk5OjkaOHCk/Pz9lZ2dr1qxZevTRRxUYGNgCV90w5ugDAKwoKChIq1atuub+qKio6y6OJqne/vnz52v+/PnXPebZZ5/Vs88+2+g4YX1+3m007I5gbSkoVsa+IiWP8nPYX1FVo4OnLkqSeodT6ANwbaYW+j/96U9VXFysefPmqaioSP3791dGRoZ9gb6jR4/Kze27QQdvvvmmKioq9MgjjzicJzU11SHhr169WoZhaMKECfVe08vLS6tXr9b8+fNVXl6u6OhozZo1y2Hefksrr6pWaXntqv/BPiwCBAAA0BzG9AmrLfT3Fyl5VFeHfYeKL6qiukZ+Xh6KCGxrUoQA0DRMX4wvOTlZycnJDe67emVdSdedQ3+16dOnO3yX7tUGDhyoHTt23EyIze5sWaUkyd3NJj9v098SAAAAS4rvGSo3217tO35B35y9pIjAdvZ9+VctxGez2cwKEQCahGlz9PGdM2W1X9sX2M5Tbm4kFgAAgOYQ7OulO6OCJEmf7ndcEPq7+fl+9Y4DAFdDoe8E6u7osxAfAABA8xpzZfX9T/c5rr6fX3RlxX3m5wOwAAp9J1B3R5+F+AAAAJpXQu/aQn/Xf0pUXFr7N5hhGKy4D8BSKPSdQEnZlRX3fSn0AQAAmtNtAW3VL6K9DEP6e37t8P2TF8p19lKl3N1s6hbK0H0Aro9C3wnUFfoM3QcAAGh+iVfu6mdcGb7/ZeF5SdIPQnzk3cbdtLgAoKlQ6DuBM1cK/cB2FPoAAADNrW6e/vZDp3XhcqXyC0slMWwfgHVQ6DuBkotX7ugzdB8AAKDZ/SDEV3d09FVltaEtB07Z5+f3otAHYBEU+k6g5NKVOfoM3QcAAGgRY64avp9fyEJ8AKyFQt8J2Bfjo9AHAABoEXXD9zMLinX4TJkkCn0A1kGh7wS+W4zPy+RIAAAAWofe4f66LaCtvq2slmFIIX5eCvHjbzEA1kChb7LqGkNnrwzdD/RpY3I0AAAArYPNZrOvvi8xPx+AtVDom+zcpQoZRu3PrLoPAADQcuqG70sM2wdgLRT6Jqu7m9++bRu1ceftAAAAaCmxXQIVfGWNpF7hFPoArMPD7ABg07A7gtW2DW8FAABAS3J3s+nFB/toy4FiJfQKNTscAGgyVJcmu6Ojr9791V1mhwEAANAqjenTSWP6dDI7DABoUowVBwAAAADAQij0AQAAAACwEAp9AAAAAAAshEIfAAAAAAALodAHAAAAAMBCKPQBAAAAALAQCn0AAAAAACyEQh8AAAAAAAuh0AcAAAAAwEIo9AEAAAAAsBAKfQAAAAAALMTD7ABclWEYkqQLFy6YHAkAAN/lo7r8hO+PXA8AcDaNzfcU+reotLRUkhQZGWlyJAAAfKe0tFTt27c3OwxLINcDAJzVjfK9zeCj/1tSU1OjEydOyM/PTzabrcE2Fy5cUGRkpI4dOyZ/f/8WjrD1ot/NQb+bg35vec7a54ZhqLS0VOHh4XJzY2ZeU2hMrpec93fCyuhzc9Dv5qDfzeGs/d7YfM8d/Vvk5uamiIiIRrX19/d3ql+O1oJ+Nwf9bg76veU5Y59zJ79p3Uyul5zzd8Lq6HNz0O/moN/N4Yz93ph8z0f+AAAAAABYCIU+AAAAAAAWQqHfjLy8vJSamiovLy+zQ2lV6Hdz0O/moN9bHn2O/8bvRMujz81Bv5uDfjeHq/c7i/EBAAAAAGAh3NEHAAAAAMBCKPQBAAAAALAQCn0AAAAAACyEQh8AAAAAAAuh0G9Gy5YtU1RUlLy9vTVkyBDt3LnT7JAs5fPPP9e4ceMUHh4um82mDRs2OOw3DEPz5s1Tp06d1LZtW8XHx+urr74yJ1iLSEtL05133ik/Pz917NhRDzzwgAoKChzaXL58WUlJSQoODpavr68efvhhnTx50qSIreHNN99Uv3795O/vL39/f8XFxemTTz6x76fPm196erpsNptmzpxp30a/QyLXNzdyfcsj15uDXO8crJTvKfSbyZo1a5SSkqLU1FTt3r1bMTExSkxM1KlTp8wOzTLKysoUExOjZcuWNbj/5Zdf1htvvKHly5crJydHPj4+SkxM1OXLl1s4UuvIyspSUlKSduzYoY0bN6qyslIJCQkqKyuzt5k1a5b++te/6v3331dWVpZOnDihhx56yMSoXV9ERITS09OVm5urL774QqNGjdL48eO1f/9+SfR5c9u1a5f+8Ic/qF+/fg7b6XeQ65sfub7lkevNQa43n+XyvYFmMXjwYCMpKcn+vLq62ggPDzfS0tJMjMq6JBnr16+3P6+pqTHCwsKMV155xb7t3LlzhpeXl/Hee++ZEKE1nTp1ypBkZGVlGYZR28dt2rQx3n//fXub/Px8Q5KRnZ1tVpiWFBgYaLz11lv0eTMrLS01unbtamzcuNEYPny4MWPGDMMw+F1HLXJ9yyLXm4Ncbx5yfcuxYr7njn4zqKioUG5uruLj4+3b3NzcFB8fr+zsbBMjaz0OHz6soqIih/egffv2GjJkCO9BEzp//rwkKSgoSJKUm5uryspKh37v0aOHOnfuTL83kerqaq1evVplZWWKi4ujz5tZUlKSfvjDHzr0r8TvOsj1zoBc3zLI9S2PXN/yrJjvPcwOwIpOnz6t6upqhYaGOmwPDQ3VgQMHTIqqdSkqKpKkBt+Dun34fmpqajRz5kwNGzZMffr0kVTb756engoICHBoS79/f3v37lVcXJwuX74sX19frV+/Xr169VJeXh593kxWr16t3bt3a9euXfX28bsOcr35yPXNj1zfssj15rBqvqfQB3BLkpKStG/fPm3dutXsUFqF7t27Ky8vT+fPn9fatWs1adIkZWVlmR2WZR07dkwzZszQxo0b5e3tbXY4AGAKcn3LIte3PCvne4buN4MOHTrI3d293mqMJ0+eVFhYmElRtS51/cx70DySk5P1t7/9TVu2bFFERIR9e1hYmCoqKnTu3DmH9vT79+fp6ak77rhDsbGxSktLU0xMjJYsWUKfN5Pc3FydOnVKAwcOlIeHhzw8PJSVlaU33nhDHh4eCg0Npd9bOXK9+cj1zYtc3/LI9S3PyvmeQr8ZeHp6KjY2Vps2bbJvq6mp0aZNmxQXF2diZK1HdHS0wsLCHN6DCxcuKCcnh/fgezAMQ8nJyVq/fr02b96s6Ohoh/2xsbFq06aNQ78XFBTo6NGj9HsTq6mpUXl5OX3eTEaPHq29e/cqLy/P/hg0aJAmTpxo/5l+b93I9eYj1zcPcr3zINc3Pyvne4buN5OUlBRNmjRJgwYN0uDBg7V48WKVlZVpypQpZodmGRcvXtTBgwftzw8fPqy8vDwFBQWpc+fOmjlzpn73u9+pa9euio6O1vPPP6/w8HA98MAD5gXt4pKSkrRq1Sp9+OGH8vPzs89Nat++vdq2bav27dtr2rRpSklJUVBQkPz9/fXkk08qLi5Od911l8nRu665c+dq7Nix6ty5s0pLS7Vq1SplZmbq008/pc+biZ+fn30+ah0fHx8FBwfbt9PvINc3P3J9yyPXm4Ncbw5L53uzl/23sqVLlxqdO3c2PD09jcGDBxs7duwwOyRL2bJliyGp3mPSpEmGYdR+7c7zzz9vhIaGGl5eXsbo0aONgoICc4N2cQ31tyTj7bfftrf59ttvjSeeeMIIDAw02rVrZzz44INGYWGheUFbwNSpU40uXboYnp6eRkhIiDF69Gjjs88+s++nz1vG1V+3Yxj0O2qR65sXub7lkevNQa53HlbJ9zbDMIyW/GABAAAAAAA0H+boAwAAAABgIRT6AAAAAABYCIU+AAAAAAAWQqEPAAAAAICFUOgDAAAAAGAhFPoAAAAAAFgIhT4AAAAAABZCoQ/gho4cOSKbzaa8vDyzQ7E7cOCA7rrrLnl7e6t///5mhwMAgMsj3wPWQaEPuIDJkyfLZrMpPT3dYfuGDRtks9lMispcqamp8vHxUUFBgTZt2tRgmxEjRmjmzJktGxgAALeIfF8f+R64NRT6gIvw9vbWwoULdfbsWbNDaTIVFRW3fOyhQ4d09913q0uXLgoODr7l8xiGoaqqqls+HgCApkS+d0S+B24NhT7gIuLj4xUWFqa0tLRrtpk/f369YW2LFy9WVFSU/fnkyZP1wAMP6KWXXlJoaKgCAgK0YMECVVVV6ZlnnlFQUJAiIiL09ttv1zv/gQMHNHToUHl7e6tPnz7Kyspy2L9v3z6NHTtWvr6+Cg0N1S9+8QudPn3avn/EiBFKTk7WzJkz1aFDByUmJjZ4HTU1NVqwYIEiIiLk5eWl/v37KyMjw77fZrMpNzdXCxYskM1m0/z58+udY/LkycrKytKSJUtks9lks9l05MgRZWZmymaz6ZNPPlFsbKy8vLy0detW1dTUKC0tTdHR0Wrbtq1iYmK0du3am7q+tWvXqm/fvmrbtq2Cg4MVHx+vsrKyBq8RAICGkO/J90BToNAHXIS7u7teeuklLV26VN988833OtfmzZt14sQJff7553rttdeUmpqqH/3oRwoMDFROTo5+/etf6/HHH6/3Os8884xmz56tf/7zn4qLi9O4ceN05swZSdK5c+c0atQoDRgwQF988YUyMjJ08uRJ/eQnP3E4x8qVK+Xp6alt27Zp+fLlDca3ZMkSLVq0SK+++qr+9a9/KTExUT/+8Y/11VdfSZIKCwvVu3dvzZ49W4WFhXr66acbPEdcXJwee+wxFRYWqrCwUJGRkfb9c+bMUXp6uvLz89WvXz+lpaXpL3/5i5YvX679+/dr1qxZevTRR+1/3Nzo+goLCzVhwgRNnTpV+fn5yszM1EMPPSTDMG7xXQIAtEbke/I90CQMAE5v0qRJxvjx4w3DMIy77rrLmDp1qmEYhrF+/Xrj6n/GqampRkxMjMOxr7/+utGlSxeHc3Xp0sWorq62b+vevbtxzz332J9XVVUZPj4+xnvvvWcYhmEcPnzYkGSkp6fb21RWVhoRERHGwoULDcMwjBdeeMFISEhweO1jx44ZkoyCggLDMAxj+PDhxoABA254veHh4caLL77osO3OO+80nnjiCfvzmJgYIzU19brnGT58uDFjxgyHbVu2bDEkGRs2bLBvu3z5stGuXTtj+/btDm2nTZtmTJgwoVHXl5uba0gyjhw5csPrAwCgIeR78j3QVDzM+XgBwK1auHChRo0a1eCn2o3Vu3dvubl9N6AnNDRUffr0sT93d3dXcHCwTp065XBcXFyc/WcPDw8NGjRI+fn5kqQ9e/Zoy5Yt8vX1rfd6hw4dUrdu3SRJsbGx143twoULOnHihIYNG+awfdiwYdqzZ08jr/DGBg0aZP/54MGDunTpku677z6HNhUVFRowYICkG19fQkKCRo8erb59+yoxMVEJCQl65JFHFBgY2GQxAwBaD/J90yDfo7Wi0AdczL333qvExETNnTtXkydPdtjn5uZWb+hYZWVlvXO0adPG4bnNZmtwW01NTaPjunjxosaNG6eFCxfW29epUyf7zz4+Po0+Z3O6Oo6LFy9Kkj7++GPddtttDu28vLzsba53fe7u7tq4caO2b9+uzz77TEuXLtX//M//KCcnR9HR0c14JQAAKyLfNw3yPVorCn3ABaWnp6t///7q3r27w/aQkBAVFRXJMAz71/A05Xfh7tixQ/fee68kqaqqSrm5uUpOTpYkDRw4UOvWrVNUVJQ8PG79vxZ/f3+Fh4dr27ZtGj58uH37tm3bNHjw4Js6l6enp6qrq2/YrlevXvLy8tLRo0cdXvNqjbk+m82mYcOGadiwYZo3b566dOmi9evXKyUl5abiBgBAIt83FvkeqI/F+AAX1LdvX02cOFFvvPGGw/YRI0aouLhYL7/8sg4dOqRly5bpk08+abLXXbZsmdavX68DBw4oKSlJZ8+e1dSpUyVJSUlJKikp0YQJE7Rr1y4dOnRIn376qaZMmdKo5Hu1Z555RgsXLtSaNWtUUFCgOXPmKC8vTzNmzLip80RFRSknJ0dHjhzR6dOnr3nHws/PT08//bRmzZqllStX6tChQ9q9e7eWLl2qlStXNur6cnJy9NJLL+mLL77Q0aNH9cEHH6i4uFg9e/a8qZgBAKhDvm8c8j1QH4U+4KIWLFhQL5H17NlTv//977Vs2TLFxMRo586d32tu339LT09Xenq6YmJitHXrVn300Ufq0KGDJNk/la+urlZCQoL69u2rmTNnKiAgwGF+YGM89dRTSklJ0ezZs9W3b19lZGToo48+UteuXW/qPE8//bTc3d3Vq1cvhYSE6OjRo9ds+8ILL+j5559XWlqaevbsqTFjxujjjz+2D8O70fX5+/vr888/1/33369u3brpueee06JFizR27NibihkAgKuR72+MfA/UZzP+e4IPAAAAAABwWdzRBwAAAADAQij0AQAAAACwEAp9AAAAAAAshEIfAAAAAAALodAHAAAAAMBCKPQBAAAAALAQCn0AAAAAACyEQh8AAAAAAAuh0AcAAAAAwEIo9AEAAAAAsBAKfQAAAAAALIRCHwAAAAAAC/n/mhXyqG1fsqcAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "logs = model.make_inspector().training_logs()\n", "\n", "plt.figure(figsize=(12, 4))\n", "\n", "plt.subplot(1, 2, 1)\n", "plt.plot([log.num_trees for log in logs], [log.evaluation.ndcg for log in logs])\n", "plt.xlabel(\"Number of trees\")\n", "plt.ylabel(\"NDCG (validation)\")\n", "\n", "plt.subplot(1, 2, 2)\n", "plt.plot([log.num_trees for log in logs], [log.evaluation.loss for log in logs])\n", "plt.xlabel(\"Number of trees\")\n", "plt.ylabel(\"Loss (validation)\")\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "eq1E_26Y8rtQ" }, "source": [ "As for all TF-DF models, you can also look at the model report (Note: The model report also contains the training logs):" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:24.277872Z", "iopub.status.busy": "2024-04-20T11:09:24.277256Z", "iopub.status.idle": "2024-04-20T11:09:24.288341Z", "shell.execute_reply": "2024-04-20T11:09:24.287764Z" }, "id": "L4N1R8fM4jFh" }, "outputs": [ { "data": { "application/javascript": [ "google.colab.output.setIframeHeight(0, true, {maxHeight: 400})" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"gradient_boosted_trees_model\"\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "_________________________________________________________________\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Layer (type) Output Shape Param # \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "=================================================================\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "=================================================================\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Total params: 1 (1.00 Byte)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Trainable params: 0 (0.00 Byte)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Non-trainable params: 1 (1.00 Byte)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "_________________________________________________________________\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Type: \"GRADIENT_BOOSTED_TREES\"\n", "Task: RANKING\n", "Label: \"__LABEL\"\n", "Rank group: \"group\"\n", "\n", "Input Features (25):\n", "\tf_1\n", "\tf_10\n", "\tf_11\n", "\tf_12\n", "\tf_13\n", "\tf_14\n", "\tf_15\n", "\tf_16\n", "\tf_17\n", "\tf_18\n", "\tf_19\n", "\tf_2\n", "\tf_20\n", "\tf_21\n", "\tf_22\n", "\tf_23\n", "\tf_24\n", "\tf_25\n", "\tf_3\n", "\tf_4\n", "\tf_5\n", "\tf_6\n", "\tf_7\n", "\tf_8\n", "\tf_9\n", "\n", "No weights\n", "\n", "Variable Importance: INV_MEAN_MIN_DEPTH:\n", " 1. \"f_9\" 0.326164 ################\n", " 2. \"f_3\" 0.318071 ###############\n", " 3. \"f_8\" 0.308922 #############\n", " 4. \"f_4\" 0.271175 #########\n", " 5. \"f_19\" 0.221570 ###\n", " 6. \"f_10\" 0.215666 ##\n", " 7. \"f_11\" 0.206509 #\n", " 8. \"f_22\" 0.204742 #\n", " 9. \"f_25\" 0.204497 #\n", " 10. \"f_23\" 0.203238 \n", " 11. \"f_21\" 0.200830 \n", " 12. \"f_24\" 0.200445 \n", " 13. \"f_12\" 0.198840 \n", " 14. \"f_18\" 0.197676 \n", " 15. \"f_20\" 0.196634 \n", " 16. \"f_6\" 0.196085 \n", " 17. \"f_16\" 0.196061 \n", " 18. \"f_2\" 0.195683 \n", " 19. \"f_5\" 0.195683 \n", " 20. \"f_13\" 0.195559 \n", " 21. \"f_17\" 0.195559 \n", "\n", "Variable Importance: NUM_AS_ROOT:\n", " 1. \"f_3\" 4.000000 ################\n", " 2. \"f_4\" 4.000000 ################\n", " 3. \"f_8\" 3.000000 ##########\n", " 4. \"f_9\" 1.000000 \n", "\n", "Variable Importance: NUM_NODES:\n", " 1. \"f_8\" 25.000000 ################\n", " 2. \"f_19\" 18.000000 ###########\n", " 3. \"f_10\" 15.000000 #########\n", " 4. \"f_9\" 14.000000 ########\n", " 5. \"f_3\" 13.000000 ########\n", " 6. \"f_23\" 7.000000 ####\n", " 7. \"f_24\" 6.000000 ###\n", " 8. \"f_11\" 5.000000 ##\n", " 9. \"f_21\" 5.000000 ##\n", " 10. \"f_25\" 5.000000 ##\n", " 11. \"f_4\" 5.000000 ##\n", " 12. \"f_22\" 4.000000 ##\n", " 13. \"f_12\" 3.000000 #\n", " 14. \"f_20\" 3.000000 #\n", " 15. \"f_16\" 2.000000 \n", " 16. \"f_6\" 2.000000 \n", " 17. \"f_13\" 1.000000 \n", " 18. \"f_17\" 1.000000 \n", " 19. \"f_18\" 1.000000 \n", " 20. \"f_2\" 1.000000 \n", " 21. \"f_5\" 1.000000 \n", "\n", "Variable Importance: SUM_SCORE:\n", " 1. \"f_8\" 10779.340861 ################\n", " 2. \"f_9\" 8831.772410 #############\n", " 3. \"f_3\" 4526.101184 ######\n", " 4. \"f_4\" 4360.245403 ######\n", " 5. \"f_19\" 2325.288894 ###\n", " 6. \"f_10\" 1881.848369 ##\n", " 7. \"f_21\" 1674.980191 ##\n", " 8. \"f_11\" 1127.632256 #\n", " 9. \"f_23\" 1021.834252 #\n", " 10. \"f_24\" 914.851512 #\n", " 11. \"f_22\" 885.619576 #\n", " 12. \"f_25\" 748.665007 #\n", " 13. \"f_20\" 310.610858 \n", " 14. \"f_16\" 298.972842 \n", " 15. \"f_6\" 212.376573 \n", " 16. \"f_12\" 130.725240 \n", " 17. \"f_2\" 112.124991 \n", " 18. \"f_18\" 86.341193 \n", " 19. \"f_5\" 65.103908 \n", " 20. \"f_13\" 57.966947 \n", " 21. \"f_17\" 21.930388 \n", "\n", "\n", "\n", "Loss: LAMBDA_MART_NDCG5\n", "Validation loss value: -0.438692\n", "Number of trees per iteration: 1\n", "Node format: NOT_SET\n", "Number of trees: 12\n", "Total number of nodes: 286\n", "\n", "Number of nodes by tree:\n", "Count: 12 Average: 23.8333 StdDev: 3.50793\n", "Min: 17 Max: 29 Ignored: 0\n", "----------------------------------------------\n", "[ 17, 18) 1 8.33% 8.33% ###\n", "[ 18, 19) 0 0.00% 8.33%\n", "[ 19, 20) 1 8.33% 16.67% ###\n", "[ 20, 21) 0 0.00% 16.67%\n", "[ 21, 22) 2 16.67% 33.33% #######\n", "[ 22, 23) 0 0.00% 33.33%\n", "[ 23, 24) 1 8.33% 41.67% ###\n", "[ 24, 25) 0 0.00% 41.67%\n", "[ 25, 26) 3 25.00% 66.67% ##########\n", "[ 26, 27) 0 0.00% 66.67%\n", "[ 27, 28) 3 25.00% 91.67% ##########\n", "[ 28, 29) 0 0.00% 91.67%\n", "[ 29, 29] 1 8.33% 100.00% ###\n", "\n", "Depth by leafs:\n", "Count: 149 Average: 4.14094 StdDev: 1.08696\n", "Min: 1 Max: 5 Ignored: 0\n", "----------------------------------------------\n", "[ 1, 2) 2 1.34% 1.34%\n", "[ 2, 3) 18 12.08% 13.42% ##\n", "[ 3, 4) 13 8.72% 22.15% ##\n", "[ 4, 5) 40 26.85% 48.99% #####\n", "[ 5, 5] 76 51.01% 100.00% ##########\n", "\n", "Number of training obs by leaf:\n", "Count: 149 Average: 673.691 StdDev: 2015.44\n", "Min: 5 Max: 8211 Ignored: 0\n", "----------------------------------------------\n", "[ 5, 415) 127 85.23% 85.23% ##########\n", "[ 415, 825) 6 4.03% 89.26%\n", "[ 825, 1236) 2 1.34% 90.60%\n", "[ 1236, 1646) 0 0.00% 90.60%\n", "[ 1646, 2056) 0 0.00% 90.60%\n", "[ 2056, 2467) 1 0.67% 91.28%\n", "[ 2467, 2877) 0 0.00% 91.28%\n", "[ 2877, 3287) 0 0.00% 91.28%\n", "[ 3287, 3698) 1 0.67% 91.95%\n", "[ 3698, 4108) 0 0.00% 91.95%\n", "[ 4108, 4518) 0 0.00% 91.95%\n", "[ 4518, 4929) 1 0.67% 92.62%\n", "[ 4929, 5339) 0 0.00% 92.62%\n", "[ 5339, 5749) 0 0.00% 92.62%\n", "[ 5749, 6160) 1 0.67% 93.29%\n", "[ 6160, 6570) 0 0.00% 93.29%\n", "[ 6570, 6980) 0 0.00% 93.29%\n", "[ 6980, 7391) 0 0.00% 93.29%\n", "[ 7391, 7801) 8 5.37% 98.66% #\n", "[ 7801, 8211] 2 1.34% 100.00%\n", "\n", "Attribute in nodes:\n", "\t25 : f_8 [NUMERICAL]\n", "\t18 : f_19 [NUMERICAL]\n", "\t15 : f_10 [NUMERICAL]\n", "\t14 : f_9 [NUMERICAL]\n", "\t13 : f_3 [NUMERICAL]\n", "\t7 : f_23 [NUMERICAL]\n", "\t6 : f_24 [NUMERICAL]\n", "\t5 : f_4 [NUMERICAL]\n", "\t5 : f_25 [NUMERICAL]\n", "\t5 : f_21 [NUMERICAL]\n", "\t5 : f_11 [NUMERICAL]\n", "\t4 : f_22 [NUMERICAL]\n", "\t3 : f_20 [NUMERICAL]\n", "\t3 : f_12 [NUMERICAL]\n", "\t2 : f_6 [NUMERICAL]\n", "\t2 : f_16 [NUMERICAL]\n", "\t1 : f_5 [NUMERICAL]\n", "\t1 : f_2 [NUMERICAL]\n", "\t1 : f_18 [NUMERICAL]\n", "\t1 : f_17 [NUMERICAL]\n", "\t1 : f_13 [NUMERICAL]\n", "\n", "Attribute in nodes with depth <= 0:\n", "\t4 : f_4 [NUMERICAL]\n", "\t4 : f_3 [NUMERICAL]\n", "\t3 : f_8 [NUMERICAL]\n", "\t1 : f_9 [NUMERICAL]\n", "\n", "Attribute in nodes with depth <= 1:\n", "\t11 : f_9 [NUMERICAL]\n", "\t9 : f_8 [NUMERICAL]\n", "\t4 : f_4 [NUMERICAL]\n", "\t4 : f_3 [NUMERICAL]\n", "\t1 : f_25 [NUMERICAL]\n", "\t1 : f_24 [NUMERICAL]\n", "\t1 : f_23 [NUMERICAL]\n", "\t1 : f_22 [NUMERICAL]\n", "\t1 : f_19 [NUMERICAL]\n", "\t1 : f_11 [NUMERICAL]\n", "\n", "Attribute in nodes with depth <= 2:\n", "\t15 : f_8 [NUMERICAL]\n", "\t12 : f_9 [NUMERICAL]\n", "\t11 : f_3 [NUMERICAL]\n", "\t6 : f_19 [NUMERICAL]\n", "\t5 : f_4 [NUMERICAL]\n", "\t2 : f_25 [NUMERICAL]\n", "\t2 : f_11 [NUMERICAL]\n", "\t2 : f_10 [NUMERICAL]\n", "\t1 : f_24 [NUMERICAL]\n", "\t1 : f_23 [NUMERICAL]\n", "\t1 : f_22 [NUMERICAL]\n", "\t1 : f_18 [NUMERICAL]\n", "\t1 : f_17 [NUMERICAL]\n", "\n", "Attribute in nodes with depth <= 3:\n", "\t22 : f_8 [NUMERICAL]\n", "\t13 : f_9 [NUMERICAL]\n", "\t11 : f_3 [NUMERICAL]\n", "\t10 : f_19 [NUMERICAL]\n", "\t9 : f_10 [NUMERICAL]\n", "\t5 : f_4 [NUMERICAL]\n", "\t5 : f_23 [NUMERICAL]\n", "\t5 : f_11 [NUMERICAL]\n", "\t4 : f_25 [NUMERICAL]\n", "\t4 : f_22 [NUMERICAL]\n", "\t4 : f_21 [NUMERICAL]\n", "\t3 : f_24 [NUMERICAL]\n", "\t2 : f_12 [NUMERICAL]\n", "\t1 : f_18 [NUMERICAL]\n", "\t1 : f_17 [NUMERICAL]\n", "\n", "Attribute in nodes with depth <= 5:\n", "\t25 : f_8 [NUMERICAL]\n", "\t18 : f_19 [NUMERICAL]\n", "\t15 : f_10 [NUMERICAL]\n", "\t14 : f_9 [NUMERICAL]\n", "\t13 : f_3 [NUMERICAL]\n", "\t7 : f_23 [NUMERICAL]\n", "\t6 : f_24 [NUMERICAL]\n", "\t5 : f_4 [NUMERICAL]\n", "\t5 : f_25 [NUMERICAL]\n", "\t5 : f_21 [NUMERICAL]\n", "\t5 : f_11 [NUMERICAL]\n", "\t4 : f_22 [NUMERICAL]\n", "\t3 : f_20 [NUMERICAL]\n", "\t3 : f_12 [NUMERICAL]\n", "\t2 : f_6 [NUMERICAL]\n", "\t2 : f_16 [NUMERICAL]\n", "\t1 : f_5 [NUMERICAL]\n", "\t1 : f_2 [NUMERICAL]\n", "\t1 : f_18 [NUMERICAL]\n", "\t1 : f_17 [NUMERICAL]\n", "\t1 : f_13 [NUMERICAL]\n", "\n", "Condition type in nodes:\n", "\t137 : HigherCondition\n", "Condition type in nodes with depth <= 0:\n", "\t12 : HigherCondition\n", "Condition type in nodes with depth <= 1:\n", "\t34 : HigherCondition\n", "Condition type in nodes with depth <= 2:\n", "\t60 : HigherCondition\n", "Condition type in nodes with depth <= 3:\n", "\t99 : HigherCondition\n", "Condition type in nodes with depth <= 5:\n", "\t137 : HigherCondition\n", "\n", "Training logs:\n", "Number of iteration to final model: 12\n", "\tIter:1 train-loss:-0.346669 valid-loss:-0.262935 train-NDCG@5:0.346669 valid-NDCG@5:0.262935\n", "\tIter:2 train-loss:-0.412635 valid-loss:-0.335301 train-NDCG@5:0.412635 valid-NDCG@5:0.335301\n", "\tIter:3 train-loss:-0.468270 valid-loss:-0.341295 train-NDCG@5:0.468270 valid-NDCG@5:0.341295\n", "\tIter:4 train-loss:-0.481511 valid-loss:-0.301897 train-NDCG@5:0.481511 valid-NDCG@5:0.301897\n", "\tIter:5 train-loss:-0.473165 valid-loss:-0.394670 train-NDCG@5:0.473165 valid-NDCG@5:0.394670\n", "\tIter:6 train-loss:-0.496260 valid-loss:-0.415201 train-NDCG@5:0.496260 valid-NDCG@5:0.415201\n", "\tIter:16 train-loss:-0.526791 valid-loss:-0.380900 train-NDCG@5:0.526791 valid-NDCG@5:0.380900\n", "\tIter:26 train-loss:-0.560398 valid-loss:-0.367496 train-NDCG@5:0.560398 valid-NDCG@5:0.367496\n", "\tIter:36 train-loss:-0.584252 valid-loss:-0.341845 train-NDCG@5:0.584252 valid-NDCG@5:0.341845\n", "\n" ] } ], "source": [ "%set_cell_height 400\n", "model.summary()" ] }, { "cell_type": "markdown", "metadata": { "id": "gCWwJwkw9aHB" }, "source": [ "And if you are curious, you can also plot the model:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:24.292907Z", "iopub.status.busy": "2024-04-20T11:09:24.292662Z", "iopub.status.idle": "2024-04-20T11:09:24.299797Z", "shell.execute_reply": "2024-04-20T11:09:24.299130Z" }, "id": "8wdUhz9X9cbI" }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tfdf.model_plotter.plot_model_in_colab(model, tree_idx=0, max_depth=3)" ] }, { "cell_type": "markdown", "metadata": { "id": "DjViB8SK1BrY" }, "source": [ "## Predicting with a ranking model\n", "\n", "For an incoming query, we can use our ranking model to predict the relevance of\n", "a stack of documents. In practice this means that for each query, we must come\n", "up with a set of documents that may or may not be relevant to the query. We call\n", "these documents our **candidate documents**. For each pair\n", "query/candidate document, we can compute the same features used during training.\n", "This is our **serving dataset**.\n", "\n", "Going back to the example from the beginning of this tutorial, the serving\n", "dataset might look like this:\n", "\n", "query | document_id | feature_1 | feature_2\n", "------ | ----------- | --------- | --------- \n", "fish | 32 | 0.3 | blue \n", "fish | 33 | 1.0 | green \n", "fish | 34 | 0.4 | blue \n", "fish | 35 | NA | brown \n", "\n", "Observe that *relevance* is not part of the serving dataset, since this is what\n", "the model is trying to predict.\n", "\n", "The serving dataset is fed to the TF-DF model and assigns a relevance score to\n", "each document.\n", "\n", "query | document_id | feature_1 | feature_2 | relevance\n", "------ | ----------- | --------- | --------- | ---------\n", "fish | 32 | 0.3 | blue | 0.325\n", "fish | 33 | 1.0 | green | 0.125\n", "fish | 34 | 0.4 | blue | 0.155\n", "fish | 35 | NA | brown | 0.593\n", "\n", "This means that the document with document_id 35 is predicted to be most\n", "relevant for query \"fish\"." ] }, { "cell_type": "markdown", "metadata": { "id": "WXBPhvfr1BrY" }, "source": [ "Let's try to do this with our real model." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:24.303079Z", "iopub.status.busy": "2024-04-20T11:09:24.302449Z", "iopub.status.idle": "2024-04-20T11:09:24.434750Z", "shell.execute_reply": "2024-04-20T11:09:24.433967Z" }, "id": "cArmmZZ81BrY" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
relevancegroupf_1f_2f_3f_4f_5f_6f_7f_8...f_16f_17f_18f_19f_20f_21f_22f_23f_24f_25
02g_13.02.0794420.2727270.26103437.33056511.43124137.299751.138657...9.34002424.8087850.39309157.4165173.29489325.02313.219799-3.87098-3.90273-3.87512
10g_13.02.0794420.4285710.40059437.33056511.43124137.299751.814480...9.34002424.8087850.34920543.2406262.65472423.49033.156588-3.96838-4.00865-3.98670
22g_10.00.0000000.0000000.00000037.33056511.43124137.299750.000000...9.34002424.8087850.24031925.8169891.55134215.86502.764115-4.28166-4.33313-4.44161
\n", "

3 rows × 27 columns

\n", "
" ], "text/plain": [ " relevance group f_1 f_2 f_3 f_4 f_5 f_6 \\\n", "0 2 g_1 3.0 2.079442 0.272727 0.261034 37.330565 11.431241 \n", "1 0 g_1 3.0 2.079442 0.428571 0.400594 37.330565 11.431241 \n", "2 2 g_1 0.0 0.000000 0.000000 0.000000 37.330565 11.431241 \n", "\n", " f_7 f_8 ... f_16 f_17 f_18 f_19 \\\n", "0 37.29975 1.138657 ... 9.340024 24.808785 0.393091 57.416517 \n", "1 37.29975 1.814480 ... 9.340024 24.808785 0.349205 43.240626 \n", "2 37.29975 0.000000 ... 9.340024 24.808785 0.240319 25.816989 \n", "\n", " f_20 f_21 f_22 f_23 f_24 f_25 \n", "0 3.294893 25.0231 3.219799 -3.87098 -3.90273 -3.87512 \n", "1 2.654724 23.4903 3.156588 -3.96838 -4.00865 -3.98670 \n", "2 1.551342 15.8650 2.764115 -4.28166 -4.33313 -4.44161 \n", "\n", "[3 rows x 27 columns]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Path to a test dataset using libsvm format.\n", "test_dataset_path = os.path.join(os.path.dirname(archive_path),\"OHSUMED/Data/Fold1/testset.txt\")\n", "# Convert the dataset.\n", "csv_test_dataset_path=\"/tmp/ohsumed_test.csv\"\n", "convert_libsvm_to_csv(raw_dataset_path, csv_test_dataset_path)\n", "\n", "# Load a dataset into a Pandas Dataframe.\n", "test_dataset_df = pd.read_csv(csv_test_dataset_path)\n", "\n", "# Display the first 3 examples.\n", "test_dataset_df.head(3)" ] }, { "cell_type": "markdown", "metadata": { "id": "SXty9G-P1BrY" }, "source": [ "Suppose our query is \"g_5\" and the test dataset already contains the candidate\n", "documents for this query." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:24.438041Z", "iopub.status.busy": "2024-04-20T11:09:24.437513Z", "iopub.status.idle": "2024-04-20T11:09:24.482092Z", "shell.execute_reply": "2024-04-20T11:09:24.481442Z" }, "id": "IBfLDHig1BrY" }, "outputs": [], "source": [ "# Filter by \"g_5\"\n", "serving_dataset_df = test_dataset_df[test_dataset_df['group'] == 'g_5']\n", "# Remove the columns for group and relevance, not needed for predictions.\n", "serving_dataset_df = serving_dataset_df.drop(['relevance', 'group'], axis=1)\n", "# Convert to a Tensorflow dataset\n", "serving_dataset_ds = tfdf.keras.pd_dataframe_to_tf_dataset(serving_dataset_df, task=tfdf.keras.Task.RANKING)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:24.485124Z", "iopub.status.busy": "2024-04-20T11:09:24.484901Z", "iopub.status.idle": "2024-04-20T11:09:24.673274Z", "shell.execute_reply": "2024-04-20T11:09:24.672361Z" }, "id": "_EzlVuk01BrY" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 176ms/step\n" ] } ], "source": [ "# Run predictions with on all candidate documents\n", "predictions = model.predict(serving_dataset_ds)" ] }, { "cell_type": "markdown", "metadata": { "id": "LRqRnruM1BrY" }, "source": [ "We can use add the predictions to the dataframe and use them to find the \n", "documents with the highest scores. " ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2024-04-20T11:09:24.676926Z", "iopub.status.busy": "2024-04-20T11:09:24.676472Z", "iopub.status.idle": "2024-04-20T11:09:24.697665Z", "shell.execute_reply": "2024-04-20T11:09:24.696816Z" }, "id": "IrkLkIWG1BrY" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
f_1f_2f_3f_4f_5f_6f_7f_8f_9f_10...f_17f_18f_19f_20f_21f_22f_23f_24f_25prediction_score
6422.01.3862940.6666670.57536429.4471178.43511629.4480212.20713512.29217010.101899...21.2087150.52384577.8521487.65910130.26603.410025-3.03908-3.19282-2.871120.965342
6853.02.0794420.7500000.66943129.4471178.43511629.4480213.06016421.79565717.652746...21.2087150.79368139.6232718.51380133.98303.525860-2.84235-2.81360-2.599200.893874
6464.02.7725890.2857140.27597129.4471178.43511629.4480211.42106324.55033814.727974...21.2087150.60296384.8681087.76793131.02683.434851-3.19269-3.31166-3.149010.258856
6844.02.4849070.3333330.31423629.4471178.43511629.4480211.73030429.29974415.114793...21.2087150.69289971.2796488.14880436.56453.599078-2.16625-2.43823-1.946580.258856
6403.02.0794420.4285710.40059429.4471178.43511629.4480212.10736121.79565715.999891...21.2087150.0000000.0000000.00000030.64223.422378-3.20997-2.59768-2.597680.258856
\n", "

5 rows × 26 columns

\n", "
" ], "text/plain": [ " f_1 f_2 f_3 f_4 f_5 f_6 f_7 \\\n", "642 2.0 1.386294 0.666667 0.575364 29.447117 8.435116 29.448021 \n", "685 3.0 2.079442 0.750000 0.669431 29.447117 8.435116 29.448021 \n", "646 4.0 2.772589 0.285714 0.275971 29.447117 8.435116 29.448021 \n", "684 4.0 2.484907 0.333333 0.314236 29.447117 8.435116 29.448021 \n", "640 3.0 2.079442 0.428571 0.400594 29.447117 8.435116 29.448021 \n", "\n", " f_8 f_9 f_10 ... f_17 f_18 f_19 \\\n", "642 2.207135 12.292170 10.101899 ... 21.208715 0.523845 77.852148 \n", "685 3.060164 21.795657 17.652746 ... 21.208715 0.793681 39.623271 \n", "646 1.421063 24.550338 14.727974 ... 21.208715 0.602963 84.868108 \n", "684 1.730304 29.299744 15.114793 ... 21.208715 0.692899 71.279648 \n", "640 2.107361 21.795657 15.999891 ... 21.208715 0.000000 0.000000 \n", "\n", " f_20 f_21 f_22 f_23 f_24 f_25 prediction_score \n", "642 7.659101 30.2660 3.410025 -3.03908 -3.19282 -2.87112 0.965342 \n", "685 8.513801 33.9830 3.525860 -2.84235 -2.81360 -2.59920 0.893874 \n", "646 7.767931 31.0268 3.434851 -3.19269 -3.31166 -3.14901 0.258856 \n", "684 8.148804 36.5645 3.599078 -2.16625 -2.43823 -1.94658 0.258856 \n", "640 0.000000 30.6422 3.422378 -3.20997 -2.59768 -2.59768 0.258856 \n", "\n", "[5 rows x 26 columns]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "serving_dataset_df['prediction_score'] = predictions\n", "serving_dataset_df.sort_values(by=['prediction_score'], ascending=False).head()" ] } ], "metadata": { "colab": { "name": "ranking_colab.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.19" } }, "nbformat": 4, "nbformat_minor": 0 }