{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "klGNgWREsvQv" }, "source": [ "##### Copyright 2023 The TF-Agents Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2024-03-09T12:47:09.993607Z", "iopub.status.busy": "2024-03-09T12:47:09.993014Z", "iopub.status.idle": "2024-03-09T12:47:09.996725Z", "shell.execute_reply": "2024-03-09T12:47:09.996144Z" }, "id": "nQnmcm0oI1Q-" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "oMaGpi7TciQs" }, "source": [ "# DQN C51/Rainbow\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " View on TensorFlow.org\n", " \n", " \n", " \n", " Run in Google Colab\n", " \n", " \n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "ZOUOQOrFs3zn" }, "source": [ "## Introduction" ] }, { "cell_type": "markdown", "metadata": { "id": "cKOCZlhUgXVK" }, "source": [ "This example shows how to train a [Categorical DQN (C51)](https://arxiv.org/pdf/1707.06887.pdf) agent on the Cartpole environment using the TF-Agents library.\n", "\n", "![Cartpole environment](https://github.com/tensorflow/agents/blob/master/docs/tutorials/images/cartpole.png?raw=1)\n", "\n", "Make sure you take a look through the [DQN tutorial](https://github.com/tensorflow/agents/blob/master/docs/tutorials/1_dqn_tutorial.ipynb) as a prerequisite. This tutorial will assume familiarity with the DQN tutorial; it will mainly focus on the differences between DQN and C51.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "lsaQlK8fFQqH" }, "source": [ "## Setup\n" ] }, { "cell_type": "markdown", "metadata": { "id": "-NzBsZzPcyBm" }, "source": [ "If you haven't installed tf-agents yet, run:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:10.000701Z", "iopub.status.busy": "2024-03-09T12:47:10.000075Z", "iopub.status.idle": "2024-03-09T12:47:29.099292Z", "shell.execute_reply": "2024-03-09T12:47:29.098447Z" }, "id": "KEHR2Ui-lo8O" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]\r", " \r", "Hit:1 http://us-central1.gce.archive.ubuntu.com/ubuntu focal InRelease\r\n", "\r", "0% [Connecting to security.ubuntu.com (91.189.91.81)] [Connecting to apt.llvm.o\r", " \r", "Hit:2 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates InRelease\r\n", "\r", " \r", "Hit:3 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-backports InRelease\r\n", "\r", "0% [Connecting to security.ubuntu.com (91.189.91.81)] [Connecting to apt.llvm.o" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Waiting for headers] [Waiting for headers] [Connecting to ppa.launchpad.net\r", " \r", "Hit:5 https://download.docker.com/linux/ubuntu focal InRelease\r\n", "\r", "0% [Waiting for headers] [Waiting for headers] [Connecting to ppa.launchpad.net\r", " \r", "Get:6 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64 InRelease [1484 B]\r\n", "\r", "0% [Waiting for headers] [Waiting for headers] [Connecting to ppa.launchpad.net\r", "0% [Waiting for headers] [Waiting for headers] [Connecting to ppa.launchpad.net\r", " \r", "Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease\r\n", "\r", "0% [Waiting for headers] [Connecting to ppa.launchpad.net (185.125.190.80)] [Wa\r", " \r", "Hit:8 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64 InRelease\r\n", "\r", " \r", "0% [Waiting for headers] [Connecting to ppa.launchpad.net (185.125.190.80)]\r", " \r", "Hit:9 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 InRelease\r\n", "\r", "0% [Waiting for headers] [Connecting to ppa.launchpad.net (185.125.190.80)]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "Hit:10 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease\r\n", "\r", " \r", "0% [Connecting to ppa.launchpad.net (185.125.190.80)]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "0% [Waiting for headers]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "Hit:11 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal InRelease\r\n", "\r", " \r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Connected to apt.llvm.org (199.232.198.49)] [Waiting for headers]\r", " \r", "Hit:4 https://apt.llvm.org/focal llvm-toolchain-focal-17 InRelease\r\n", "\r", " \r", "0% [Waiting for headers]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "Hit:12 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu focal InRelease\r\n", "\r", " \r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Waiting for headers]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "Hit:13 http://ppa.launchpad.net/openjdk-r/ppa/ubuntu focal InRelease\r\n", "\r", " \r", "0% [Working]\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "100% [Working]\r", " \r", "Fetched 1484 B in 1s (1065 B/s)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 0%\r", "\r", "Reading package lists... 0%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 0%\r", "\r", "Reading package lists... 2%\r", "\r", "Reading package lists... 2%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 27%\r", "\r", "Reading package lists... 27%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 39%\r", "\r", "Reading package lists... 39%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 40%\r", "\r", "Reading package lists... 40%\r", "\r", "Reading package lists... 40%\r", "\r", "Reading package lists... 40%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 48%\r", "\r", "Reading package lists... 48%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 53%\r", "\r", "Reading package lists... 53%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 56%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 61%\r", "\r", "Reading package lists... 61%\r", "\r", "Reading package lists... 64%\r", "\r", "Reading package lists... 64%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 68%\r", "\r", "Reading package lists... 68%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 76%\r", "\r", "Reading package lists... 76%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 81%\r", "\r", "Reading package lists... 81%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 88%\r", "\r", "Reading package lists... 88%\r", "\r", "Reading package lists... 92%\r", "\r", "Reading package lists... 92%\r", "\r", "Reading package lists... 94%\r", "\r", "Reading package lists... 94%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... Done\r", "\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 0%\r", "\r", "Reading package lists... 100%\r", "\r", "Reading package lists... Done\r", "\r\n", "\r", "Building dependency tree... 0%\r", "\r", "Building dependency tree... 0%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Building dependency tree... 50%\r", "\r", "Building dependency tree... 50%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Building dependency tree \r", "\r\n", "\r", "Reading state information... 0%\r", "\r", "Reading state information... 0%\r", "\r", "Reading state information... Done\r", "\r\n", "freeglut3-dev is already the newest version (2.8.1-3).\r\n", "ffmpeg is already the newest version (7:4.2.7-0ubuntu0.1).\r\n", "xvfb is already the newest version (2:1.20.13-1ubuntu1~20.04.15).\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "The following packages were automatically installed and are no longer required:\r\n", " libatasmart4 libblockdev-fs2 libblockdev-loop2 libblockdev-part-err2\r\n", " libblockdev-part2 libblockdev-swap2 libblockdev-utils2 libblockdev2\r\n", " libparted-fs-resize0 libxmlb2\r\n", "Use 'sudo apt autoremove' to remove them.\r\n", "0 upgraded, 0 newly installed, 0 to remove and 187 not upgraded.\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting imageio==2.4.0\r\n", " Using cached imageio-2.4.0-py3-none-any.whl\r\n", "Requirement already satisfied: numpy in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from imageio==2.4.0) (1.26.4)\r\n", "Requirement already satisfied: pillow in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from imageio==2.4.0) (10.2.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: imageio\r\n", " Attempting uninstall: imageio\r\n", " Found existing installation: imageio 2.34.0\r\n", " Uninstalling imageio-2.34.0:\r\n", " Successfully uninstalled imageio-2.34.0\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n", "scikit-image 0.22.0 requires imageio>=2.27, but you have imageio 2.4.0 which is incompatible.\u001b[0m\u001b[31m\r\n", "\u001b[0mSuccessfully installed imageio-2.4.0\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting pyvirtualdisplay\r\n", " Using cached PyVirtualDisplay-3.0-py3-none-any.whl.metadata (943 bytes)\r\n", "Using cached PyVirtualDisplay-3.0-py3-none-any.whl (15 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: pyvirtualdisplay\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed pyvirtualdisplay-3.0\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting tf-agents\r\n", " Using cached tf_agents-0.19.0-py3-none-any.whl.metadata (12 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: absl-py>=0.6.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents) (1.4.0)\r\n", "Collecting cloudpickle>=1.3 (from tf-agents)\r\n", " Using cached cloudpickle-3.0.0-py3-none-any.whl.metadata (7.0 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting gin-config>=0.4.0 (from tf-agents)\r\n", " Using cached gin_config-0.5.0-py3-none-any.whl.metadata (2.9 kB)\r\n", "Collecting gym<=0.23.0,>=0.17.0 (from tf-agents)\r\n", " Using cached gym-0.23.0-py3-none-any.whl\r\n", "Requirement already satisfied: numpy>=1.19.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents) (1.26.4)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pillow in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents) (10.2.0)\r\n", "Requirement already satisfied: six>=1.10.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents) (1.16.0)\r\n", "Requirement already satisfied: protobuf>=3.11.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents) (3.20.3)\r\n", "Requirement already satisfied: wrapt>=1.11.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents) (1.16.0)\r\n", "Collecting typing-extensions==4.5.0 (from tf-agents)\r\n", " Using cached typing_extensions-4.5.0-py3-none-any.whl.metadata (8.5 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting pygame==2.1.3 (from tf-agents)\r\n", " Using cached pygame-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.3 kB)\r\n", "Collecting tensorflow-probability~=0.23.0 (from tf-agents)\r\n", " Using cached tensorflow_probability-0.23.0-py2.py3-none-any.whl.metadata (13 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting gym-notices>=0.0.4 (from gym<=0.23.0,>=0.17.0->tf-agents)\r\n", " Using cached gym_notices-0.0.8-py3-none-any.whl.metadata (1.0 kB)\r\n", "Requirement already satisfied: importlib-metadata>=4.10.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from gym<=0.23.0,>=0.17.0->tf-agents) (7.0.2)\r\n", "Requirement already satisfied: decorator in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-probability~=0.23.0->tf-agents) (5.1.1)\r\n", "Requirement already satisfied: gast>=0.3.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-probability~=0.23.0->tf-agents) (0.5.4)\r\n", "Requirement already satisfied: dm-tree in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-probability~=0.23.0->tf-agents) (0.1.8)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: zipp>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from importlib-metadata>=4.10.0->gym<=0.23.0,>=0.17.0->tf-agents) (3.17.0)\r\n", "Using cached tf_agents-0.19.0-py3-none-any.whl (1.4 MB)\r\n", "Using cached pygame-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.7 MB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)\r\n", "Using cached cloudpickle-3.0.0-py3-none-any.whl (20 kB)\r\n", "Using cached gin_config-0.5.0-py3-none-any.whl (61 kB)\r\n", "Using cached tensorflow_probability-0.23.0-py2.py3-none-any.whl (6.9 MB)\r\n", "Using cached gym_notices-0.0.8-py3-none-any.whl (3.0 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: gym-notices, gin-config, typing-extensions, pygame, cloudpickle, tensorflow-probability, gym, tf-agents\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Attempting uninstall: typing-extensions\r\n", " Found existing installation: typing_extensions 4.10.0\r\n", " Uninstalling typing_extensions-4.10.0:\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Successfully uninstalled typing_extensions-4.10.0\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed cloudpickle-3.0.0 gin-config-0.5.0 gym-0.23.0 gym-notices-0.0.8 pygame-2.1.3 tensorflow-probability-0.23.0 tf-agents-0.19.0 typing-extensions-4.5.0\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pyglet in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (2.0.14)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: tf-keras in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (2.16.0)\r\n", "Requirement already satisfied: tensorflow<2.17,>=2.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-keras) (2.16.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: absl-py>=1.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (1.4.0)\r\n", "Requirement already satisfied: astunparse>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (1.6.3)\r\n", "Requirement already satisfied: flatbuffers>=23.5.26 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (24.3.7)\r\n", "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (0.5.4)\r\n", "Requirement already satisfied: google-pasta>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (0.2.0)\r\n", "Requirement already satisfied: h5py>=3.10.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (3.10.0)\r\n", "Requirement already satisfied: libclang>=13.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (16.0.6)\r\n", "Requirement already satisfied: ml-dtypes~=0.3.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (0.3.2)\r\n", "Requirement already satisfied: opt-einsum>=2.3.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (3.3.0)\r\n", "Requirement already satisfied: packaging in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (23.2)\r\n", "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (3.20.3)\r\n", "Requirement already satisfied: requests<3,>=2.21.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (2.31.0)\r\n", "Requirement already satisfied: setuptools in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (69.1.1)\r\n", "Requirement already satisfied: six>=1.12.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (1.16.0)\r\n", "Requirement already satisfied: termcolor>=1.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (2.4.0)\r\n", "Requirement already satisfied: typing-extensions>=3.6.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (4.5.0)\r\n", "Requirement already satisfied: wrapt>=1.11.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (1.16.0)\r\n", "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (1.62.1)\r\n", "Requirement already satisfied: tensorboard<2.17,>=2.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (2.16.2)\r\n", "Requirement already satisfied: keras>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (3.0.5)\r\n", "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (0.36.0)\r\n", "Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.17,>=2.16->tf-keras) (1.26.4)\r\n", "Requirement already satisfied: wheel<1.0,>=0.23.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from astunparse>=1.6.0->tensorflow<2.17,>=2.16->tf-keras) (0.41.2)\r\n", "Requirement already satisfied: rich in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras>=3.0.0->tensorflow<2.17,>=2.16->tf-keras) (13.7.1)\r\n", "Requirement already satisfied: namex in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras>=3.0.0->tensorflow<2.17,>=2.16->tf-keras) (0.0.7)\r\n", "Requirement already satisfied: dm-tree in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras>=3.0.0->tensorflow<2.17,>=2.16->tf-keras) (0.1.8)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: charset-normalizer<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow<2.17,>=2.16->tf-keras) (3.3.2)\r\n", "Requirement already satisfied: idna<4,>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow<2.17,>=2.16->tf-keras) (3.6)\r\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow<2.17,>=2.16->tf-keras) (2.2.1)\r\n", "Requirement already satisfied: certifi>=2017.4.17 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow<2.17,>=2.16->tf-keras) (2024.2.2)\r\n", "Requirement already satisfied: markdown>=2.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16->tf-keras) (3.5.2)\r\n", "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16->tf-keras) (0.7.2)\r\n", "Requirement already satisfied: werkzeug>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16->tf-keras) (3.0.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: importlib-metadata>=4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from markdown>=2.6.8->tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16->tf-keras) (7.0.2)\r\n", "Requirement already satisfied: MarkupSafe>=2.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16->tf-keras) (2.1.5)\r\n", "Requirement already satisfied: markdown-it-py>=2.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from rich->keras>=3.0.0->tensorflow<2.17,>=2.16->tf-keras) (3.0.0)\r\n", "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from rich->keras>=3.0.0->tensorflow<2.17,>=2.16->tf-keras) (2.17.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: zipp>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16->tf-keras) (3.17.0)\r\n", "Requirement already satisfied: mdurl~=0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.0.0->tensorflow<2.17,>=2.16->tf-keras) (0.1.2)\r\n" ] } ], "source": [ "!sudo apt-get update\n", "!sudo apt-get install -y xvfb ffmpeg freeglut3-dev\n", "!pip install 'imageio==2.4.0'\n", "!pip install pyvirtualdisplay\n", "!pip install tf-agents\n", "!pip install pyglet\n", "!pip install tf-keras" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:29.103366Z", "iopub.status.busy": "2024-03-09T12:47:29.103101Z", "iopub.status.idle": "2024-03-09T12:47:29.106787Z", "shell.execute_reply": "2024-03-09T12:47:29.106118Z" }, "id": "WPuD0bMEY9Iz" }, "outputs": [], "source": [ "import os\n", "# Keep using keras-2 (tf-keras) rather than keras-3 (keras).\n", "os.environ['TF_USE_LEGACY_KERAS'] = '1'" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:29.109806Z", "iopub.status.busy": "2024-03-09T12:47:29.109441Z", "iopub.status.idle": "2024-03-09T12:47:32.547395Z", "shell.execute_reply": "2024-03-09T12:47:32.546038Z" }, "id": "sMitx5qSgJk1" }, "outputs": [], "source": [ "from __future__ import absolute_import\n", "from __future__ import division\n", "from __future__ import print_function\n", "\n", "import base64\n", "import imageio\n", "import IPython\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "import PIL.Image\n", "import pyvirtualdisplay\n", "\n", "import tensorflow as tf\n", "\n", "from tf_agents.agents.categorical_dqn import categorical_dqn_agent\n", "from tf_agents.drivers import dynamic_step_driver\n", "from tf_agents.environments import suite_gym\n", "from tf_agents.environments import tf_py_environment\n", "from tf_agents.eval import metric_utils\n", "from tf_agents.metrics import tf_metrics\n", "from tf_agents.networks import categorical_q_network\n", "from tf_agents.policies import random_tf_policy\n", "from tf_agents.replay_buffers import tf_uniform_replay_buffer\n", "from tf_agents.trajectories import trajectory\n", "from tf_agents.utils import common\n", "\n", "# Set up a virtual display for rendering OpenAI gym environments.\n", "display = pyvirtualdisplay.Display(visible=0, size=(1400, 900)).start()" ] }, { "cell_type": "markdown", "metadata": { "id": "LmC0NDhdLIKY" }, "source": [ "## Hyperparameters" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:32.552900Z", "iopub.status.busy": "2024-03-09T12:47:32.551774Z", "iopub.status.idle": "2024-03-09T12:47:32.557770Z", "shell.execute_reply": "2024-03-09T12:47:32.557060Z" }, "id": "HC1kNrOsLSIZ" }, "outputs": [], "source": [ "env_name = \"CartPole-v1\" # @param {type:\"string\"}\n", "num_iterations = 15000 # @param {type:\"integer\"}\n", "\n", "initial_collect_steps = 1000 # @param {type:\"integer\"} \n", "collect_steps_per_iteration = 1 # @param {type:\"integer\"}\n", "replay_buffer_capacity = 100000 # @param {type:\"integer\"}\n", "\n", "fc_layer_params = (100,)\n", "\n", "batch_size = 64 # @param {type:\"integer\"}\n", "learning_rate = 1e-3 # @param {type:\"number\"}\n", "gamma = 0.99\n", "log_interval = 200 # @param {type:\"integer\"}\n", "\n", "num_atoms = 51 # @param {type:\"integer\"}\n", "min_q_value = -20 # @param {type:\"integer\"}\n", "max_q_value = 20 # @param {type:\"integer\"}\n", "n_step_update = 2 # @param {type:\"integer\"}\n", "\n", "num_eval_episodes = 10 # @param {type:\"integer\"}\n", "eval_interval = 1000 # @param {type:\"integer\"}" ] }, { "cell_type": "markdown", "metadata": { "id": "VMsJC3DEgI0x" }, "source": [ "## Environment\n", "\n", "Load the environment as before, with one for training and one for evaluation. Here we use CartPole-v1 (vs. CartPole-v0 in the DQN tutorial), which has a larger max reward of 500 rather than 200." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:32.560829Z", "iopub.status.busy": "2024-03-09T12:47:32.560567Z", "iopub.status.idle": "2024-03-09T12:47:32.602216Z", "shell.execute_reply": "2024-03-09T12:47:32.601553Z" }, "id": "Xp-Y4mD6eDhF" }, "outputs": [], "source": [ "train_py_env = suite_gym.load(env_name)\n", "eval_py_env = suite_gym.load(env_name)\n", "\n", "train_env = tf_py_environment.TFPyEnvironment(train_py_env)\n", "eval_env = tf_py_environment.TFPyEnvironment(eval_py_env)" ] }, { "cell_type": "markdown", "metadata": { "id": "E9lW_OZYFR8A" }, "source": [ "## Agent\n", "\n", "C51 is a Q-learning algorithm based on DQN. Like DQN, it can be used on any environment with a discrete action space.\n", "\n", "The main difference between C51 and DQN is that rather than simply predicting the Q-value for each state-action pair, C51 predicts a histogram model for the probability distribution of the Q-value:\n", "\n", "![Example C51 Distribution](images/c51_distribution.png)\n", "\n", "By learning the distribution rather than simply the expected value, the algorithm is able to stay more stable during training, leading to improved final performance. This is particularly true in situations with bimodal or even multimodal value distributions, where a single average does not provide an accurate picture.\n", "\n", "In order to train on probability distributions rather than on values, C51 must perform some complex distributional computations in order to calculate its loss function. But don't worry, all of this is taken care of for you in TF-Agents!\n", "\n", "To create a C51 Agent, we first need to create a `CategoricalQNetwork`. The API of the `CategoricalQNetwork` is the same as that of the `QNetwork`, except that there is an additional argument `num_atoms`. This represents the number of support points in our probability distribution estimates. (The above image includes 10 support points, each represented by a vertical blue bar.) As you can tell from the name, the default number of atoms is 51.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:32.605688Z", "iopub.status.busy": "2024-03-09T12:47:32.605217Z", "iopub.status.idle": "2024-03-09T12:47:32.633217Z", "shell.execute_reply": "2024-03-09T12:47:32.632572Z" }, "id": "TgkdEPg_muzV" }, "outputs": [], "source": [ "categorical_q_net = categorical_q_network.CategoricalQNetwork(\n", " train_env.observation_spec(),\n", " train_env.action_spec(),\n", " num_atoms=num_atoms,\n", " fc_layer_params=fc_layer_params)" ] }, { "cell_type": "markdown", "metadata": { "id": "z62u55hSmviJ" }, "source": [ "We also need an `optimizer` to train the network we just created, and a `train_step_counter` variable to keep track of how many times the network was updated.\n", "\n", "Note that one other significant difference from vanilla `DqnAgent` is that we now need to specify `min_q_value` and `max_q_value` as arguments. These specify the most extreme values of the support (in other words, the most extreme of the 51 atoms on either side). Make sure to choose these appropriately for your particular environment. Here we use -20 and 20." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:32.637145Z", "iopub.status.busy": "2024-03-09T12:47:32.636382Z", "iopub.status.idle": "2024-03-09T12:47:36.252323Z", "shell.execute_reply": "2024-03-09T12:47:36.251611Z" }, "id": "jbY4yrjTEyc9" }, "outputs": [], "source": [ "optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=learning_rate)\n", "\n", "train_step_counter = tf.Variable(0)\n", "\n", "agent = categorical_dqn_agent.CategoricalDqnAgent(\n", " train_env.time_step_spec(),\n", " train_env.action_spec(),\n", " categorical_q_network=categorical_q_net,\n", " optimizer=optimizer,\n", " min_q_value=min_q_value,\n", " max_q_value=max_q_value,\n", " n_step_update=n_step_update,\n", " td_errors_loss_fn=common.element_wise_squared_loss,\n", " gamma=gamma,\n", " train_step_counter=train_step_counter)\n", "agent.initialize()" ] }, { "cell_type": "markdown", "metadata": { "id": "L7O7F_HqiQ1G" }, "source": [ "One last thing to note is that we also added an argument to use n-step updates with $n$ = 2. In single-step Q-learning ($n$ = 1), we only compute the error between the Q-values at the current time step and the next time step using the single-step return (based on the Bellman optimality equation). The single-step return is defined as:\n", "\n", "$G_t = R_{t + 1} + \\gamma V(s_{t + 1})$\n", "\n", "where we define $V(s) = \\max_a{Q(s, a)}$.\n", "\n", "N-step updates involve expanding the standard single-step return function $n$ times:\n", "\n", "$G_t^n = R_{t + 1} + \\gamma R_{t + 2} + \\gamma^2 R_{t + 3} + \\dots + \\gamma^n V(s_{t + n})$\n", "\n", "N-step updates enable the agent to bootstrap from further in the future, and with the right value of $n$, this often leads to faster learning.\n", "\n", "Although C51 and n-step updates are often combined with prioritized replay to form the core of the [Rainbow agent](https://arxiv.org/pdf/1710.02298.pdf), we saw no measurable improvement from implementing prioritized replay. Moreover, we find that when combining our C51 agent with n-step updates alone, our agent performs as well as other Rainbow agents on the sample of Atari environments we've tested." ] }, { "cell_type": "markdown", "metadata": { "id": "94rCXQtbUbXv" }, "source": [ "## Metrics and Evaluation\n", "\n", "The most common metric used to evaluate a policy is the average return. The return is the sum of rewards obtained while running a policy in an environment for an episode, and we usually average this over a few episodes. We can compute the average return metric as follows.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:36.256573Z", "iopub.status.busy": "2024-03-09T12:47:36.256058Z", "iopub.status.idle": "2024-03-09T12:47:38.637806Z", "shell.execute_reply": "2024-03-09T12:47:38.637056Z" }, "id": "bitzHo5_UbXy" }, "outputs": [ { "data": { "text/plain": [ "37.7" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#@test {\"skip\": true}\n", "def compute_avg_return(environment, policy, num_episodes=10):\n", "\n", " total_return = 0.0\n", " for _ in range(num_episodes):\n", "\n", " time_step = environment.reset()\n", " episode_return = 0.0\n", "\n", " while not time_step.is_last():\n", " action_step = policy.action(time_step)\n", " time_step = environment.step(action_step.action)\n", " episode_return += time_step.reward\n", " total_return += episode_return\n", "\n", " avg_return = total_return / num_episodes\n", " return avg_return.numpy()[0]\n", "\n", "\n", "random_policy = random_tf_policy.RandomTFPolicy(train_env.time_step_spec(),\n", " train_env.action_spec())\n", "\n", "compute_avg_return(eval_env, random_policy, num_eval_episodes)\n", "\n", "# Please also see the metrics module for standard implementations of different\n", "# metrics." ] }, { "cell_type": "markdown", "metadata": { "id": "NLva6g2jdWgr" }, "source": [ "## Data Collection\n", "\n", "As in the DQN tutorial, set up the replay buffer and the initial data collection with the random policy." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:38.641633Z", "iopub.status.busy": "2024-03-09T12:47:38.640989Z", "iopub.status.idle": "2024-03-09T12:47:46.000946Z", "shell.execute_reply": "2024-03-09T12:47:46.000189Z" }, "id": "wr1KSAEGG4h9" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py:377: ReplayBuffer.get_next (from tf_agents.replay_buffers.replay_buffer) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use `as_dataset(..., single_deterministic_pass=False) instead.\n" ] } ], "source": [ "#@test {\"skip\": true}\n", "replay_buffer = tf_uniform_replay_buffer.TFUniformReplayBuffer(\n", " data_spec=agent.collect_data_spec,\n", " batch_size=train_env.batch_size,\n", " max_length=replay_buffer_capacity)\n", "\n", "def collect_step(environment, policy):\n", " time_step = environment.current_time_step()\n", " action_step = policy.action(time_step)\n", " next_time_step = environment.step(action_step.action)\n", " traj = trajectory.from_transition(time_step, action_step, next_time_step)\n", "\n", " # Add trajectory to the replay buffer\n", " replay_buffer.add_batch(traj)\n", "\n", "for _ in range(initial_collect_steps):\n", " collect_step(train_env, random_policy)\n", "\n", "# This loop is so common in RL, that we provide standard implementations of\n", "# these. For more details see the drivers module.\n", "\n", "# Dataset generates trajectories with shape [BxTx...] where\n", "# T = n_step_update + 1.\n", "dataset = replay_buffer.as_dataset(\n", " num_parallel_calls=3, sample_batch_size=batch_size,\n", " num_steps=n_step_update + 1).prefetch(3)\n", "\n", "iterator = iter(dataset)" ] }, { "cell_type": "markdown", "metadata": { "id": "hBc9lj9VWWtZ" }, "source": [ "## Training the agent\n", "\n", "The training loop involves both collecting data from the environment and optimizing the agent's networks. Along the way, we will occasionally evaluate the agent's policy to see how we are doing.\n", "\n", "The following will take ~7 minutes to run." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:47:46.004870Z", "iopub.status.busy": "2024-03-09T12:47:46.004589Z", "iopub.status.idle": "2024-03-09T12:59:36.892398Z", "shell.execute_reply": "2024-03-09T12:59:36.891625Z" }, "id": "0pTbJ3PeyF-u" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py:1260: calling foldr_v2 (from tensorflow.python.ops.functional_ops) with back_prop=False is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "back_prop=False is deprecated. Consider using tf.stop_gradient instead.\n", "Instead of:\n", "results = tf.foldr(fn, elems, back_prop=False)\n", "Use:\n", "results = tf.nest.map_structure(tf.stop_gradient, tf.foldr(fn, elems))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 200: loss = 3.2159409523010254\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 400: loss = 2.422974109649658\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 600: loss = 1.9803032875061035\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 800: loss = 1.733839750289917\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 1000: loss = 1.705157995223999\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 1000: Average Return = 88.60\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 1200: loss = 1.655350923538208\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 1400: loss = 1.419114351272583\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 1600: loss = 1.2578476667404175\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 1800: loss = 1.3189895153045654\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 2000: loss = 0.9676651954650879\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 2000: Average Return = 130.80\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 2200: loss = 0.7909003496170044\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 2400: loss = 0.9291537404060364\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 2600: loss = 0.8300429582595825\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 2800: loss = 0.9739845991134644\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 3000: loss = 0.5435967445373535\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 3000: Average Return = 261.40\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 3200: loss = 0.7065144777297974\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 3400: loss = 0.8492055535316467\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 3600: loss = 0.808651864528656\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 3800: loss = 0.48259130120277405\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 4000: loss = 0.9187874794006348\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 4000: Average Return = 280.90\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 4200: loss = 0.7415772676467896\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 4400: loss = 0.621947169303894\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 4600: loss = 0.5226543545722961\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 4800: loss = 0.7011302709579468\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 5000: loss = 0.7732619047164917\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 5000: Average Return = 271.70\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 5200: loss = 0.8493011593818665\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 5400: loss = 0.6786139011383057\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 5600: loss = 0.5639233589172363\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 5800: loss = 0.48468759655952454\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 6000: loss = 0.6366198062896729\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 6000: Average Return = 350.70\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 6200: loss = 0.4855012893676758\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 6400: loss = 0.4458327889442444\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 6600: loss = 0.6745614409446716\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 6800: loss = 0.5021890997886658\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 7000: loss = 0.4639193117618561\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 7000: Average Return = 343.00\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 7200: loss = 0.4711253345012665\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 7400: loss = 0.5891958475112915\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 7600: loss = 0.3957907557487488\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 7800: loss = 0.4868921637535095\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 8000: loss = 0.5140666365623474\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 8000: Average Return = 396.10\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 8200: loss = 0.6051771640777588\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 8400: loss = 0.6179391741752625\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 8600: loss = 0.5253893733024597\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 8800: loss = 0.3697047531604767\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 9000: loss = 0.7271263599395752\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 9000: Average Return = 320.20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 9200: loss = 0.5285177826881409\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 9400: loss = 0.4590812921524048\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 9600: loss = 0.4743385910987854\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 9800: loss = 0.47938746213912964\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 10000: loss = 0.5290409326553345\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 10000: Average Return = 433.00\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 10200: loss = 0.4573556184768677\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 10400: loss = 0.352144718170166\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 10600: loss = 0.39160820841789246\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 10800: loss = 0.3254846930503845\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 11000: loss = 0.37145161628723145\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 11000: Average Return = 414.60\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 11200: loss = 0.382583349943161\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 11400: loss = 0.44465434551239014\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 11600: loss = 0.4484185576438904\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 11800: loss = 0.248131662607193\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 12000: loss = 0.5516679883003235\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 12000: Average Return = 375.40\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 12200: loss = 0.3307253420352936\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 12400: loss = 0.19486135244369507\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 12600: loss = 0.31668007373809814\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 12800: loss = 0.4462052285671234\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 13000: loss = 0.241848886013031\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 13000: Average Return = 326.80\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 13200: loss = 0.20919030904769897\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 13400: loss = 0.2044396996498108\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 13600: loss = 0.428558886051178\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 13800: loss = 0.1880824714899063\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 14000: loss = 0.34256821870803833\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 14000: Average Return = 345.50\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 14200: loss = 0.22452744841575623\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 14400: loss = 0.29694461822509766\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 14600: loss = 0.4149337410926819\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 14800: loss = 0.41922691464424133\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 15000: loss = 0.4064670205116272\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 15000: Average Return = 242.10\n" ] } ], "source": [ "#@test {\"skip\": true}\n", "try:\n", " %%time\n", "except:\n", " pass\n", "\n", "# (Optional) Optimize by wrapping some of the code in a graph using TF function.\n", "agent.train = common.function(agent.train)\n", "\n", "# Reset the train step\n", "agent.train_step_counter.assign(0)\n", "\n", "# Evaluate the agent's policy once before training.\n", "avg_return = compute_avg_return(eval_env, agent.policy, num_eval_episodes)\n", "returns = [avg_return]\n", "\n", "for _ in range(num_iterations):\n", "\n", " # Collect a few steps using collect_policy and save to the replay buffer.\n", " for _ in range(collect_steps_per_iteration):\n", " collect_step(train_env, agent.collect_policy)\n", "\n", " # Sample a batch of data from the buffer and update the agent's network.\n", " experience, unused_info = next(iterator)\n", " train_loss = agent.train(experience)\n", "\n", " step = agent.train_step_counter.numpy()\n", "\n", " if step % log_interval == 0:\n", " print('step = {0}: loss = {1}'.format(step, train_loss.loss))\n", "\n", " if step % eval_interval == 0:\n", " avg_return = compute_avg_return(eval_env, agent.policy, num_eval_episodes)\n", " print('step = {0}: Average Return = {1:.2f}'.format(step, avg_return))\n", " returns.append(avg_return)" ] }, { "cell_type": "markdown", "metadata": { "id": "68jNcA_TiJDq" }, "source": [ "## Visualization\n" ] }, { "cell_type": "markdown", "metadata": { "id": "aO-LWCdbbOIC" }, "source": [ "### Plots\n", "\n", "We can plot return vs global steps to see the performance of our agent. In `Cartpole-v1`, the environment gives a reward of +1 for every time step the pole stays up, and since the maximum number of steps is 500, the maximum possible return is also 500." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:59:36.896462Z", "iopub.status.busy": "2024-03-09T12:59:36.896172Z", "iopub.status.idle": "2024-03-09T12:59:37.121197Z", "shell.execute_reply": "2024-03-09T12:59:37.120473Z" }, "id": "NxtL1mbOYCVO" }, "outputs": [ { "data": { "text/plain": [ "(-11.255000400543214, 550.0)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAGwCAYAAABPSaTdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABeOklEQVR4nO3deVhUZf8G8HsWGPZBdtlBUURxAwXUNJU010qzVHLLrMzKtMz8lVq2aLbYZtnba2qvW1paai4Z7oqouOGGoiIgqygzrAPMnN8fyBS5McpwZob7c11zXXHOmZnvEzpz+5xnkQiCIICIiIjIQknFLoCIiIjImBh2iIiIyKIx7BAREZFFY9ghIiIii8awQ0RERBaNYYeIiIgsGsMOERERWTS52AWYAp1Oh6ysLDg6OkIikYhdDhEREdWBIAgoKiqCt7c3pNI7998w7ADIysqCn5+f2GUQERHRfcjIyICvr+8dzzPsAHB0dARQ/T/LyclJ5GqIiIioLtRqNfz8/PTf43fCsAPob105OTkx7BAREZmZew1B4QBlIiIismgMO0RERGTRGHaIiIjIojHsEBERkUVj2CEiIiKLxrBDREREFo1hh4iIiCwaww4RERFZNIYdIiIismgMO0RERGTRGHaIiIjIojHsEBERkUVj2CEiIiKLxrBDREREFo1hh4iIiCwaww4RERFZNIYdIiIismgMO0RERGTRGHaIiIjIojHsEBERkUVj2CEiIiKLxrBDREREFo1hh4iIiCwaww4RERFZNIYdIiIismgMO0RERGTRRA077777LiQSSa1HaGio/nx5eTkmTZoEV1dXODg4YOjQocjNza31Gunp6RgwYADs7Ozg4eGBadOmoaqqqqGbQkRERCZKLnYBrVu3xl9//aX/WS7/u6QpU6bgjz/+wNq1a6FUKvHyyy9jyJAh2L9/PwBAq9ViwIAB8PLywoEDB5CdnY3Ro0fDysoKH330UYO3hYiIiEyP6GFHLpfDy8vrluMqlQqLFy/GypUr0atXLwDAkiVL0KpVKxw8eBDR0dH4888/cebMGfz111/w9PRE+/bt8f7772P69Ol49913YW1t3dDNISIiIhMj+pidCxcuwNvbG8HBwYiLi0N6ejoAICkpCZWVlYiNjdVfGxoaCn9/fyQkJAAAEhISEB4eDk9PT/01ffv2hVqtxunTp+/4nhqNBmq1utaDiIiILJOoYScqKgpLly7F1q1b8d133+Hy5ct46KGHUFRUhJycHFhbW8PZ2bnWczw9PZGTkwMAyMnJqRV0as7XnLuTuXPnQqlU6h9+fn712zAiIiIyGaLexurXr5/+v9u2bYuoqCgEBARgzZo1sLW1Ndr7zpgxA1OnTtX/rFarGXiIiIgslOi3sf7J2dkZLVq0QGpqKry8vFBRUYHCwsJa1+Tm5urH+Hh5ed0yO6vm59uNA6qhUCjg5ORU60FERESWyaTCTnFxMS5evIimTZsiIiICVlZWiI+P159PSUlBeno6YmJiAAAxMTFITk5GXl6e/prt27fDyckJYWFhDV4/ERERmR5Rb2O98cYbGDRoEAICApCVlYXZs2dDJpNhxIgRUCqVGD9+PKZOnQoXFxc4OTnhlVdeQUxMDKKjowEAffr0QVhYGEaNGoX58+cjJycH77zzDiZNmgSFQiFm04iIiMhEiBp2MjMzMWLECBQUFMDd3R3dunXDwYMH4e7uDgBYsGABpFIphg4dCo1Gg759++Lbb7/VP18mk2HTpk2YOHEiYmJiYG9vjzFjxmDOnDliNYmIiIhMjEQQBEHsIsSmVquhVCqhUqk4foeIiMhM1PX726TG7BARERHVN4YdIiIismgMO0RERGTRGHaIiIjIojHsEBERkUVj2CEiIiKLxrBDREREFo1hh4iIiCwaww4RERFZNIYdIiIismgMO0RERGTRGHaIiIjIojHsEBERkUVj2CEiIiKLxrBDREREFo1hh4iIiCwaww4RERFZNIYdIiIismgMO0RERGTRGHaIiIjIojHsEBERkUVj2CEiIiKLxrBDREREFo1hh4iIiCwaww4RERFZNIYdIiIismgMO0RERGTRGHaIiIjIojHsEBERkUVj2CEiIiKLxrBDREREFo1hh4iIiCwaww4RERFZNIYdIiIismgMO0RERGTRGHaIiIjIosnFLoCIiMSnKqvEq6uOwV4hw/BO/ujW3A1SqUTssojqBcMOERFh9aF07D6fDwDYnJwDfxc7DO/sh2ERfnB3VIhcHdGD4W0sIqJGTqcTsOpQOgCgSzNXONrIkX69FPO3pqDLvHhMWnkUB1KvQRAEkSsluj/s2SEiauQOXCxAWkEpHBVy/HdMJCSQYOPJLKxMTMfxjEL8cTIbf5zMRpCbPUZ09sOTEX5wsbcWu2yiOpMIjOpQq9VQKpVQqVRwcnISuxwiogb10ookbE7OweiYAMx5rE2tc6ezVFiZmI7fj2ehWFMFALCWSdEv3AsjO/ujc5ALJBKO7SFx1PX7m2EHDDtE1HjlqcvRZd4OVOkEbH3tIYR63f4zsERThQ0nqnt7kq+q9MebezhgRGd/DO3oA2c79vZQw2LYMQDDDhE1Vt/suIBP/zyPiIAm+HVilzo9JzlThZWHruD341kordACABRyKQa0bYq4KH909G/C3h5qEAw7BmDYIaLGSKsT0H3+TlwtLMPnT7XDkI6+Bj2/qLwSvx2v7u05m63WH2/p6YiRUf54oqMPnGys6rtsIj2GHQMw7BBRY7TzXB7GLT0Mpa0VEv+vN2ysZPf1OoIg4HhGIVYmpmPjySyUV+oAADZWUgxu542RUQFo56tkbw/Vu7p+f3M2FhFRI7UisXq6+ZMRvvcddABAIpGgg38TdPBvgncGhmH90UysPJSO87nFWHMkE2uOZCKsqRNGRvnj8Q4+cFDwq4caFnt2wJ4dImp8sgrL0O3jHdAJQPzrPdDM3aFeX18QBCRduYGVienYlJyNiqrq3h57axkGt/dBXJQ/2vgo6/U9qfFhzw4REd3R6sMZ0AlAdLBLvQcdoLq3JzLQBZGBLpg5MAy/3uztuZRfglWH0rHqUDra+ioxsrM/Brf3hp01v47IeNizA/bsEFHjUqXVoevHO5Cr1uDrER0wqJ13g7yvIAhIvHwdKxPTseVUNiq11V8/Dgo5xncLwuTeIdyPiwzCnh0iIrqt+HN5yFVr4Gpvjb6tvRrsfSUSCaKDXREd7IqC4jD8kpSJVYfSkVZQii/jLyAlpwgLnm4PW+v7Hz9EdDvcG4uIqJFZeXNg8rBIP1jLxfkacHVQ4IUezbDj9Yfx6bB2sJZJsfV0Dob/JwF5ReWi1ESWi2GHiKgRSS8oxZ4L1bubj+jsJ3I1gFQqwZMRvlgxIQpN7KxwIlOFJxYewPncIrFLIwvCsENE1IisOpwOQQAeCnFDgKu92OXodQp0wbqXuiLIzR5XC8sw9NsD2HfhmthlkYVg2CEiaiQqqnRYeyQDABAXFSByNbcKcrPHuold0DnQBUWaKoxdcgirD6WLXRZZAIYdIqJG4s8zObhWXAEPRwV6t/IQu5zbamJvjf891xmPt/dGlU7AW+uSMW/LOeh0jX7iMD0Akwk78+bNg0QiwWuvvaY/Vl5ejkmTJsHV1RUODg4YOnQocnNzaz0vPT0dAwYMgJ2dHTw8PDBt2jRUVVU1cPVERKavZmDy8E5+sJKZzMf/LRRyGRY83R6Te4cAABbtvoiXVx1FeaVW5MrIXJnEn/bDhw/j+++/R9u2bWsdnzJlCjZu3Ii1a9di9+7dyMrKwpAhQ/TntVotBgwYgIqKChw4cADLli3D0qVLMWvWrIZuAhGRSbuYX4wDFwsglQBPd/YXu5x7kkgkmPJIC3z+VDtYySTYnJyD4f85iGvFGrFLIzMketgpLi5GXFwcfvjhBzRp0kR/XKVSYfHixfj888/Rq1cvREREYMmSJThw4AAOHjwIAPjzzz9x5swZLF++HO3bt0e/fv3w/vvvY+HChaioqBCrSUREJmfVzV6dni094ONsK3I1dTekoy/+Nz4KSlsrHM8oxOML9+MCZ2qRgUQPO5MmTcKAAQMQGxtb63hSUhIqKytrHQ8NDYW/vz8SEhIAAAkJCQgPD4enp6f+mr59+0KtVuP06dN3fE+NRgO1Wl3rQURkqcortfjlaCYAIC7a9Ht1/i062BXrXuqCAFc7ZN4ow5DvDmB/KmdqUd2JGnZWr16No0ePYu7cubecy8nJgbW1NZydnWsd9/T0RE5Ojv6afwadmvM15+5k7ty5UCqV+oefn/hrTRARGcuWU9koLK2Ej7MterQwzYHJ99LM3QHrX+qKyIAmKCqvwpgfD2HN4QyxyyIzIVrYycjIwOTJk7FixQrY2Ng06HvPmDEDKpVK/8jI4F8YIqqtRFOFhTtTcSbL/Ht+/zkwWWbGe0+52Ftj+XNRGNSueqbWm7+exPytnKlF9yZa2ElKSkJeXh46duwIuVwOuVyO3bt346uvvoJcLoenpycqKipQWFhY63m5ubnw8qrey8XLy+uW2Vk1P9dcczsKhQJOTk61HkRENaq0Ory88ig+2ZaCCT8dQVmF+c4COp9bhMNpNyCTSvBUJ/PvxbaxkuHLp9vjlV7NAQDf7rqIV1Yf40wtuivRwk7v3r2RnJyM48eP6x+RkZGIi4vT/7eVlRXi4+P1z0lJSUF6ejpiYmIAADExMUhOTkZeXp7+mu3bt8PJyQlhYWEN3iYiMn+CIOC9jWewM6V6S4WrhWVYuDNV5KruX02vziOtPOHp1LC96MYilUrwep+W+HRY9UytP05mY+QPB1HAmVp0B6KFHUdHR7Rp06bWw97eHq6urmjTpg2USiXGjx+PqVOnYufOnUhKSsK4ceMQExOD6OhoAECfPn0QFhaGUaNG4cSJE9i2bRveeecdTJo0CQqFQqymEZEZ+3F/Gv538AokEiAuqnow73/2XMKl/GKRKzNcaUUVfjXjgcn38mSEL5Y92xlONnIcTS/EE98eQGqe+f2eyPhEn411NwsWLMDAgQMxdOhQdO/eHV5eXli3bp3+vEwmw6ZNmyCTyRATE4NnnnkGo0ePxpw5c0SsmojM1Z+nc/DBH2cAADP6heKDx9vg4ZbuqNDqMHvDaQiCeY0N2XQiG0XlVfB3sUPXZm5il2MUXZq5Yd1LXeHnYov066UY8u1+JFwsELssMjESwdz+9hqBWq2GUqmESqXi+B2iRupkZiGe/v4gyiq1GBnljw8fbwOJRIK0ayXos2APKrQ6LHqmIx5t01TsUuvssYX7cSKjEG/1C8WLPZqJXY5RFRRrMOGnIziaXggrmQRzh7TFkxG+YpdFRlbX72+T7tkhImoIVwvLMH7ZEZRVatG9hTvmDG4NiaR61lKgmz1e7BEMAJiz8QxKK8xjO5pTV1U4kVH9xd8YvvRdHRRYOSEaA9o2RaVWwBtrT+DzP1PMrjeOjINhh4gaNXV5JZ5dchj5RRqEejli4cgOkP9r36iJDzeHbxNbZKnK8c0O8xisvPLmbuGPtmkKN4fGMYbRxkqGr4d3wEsPV/difbUjFZNXH+dMLWLYIaLGq1Krw6QVR5GSWwQPRwV+HNsJjjZWt1xnay3D7EGtAQA/7L2EiyY+WLlYU4Xfj10FAIw0g32w6pNUKsGbj4Zi/tC2kEsl2HAiC8/8NxHXS7iFUGPGsENEjZIgCJj1+ynsvXANtlYyLB7TCd532TMqtpUHeoV6oFIr4F0TH6z827GrKKnQItjdHtHBLmKXI4qnOvlh2bOd4Wgjx5ErN/DEt/vNckYd1Q+GHSJqlP6z5xJWHcqARAJ8NaIDwn2Vd71eIpFg9qAwWMul2HvhGracuvOWNGISBAErbq6tExcVoB971Bh1be6GdRO7wLeJLa4UlOKJbw8g8RJnajVGDDtE1OhsTs7G3C3nAAAzB4ThkTDPezyjWoCrPSbenNX0/qYzKNGY3mDl4xmFOJuthrVciqEdfcQuR3Qhno5Y/1JXtPNzhqqsEs8sTsS6m2sPUePBsENEjcqx9BuY8vNxAMCYmACM6xpo0PMnPtwMfi62yFaV42sTHKxcs2LywLZN4WxnLXI1psHdUYHVE6LRr40XKrUCpq45gQXbz5v0rUiqXww7RNRoZFwvxYSfjkBTpUOvUA/MHBhm8G0eGysZ3r05WPm/ey8hNa/IGKXeF1VpJTaezALw9+rPVM3WWoaFIzvihZvLCHwZfwFT15yApooztRoDhh0iahRUZZUYt/QwrhVXIKypE74ecesU87rq3coTsa08UKUTTGpl5XXHMlFeqUOolyM6+jcRuxyTI5VKMKNfK8wdEg6ZVIL1x65i7I+HzXqjV6obhh0isngVVTpMXJ6E1LxieDnZ4MexnWCvkD/Qa84e1BoKuRT7UwvwR3J2PVV6/wRB0N/Ciovyb9QDk+9lRGd/LB3XCQ4KORIuFWDiiiRUVOnELouMiGGHiCyaIAh457dkHLhYADtrGRaPjYSX8sF3//ZzscNLDzcHUD1YuVjkwcqH027gQl4xbK1keKwDBybfy0Mh7lgyrhNsrKTYlZKPKWuOQ6szjR46qn8MO0Rk0b7ddRFrjmRCKgG+GdkBrb3vPsXcEC/0CIa/ix1y1Rp8HX+h3l73fqxMvAIAeKy9N5xuszAi3apToAsWPRMBK5kEf5zMxtvrk03mliTVL4YdIrJYG09k4ZNtKQCAdwe3Rq/Quk0xrysbKxneHRwGAFi87zIu5IozWPl6SQU2J1ev+zOSA5MN8nBLD3w5vAOkEmD14Qx8tPksA48FYtghIouUdOU6Xl97AgDwbNcgjI4JNMr79Ar1xCNhnqjSCZj1uziDlX9JykCFVodwHyXa+jo3+Pubu/7hTTFvSFsAwA97L5vN/mdUdww7RGRxrhSUYMJP1YNOHwnzxNsDWhn1/WYNDINCLkXCpQJsPNmwg5V1OgGrDmUA4HTzB/FUJz/MHFjdS/fZ9vNYsv+yyBVRfWLYISKLUlhagXFLDuN6SQXCfZT4cnh7yKTGnZnk52KHST2rByt/0MCDlRMuFeDytRI4KOQY1M67wd7XEo3vFoTXYkMAAO9tPINfkrjSsqVg2CEii6Gp0uL5/yXh0rUSeCttsHhMJOysH2yKeV093z0YAa52yCvS4Mu/zjfIewJ/r5j8RAefB55OT8Dk3iF4tmsQAODNX05g6ynxlxWgB8ewQ0QWQRAEzPg1GYcuX4eDQo4fx3WCh9ODTzGvq+rBytUrK/+4Pw3nG2Cwcl5RObad5sDk+iSRSDBzYCs8FekLnQC8uuo49l7IF7ssekAMO0RkEb6KT8W6Y1chk0qwMK4jQr2cGryGni090CfME1qdgFm/nzL6YOW1RzJRpRPQ0d8ZrZo2fHstlUQiwdwhbdE/3AsVWh2e/ykJSVeui10WPQCGHSIye+uPZWLBzVtH7z/WBj1auItWy8yBYbCxkuLgpevYcCLLaO+j1QlYdaj6FtbIqACjvU9jJZNK8MXTHdCjhTvKKrUYu+QwTmepxC6L7hPDDhGZtcRLBZj+SzIA4IXuwaLfzvFzscPLNwcrf/jHWRSVVxrlffZcyEfmjTI42cgxsG1To7xHY2ctl2LRMxHoFNgEReVVGL34EC7lF4tdFt0Hhh0iMluX8ovxwvIkVGh16NfGC9MfDRW7JADAhO7BCNQPVjbOyso1A5OfjPCDjZXMKO9B1bulLx7bCa29nVBQUoFn/puIq4VlYpdFBmLYISKzdL2kAs8uPYzC0kq083PG50+1h9TIU8zrSiH/e7DykgNpSMmp38HK2aoyxJ/NBQCMjPKr19emWznZWOGnZzujmbs9slTleOa/icgv0ohdFhmAYYeIzE55pRbP/3QEaQWl8HG2xX9HR8LW2rR6Nx5u6YFHW3tBqxMws54HK68+lAGdAEQFuaC5h2O9vS7dmauDAsufi4KPsy0uXyvBqMWJUJUa5xYl1T+GHSKqk6uFZZi29gTmbjmLX5MyceqqCmUV2gavQxAEvPnLSRy5cgOONnIsHdcJ7o6KBq+jLmYOqh6sfOjydfx+vH4GK1dpdfj5cPWKyWKPT2psmiptseK5KLg5KHAupwjjlh5Cici73VPdcAUqIronQRDw1q8nsffCtVrHJRLA38UOIR6OaOHpgJZejgjxcESwu73RxpEs2H4eG05kQS6V4Lu4CIR4mm7Pho+zLV7pFYJPtqXgw81n0auVxwPvSL7jXB5y1OVwsbfGo2286qlSqqtAN3ssf64znv7+II6mF+KF/yVh8dhIKOSm1bNItTHsENE9xZ/Nw94L12Atk2JYpC8u5hfjfG4xrpdU4EpBKa4UlOKvm2NIAEAqAQJd7RHi6YAWno76R5CbPazl99+hvPZIBr66uUnjh0+0QbcQtwdum7E991AQfk3KxKVrJfhi+wXMGhT2QK+38uZ082GRvvyCFUmolxOWjuuEuP8mYl/qNby66hgWjuwIuYw3S0yVROBe9lCr1VAqlVCpVHBy4sJcRP9UUaVDnwW7kVZQiokPN6s14+lasQbnc4twIbcY53OLbj6KoSq7/VgGuVSCQDd7tPB0QIiHI1p6VfcIBbjaw+oeXxQHUq9h9I+HUKUT8NLDzfCmicy8qos95/Mx+sdDkEkl2PRKt/teADDjeim6f7ITggDseuNhBLrZ13OlZIgDqdcwdulhVFTpMKSjDz59sp3JDJK/nTx1OSABPBwbbmVxY6vr9/d99excuHABO3fuRF5eHnQ6Xa1zs2bNup+XJCITtfTAZaQVlMLdUaHf7LKGm4MCbg4KdGn2dw+LIAjIL9LgfK0AVB2IijRVSM0rRmpeMYAc/XOsZBIEuzkgxNMBLT0dEeL5dwiSSSVIzSvCi8uTUKUTMLBtU7zRp2VDNb9edG/hjv7hXticnINZv5/CmhdiIJEY/qW46lA6BAF4KMSNQccEdGnuhoUjO+LF5UlYd/QqHBVyvDu49X39bo1FqxOw+3welh9Mx86UPDjZWOGvqT1MdpybsRjcs/PDDz9g4sSJcHNzg5eXV61fqkQiwdGjR+u9SGNjzw7R7eUXadDz010o1lThkyfbYljk/U9zFgQBOery6hCUczME5RUjNbcIJXcY6Gwtl6KZuwMKijXIK9IgIqAJVjwXZZbrymQVlqH3Z7tRVqnF50+1w5COvgY9v6JKhy7zduBasQbfxXVEv3AuJGgqfjt2FVPWHIcgAK/0ao7XTSCM5xdpsOZIBlYmpt+yLtDk3iGY8kgLkSqrX0br2fnggw/w4YcfYvr06Q9UIBGZvk+3paBYU4W2vkoMNfDL+d8kEgmaKm3RVGlbazsHnU5AlqpMfwusphfoQl4Ryit1OJutBlA9EPo/oyLMMugAgLezLV7tHYKPt57DR5vPoncrTyht6z5YefuZXFwr1sDdUYHYME8jVkqGeryDD4o0VZj52yl8vSMVjjZyPN+9WYPXIQgCEi9fx/KDV7DtdA4qtdV9GUpbKwyL8IW7owJzt5zD8oNXMPHhZmb7d+l+GBx2bty4gWHDhhmjFiIyIcmZKqxJqp7iPHtQa6ONRZBKJfBtYgffJnboFfr3l7hWJyDzRinO5xYj/Xop+rXxgquDeXe9j+8WhLVJGbiUX4IF28/rFx6si5WHrgAAno70u+f4Jmp4o6IDUFReiflbU/DR5nNwtLHCiM4NszSAqqwS649mYkViOi7k/b2dRXs/ZzwTHYCBbZvCxkqGKq0Oyw6kIUtVjg3Hs/BUp8azIKXBYWfYsGH4888/8eKLLxqjHiIyAYIg4L2NpyEIwOPtvRER0KTBa5BJJQhwtUeAq+WMTbGWSzFncBs8szgRPyWk4alIP4R53/vW+aX8YuxPLYBEAgzv3Hi+oMzNSw83R1F5Fb7bdRH/tz4ZDgo5BrXzNtr7JWeqsPzgFWw4kYWyyupbwbZWMjzewRtxUQFo46Osdb1cJsWYLoGYu+Ucftx/GcMifU1qfJExGRx2mjdvjpkzZ+LgwYMIDw+HlVXtbthXX3213oojInFsPJmNI1duwNZKhun9zGfWkznoFuKGAW2b4o+T2frByvfqNavZ3bxnSw/4NrFriDLpPr3ZtyWKyiux/GA6pvx8HPYKWa0eywdVVqHFxpNZWHHwCk5k/r0LewtPBzwTHYDHO/jcdS2n4Z398WX8BZzLKcL+1AKzWL6hPhg8QDkoKOjOLyaR4NKlSw9cVEPjAGWiv5VVaNHrs13IVpXj9Uda4JXeIWKXZHGyVdWDlUsrtPh0WDs8GXHn8VDllVrEzI3HjdJK/Hd0JMfrmAGdTsCUNcfx+/EsKORSLHu2M6KDXR/oNS/mF2PFwXT8kpQBdXn1qs1WMgn6tWmKZ6ID0CmwSZ17aWb/fgrLEq6gZ0t3LBnX+YHqEptRBigLgoBdu3bBw8MDtra2D1wkEZmeRbsvIltVDh9nW0zoHix2ORapqdIWk3uHYO6Wc5i7+SweCbvzYOWtp3Jwo7QS3kob9Az1aOBK6X5IpRJ8OqwdSjRa/HU2F88tO4KVE6LQ1tfZoNep1Oqw/Uwulh+8ggMXC/THfZvYYmSUP56K9IPbfYxjG9c1CD8dvIKdKflIzStGcw8Hg1/D3Bg0yk0QBISEhCAzM9NY9RCRiK4WlmHR7osAgLcHtGpUszUa2riuQWju4YCCkgp8/mfKHa9bmVh9C+vpTv6QmfCCdVSblUyKb0Z2QEywK4o1VRj94yGczy2q03OzCsvw2Z8p6DJvB15acRQHLhZAKgFiW3lgybhO2DOtJ156uPl9BR2gesuL3jdvrS3Zf/m+XsPcGBR2pFIpQkJCUFBQcO+LicjszN18FpoqHaKCXNCP+y4ZVfVg5erZWP87eAWnrqpuueZ8bhEOpV2HTCrB041o5oylsLGS4YcxkWjv54zC0ko8899EpBeU3vZanU7ArpQ8PLfsCLp9vANf70hFfpEGbg4KvNyzOfZO74X/jumEni096mVm5Phu1UNSfj2aiRslFQ/8eqbO4PmL8+bNw7Rp03Dq1Clj1ENEIjl0+To2ncyGVALMGhTWaGZpiKlLczcMaucNnQDM+v0UdLraQyhrenViW3nAS2k5S/w3Jg4KOZaO64SWno7IK9IgbvFB5KrL9ecLijVYtPsiHv50F8YuOYy/zuZCJwAxwa74ZmQHHHirF97o2xI+zvU7dCQ62AVhTZ1QXqnT77dmyQweoNykSROUlpaiqqoK1tbWt4zduX79er0W2BA4QJkaO61OwOBv9uF0lhojo/zx0RPhYpfUaOSoytH7s10oqdBi/pNt8dTNVarLKrTo/NFfKCqvwrJnO9daiJHMT566HMO+T8CVglKEeDjg7QGtsP7YVWxJzkGFtnrbJUcbOZ6M8EVcVECDjKP5NSkTr689AU8nBfa+2euBNukVi9FWUP7iiy8epC4iMkFrj2TgdJYajjZyvG4hy8ibCy+lDSbHhuCjzecwb8s59A3zgtLOChtPZqGovAp+LrZ4qHnjmB5syTycbLB8fBSGLUrAhbxijF1yWH+ura8Sz0QFYFA7b9haN9w4uUHtvDFv6znkqjXYnJyNxzv4NNh7NzSDw86YMWOMUQcRiURdXolPtlUPkH0ttoXZr1JsjsZ1DcLaI5m4kFeMT/9MwfuPt9HfwhrR2d+kd9KmuvNzscPy56Iw4oeDKCqvxGPtfBAX7W/wLK36Yi2XYnR0AD7bfh6L913GY+29Lfb2tcFhJz397vf2/P0bZnlsIqof3+xIRUFJBYLd7TE6JkDscholK5kU7z3WGiN/SMTyxCto4+OE4xmFsJJJMCyCA5MtSXMPB+ye9jAkkDRoL86dxEUH4JudqUi+qsLhtBvoHOQidklGYXDYCQwMvGvy02pvv3sxEZmeS/nF+qmnMweGcc8lEXVp5obB7byx4UQW3lqXDADo29oL7o7sabM0dtYGf/UajYu9NYZ09MGqQxlYvO8Sw06NY8eO1fq5srISx44dw+eff44PP/yw3gojIuP78I+zqNQK6NnSHT1bcsE6sb09oBXiz+aipKL6H40jo9hTTsb3bNcgrDqUgT/P5CK9oBT+rpa3JYnBYaddu3a3HIuMjIS3tzc++eQTDBkypF4KIyLj2pWSh/hzeZBLJXhnYJjY5RAATycbTHmkBT744yyaudsj5gG3GCCqixBPR3Rv4Y495/Ox5MBlzB7UWuyS6l299aW1bNkShw8fvveFRCS6Sq0O7286AwAY2yUQzdwtf7l4c/Fs1yC42FujnZ+zxQ4WJdMzvlsQ9pzPx5rDGZjySIu7biZqjgwOO2q1utbPgiAgOzsb7777LkJCuGEgkTn4X8IVXMwvgau9NTf6NDFSqQRDOt55Y1AiY+ge4oYQDwdcyCvGmsMZeO4hy9oXz+Cw4+x86782BEGAn58fVq9eXW+FEZFxFBRrsOCv8wCAN/q2vOMGlETUeEgkEjzbLQgz1iVjyf40jO0SCLkFTVgwOOzs3Lmz1s9SqRTu7u5o3rw55HLTGWFORLf3+fbzKCqvQlhTJ/1qvURET3TwwSfbUnC1sAx/nslF//CmYpdUbwxOJxKJBF26dLkl2FRVVWHPnj3o3r17vRVHRPXrTJYaq27ugzN7UBh30SYiPRsrGeKi/PH1jlQs3nfZosKOwX1UPXv2vO3+VyqVCj179qyXooio/gmCgDmbTkMnAAPaNkUUZ/oQ0b+Mig6AlUyCpCs3cDyjUOxy6o3BYUcQhNvOECgoKIC9vX29FEVE9W/rqRwcvHQdCrkUM/qFil0OEZkgDycbDGrnDQBYvO+yyNXUnzrfxqpZP0cikWDs2LFQKP5e1VOr1eLkyZPo0qVL/VdIRA+svFKLDzefBQC80KMZfJtY3qJhRFQ/xncLwrqjV7E5ORsz+oXC29lW7JIeWJ17dpRKJZRKJQRBgKOjo/5npVIJLy8vPP/881i+fLkxayWi+/TfvZeQeaMMTZU2eLGHZU0pJaL61dpbiehgF2h1ApYlpIldTr2oc8/OkiVLAFTvjfXGG2/Uyy2r7777Dt999x3S0tIAAK1bt8asWbPQr18/AEB5eTlef/11rF69GhqNBn379sW3334LT09P/Wukp6dj4sSJ2LlzJxwcHDBmzBjMnTuXM8OIbspRlWPhzosAgLf6hZrUvjxEZJrGdwvGwUvXsSoxHa/2CoG9wrw/NwweszN79mwoFAr89ddf+P7771FUVAQAyMrKQnFxsUGv5evri3nz5iEpKQlHjhxBr1698Nhjj+H06dMAgClTpmDjxo1Yu3Ytdu/ejaysrFrbUWi1WgwYMAAVFRU4cOAAli1bhqVLl2LWrFmGNovIYn289RzKKrWICGiCwTfvxRMR3U3vUA8EutpBXV6FX49mil3OA5MIgiAY8oQrV67g0UcfRXp6OjQaDc6fP4/g4GBMnjwZGo0GixYteqCCXFxc8Mknn+DJJ5+Eu7s7Vq5ciSeffBIAcO7cObRq1QoJCQmIjo7Gli1bMHDgQGRlZel7exYtWoTp06cjPz8f1tbWdXpPtVoNpVIJlUoFJyenB6qfyJQcTb+BId8eAABseLkr2vo6i1sQEZmNZQfSMHvDaQS52SN+ag9ITXCpirp+fxvcszN58mRERkbixo0bsLX9e9DSE088gfj4+PurFtW9NKtXr0ZJSQliYmKQlJSEyspKxMbG6q8JDQ2Fv78/EhISAAAJCQkIDw+vdVurb9++UKvV+t6h29FoNFCr1bUeRJZGpxPw3sbq/a+GRfgy6BCRQZ6M8IWTjRyXr5Vgx7k8sct5IAaHnb179+Kdd965pdckMDAQV69eNbiA5ORkODg4QKFQ4MUXX8T69esRFhaGnJwcWFtbw9nZudb1np6eyMnJAQDk5OTUCjo152vO3cncuXNrDbD28+MqsmR51h+7ihMZhbC3lmHaoy3FLoeIzIy9Qo4Rnf0BmP80dIPDjk6ng1arveV4ZmYmHB0dDS6gZcuWOH78OBITEzFx4kSMGTMGZ86cMfh1DDFjxgyoVCr9IyMjw6jvR9TQijVV+HjrOQDAK71D4OFoI3JFRGSOxnQJhEwqQcKlApzOUoldzn0zOOz06dMHX3zxhf5niUSC4uJizJ49G/379ze4AGtrazRv3hwRERGYO3cu2rVrhy+//BJeXl6oqKhAYWFhretzc3Ph5eUFAPDy8kJubu4t52vO3YlCoYCTk1OtB5El+XZnKvKKNAhwtcO4roFil0NEZsrb2Rb92lR/n/64L03cYh6AwWHns88+w/79+xEWFoby8nKMHDlSfwvr448/fuCCdDodNBoNIiIiYGVlVWscUEpKCtLT0xETEwMAiImJQXJyMvLy/r6XuH37djg5OSEsLOyBayEyR+kFpfjv3uou53cGhEEhl4lcERGZs/HdggAAG09kIa+oXORq7o/BE+d9fX1x4sQJ/Pzzzzhx4gSKi4sxfvx4xMXF1RqwXBczZsxAv3794O/vj6KiIqxcuRK7du3Ctm3boFQqMX78eEydOhUuLi5wcnLCK6+8gpiYGERHRwOo7mUKCwvDqFGjMH/+fOTk5OCdd97BpEmTaq3wTNSYfLj5DCq0OjwU4obYVh5il0NEZq6DfxN09HfG0fRCLE+4gql9zG8M4H2tEiSXyxEXF4e4uDj9sezsbEybNg3ffPNNnV8nLy8Po0ePRnZ2NpRKJdq2bYtt27bhkUceAQAsWLAAUqkUQ4cOrbWoYA2ZTIZNmzZh4sSJiImJgb29PcaMGYM5c+bcT7OIzN7+1GvYdjoXMqkEMweG3XYfOyIiQ43vFoyjK49ieWI6XurZHDZW5tVjbNA6O6dPn8bOnTthbW2Np556Cs7Ozrh27Ro+/PBDLFq0CMHBwXed8m2quM4OWYIqrQ4DvtqHlNwijO0SiHcHtxa7JCKyEFVaHXp8sgtXC8swb0g4ht+cpSW2el9nZ8OGDejQoQNeffVVvPjii4iMjMTOnTvRqlUrnD17FuvXrzfLoENkKVYdSkdKbhGc7azwWmyI2OUQkQWRy6QY2yUQAPDj/sswcD1i0dU57HzwwQeYNGkS1Go1Pv/8c1y6dAmvvvoqNm/ejK1bt+LRRx81Zp1EdBeFpRX4bPt5AMDrj7SAs13dVg8nIqqrpzv7wd5ahvO5xdh74ZrY5RikzmEnJSUFkyZNgoODA1555RVIpVIsWLAAnTp1MmZ9RFQHX/x1AYWllWjp6ahfBIyIqD452VhhWGT1IrzmtshgncNOUVGR/n6YTCaDra0tgoODjVYYEdXN+dwi/O/gFQDArEFhkMsMXlGCiKhOxnUNhEQC7D6fj9S8IrHLqTODZmPVTAkHqtfDiY+Px6lTp2pdM3jw4PqrjshElVZUYf2xqzhwsQABLnZo66tEuK8zvJU2DToDShAEvL/pDLQ6AX3CPNG1uVuDvTcRNT4BrvZ4pJUn/jyTi8X70jB3SLjYJdVJnWdjSaX3/teiRCK57VYSpo6zsaiuMq6X4qeENPx8OAPq8qpbzrvYWyPcR1n98FWira8SXk7GC0B/ncnFcz8dgbVMiu1TuyPA1d4o70NEVCPxUgGe/s9BKORSJMzoDRd78cYI1vX7u849Ozqdrl4KIzI3giBgf2oBlh64jPhzeaj550GAqx0ea++DXFU5kq+qcD63CNdLKrD7fD52n8/XP9/N4Z8ByBltfZXwdHrwvao0VVp88Ef1PnLjHwpi0CGiBtE5yAVtfJxw6qoaKxOv4OVepj/7874WFSRqDEo0VVh3NBPLEq4gNa9Yf7x7C3eM7RKAh1t4QCr9u8emvFKLs9lqnLqqwslMFZKvqnAhrxjXiiuwMyUfO1P+DkDujgq09VGijU9170+4jxIeBgagpfvTkFZQCndHBSb1bP7gDSYiqgOJRILx3YIw5ecT+CnhCp7v3gzWctMeK8iwQ/QvaddK8FPCFaxNykDRzVtV9tYyPBnhi9FdAtHM3eG2z7OxkqGDfxN08G+iP1ZeqcWZbDWSb4af5EwVLuQVIb9Ig/hzeYg/9/e+bp5OCoT7OCP8ZgBq46OEu+Pttz3JKyrH1ztSAQDTHw2Fg4J/lYmo4QwI98bczeeQV6TBppNZGNLRV+yS7oqfkEQAdDoBe1OvYdmBNOxM+ftWVZCbPUbHBODJCF842lgZ/Lo2VjJ09G+Cjv8IQKUVVTibrdb3/iRnqpCaX4xctQa56lz8dTZXf21TpU2tMUDhPkq4Oijw6bYUFGuq0M5XiSEdfB64/UREhrCWSzGmSyA+2ZaCxfsu44kOPia9PY1B20VYKg5QbryKNVX4NSkTyxLScCm/RH+8Z0t3jOkSiO4h7rVuVRlLiaYKZ24GoOrbYIW4dK0Et/vb6eNsiyxVGQQB+HViF0QENLn1IiIiI7tRUoGYefEor9Rh9fPRiA52bfAa6n2AMpEluXytBMsOpOGXpEwUa6pvVTko5BgW6YvRMYEIcmvYwb72Cjk6BbqgU6CL/lixpgqnr97s/bnZA3TpWgmuFpYBAJ7o4MOgQ0SiaWJvjSEdfbEyMR2L910WJezU1X2FncLCQvzyyy+4ePEipk2bBhcXFxw9ehSenp7w8WGXOpkmnU7A7gv5WLo/rdZsqWbu9hjTJRBDOvqa1NgXB4UcUcGuiPrHB4i6vBKnr6qReaMUA9o2FbE6IiLg2a5BWJmYjr/O5iLtWgkCG/gfinVl8Cf7yZMnERsbC6VSibS0NEyYMAEuLi5Yt24d0tPT8dNPPxmjTqL7pi6vxC9HMvG/g1dw+Vr1rSqJBOjV0gNjuwaiW3M3k77X/E9ONlaIaeYKwHT/BUVEjUdzDwc83NIdu1LysfRAGt4d3Frskm7L4LAzdepUjB07FvPnz4ejo6P+eP/+/TFy5Mh6LY7oQaTmFeOnhDT8mpSJkorqxS4dbeR4KtIPo2MCuC4NEVE9eK5bMHal5GPNkQxMeaQFlLaGT+YwNoPDzuHDh/H999/fctzHxwc5OTn1UhTR/dLpBOxMycPSA2m1duUN8XDAmC6BeKKDD+xN6FYVEZG569rcFaFejjiXU4SfD6fj+e7NxC7pFgZ/6isUCqjV6luOnz9/Hu7u7vVSFJGhVGWVWHskA/87eAVXCkoBVN+qim3libFdAtGlmavZ3KoiIjInEokEz3YNwpu/nsSyA1fwbNcgk9uQ2OCwM3jwYMyZMwdr1qwBUN3I9PR0TJ8+HUOHDq33AonuZfuZXExefQylN29VOdnIMbyzP0ZFB8DPxU7k6oiILN/g9t6Yv+0crhaWYevpHAxs6y12SbUYHL0+++wzFBcXw8PDA2VlZejRoweaN28OR0dHfPjhh8aokeiOBEHAvC1nUVqhRQtPB3z0RDgO/l9v/F//Vgw6REQNxMZKhrioAADA4n2XRa7mVgb37CiVSmzfvh379u3DyZMnUVxcjI4dOyI2NtYY9RHd1ZlsNS7ml0Ahl+LXiV3ua5VjIiJ6cM9EB+C7XRdxLL0QSVdumNQ6YPc9UrNbt27o1q1bfdZCZLANx7MAAL1beTDoEBGJyN1Rgcfae2NtUiZ+3HfZvMPOV199ddvjEokENjY2aN68Obp37w6ZTPbAxRHdjU4nYOOJ6rAzuJ1p3R8mImqMxj8UhLVJmdhyKhuZN0rh28Q0hhMYHHYWLFiA/Px8lJaWokmT6tR248YN2NnZwcHBAXl5eQgODsbOnTvh5+dX7wUT1Thy5QayVOVwVMjxcEsPscshImr0Qr2c0LW5K/anFmDZgTS8PSBM7JIA3McA5Y8++gidOnXChQsXUFBQgIKCApw/fx5RUVH48ssvkZ6eDi8vL0yZMsUY9RLpbThxFQDQt40XbKzYk0hEZArGdwsCAKw+lKHfe1BsBoedd955BwsWLECzZn8vGtS8eXN8+umnmDFjBnx9fTF//nzs37+/Xgsl+qdKrQ5/nMwGADzWnrewiIhMxcMtPBDsbo8iTRXWHskQuxwA9xF2srOzUVV1a1KrqqrSr6Ds7e2NoqKiB6+O6A72pV7DjdJKuDlYI8aEd9olImpspFIJxnWt7t1Zsj8NWp0gckX3EXZ69uyJF154AceOHdMfO3bsGCZOnIhevXoBAJKTkxEUFFR/VRL9S80srIFtvU1upU4iosZuaEcfKG2tkH69FH+dzRW7HMPDzuLFi+Hi4oKIiAgoFAooFApERkbCxcUFixcvBgA4ODjgs88+q/diiQCgrEKLP09X9yIO4iwsIiKTY2ctx8gofwCmscigwbOxvLy8sH37dpw7dw7nz58HALRs2RItW7bUX9OzZ8/6q5DoX+LP5aKkQgvfJrbo6O8sdjlERHQbY2IC8cOeSzh0+TpOXVWhjY9StFrue1HB0NBQhIaG1mctRHVScwtrcDtvbu5JRGSivJQ2GNC2KX4/noXF+y5jwdPtRavlvsJOZmYmNmzYgPT0dFRUVNQ69/nnn9dLYUS3oyqrxK6UfADAY+19RK6GiIjuZny3IPx+PAsbT2ThrX6h8HSyEaUOg8NOfHw8Bg8ejODgYJw7dw5t2rRBWloaBEFAx44djVEjkd62Uzmo0OrQ0tMRLb0cxS6HiIjuoq2vMzoFNkFBSQWyCsvMJ+zMmDEDb7zxBt577z04Ojri119/hYeHB+Li4vDoo48ao0Yivd9vLiQ4mGvrEBGZhUXPRKCJnTWkUvGGHRg8G+vs2bMYPXo0AEAul6OsrAwODg6YM2cOPv7443ovkKhGnrocCRcLAHAvLCIic+HqoBA16AD3EXbs7e3143SaNm2Kixcv6s9du3at/ioj+pdNJ7OhE4CO/s7wczGNzeWIiMj0GXwbKzo6Gvv27UOrVq3Qv39/vP7660hOTsa6desQHR1tjBqJAAAbuMM5ERHdB4PDzueff47i4mIAwHvvvYfi4mL8/PPPCAkJ4UwsMporBSU4nlEIqQQY0JZhh4iI6s6gsKPVapGZmYm2bdsCqL6ltWjRIqMURvRPG2/26nRt7gZ3R4XI1RARkTkxaMyOTCZDnz59cOPGDWPVQ3QLQRDw+82FBLk9BBERGcrgAcpt2rTBpUuXjFEL0W2dyynChbxiWMuleLSNl9jlEBGRmTE47HzwwQd44403sGnTJmRnZ0OtVtd6ENW3ml6dni3d4WRjJXI1RERkbgweoNy/f38AwODBg2vtSyQIAiQSCbRabf1VR42eTifox+twewgiIrofBoednTt3GqMOots6mn4DVwvL4KCQo1eoh9jlEBGRGTI47PTo0cMYdRDdVs3aOn1ae8LGSiZyNUREZI4MHrMDAHv37sUzzzyDLl264OrV6r2K/ve//2Hfvn31Whw1blVaHf44mQ2ACwkSEdH9Mzjs/Prrr+jbty9sbW1x9OhRaDQaAIBKpcJHH31U7wVS47X/YgEKSirgam+Nrs3dxC6HiIjM1H3Nxlq0aBF++OEHWFn9PTOma9euOHr0aL0WR43bhpuzsPqHN4WV7L46IYmIiAwPOykpKejevfstx5VKJQoLC+ujJiKUV2qx7XQOAOCx9ryFRURE98/gsOPl5YXU1NRbju/btw/BwcH1UhTRznN5KNZUwcfZFh39m4hdDhERmTGDw86ECRMwefJkJCYmQiKRICsrCytWrMAbb7yBiRMnGqNGaoT+uT2EVCq5x9VERER3ZvDU87feegs6nQ69e/dGaWkpunfvDoVCgTfeeAOvvPKKMWqkRkZdXokdKXkAOAuLiIgenMFhRyKR4O2338a0adOQmpqK4uJihIWFwcHBwRj1USO07VQOKqp0CPFwQKumjmKXQ0REZs7g21jLly9HaWkprK2tERYWhs6dOzPoUL2qWUhwcDvvWluSEBER3Q+Dw86UKVPg4eGBkSNHYvPmzdwLi+pVfpEG+1OvAQAGcxYWERHVA4PDTnZ2NlavXg2JRIKnnnoKTZs2xaRJk3DgwAFj1EeNzObkbOgEoJ2fMwJc7cUuh4iILIDBYUcul2PgwIFYsWIF8vLysGDBAqSlpaFnz55o1qyZQa81d+5cdOrUCY6OjvDw8MDjjz+OlJSUWteUl5dj0qRJcHV1hYODA4YOHYrc3Nxa16Snp2PAgAGws7ODh4cHpk2bhqqqKkObRibg9+PV249wYDIREdWXB1qW1s7ODn379kW/fv0QEhKCtLQ0g56/e/duTJo0CQcPHsT27dtRWVmJPn36oKSkRH/NlClTsHHjRqxduxa7d+9GVlYWhgwZoj+v1WoxYMAAVFRU4MCBA1i2bBmWLl2KWbNmPUjTSAQZ10txNL0QEgkwqG1TscshIiILIREEQTD0SaWlpVi/fj1WrFiB+Ph4+Pn5YcSIEYiLi0NoaOh9F5Ofnw8PDw/s3r0b3bt3h0qlgru7O1auXIknn3wSAHDu3Dm0atUKCQkJiI6OxpYtWzBw4EBkZWXB09MTALBo0SJMnz4d+fn5sLa2vuf7qtVqKJVKqFQqODk53Xf99GAW7kzFJ9tS0KWZK1ZOiBa7HCIiMnF1/f42uGdn+PDh8PDwwJQpUxAcHIxdu3YhNTUV77///gMFHaB6M1EAcHFxAQAkJSWhsrISsbGx+mtCQ0Ph7++PhIQEAEBCQgLCw8P1QQcA+vbtC7VajdOnT9/2fTQaDdRqda0HiW/jzVlY3B6CiIjqk8Hr7MhkMqxZswZ9+/aFTCarde7UqVNo06bNfRWi0+nw2muvoWvXrvrXyMnJgbW1NZydnWtd6+npiZycHP01/ww6Nedrzt3O3Llz8d57791XnWQcKTlFOJdTBCuZBI+25i0sIiKqPwb37KxYsQL9+/fXB52ioiL85z//QefOndGuXbv7LmTSpEk4deoUVq9efd+vUVczZsyASqXSPzIyMoz+nnR3G05UD0x+uKUHlHZWIldDRESW5L4HKO/ZswdjxoxB06ZN8emnn6JXr144ePDgfb3Wyy+/jE2bNmHnzp3w9fXVH/fy8kJFRcUtu6nn5ubCy8tLf82/Z2fV/Fxzzb8pFAo4OTnVepB4BEGotZAgERFRfTIo7OTk5GDevHkICQnBsGHD4OTkBI1Gg99++w3z5s1Dp06dDHpzQRDw8ssvY/369dixYweCgoJqnY+IiICVlRXi4+P1x1JSUpCeno6YmBgAQExMDJKTk5GXl6e/Zvv27XByckJYWJhB9ZA4jmUUIuN6GeysZYht5XnvJxARERmgzmFn0KBBaNmyJU6ePIkvvvgCWVlZ+Prrrx/ozSdNmoTly5dj5cqVcHR0RE5ODnJyclBWVgYAUCqVGD9+PKZOnYqdO3ciKSkJ48aNQ0xMDKKjq2fr9OnTB2FhYRg1ahROnDiBbdu24Z133sGkSZOgUCgeqD5qGBtu7nDeJ8wTttaye1xNRERkmDoPUN6yZQteffVVTJw4ESEhIfXy5t999x0A4OGHH651fMmSJRg7diwAYMGCBZBKpRg6dCg0Gg369u2Lb7/9Vn+tTCbDpk2bMHHiRMTExMDe3h5jxozBnDlz6qVGMq4qrQ6bTmYDAB5r7yNyNUREZInqHHb27duHxYsXIyIiAq1atcKoUaMwfPjwB3rzuizxY2Njg4ULF2LhwoV3vCYgIACbN29+oFpIHAmXCnCtWIMmdlboFuImdjlERGSB6nwbKzo6Gj/88AOys7PxwgsvYPXq1fD29oZOp8P27dtRVFRkzDrJQtXcwuof3hRWsgda0JuIiOi2DP52sbe3x7PPPot9+/YhOTkZr7/+OubNmwcPDw8MHjzYGDWShSqv1GLrqeq1kDgLi4iIjOWB/indsmVLzJ8/H5mZmVi1alV91USNxK6UfBRpqtBUaYNOgS5il0NERBaqXu4byGQyPP7449iwYUN9vBw1EjULCQ5q5w2pVCJyNUREZKk4SIJEUVReifiz1Wsj8RYWEREZE8MOieLP07nQVOkQ7G6P1t5cwZqIiIyHYYdE8c/tISQS3sIiIiLjYdihBldQrMG+1GsAeAuLiIiMj2GHGtzm5GxodQLCfZQIdncQuxwiIrJwDDvU4GpuYT3Wnr06RERkfAw71KCuFpbhcNoNSCTAwLYMO0REZHwMO9SgNt7s1YkKcoGX0kbkaoiIqDFg2KEG9fvxmllY3OGciIgaBsMONZgLuUU4m62GlUyCfm28xC6HiIgaCYYdajA1A5O7h7ijib21yNUQEVFjwbBDDUIQhL8XEuQsLCIiakAMO9QgTmSqcKWgFLZWMjwS5il2OURE1Igw7FCD2HBzYPIjYZ6ws5aLXA0RETUmDDtkdFqdgI0n/94Li4iIqCEx7JDRJV4qQH6RBkpbK3Rv4S52OURE1Mgw7JDR1ayt0z/cC9Zy/pEjIqKGxW8eMipNlRZbTmUDAAbxFhYREYmAYYeMandKPtTlVfB0UiAqyFXscoiIqBFi2CGjqllbZ2Bbb8ikEpGrISKixohhh4ymRFOFv87mAgAe40KCREQkEoYdMprtZ3JRXqlDoKsdwn2UYpdDRESNFMMOGc3vx68CAAa394FEwltYREQkDoYdMorrJRXYe+EaAC4kSERE4mLYIaPYnJyNKp2A1t5OaO7hIHY5RETUiDHskFHodzhnrw4REYmMYYfqXVZhGQ5dvg6ACwkSEZH4GHao3m26ueln50AXeDvbilwNERE1dgw7VO9q9sIazLV1iIjIBDDsUL1KzSvG6Sw15FIJ+oc3FbscIiIihh2qXzUDkx8KcYOLvbXI1RARETHsUD0SBAEbT/AWFhERmRaGHao3yVdVuHytBDZWUjwS5iV2OURERAAYdqieCIKA5QevAAB6t/KEg0IuckVERETV+I1ED0ynEzBn0xmsOZIJAHgq0k/kioiIiP7GsEMPpFKrw5u/nMT6Y9Wbfs4eFIYeLdxFroqIiOhvDDt038ortZi04ijiz+VBJpXgs2Ht8HgHH7HLIiIiqoVhh+6LqqwSE5YdwaG061DIpfg2riN6t/IUuywiIqJbMOyQwfKLNBj94yGczVbDUSHHf8dEIirYVeyyiIiIbothhwyScb0UoxYnIq2gFG4O1lj2bGe09laKXRYREdEdMexQnaXkFGH0j4nIVWvg42yL5c9FIcjNXuyyiIiI7ophh+rkaPoNjFtyGKqySrTwdMBPz0bBS2kjdllERET3xLBD97TnfD5e+F8Syiq1aO/njKXjOsHZjvteERGReWDYobv642Q2Xvv5GCq1Ah4KccOiZyJgz9WRiYjIjPBbi+5oZWI63v4tGYIADAhvis+fbgeFXCZ2WURERAZh2KFbCIKAb3ddxCfbUgAAI6P88f5jbSCTSkSujIiIyHAMO1SLIAj4aPNZ/LD3MgBgUs9meKNPS0gkDDpERGSeGHZIr0qrw1vrkvFLUvWGnu8MaIXnHgoWuSoiIqIHw7BDAKr3uXpl1TFsP5MLmVSCeUPCMYy7lxMRkQVg2CEUlVdiwk9HcPDSdVjLpfhmRAf0ae0ldllERET1gmGnkSso1mDsksNIvqqCg0KO/4yOQJdmbmKXRUREVG+kYr75nj17MGjQIHh7e0MikeC3336rdV4QBMyaNQtNmzaFra0tYmNjceHChVrXXL9+HXFxcXBycoKzszPGjx+P4uLiBmyF+bpaWIZhixKQfFUFF3trrJoQzaBDREQWR9SwU1JSgnbt2mHhwoW3PT9//nx89dVXWLRoERITE2Fvb4++ffuivLxcf01cXBxOnz6N7du3Y9OmTdizZw+ef/75hmqC2UrNK8KT3x3ApWsl8FbaYO2LMQj35YaeRERkeSSCIAhiFwEAEokE69evx+OPPw6gulfH29sbr7/+Ot544w0AgEqlgqenJ5YuXYrhw4fj7NmzCAsLw+HDhxEZGQkA2Lp1K/r374/MzEx4e3vf9r00Gg00Go3+Z7VaDT8/P6hUKjg5ORm3oSbgREYhxi45hBullWjmbo//jY+Ct7Ot2GUREREZRK1WQ6lU3vP7W9Senbu5fPkycnJyEBsbqz+mVCoRFRWFhIQEAEBCQgKcnZ31QQcAYmNjIZVKkZiYeMfXnjt3LpRKpf7h59d4Zh3tT72GkT8cxI3SSrT1VWLti10YdIiIyKKZbNjJyckBAHh6etY67unpqT+Xk5MDDw+PWuflcjlcXFz019zOjBkzoFKp9I+MjIx6rt40bT2VjXFLDqOkQosuzVyxckI0XOy5oScREVm2RjkbS6FQQKFQiF1Gg/r5cDpmrEuGTgD6tvbEl8M7wMaK+1wREZHlM9meHS+v6nVecnNzax3Pzc3Vn/Py8kJeXl6t81VVVbh+/br+GgK+330R03+tDjpPR/ph4ciODDpERNRomGzYCQoKgpeXF+Lj4/XH1Go1EhMTERMTAwCIiYlBYWEhkpKS9Nfs2LEDOp0OUVFRDV6zqREEAfO2nMPcLecAAC90D8a8oeGQy0z2105ERFTvRL2NVVxcjNTUVP3Ply9fxvHjx+Hi4gJ/f3+89tpr+OCDDxASEoKgoCDMnDkT3t7e+hlbrVq1wqOPPooJEyZg0aJFqKysxMsvv4zhw4ffcSZWY6HVCXh7fTJWH64ej/RWv1C82KOZyFURERE1PFHDzpEjR9CzZ0/9z1OnTgUAjBkzBkuXLsWbb76JkpISPP/88ygsLES3bt2wdetW2NjY6J+zYsUKvPzyy+jduzekUimGDh2Kr776qsHbYko0VVq8tvo4tpzKgVQCfPREOIZ39he7LCIiIlGYzDo7YqrrPH1z8cqqY9h4IgvWMim+HN4e/cKbil0SERFRvavr93ejnI1lyQ6kXsPGE1mQSSVYPDYSD4W4i10SERGRqDhS1YJUaXV4b+MZAMAzUf4MOkRERGDYsSgrD6UjJbcIznZWmPJIC7HLISIiMgkMOxbiRkkFPvvzPADg9UdawNmOKyMTEREBDDsWY8Ff56Eqq0SolyNGcOYVERGRHsOOBTiXo8byg1cAALMGhXHRQCIion/gt6KZEwQBczaegU4A+rXxQpdmbmKXREREZFIYdszcttO5OHCxANZyKf6vfyuxyyEiIjI5DDtmrLxSiw83V081f6F7MPxc7ESuiIiIyPQw7JixxfsuI+N6GbycbDDxYe57RUREdDsMO2YqR1WOhTurN1Gd0T8UdtZcDJuIiOh2GHbM1Mdbz6G0QouIgCYY3K5x7/BORER0Nww7Zijpyg2sP3YVEgnw7qDWkEgkYpdERERkshh2zIxOJ+C9jacBAMMifBHuqxS5IiIiItPGsGNmfjmaiZOZKjgo5JjWN1TscoiIiEwew44ZKSqvxPytKQCAV3s3h7ujQuSKiIiITB/Djhn5ZkcqrhVrEORmj7FdgsQuh4iIyCww7JiJy9dK8OP+ywCAmQNbwVrOXx0REVFd8BvTTHyw6QwqtQJ6tHBHz5YeYpdDRERkNhh2zMCulDzEn8uDXCrBzIFhnGpORERkAIYdE1ep1eH9TdX7X43pEojmHg4iV0RERGReGHZM3E8JV3AxvwSu9tZ4tXeI2OUQERGZHYYdE1ZQrMEXf50HALzRtyWUtlYiV0RERGR+GHZM2Kd/pqCovAqtvZ3wVKSf2OUQERGZJYYdE3XqqgqrD2cAAGYPag2ZlIOSiYiI7gfDjgkShOr9rwQBGNTOG52DXMQuiYiIyGwx7JigTSezcTjtBmyspJjRj/tfERERPQiGHRNTVqHF3M1nAQATezSHt7OtyBURERGZN4YdE7No90Vkqcrh42yL57sHi10OERGR2WPYMSGZN0qxaPdFAMD/9W8FW2uZyBURERGZP4YdEzJ3yzloqnSICnJB/3AvscshIiKyCAw7JuLgpQL8cTIbUgkwaxD3vyIiIqovDDsmQKsT8N7G6v2vhnf2R2tvpcgVERERWQ6GHROw+nA6zmar4WQjx+uPtBC7HCIiIovCsCMyVWklPt2WAgB4LbYFXB0UIldERERkWRh2RPZF/HncKK1Ecw8HjIoJELscIiIii8OwI6ILuUX4KeEKAGDWwDBYyfjrICIiqm/8dhWJIAiYs+kMtDoBsa080b2Fu9glERERWSSGHZHEn83D3gvXYC2T4p0BrcQuh4iIyGIx7IhAU6XF+39UTzV/tlsQAt3sRa6IiIjIcjHsiGDJ/jRcKSiFu6MCL/dqLnY5REREFo1hp4HlqcvxdfwFAMD0R0PhoJCLXBEREZFlY9hpYPO3paCkQot2fs4Y0sFH7HKIiIgsHsNOAzqeUYhfkjIBALMHhUEq5f5XRERExsaw00B0OgHvbjgNABjSwQcd/ZuIXBEREVHjwLDTQH47fhXHMwphZy3D9H6hYpdDRETUaDDsNIASTRXmbTkHAJjUszk8nWxEroiIiKjxYNhpAAt3piKvSAN/FzuM7xYkdjlERESNCsOOkaUXlOK/ey8DAN4e0Ao2VjKRKyIiImpcGHaM7IM/zqBCq0O35m7oE+YpdjlERESNDsOOEe27cA1/nsmFTCrBrEFhkEg41ZyIiKihMewYSZVWhzmbqqeaj4oOQAtPR5ErIiIiapy4V4GRaKp06BToguslFXgtNkTscoiIiBotiSAIgthFiE2tVkOpVEKlUsHJyaleX7uovBKONlb1+ppERERU9+9v3sYyMgYdIiIicVlM2Fm4cCECAwNhY2ODqKgoHDp0SOySiIiIyARYRNj5+eefMXXqVMyePRtHjx5Fu3bt0LdvX+Tl5YldGhEREYnMIsbsREVFoVOnTvjmm28AADqdDn5+fnjllVfw1ltv3XK9RqOBRqPR/6xWq+Hn52eUMTtERERkHI1mzE5FRQWSkpIQGxurPyaVShEbG4uEhITbPmfu3LlQKpX6h5+fX0OVS0RERA3M7MPOtWvXoNVq4elZe3ViT09P5OTk3PY5M2bMgEql0j8yMjIaolQiIiISQaNcZ0ehUEChUIhdBhERETUAs+/ZcXNzg0wmQ25ubq3jubm58PLyEqkqIiIiMhVmH3asra0RERGB+Ph4/TGdTof4+HjExMSIWBkRERGZAou4jTV16lSMGTMGkZGR6Ny5M7744guUlJRg3LhxYpdGREREIrOIsPP0008jPz8fs2bNQk5ODtq3b4+tW7feMmiZiIiIGh+LWGfnQalUKjg7OyMjI4Pr7BAREZmJmnXyCgsLoVQq73idRfTsPKiioiIA4Ho7REREZqioqOiuYYc9O6ge0JyVlQVHR0dIJJJ6e92axNlYeozYXsvX2NrM9lo2ttf8CYKAoqIieHt7Qyq985wr9uygesVlX19fo72+k5OTxfzBqgu21/I1tjazvZaN7TVvd+vRqWH2U8+JiIiI7oZhh4iIiCwaw44RKRQKzJ49u9FsTcH2Wr7G1ma217KxvY0HBygTERGRRWPPDhEREVk0hh0iIiKyaAw7REREZNEYdoiIiMiiMewY0cKFCxEYGAgbGxtERUXh0KFDYpd0T3PnzkWnTp3g6OgIDw8PPP7440hJSal1TXl5OSZNmgRXV1c4ODhg6NChyM3NrXVNeno6BgwYADs7O3h4eGDatGmoqqqqdc2uXbvQsWNHKBQKNG/eHEuXLjV28+5p3rx5kEgkeO211/THLK29V69exTPPPANXV1fY2toiPDwcR44c0Z8XBAGzZs1C06ZNYWtri9jYWFy4cKHWa1y/fh1xcXFwcnKCs7Mzxo8fj+Li4lrXnDx5Eg899BBsbGzg5+eH+fPnN0j7/kmr1WLmzJkICgqCra0tmjVrhvfffx//nJdhzu3ds2cPBg0aBG9vb0gkEvz222+1zjdk29auXYvQ0FDY2NggPDwcmzdvrvf2Andvc2VlJaZPn47w8HDY29vD29sbo0ePRlZWltm2+V6/43968cUXIZFI8MUXX9Q6bk7tNRqBjGL16tWCtbW18OOPPwqnT58WJkyYIDg7Owu5ublil3ZXffv2FZYsWSKcOnVKOH78uNC/f3/B399fKC4u1l/z4osvCn5+fkJ8fLxw5MgRITo6WujSpYv+fFVVldCmTRshNjZWOHbsmLB582bBzc1NmDFjhv6aS5cuCXZ2dsLUqVOFM2fOCF9//bUgk8mErVu3Nmh7/+nQoUNCYGCg0LZtW2Hy5Mn645bU3uvXrwsBAQHC2LFjhcTEROHSpUvCtm3bhNTUVP018+bNE5RKpfDbb78JJ06cEAYPHiwEBQUJZWVl+mseffRRoV27dsLBgweFvXv3Cs2bNxdGjBihP69SqQRPT08hLi5OOHXqlLBq1SrB1tZW+P777xu0vR9++KHg6uoqbNq0Sbh8+bKwdu1awcHBQfjyyy8tor2bN28W3n77bWHdunUCAGH9+vW1zjdU2/bv3y/IZDJh/vz5wpkzZ4R33nlHsLKyEpKTkxu0zYWFhUJsbKzw888/C+fOnRMSEhKEzp07CxEREbVew5zafK/fcY1169YJ7dq1E7y9vYUFCxaYbXuNhWHHSDp37ixMmjRJ/7NWqxW8vb2FuXPniliV4fLy8gQAwu7duwVBqP4wsbKyEtauXau/5uzZswIAISEhQRCE6r+cUqlUyMnJ0V/z3XffCU5OToJGoxEEQRDefPNNoXXr1rXe6+mnnxb69u1r7CbdVlFRkRASEiJs375d6NGjhz7sWFp7p0+fLnTr1u2O53U6neDl5SV88skn+mOFhYWCQqEQVq1aJQiCIJw5c0YAIBw+fFh/zZYtWwSJRCJcvXpVEARB+Pbbb4UmTZro21/z3i1btqzvJt3VgAEDhGeffbbWsSFDhghxcXGCIFhWe//9RdiQbXvqqaeEAQMG1KonKipKeOGFF+q1jf92ty//GocOHRIACFeuXBEEwbzbfKf2ZmZmCj4+PsKpU6eEgICAWmHHnNtbn3gbywgqKiqQlJSE2NhY/TGpVIrY2FgkJCSIWJnhVCoVAMDFxQUAkJSUhMrKylptCw0Nhb+/v75tCQkJCA8Ph6enp/6avn37Qq1W4/Tp0/pr/vkaNdeI9f9n0qRJGDBgwC01WVp7N2zYgMjISAwbNgweHh7o0KEDfvjhB/35y5cvIycnp1atSqUSUVFRtdrr7OyMyMhI/TWxsbGQSqVITEzUX9O9e3dYW1vrr+nbty9SUlJw48YNYzdTr0uXLoiPj8f58+cBACdOnMC+ffvQr18/AJbX3n9qyLaZyp/v21GpVJBIJHB2dgZgeW3W6XQYNWoUpk2bhtatW99y3tLae78Ydozg2rVr0Gq1tb78AMDT0xM5OTkiVWU4nU6H1157DV27dkWbNm0AADk5ObC2ttZ/cNT4Z9tycnJu2/aac3e7Rq1Wo6yszBjNuaPVq1fj6NGjmDt37i3nLK29ly5dwnfffYeQkBBs27YNEydOxKuvvoply5bVqvduf3ZzcnLg4eFR67xcLoeLi4tB/08awltvvYXhw4cjNDQUVlZW6NChA1577TXExcXVqsVS2vtPDdm2O10j9uddeXk5pk+fjhEjRug3vrS0Nn/88ceQy+V49dVXb3ve0tp7v7jrOd3RpEmTcOrUKezbt0/sUowmIyMDkydPxvbt22FjYyN2OUan0+kQGRmJjz76CADQoUMHnDp1CosWLcKYMWNErq7+rVmzBitWrMDKlSvRunVrHD9+HK+99hq8vb0tsr30t8rKSjz11FMQBAHfffed2OUYRVJSEr788kscPXoUEolE7HJMGnt2jMDNzQ0ymeyWGTu5ubnw8vISqSrDvPzyy9i0aRN27twJX19f/XEvLy9UVFSgsLCw1vX/bJuXl9dt215z7m7XODk5wdbWtr6bc0dJSUnIy8tDx44dIZfLIZfLsXv3bnz11VeQy+Xw9PS0qPY2bdoUYWFhtY61atUK6enp+jpravunf7c3Ly+v1vmqqipcv37doP8nDWHatGn63p3w8HCMGjUKU6ZM0ffiWVp7/6kh23ana8Rqe03QuXLlCrZv367v1QEsq8179+5FXl4e/P399Z9fV65cweuvv47AwEB9nZbS3gfBsGME1tbWiIiIQHx8vP6YTqdDfHw8YmJiRKzs3gRBwMsvv4z169djx44dCAoKqnU+IiICVlZWtdqWkpKC9PR0fdtiYmKQnJxc6y9YzQdOzRdtTExMrdeouaah///07t0bycnJOH78uP4RGRmJuLg4/X9bUnu7du16y1IC58+fR0BAAAAgKCgIXl5etWpVq9VITEys1d7CwkIkJSXpr9mxYwd0Oh2ioqL01+zZsweVlZX6a7Zv346WLVuiSZMmRmvfv5WWlkIqrf0xJ5PJoNPpAFhee/+pIdtmKn++gb+DzoULF/DXX3/B1dW11nlLavOoUaNw8uTJWp9f3t7emDZtGrZt26av01La+0DEHiFtqVavXi0oFAph6dKlwpkzZ4Tnn39ecHZ2rjVjxxRNnDhRUCqVwq5du4Ts7Gz9o7S0VH/Niy++KPj7+ws7duwQjhw5IsTExAgxMTH68zVTsfv06SMcP35c2Lp1q+Du7n7bqdjTpk0Tzp49KyxcuFD0qec1/jkbSxAsq72HDh0S5HK58OGHHwoXLlwQVqxYIdjZ2QnLly/XXzNv3jzB2dlZ+P3334WTJ08Kjz322G2nK3fo0EFITEwU9u3bJ4SEhNSaylpYWCh4enoKo0aNEk6dOiWsXr1asLOza/Cp52PGjBF8fHz0U8/XrVsnuLm5CW+++aZFtLeoqEg4duyYcOzYMQGA8PnnnwvHjh3TzzxqqLbt379fkMvlwqeffiqcPXtWmD17ttGmJd+tzRUVFcLgwYMFX19f4fjx47U+w/4508ic2nyv3/G//Xs2lrm111gYdozo66+/Fvz9/QVra2uhc+fOwsGDB8Uu6Z4A3PaxZMkS/TVlZWXCSy+9JDRp0kSws7MTnnjiCSE7O7vW66SlpQn9+vUTbG1tBTc3N+H1118XKisra12zc+dOoX379oK1tbUQHBxc6z3E9O+wY2nt3bhxo9CmTRtBoVAIoaGhwn/+859a53U6nTBz5kzB09NTUCgUQu/evYWUlJRa1xQUFAgjRowQHBwcBCcnJ2HcuHFCUVFRrWtOnDghdOvWTVAoFIKPj48wb948o7ft39RqtTB58mTB399fsLGxEYKDg4W333671hefObd3586dt/37OmbMmAZv25o1a4QWLVoI1tbWQuvWrYU//vijwdt8+fLlO36G7dy50yzbfK/f8b/dLuyYU3uNRSII/1hKlIiIiMjCcMwOERERWTSGHSIiIrJoDDtERERk0Rh2iIiIyKIx7BAREZFFY9ghIiIii8awQ0RERBaNYYeIiIgsGsMOERERWTSGHSIyG/n5+Zg4cSL8/f2hUCjg5eWFvn37Yv/+/QAAiUSC3377TdwiicjkyMUugIioroYOHYqKigosW7YMwcHByM3NRXx8PAoKCsQujYhMGPfGIiKzUFhYiCZNmmDXrl3o0aPHLecDAwNx5coV/c8BAQFIS0sDAPz+++947733cObMGXh7e2PMmDF4++23IZdX/3tPIpHg22+/xYYNG7Br1y40bdoU8+fPx5NPPtkgbSMi4+JtLCIyCw4ODnBwcMBvv/0GjUZzy/nDhw8DAJYsWYLs7Gz9z3v37sXo0aMxefJknDlzBt9//z2WLl2KDz/8sNbzZ86ciaFDh+LEiROIi4vD8OHDcfbsWeM3jIiMjj07RGQ2fv31V0yYMAFlZWXo2LEjevTogeHDh6Nt27YAqnto1q9fj8cff1z/nNjYWPTu3RszZszQH1u+fDnefPNNZGVl6Z/34osv4rvvvtNfEx0djY4dO+Lbb79tmMYRkdGwZ4eIzMbQoUORlZWFDRs24NFHH8WuXbvQsWNHLF269I7POXHiBObMmaPvGXJwcMCECROQnZ2N0tJS/XUxMTG1nhcTE8OeHSILwQHKRGRWbGxs8Mgjj+CRRx7BzJkz8dxzz2H27NkYO3bsba8vLi7Ge++9hyFDhtz2tYjI8rFnh4jMWlhYGEpKSgAAVlZW0Gq1tc537NgRKSkpaN68+S0PqfTvj8CDBw/Wet7BgwfRqlUr4zeAiIyOPTtEZBYKCgowbNgwPPvss2jbti0cHR1x5MgRzJ8/H4899hiA6hlZ8fHx6Nq1KxQKBZo0aYJZs2Zh4MCB8Pf3x5NPPgmpVIoTJ07g1KlT+OCDD/Svv3btWkRGRqJbt25YsWIFDh06hMWLF4vVXCKqRxygTERmQaPR4N1338Wff/6JixcvorKyEn5+fhg2bBj+7//+D7a2tti4cSOmTp2KtLQ0+Pj46Keeb9u2DXPmzMGxY8dgZWWF0NBQPPfcc5gwYQKA6gHKCxcuxG+//YY9e/agadOm+Pjjj/HUU0+J2GIiqi8MO0TU6N1uFhcRWQ6O2SEiIiKLxrBDREREFo0DlImo0ePdfCLLxp4dIiIismgMO0RERGTRGHaIiIjIojHsEBERkUVj2CEiIiKLxrBDREREFo1hh4iIiCwaww4RERFZtP8H6cH2kIfhd/MAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "#@test {\"skip\": true}\n", "\n", "steps = range(0, num_iterations + 1, eval_interval)\n", "plt.plot(steps, returns)\n", "plt.ylabel('Average Return')\n", "plt.xlabel('Step')\n", "plt.ylim(top=550)" ] }, { "cell_type": "markdown", "metadata": { "id": "M7-XpPP99Cy7" }, "source": [ "### Videos" ] }, { "cell_type": "markdown", "metadata": { "id": "9pGfGxSH32gn" }, "source": [ "It is helpful to visualize the performance of an agent by rendering the environment at each step. Before we do that, let us first create a function to embed videos in this colab." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:59:37.124663Z", "iopub.status.busy": "2024-03-09T12:59:37.124398Z", "iopub.status.idle": "2024-03-09T12:59:37.128794Z", "shell.execute_reply": "2024-03-09T12:59:37.128102Z" }, "id": "ULaGr8pvOKbl" }, "outputs": [], "source": [ "def embed_mp4(filename):\n", " \"\"\"Embeds an mp4 file in the notebook.\"\"\"\n", " video = open(filename,'rb').read()\n", " b64 = base64.b64encode(video)\n", " tag = '''\n", " '''.format(b64.decode())\n", "\n", " return IPython.display.HTML(tag)" ] }, { "cell_type": "markdown", "metadata": { "id": "9c_PH-pX4Pr5" }, "source": [ "The following code visualizes the agent's policy for a few episodes:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2024-03-09T12:59:37.132462Z", "iopub.status.busy": "2024-03-09T12:59:37.131935Z", "iopub.status.idle": "2024-03-09T12:59:44.221913Z", "shell.execute_reply": "2024-03-09T12:59:44.221017Z" }, "id": "owOVWB158NlF" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (400, 600) to (400, 608) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[swscaler @ 0x55f48c41d880] Warning: data is not aligned! This can lead to a speed loss\n" ] }, { "data": { "text/html": [ "\n", " " ], "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "num_episodes = 3\n", "video_filename = 'imageio.mp4'\n", "with imageio.get_writer(video_filename, fps=60) as video:\n", " for _ in range(num_episodes):\n", " time_step = eval_env.reset()\n", " video.append_data(eval_py_env.render())\n", " while not time_step.is_last():\n", " action_step = agent.policy.action(time_step)\n", " time_step = eval_env.step(action_step.action)\n", " video.append_data(eval_py_env.render())\n", "\n", "embed_mp4(video_filename)" ] }, { "cell_type": "markdown", "metadata": { "id": "exziB27hY8ia" }, "source": [ "C51 tends to do slightly better than DQN on CartPole-v1, but the difference between the two agents becomes more and more significant in increasingly complex environments. For example, on the full Atari 2600 benchmark, C51 demonstrates a mean score improvement of 126% over DQN after normalizing with respect to a random agent. Additional improvements can be gained by including n-step updates.\n", "\n", "For a deeper dive into the C51 algorithm, see [A Distributional Perspective on Reinforcement Learning (2017)](https://arxiv.org/pdf/1707.06887.pdf)." ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "DQN C51/Rainbow Tutorial.ipynb", "private_outputs": true, "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" }, "pycharm": { "stem_cell": { "cell_type": "raw", "metadata": { "collapsed": false }, "source": [] } } }, "nbformat": 4, "nbformat_minor": 0 }