{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "klGNgWREsvQv" }, "source": [ "##### Copyright 2023 The TF-Agents Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2023-12-22T14:04:42.484544Z", "iopub.status.busy": "2023-12-22T14:04:42.484329Z", "iopub.status.idle": "2023-12-22T14:04:42.487821Z", "shell.execute_reply": "2023-12-22T14:04:42.487255Z" }, "id": "nQnmcm0oI1Q-" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "HNtBC6Bbb1YU" }, "source": [ "# REINFORCE agent\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " View on TensorFlow.org\n", " \n", " \n", " \n", " Run in Google Colab\n", " \n", " \n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "ZOUOQOrFs3zn" }, "source": [ "## Introduction" ] }, { "cell_type": "markdown", "metadata": { "id": "cKOCZlhUgXVK" }, "source": [ "This example shows how to train a [REINFORCE](https://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf) agent on the Cartpole environment using the TF-Agents library, similar to the [DQN tutorial](1_dqn_tutorial.ipynb).\n", "\n", "![Cartpole environment](images/cartpole.png)\n", "\n", "We will walk you through all the components in a Reinforcement Learning (RL) pipeline for training, evaluation and data collection.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "1u9QVVsShC9X" }, "source": [ "## Setup" ] }, { "cell_type": "markdown", "metadata": { "id": "I5PNmEzIb9t4" }, "source": [ "If you haven't installed the following dependencies, run:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:04:42.491556Z", "iopub.status.busy": "2023-12-22T14:04:42.491001Z", "iopub.status.idle": "2023-12-22T14:05:02.486448Z", "shell.execute_reply": "2023-12-22T14:05:02.485617Z" }, "id": "KEHR2Ui-lo8O" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]\r", " \r", "Hit:1 http://us-central1.gce.archive.ubuntu.com/ubuntu focal InRelease\r\n", "\r", "0% [Connecting to security.ubuntu.com (185.125.190.36)] [Connecting to apt.llvm\r", " \r", "Hit:2 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates InRelease\r\n", "\r", " \r", "Hit:3 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-backports InRelease\r\n", "\r", "0% [Connecting to security.ubuntu.com (185.125.190.36)] [Waiting for headers] [\r", "0% [Connecting to security.ubuntu.com (185.125.190.36)] [Connected to developer" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "Get:5 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64 InRelease [1484 B]\r\n", "\r", "0% [Connected to apt.llvm.org (199.232.198.49)] [Connecting to security.ubuntu.\r", "0% [Connected to apt.llvm.org (199.232.198.49)] [Connecting to security.ubuntu.\r", " \r", "Hit:6 https://download.docker.com/linux/ubuntu focal InRelease\r\n", "\r", "0% [Connected to apt.llvm.org (199.232.198.49)] [Connecting to security.ubuntu.\r", " \r", "Hit:7 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64 InRelease\r\n", "\r", "0% [Connected to apt.llvm.org (199.232.198.49)] [Connecting to security.ubuntu.\r", " \r", "Hit:8 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 InRelease\r\n", "\r", "0% [Connected to apt.llvm.org (199.232.198.49)] [Connecting to security.ubuntu." ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "Hit:4 https://apt.llvm.org/focal llvm-toolchain-focal-17 InRelease\r\n", "\r", "0% [Connecting to security.ubuntu.com (185.125.190.36)] [Waiting for headers] [\r", " \r", "Hit:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease\r\n", "\r", "0% [Connecting to security.ubuntu.com (185.125.190.36)] [Connecting to ppa.laun" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "0% [Waiting for headers] [Waiting for headers]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "Hit:10 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal InRelease\r\n", "\r", " \r", "0% [Waiting for headers]\r", " \r", "Hit:11 http://security.ubuntu.com/ubuntu focal-security InRelease\r\n", "\r", " \r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Waiting for headers]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "Hit:12 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu focal InRelease\r\n", "\r", " \r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Waiting for headers]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " \r", "Hit:13 http://ppa.launchpad.net/openjdk-r/ppa/ubuntu focal InRelease\r\n", "\r", " \r", "0% [Working]\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "0% [Working]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "100% [Working]\r", " \r", "Fetched 1484 B in 1s (1075 B/s)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 0%\r", "\r", "Reading package lists... 0%\r", "\r", "Reading package lists... 0%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 2%\r", "\r", "Reading package lists... 2%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r", "\r", "Reading package lists... 4%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 28%\r", "\r", "Reading package lists... 28%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 41%\r", "\r", "Reading package lists... 41%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 41%\r", "\r", "Reading package lists... 41%\r", "\r", "Reading package lists... 41%\r", "\r", "Reading package lists... 41%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 49%\r", "\r", "Reading package lists... 49%\r", "\r", "Reading package lists... 55%\r", "\r", "Reading package lists... 55%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 57%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 62%\r", "\r", "Reading package lists... 62%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 65%\r", "\r", "Reading package lists... 65%\r", "\r", "Reading package lists... 68%\r", "\r", "Reading package lists... 68%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 69%\r", "\r", "Reading package lists... 70%\r", "\r", "Reading package lists... 70%\r", "\r", "Reading package lists... 70%\r", "\r", "Reading package lists... 70%\r", "\r", "Reading package lists... 70%\r", "\r", "Reading package lists... 70%\r", "\r", "Reading package lists... 70%\r", "\r", "Reading package lists... 70%\r", "\r", "Reading package lists... 77%\r", "\r", "Reading package lists... 77%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 82%\r", "\r", "Reading package lists... 82%\r", "\r", "Reading package lists... 89%\r", "\r", "Reading package lists... 89%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 92%\r", "\r", "Reading package lists... 92%\r", "\r", "Reading package lists... 94%\r", "\r", "Reading package lists... 94%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r", "\r", "Reading package lists... 95%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 98%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 99%\r", "\r", "Reading package lists... 99%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... Done\r", "\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Reading package lists... 0%\r", "\r", "Reading package lists... 100%\r", "\r", "Reading package lists... Done\r", "\r\n", "\r", "Building dependency tree... 0%\r", "\r", "Building dependency tree... 0%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Building dependency tree... 50%\r", "\r", "Building dependency tree... 50%\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "Building dependency tree \r", "\r\n", "\r", "Reading state information... 0%\r", "\r", "Reading state information... 0%\r", "\r", "Reading state information... Done\r", "\r\n", "freeglut3-dev is already the newest version (2.8.1-3).\r\n", "ffmpeg is already the newest version (7:4.2.7-0ubuntu0.1).\r\n", "xvfb is already the newest version (2:1.20.13-1ubuntu1~20.04.12).\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "The following packages were automatically installed and are no longer required:\r\n", " libatasmart4 libblockdev-fs2 libblockdev-loop2 libblockdev-part-err2\r\n", " libblockdev-part2 libblockdev-swap2 libblockdev-utils2 libblockdev2\r\n", " libparted-fs-resize0 libxmlb2\r\n", "Use 'sudo apt autoremove' to remove them.\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0 upgraded, 0 newly installed, 0 to remove and 115 not upgraded.\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting imageio==2.4.0\r\n", " Using cached imageio-2.4.0-py3-none-any.whl\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: numpy in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from imageio==2.4.0) (1.26.2)\r\n", "Requirement already satisfied: pillow in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from imageio==2.4.0) (10.1.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: imageio\r\n", " Attempting uninstall: imageio\r\n", " Found existing installation: imageio 2.33.1\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Uninstalling imageio-2.33.1:\r\n", " Successfully uninstalled imageio-2.33.1\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n", "scikit-image 0.22.0 requires imageio>=2.27, but you have imageio 2.4.0 which is incompatible.\u001b[0m\u001b[31m\r\n", "\u001b[0mSuccessfully installed imageio-2.4.0\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting pyvirtualdisplay\r\n", " Using cached PyVirtualDisplay-3.0-py3-none-any.whl (15 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: pyvirtualdisplay\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed pyvirtualdisplay-3.0\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting tf-agents[reverb]\r\n", " Using cached tf_agents-0.19.0-py3-none-any.whl.metadata (12 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: absl-py>=0.6.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents[reverb]) (1.4.0)\r\n", "Collecting cloudpickle>=1.3 (from tf-agents[reverb])\r\n", " Using cached cloudpickle-3.0.0-py3-none-any.whl.metadata (7.0 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting gin-config>=0.4.0 (from tf-agents[reverb])\r\n", " Using cached gin_config-0.5.0-py3-none-any.whl (61 kB)\r\n", "Collecting gym<=0.23.0,>=0.17.0 (from tf-agents[reverb])\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Using cached gym-0.23.0-py3-none-any.whl\r\n", "Requirement already satisfied: numpy>=1.19.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents[reverb]) (1.26.2)\r\n", "Requirement already satisfied: pillow in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents[reverb]) (10.1.0)\r\n", "Requirement already satisfied: six>=1.10.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents[reverb]) (1.16.0)\r\n", "Requirement already satisfied: protobuf>=3.11.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents[reverb]) (3.20.3)\r\n", "Requirement already satisfied: wrapt>=1.11.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents[reverb]) (1.14.1)\r\n", "Collecting typing-extensions==4.5.0 (from tf-agents[reverb])\r\n", " Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting pygame==2.1.3 (from tf-agents[reverb])\r\n", " Using cached pygame-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.7 MB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting tensorflow-probability~=0.23.0 (from tf-agents[reverb])\r\n", " Using cached tensorflow_probability-0.23.0-py2.py3-none-any.whl.metadata (13 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting rlds (from tf-agents[reverb])\r\n", " Using cached rlds-0.1.8-py3-none-manylinux2010_x86_64.whl (48 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting dm-reverb~=0.14.0 (from tf-agents[reverb])\r\n", " Using cached dm_reverb-0.14.0-cp39-cp39-manylinux2014_x86_64.whl.metadata (17 kB)\r\n", "Requirement already satisfied: tensorflow~=2.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tf-agents[reverb]) (2.15.0.post1)\r\n", "Requirement already satisfied: dm-tree in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from dm-reverb~=0.14.0->tf-agents[reverb]) (0.1.8)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting portpicker (from dm-reverb~=0.14.0->tf-agents[reverb])\r\n", " Using cached portpicker-1.6.0-py3-none-any.whl.metadata (1.5 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting gym-notices>=0.0.4 (from gym<=0.23.0,>=0.17.0->tf-agents[reverb])\r\n", " Using cached gym_notices-0.0.8-py3-none-any.whl (3.0 kB)\r\n", "Requirement already satisfied: importlib-metadata>=4.10.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from gym<=0.23.0,>=0.17.0->tf-agents[reverb]) (7.0.0)\r\n", "Requirement already satisfied: astunparse>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (1.6.3)\r\n", "Requirement already satisfied: flatbuffers>=23.5.26 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (23.5.26)\r\n", "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (0.5.4)\r\n", "Requirement already satisfied: google-pasta>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (0.2.0)\r\n", "Requirement already satisfied: h5py>=2.9.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (3.10.0)\r\n", "Requirement already satisfied: libclang>=13.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (16.0.6)\r\n", "Requirement already satisfied: ml-dtypes~=0.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (0.2.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: opt-einsum>=2.3.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (3.3.0)\r\n", "Requirement already satisfied: packaging in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (23.2)\r\n", "Requirement already satisfied: setuptools in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (69.0.2)\r\n", "Requirement already satisfied: termcolor>=1.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (2.4.0)\r\n", "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (0.35.0)\r\n", "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (1.60.0)\r\n", "Requirement already satisfied: tensorboard<2.16,>=2.15 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (2.15.1)\r\n", "Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (2.15.0)\r\n", "Requirement already satisfied: keras<2.16,>=2.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.15.0->tf-agents[reverb]) (2.15.0)\r\n", "Requirement already satisfied: decorator in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-probability~=0.23.0->tf-agents[reverb]) (5.1.1)\r\n", "Requirement already satisfied: wheel<1.0,>=0.23.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from astunparse>=1.6.0->tensorflow~=2.15.0->tf-agents[reverb]) (0.41.3)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: zipp>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from importlib-metadata>=4.10.0->gym<=0.23.0,>=0.17.0->tf-agents[reverb]) (3.17.0)\r\n", "Requirement already satisfied: google-auth<3,>=1.6.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (2.25.2)\r\n", "Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (1.2.0)\r\n", "Requirement already satisfied: markdown>=2.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (3.5.1)\r\n", "Requirement already satisfied: requests<3,>=2.21.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (2.31.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (0.7.2)\r\n", "Requirement already satisfied: werkzeug>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (3.0.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: psutil in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from portpicker->dm-reverb~=0.14.0->tf-agents[reverb]) (5.9.7)\r\n", "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (5.3.2)\r\n", "Requirement already satisfied: pyasn1-modules>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (0.3.0)\r\n", "Requirement already satisfied: rsa<5,>=3.1.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (4.9)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: requests-oauthlib>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (1.3.1)\r\n", "Requirement already satisfied: charset-normalizer<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (3.3.2)\r\n", "Requirement already satisfied: idna<4,>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (3.6)\r\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (2.1.0)\r\n", "Requirement already satisfied: certifi>=2017.4.17 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (2023.11.17)\r\n", "Requirement already satisfied: MarkupSafe>=2.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (2.1.3)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (0.5.1)\r\n", "Requirement already satisfied: oauthlib>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow~=2.15.0->tf-agents[reverb]) (3.2.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Using cached cloudpickle-3.0.0-py3-none-any.whl (20 kB)\r\n", "Using cached dm_reverb-0.14.0-cp39-cp39-manylinux2014_x86_64.whl (6.4 MB)\r\n", "Using cached tensorflow_probability-0.23.0-py2.py3-none-any.whl (6.9 MB)\r\n", "Using cached tf_agents-0.19.0-py3-none-any.whl (1.4 MB)\r\n", "Using cached portpicker-1.6.0-py3-none-any.whl (16 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: gym-notices, gin-config, typing-extensions, rlds, pygame, portpicker, cloudpickle, tensorflow-probability, gym, dm-reverb, tf-agents\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Attempting uninstall: typing-extensions\r\n", " Found existing installation: typing_extensions 4.9.0\r\n", " Uninstalling typing_extensions-4.9.0:\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Successfully uninstalled typing_extensions-4.9.0\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed cloudpickle-3.0.0 dm-reverb-0.14.0 gin-config-0.5.0 gym-0.23.0 gym-notices-0.0.8 portpicker-1.6.0 pygame-2.1.3 rlds-0.1.8 tensorflow-probability-0.23.0 tf-agents-0.19.0 typing-extensions-4.5.0\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pyglet in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (2.0.10)\r\n", "Requirement already satisfied: xvfbwrapper in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (0.2.9)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting tf-keras\r\n", " Using cached tf_keras-2.15.0-py3-none-any.whl.metadata (1.6 kB)\r\n", "Using cached tf_keras-2.15.0-py3-none-any.whl (1.7 MB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: tf-keras\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed tf-keras-2.15.0\r\n" ] } ], "source": [ "!sudo apt-get update\n", "!sudo apt-get install -y xvfb ffmpeg freeglut3-dev\n", "!pip install 'imageio==2.4.0'\n", "!pip install pyvirtualdisplay\n", "!pip install tf-agents[reverb]\n", "!pip install pyglet xvfbwrapper\n", "!pip install tf-keras" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:02.490785Z", "iopub.status.busy": "2023-12-22T14:05:02.490492Z", "iopub.status.idle": "2023-12-22T14:05:02.494190Z", "shell.execute_reply": "2023-12-22T14:05:02.493660Z" }, "id": "WPuD0bMEY9Iz" }, "outputs": [], "source": [ "import os\n", "# Keep using keras-2 (tf-keras) rather than keras-3 (keras).\n", "os.environ['TF_USE_LEGACY_KERAS'] = '1'" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:02.497273Z", "iopub.status.busy": "2023-12-22T14:05:02.497032Z", "iopub.status.idle": "2023-12-22T14:05:05.889818Z", "shell.execute_reply": "2023-12-22T14:05:05.888543Z" }, "id": "sMitx5qSgJk1" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-12-22 14:05:03.363396: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", "2023-12-22 14:05:03.363443: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", "2023-12-22 14:05:03.365008: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n" ] } ], "source": [ "from __future__ import absolute_import\n", "from __future__ import division\n", "from __future__ import print_function\n", "\n", "import base64\n", "import imageio\n", "import IPython\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import PIL.Image\n", "import pyvirtualdisplay\n", "import reverb\n", "\n", "import tensorflow as tf\n", "\n", "from tf_agents.agents.reinforce import reinforce_agent\n", "from tf_agents.drivers import py_driver\n", "from tf_agents.environments import suite_gym\n", "from tf_agents.environments import tf_py_environment\n", "from tf_agents.networks import actor_distribution_network\n", "from tf_agents.policies import py_tf_eager_policy\n", "from tf_agents.replay_buffers import reverb_replay_buffer\n", "from tf_agents.replay_buffers import reverb_utils\n", "from tf_agents.specs import tensor_spec\n", "from tf_agents.trajectories import trajectory\n", "from tf_agents.utils import common\n", "\n", "# Set up a virtual display for rendering OpenAI gym environments.\n", "display = pyvirtualdisplay.Display(visible=0, size=(1400, 900)).start()" ] }, { "cell_type": "markdown", "metadata": { "id": "LmC0NDhdLIKY" }, "source": [ "## Hyperparameters" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:05.895225Z", "iopub.status.busy": "2023-12-22T14:05:05.894401Z", "iopub.status.idle": "2023-12-22T14:05:05.899775Z", "shell.execute_reply": "2023-12-22T14:05:05.899179Z" }, "id": "HC1kNrOsLSIZ" }, "outputs": [], "source": [ "env_name = \"CartPole-v0\" # @param {type:\"string\"}\n", "num_iterations = 250 # @param {type:\"integer\"}\n", "collect_episodes_per_iteration = 2 # @param {type:\"integer\"}\n", "replay_buffer_capacity = 2000 # @param {type:\"integer\"}\n", "\n", "fc_layer_params = (100,)\n", "\n", "learning_rate = 1e-3 # @param {type:\"number\"}\n", "log_interval = 25 # @param {type:\"integer\"}\n", "num_eval_episodes = 10 # @param {type:\"integer\"}\n", "eval_interval = 50 # @param {type:\"integer\"}" ] }, { "cell_type": "markdown", "metadata": { "id": "VMsJC3DEgI0x" }, "source": [ "## Environment\n", "\n", "Environments in RL represent the task or problem that we are trying to solve. Standard environments can be easily created in TF-Agents using `suites`. We have different `suites` for loading environments from sources such as the OpenAI Gym, Atari, DM Control, etc., given a string environment name.\n", "\n", "Now let us load the CartPole environment from the OpenAI Gym suite." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:05.902973Z", "iopub.status.busy": "2023-12-22T14:05:05.902734Z", "iopub.status.idle": "2023-12-22T14:05:05.933129Z", "shell.execute_reply": "2023-12-22T14:05:05.932580Z" }, "id": "pYEz-S9gEv2-" }, "outputs": [], "source": [ "env = suite_gym.load(env_name)" ] }, { "cell_type": "markdown", "metadata": { "id": "IIHYVBkuvPNw" }, "source": [ "We can render this environment to see how it looks. A free-swinging pole is attached to a cart. The goal is to move the cart right or left in order to keep the pole pointing up." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:05.936382Z", "iopub.status.busy": "2023-12-22T14:05:05.936138Z", "iopub.status.idle": "2023-12-22T14:05:06.079783Z", "shell.execute_reply": "2023-12-22T14:05:06.079193Z" }, "id": "RlO7WIQHu_7D" }, "outputs": [ { "data": { "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAGQAlgDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD3+iiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKyNc1z+xfI/0bzvN3fx7cYx7H1rNtPGX2q8gt/sG3zZFTd52cZOM/drKVenGXK3qaRozkuZLQ6miiitTMKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5Dxz/y4f9tP/Za5rTDjVrM/9N0/9CFdL45/5cP+2n/stczp5xqdqf8Apsn8xXlYj+O/kelR/gr5nqtFFFeqeaFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAch45/5cP8Atp/7LXL2Jxf2x/6ar/MV1fjiPMFnJn7rOuPqB/hXIQyeVNHJjOxg2PXBrycTpWb9D0qGtJHrdFFFeseaFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAcv42H/Evtj/01/pXE13HjYf8Sy3P/Tb+hrh68rF/xWelhv4Z69RRRXqnmhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHNeNR/xKYD/ANNx/wCgtXDV3fjX/kDw/wDXwP8A0Fq4SvKxf8U9LDfwz16igciivVPNCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOc8af8gaH/r4X/0Fq4Su78af8gaH/r4X/wBBauErysX/ABD0sL/DPXEOY1PsKdTITmCM/wCyP5U+vVR5oUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBznjT/kDQ/8AXwv/AKC1cJXoHi9A2hEkcrKpHseR/WvP68vF/wAQ9HC/wz1m2ObWE/7A/lUtVdMdpNKs3Y5ZoEJPvtFWq9OOqR573CiiimIKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACivH/G3x0/4Q7xffaB/wjn2z7L5f7/7d5e7dGr/d8s4xux17Vz//AA01/wBSj/5Uv/tVAHsniwf8SCX/AH1/nXntY1h8cP8AhNLxNC/4R37H5+W877b5m3aN33fLGc49a2N6ltoYZ9M15uLTdTTsehhWuT5nqek/8gax/wCveP8A9BFXK8EuP2h/7EuZdJ/4RbzvsLm283+0Nu/Ydu7HlnGcZxk1F/w01/1KP/lS/wDtVehD4UcMviZ9AUV5f8OPjB/wsDxDcaT/AGF9g8m0a5837X5ucOi7cbF/v5zntXqFUSFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHQUVz/wBj8Yf9B3Q//BNN/wDJVH2Pxh/0HdD/APBNN/8AJVAHzB8bf+Sva7/27/8ApPHXn9dx8X0vI/ilrK388E90PI3yQQmJG/cR4wpZiOMfxH146Vw9AHTeAGZfGliVznbJ0/65tXsoJByODXz9p2o3ek38V9Yy+VcxZ2PtDYyCDwQR0JraHj7xMG3DUhn/AK94v/ia6aGIVJNNCavbUzfERJ8TasW6/bJs/wDfZrNqW5uZby7mup33zTO0kjYAyxOScDjqairmGewfs4/8lD1D/sFSf+jYq+n6+VPgFHqUvjq+XS7u0tp/7MkLPdWzTqV82LgKsiEHOOc9jxzx9F/Y/GH/AEHdD/8ABNN/8lUAdBRXP/Y/GH/Qd0P/AME03/yVR9j8Yf8AQd0P/wAE03/yVQB0FFc/9j8Yf9B3Q/8AwTTf/JVH2Pxh/wBB3Q//AATTf/JVAHQUVz/2Pxh/0HdD/wDBNN/8lUfY/GH/AEHdD/8ABNN/8lUAdBRXP/Y/GH/Qd0P/AME03/yVR9j8Yf8AQd0P/wAE03/yVQB0FFc/9j8Yf9B3Q/8AwTTf/JVH2Pxh/wBB3Q//AATTf/JVAHQUVz/2Pxh/0HdD/wDBNN/8lUfY/GH/AEHdD/8ABNN/8lUAdBRXP/Y/GH/Qd0P/AME03/yVR9j8Yf8AQd0P/wAE03/yVQB0FFc/9j8Yf9B3Q/8AwTTf/JVH2Pxh/wBB3Q//AATTf/JVAHQUVz/2Pxh/0HdD/wDBNN/8lUfY/GH/AEHdD/8ABNN/8lUAdBRXP/Y/GH/Qd0P/AME03/yVR9j8Yf8AQd0P/wAE03/yVQB0FFc/9j8Yf9B3Q/8AwTTf/JVH2Pxh/wBB3Q//AATTf/JVAHQUVz/2Pxh/0HdD/wDBNN/8lUfY/GH/AEHdD/8ABNN/8lUAdBRXP/Y/GH/Qd0P/AME03/yVR9j8Yf8AQd0P/wAE03/yVQB0FFc/9j8Yf9B3Q/8AwTTf/JVH2Pxh/wBB3Q//AATTf/JVAHQUVz/2Pxh/0HdD/wDBNN/8lUfY/GH/AEHdD/8ABNN/8lUAdBRXP/Y/GH/Qd0P/AME03/yVR9j8Yf8AQd0P/wAE03/yVQB0FFc/9j8Yf9B3Q/8AwTTf/JVFAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHyB8bf+Sva7/wBu/wD6Tx15/X1v4p+Cnhvxd4ju9cv73VY7q62b0gljCDaioMAxk9FHesf/AIZx8H/9BLXP+/8AD/8AGqAPmCivp/8A4Zx8H/8AQS1z/v8Aw/8Axqj/AIZx8H/9BLXP+/8AD/8AGqAPmCivp/8A4Zx8H/8AQS1z/v8Aw/8Axqj/AIZx8H/9BLXP+/8AD/8AGqAOA/Zx/wCSh6h/2CpP/RsVfT9cH4K+E2g+A9Zm1TS7vUpp5bdrdlupEZQpZWyNqKc5Qd/Wu8oAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA/9k=", "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAUu0lEQVR4Ae3dvY5kVxUFYFfPCAIIEYkDckSITcILkCCewjwTfgpEwguQmZAXQAgCRAgSIHcVY3tm6HZX1dTPvfvuc9ZnITFTXXXP2d86raWqvjOzOxwOH/mPAAECBAikCjykDm5uAgQIECDwlYAidA4IECBAIFpAEUbHb3gCBAgQUITOAAECBAhECyjC6PgNT4AAAQKK0BkgQIAAgWgBRRgdv+EJECBAQBE6AwQIECAQLaAIo+M3PAECBAgoQmeAAAECBKIFFGF0/IYnQIAAAUXoDBAgQIBAtIAijI7f8AQIECCgCJ0BAgQIEIgWUITR8RueAAECBBShM0CAAAEC0QKKMDp+wxMgQICAInQGCBAgQCBaQBFGx294AgQIEFCEzgABAgQIRAsowuj4DU+AAAECitAZIECAAIFoAUUYHb/hCRAgQEAROgMECBAgEC2gCKPjNzwBAgQIKEJngAABAgSiBRRhdPyGJ0CAAAFF6AwQIECAQLSAIoyO3/AECBAgoAidAQIECBCIFlCE0fEbngABAgQUoTNAgAABAtECijA6fsMTIECAgCJ0BggQIEAgWkARRsdveAIECBBQhM4AAQIECEQLKMLo+A1PgAABAorQGSBAgACBaAFFGB2/4QkQIEBAEToDBAgQIBAtoAij4zc8AQIECChCZ4AAAQIEogUUYXT8hidAgAABRegMECBAgEC0gCKMjt/wBAgQIKAInQECBAgQiBZQhNHxG54AAQIEFKEzQIAAAQLRAoowOn7DEyBAgIAidAYIECBAIFpAEUbHb3gCBAgQUITOAAECBAhECyjC6PgNT4AAAQKK0BkgQIAAgWgBRRgdv+EJECBAQBE6AwQIECAQLaAIo+M3PAECBAgoQmeAAAECBKIFFGF0/IYnQIAAAUXoDBAgQIBAtIAijI7f8AQIECCgCJ0BAgQIEIgWUITR8RueAAECBBShM0CAAAEC0QKKMDp+wxMgQICAInQGCBAgQCBaQBFGx294AgQIEFCEzgABAgQIRAsowuj4DU+AAAECitAZIECAAIFoAUUYHb/hCRAgQEAROgMECBAgEC2gCKPjNzwBAgQIKEJngAABAgSiBRRhdPyGJ0CAAAFF6AwQIECAQLSAIoyO3/AECBAgoAidAQIECBCIFlCE0fEbngABAgQUoTNAgAABAtECijA6fsMTIECAgCJ0BggQIEAgWkARRsdveAIECBBQhM4AAQIECEQLKMLo+A1PgAABAorQGSBAgACBaAFFGB2/4QkQIEBAEToDBAgQIBAtoAij4zc8AQIECChCZ4AAAQIEogUUYXT8hidAgAABRegMECBAgEC0gCKMjt/wBAgQIKAInQECBAgQiBZQhNHxG54AAQIEFKEzQIAAAQLRAoowOn7DEyBAgIAidAYIECBAIFpAEUbHb3gCBAgQUITOAAECBAhECyjC6PgNT4AAAQKK0BkgQIAAgWgBRRgdv+EJECBAQBE6AwQIECAQLaAIo+M3PAECBAgoQmeAAAECBKIFFGF0/IYnQIAAAUXoDBAgQIBAtIAijI7f8AQIECCgCJ0BAgQIEIgWUITR8RueAAECBBShM0CAAAEC0QKKMDp+wxMgQICAInQGCBAgQCBaQBFGx294AgQIEFCEzgABAgQIRAsowuj4DU+AAAECitAZIECAAIFoAUUYHb/hCRAgQEAROgMECBAgEC2gCKPjNzwBAgQIKEJngAABAgSiBRRhdPyGJ0CAAAFF6AwQIECAQLSAIoyO3/AECBAgoAidAQIECBCIFlCE0fEbngABAgQUoTNAgAABAtECijA6fsMTIECAgCJ0BggQIEAgWkARRsdveAIECBBQhM4AAQIECEQLKMLo+A1PgAABAorQGSBAgACBaAFFGB2/4QkQIEBAEToDBAgQIBAtoAij4zc8AQIECChCZ4AAAQIEogUUYXT8hidAgAABRegMECBAgEC0gCKMjt/wBAgQIKAInQECBAgQiBZQhNHxG54AAQIEFKEzQIAAAQLRAoowOn7DEyBAgIAidAYIECBAIFpAEUbHb3gCBAgQUITOAAECBAhECyjC6PgNT4AAAQKK0BkgQIAAgWgBRRgdv+EJECBAQBE6AwQIECAQLaAIo+M3PAECBAgoQmeAAAECBKIFFGF0/IYnQIAAAUXoDBAgQIBAtIAijI7f8AQIECCgCJ0BAgQIEIgWUITR8RueAAECBBShM0CAAAEC0QKKMDp+wxMgQICAInQGCBAgQCBaQBFGx294AgQIEFCEzgABAgQIRAsowuj4DU+AAAECitAZIECAAIFoAUUYHb/hCRAgQEAROgMECBAgEC2gCKPjNzwBAgQIKEJngAABAgSiBRRhdPyGJ0CAAIHXCAgQqBH44jefnV/ok19/fv4JvkqAwBoC3hGuoeqaBG4ROOz3t7zMawgQuE9AEd7n59UElhM4HB6Xu5grESBwqYAivFTK8wisLeAd4drCrk/gqIAiPMriQQIbCBz23hFuwG5JAorQGSDQRuDgZ4RtsrCRJAFFmJS2WXsL+Gi0dz52N62AIpw2WoMNJ+BmmeEis+E5BBThHDmaYgYBPyOcIUUzDCigCAcMzZYnFfDR6KTBGqu7gCLsnpD95Qh4R5iTtUlbCSjCVnHYTLTAwV2j0fkbfjMBRbgZvYUJfFvAX7H2bRG/J1AhoAgrlK1B4BIBd41eouQ5BBYXUISLk7oggRsF3CxzI5yXEbhPQBHe5+fVBJYTcLPMcpauROAKAUV4BZanElhVwM0yq/K6OIFTAorwlIzHCVQL+Gi0Wtx6BL4WUIQOAoE2Av49wjZR2EiUgCKMituwrQX8jLB1PDY3r4AinDdbk40m4KPR0RKz30kEFOEkQRpjAgHvCCcI0QgjCijCEVOz5zkF3DU6Z66mai+gCNtHZIMxAj4ajYnaoL0EFGGvPOwmWcBHo8npm31DAUW4Ib6lCTwX8K9PPPfwOwI1AoqwxtkqBD4s4B3hh408g8AKAopwBVSXJHCTgJtlbmLzIgL3CijCewW9nsBSAm6WWUrSdQhcJaAIr+LyZAIrCvhodEVclyZwWkARnrbxFQK1Aoqw1ttqBN4KKEJHgUAXAT8j7JKEfYQJKMKwwI27ncDHn/zq/OJ/++Pvzj/BVwkQWENAEa6h6poEjgjsHny7HWHxEIHNBXxnbh6BDaQI7Ha+3VKyNudYAr4zx8rLbgcW2D28Gnj3tk5gXgFFOG+2Jusm4B1ht0Tsh8DXAorQQSBQJOAdYRG0ZQhcKaAIrwTzdAK3CrhZ5lY5ryOwroAiXNfX1Qm8F3CzzHsKvyDQSkARtorDZmYW8NHozOmabWQBRThyevY+lIAiHCoumw0SUIRBYRt1WwE/I9zW3+oETgkowlMyHiewtMDOnyNcmtT1CCwhoAiXUHQNAhcIeEd4AZKnENhAQBFugG7JTAF3jWbmbur+Aoqwf0Z2OImAm2UmCdIY0wkowukiNVBXAR+Ndk3GvtIFFGH6CTB/nYCbZeqsrUTgCgFFeAWWpxK4R8A7wnv0vJbAegKKcD1bVybwTMDNMs84/IZAGwFF2CYKG5ldwM0ysydsvlEFFOGoydn3cAKKcLjIbDhEQBGGBG3M7QX8jHD7DOyAwDEBRXhMxWMEVhDYuWt0BVWXJHC/gCK839AVCFwm8ODb7TIozyJQK+A7s9bbasEC7hoNDt/orQUUYet4bG4mATfLzJSmWWYSUIQzpWmW1gJulmkdj80FCyjC4PCNXivgZplab6sRuFRAEV4q5XkE7hTwjvBOQC8nsJKAIlwJ1mUJvBDY+XZ7YeIBAg0EfGc2CMEWMgQeXr3OGNSUBAYTUISDBWa7cwscDoe5BzQdgYYCirBhKLaUK3DYP+YOb3ICGwkowo3gLUvgqMBhf/RhDxIgsJ6AIlzP1pUJXC1w2CvCq9G8gMCdAorwTkAvJ7CkwOHgo9ElPV2LwCUCivASJc8hUCTgHWERtGUIPBFQhE8w/JLA1gJultk6AesnCijCxNTN3Fbg4GaZttnY2LwCinDebE02ooCbZUZMzZ4HF1CEgwdo+3MJ+Gh0rjxNM4aAIhwjJ7sMEXDXaEjQxmwloAhbxWEz6QLuGk0/AebfQkARbqFuTQInBHw0egLGwwRWFFCEK+K6NIFrBdw1eq2Y5xO4X0AR3m/oCgQWE/DR6GKULkTgYgFFeDGVJxIoEPBXrBUgW4LAcwFF+NzD7whsKuAd4ab8Fg8VUIShwRu7p4CbZXrmYldzCyjCufM13WACbpYZLDDbnUJAEU4RoyFmEfDR6CxJmmMkAUU4Ulr2Or2Aj0anj9iADQUUYcNQbClXQBHmZm/y7QQU4Xb2VibwUsA/w/TSxCMEVhZQhCsDuzyBawS8I7xGy3MJLCOgCJdxdBUCiwi4a3QRRhchcJWAIryKy5MJrCvgrtF1fV2dwDEBRXhMxWMENhLw0ehG8JaNFlCE0fEbvp2Am2XaRWJD8wsowvkzNuFAAt4RDhSWrU4joAinidIgMwi4WWaGFM0wmoAiHC0x+51awM0yU8druKYCirBpMLY1pcDHP/3l+bn++sVvzz/BVwkQWFxAES5O6oIETgs8vDr9NV8hQGAbAUW4jbtVMwV2O99xmcmburWAb8vW8djcZAI77wgnS9Q4UwgowiliNMQgArsH33GDRGWbSQK+LZPSNuvWAj4a3ToB6xM4IqAIj6B4iMBKAj4aXQnWZQncI6AI79HzWgJXCrhZ5kowTydQIKAIC5AtQeCtgHeEjgKBhgKKsGEotjStgJtlpo3WYCMLKMKR07P30QS8IxwtMfuNEFCEETEbsomAu0abBGEbBJ4KKMKnGn5NYF0B7wjX9XV1AjcJKMKb2LyIwE0CfkZ4E5sXEVhXQBGu6+vqBJ4J7Pyl2888/IZABwFF2CEFe0gR8I4wJWlzDiWgCIeKy2YHF3CzzOAB2v6cAopwzlxN1VPAzTI9c7GrcAFFGH4AjH+LwO7W/z799GcfXO/Wa3/1ug9e3BMIEHgpoAhfmniEwFoCX+73a13adQkQuFVAEd4q53UErhd43B+uf5FXECCwrsDrdS/v6gQIPBH48vH/7wj/9M+f//2/P/rP/nvfffjXD7/z5598/w9PnuiXBAjUCSjCOmsrEXh8V4S//8dn7zXedOFf/v3jN//7xQ8+f/+gXxAgUCbgo9EyagsR+Oibj0aftuBTlFOPP32OXxMgsLiAIlyc1AUJnBR43O/Pt935r568ri8QIHCHgCK8A89LCVwp8PjoZpkryTydwPoCinB9YysQeCfgj0+8k/D/BBoJKMJGYdjK9ALvb5aZflIDEhhIQBEOFJatDi/gzxEOH6EBZhRQhDOmaqauAm9uljn/ZyTOf7XrWPZFYGwBRTh2fnY/lsA3N8ucartTj481o90SGE7AH6gfLjIbHlhgf3h71+ibzvM3ywwcpK3PJbA7vPvOnGsu0xBYUaDtv/Pg23nF1F2aAAECBAgQIDClgHeEU8ZqqHUFvCNc19fVCdQKuFmm1ttqBAgQINBMQBE2C8R2CBAgQKBWQBHWeluNAAECBJoJKMJmgdgOAQIECNQKKMJab6sRIECAQDMBRdgsENshQIAAgVoBRVjrbTUCBAgQaCagCJsFYjsECBAgUCugCGu9rUaAAAECzQQUYbNAbIcAAQIEagUUYa231QgQIECgmYAibBaI7RAgQIBArYAirPW2GgECBAg0E1CEzQKxHQIECBCoFfDPMNV6W40AAQIEmgl4R9gsENshQIAAgVoBRVjrbTUCBAgQaCagCJsFYjsECBAgUCugCGu9rUaAAAECzQQUYbNAbIcAAQIEagUUYa231QgQIECgmYAibBaI7RAgQIBArYAirPW2GgECBAg0E1CEzQKxHQIECBCoFVCEtd5WI0CAAIFmAoqwWSC2Q4AAAQK1Aoqw1ttqBAgQINBMQBE2C8R2CBAgQKBWQBHWeluNAAECBJoJKMJmgdgOAQIECNQKKMJab6sRIECAQDMBRdgsENshQIAAgVoBRVjrbTUCBAgQaCagCJsFYjsECBAgUCugCGu9rUaAAAECzQQUYbNAbIcAAQIEagUUYa231QgQIECgmYAibBaI7RAgQIBArYAirPW2GgECBAg0E1CEzQKxHQIECBCoFVCEtd5WI0CAAIFmAoqwWSC2Q4AAAQK1Aoqw1ttqBAgQINBMQBE2C8R2CBAgQKBWQBHWeluNAAECBJoJKMJmgdgOAQIECNQKKMJab6sRIECAQDMBRdgsENshQIAAgVoBRVjrbTUCBAgQaCagCJsFYjsECBAgUCugCGu9rUaAAAECzQQUYbNAbIcAAQIEagUUYa231QgQIECgmYAibBaI7RAgQIBArYAirPW2GgECBAg0E1CEzQKxHQIECBCoFVCEtd5WI0CAAIFmAoqwWSC2Q4AAAQK1Aoqw1ttqBAgQINBMQBE2C8R2CBAgQKBWQBHWeluNAAECBJoJKMJmgdgOAQIECNQKKMJab6sRIECAQDMBRdgsENshQIAAgVoBRVjrbTUCBAgQaCagCJsFYjsECBAgUCugCGu9rUaAAAECzQQUYbNAbIcAAQIEagUUYa231QgQIECgmYAibBaI7RAgQIBArYAirPW2GgECBAg0E1CEzQKxHQIECBCoFVCEtd5WI0CAAIFmAoqwWSC2Q4AAAQK1Aoqw1ttqBAgQINBMQBE2C8R2CBAgQKBWQBHWeluNAAECBJoJKMJmgdgOAQIECNQKKMJab6sRIECAQDMBRdgsENshQIAAgVoBRVjrbTUCBAgQaCagCJsFYjsECBAgUCugCGu9rUaAAAECzQQUYbNAbIcAAQIEagUUYa231QgQIECgmYAibBaI7RAgQIBArYAirPW2GgECBAg0E1CEzQKxHQIECBCoFVCEtd5WI0CAAIFmAoqwWSC2Q4AAAQK1Aoqw1ttqBAgQINBMQBE2C8R2CBAgQKBWQBHWeluNAAECBJoJKMJmgdgOAQIECNQKKMJab6sRIECAQDMBRdgsENshQIAAgVoBRVjrbTUCBAgQaCagCJsFYjsECBAgUCugCGu9rUaAAAECzQT+BwOnHqGjaJ6pAAAAAElFTkSuQmCC", "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#@test {\"skip\": true}\n", "env.reset()\n", "PIL.Image.fromarray(env.render())" ] }, { "cell_type": "markdown", "metadata": { "id": "B9_lskPOey18" }, "source": [ "The `time_step = environment.step(action)` statement takes `action` in the environment. The `TimeStep` tuple returned contains the environment's next observation and reward for that action. The `time_step_spec()` and `action_spec()` methods in the environment return the specifications (types, shapes, bounds) of the `time_step` and `action` respectively." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:06.083142Z", "iopub.status.busy": "2023-12-22T14:05:06.082907Z", "iopub.status.idle": "2023-12-22T14:05:06.090439Z", "shell.execute_reply": "2023-12-22T14:05:06.089883Z" }, "id": "exDv57iHfwQV" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Observation Spec:\n", "BoundedArraySpec(shape=(4,), dtype=dtype('float32'), name='observation', minimum=[-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], maximum=[4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38])\n", "Action Spec:\n", "BoundedArraySpec(shape=(), dtype=dtype('int64'), name='action', minimum=0, maximum=1)\n" ] } ], "source": [ "print('Observation Spec:')\n", "print(env.time_step_spec().observation)\n", "print('Action Spec:')\n", "print(env.action_spec())" ] }, { "cell_type": "markdown", "metadata": { "id": "eJCgJnx3g0yY" }, "source": [ "So, we see that observation is an array of 4 floats: the position and velocity of the cart, and the angular position and velocity of the pole. Since only two actions are possible (move left or move right), the `action_spec` is a scalar where 0 means \"move left\" and 1 means \"move right.\"" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:06.093731Z", "iopub.status.busy": "2023-12-22T14:05:06.093281Z", "iopub.status.idle": "2023-12-22T14:05:06.099811Z", "shell.execute_reply": "2023-12-22T14:05:06.099243Z" }, "id": "V2UGR5t_iZX-" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Time step:\n", "TimeStep(\n", "{'step_type': array(0, dtype=int32),\n", " 'reward': array(0., dtype=float32),\n", " 'discount': array(1., dtype=float32),\n", " 'observation': array([-0.00907558, 0.02627698, -0.01019297, 0.04808202], dtype=float32)})\n", "Next time step:\n", "TimeStep(\n", "{'step_type': array(1, dtype=int32),\n", " 'reward': array(1., dtype=float32),\n", " 'discount': array(1., dtype=float32),\n", " 'observation': array([-0.00855004, 0.2215436 , -0.00923133, -0.24779937], dtype=float32)})\n" ] } ], "source": [ "time_step = env.reset()\n", "print('Time step:')\n", "print(time_step)\n", "\n", "action = np.array(1, dtype=np.int32)\n", "\n", "next_time_step = env.step(action)\n", "print('Next time step:')\n", "print(next_time_step)" ] }, { "cell_type": "markdown", "metadata": { "id": "zuUqXAVmecTU" }, "source": [ "Usually we create two environments: one for training and one for evaluation. Most environments are written in pure python, but they can be easily converted to TensorFlow using the `TFPyEnvironment` wrapper. The original environment's API uses numpy arrays, the `TFPyEnvironment` converts these to/from `Tensors` for you to more easily interact with TensorFlow policies and agents.\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:06.103106Z", "iopub.status.busy": "2023-12-22T14:05:06.102593Z", "iopub.status.idle": "2023-12-22T14:05:06.113658Z", "shell.execute_reply": "2023-12-22T14:05:06.113116Z" }, "id": "Xp-Y4mD6eDhF" }, "outputs": [], "source": [ "train_py_env = suite_gym.load(env_name)\n", "eval_py_env = suite_gym.load(env_name)\n", "\n", "train_env = tf_py_environment.TFPyEnvironment(train_py_env)\n", "eval_env = tf_py_environment.TFPyEnvironment(eval_py_env)" ] }, { "cell_type": "markdown", "metadata": { "id": "E9lW_OZYFR8A" }, "source": [ "## Agent\n", "\n", "The algorithm that we use to solve an RL problem is represented as an `Agent`. In addition to the REINFORCE agent, TF-Agents provides standard implementations of a variety of `Agents` such as [DQN](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf), [DDPG](https://arxiv.org/pdf/1509.02971.pdf), [TD3](https://arxiv.org/pdf/1802.09477.pdf), [PPO](https://arxiv.org/abs/1707.06347) and [SAC](https://arxiv.org/abs/1801.01290).\n", "\n", "To create a REINFORCE Agent, we first need an `Actor Network` that can learn to predict the action given an observation from the environment.\n", "\n", "We can easily create an `Actor Network` using the specs of the observations and actions. We can specify the layers in the network which, in this example, is the `fc_layer_params` argument set to a tuple of `ints` representing the sizes of each hidden layer (see the Hyperparameters section above).\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:06.117206Z", "iopub.status.busy": "2023-12-22T14:05:06.116765Z", "iopub.status.idle": "2023-12-22T14:05:06.161414Z", "shell.execute_reply": "2023-12-22T14:05:06.160841Z" }, "id": "TgkdEPg_muzV" }, "outputs": [], "source": [ "actor_net = actor_distribution_network.ActorDistributionNetwork(\n", " train_env.observation_spec(),\n", " train_env.action_spec(),\n", " fc_layer_params=fc_layer_params)" ] }, { "cell_type": "markdown", "metadata": { "id": "z62u55hSmviJ" }, "source": [ "We also need an `optimizer` to train the network we just created, and a `train_step_counter` variable to keep track of how many times the network was updated.\n" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:06.164948Z", "iopub.status.busy": "2023-12-22T14:05:06.164368Z", "iopub.status.idle": "2023-12-22T14:05:08.626259Z", "shell.execute_reply": "2023-12-22T14:05:08.625470Z" }, "id": "jbY4yrjTEyc9" }, "outputs": [], "source": [ "optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n", "\n", "train_step_counter = tf.Variable(0)\n", "\n", "tf_agent = reinforce_agent.ReinforceAgent(\n", " train_env.time_step_spec(),\n", " train_env.action_spec(),\n", " actor_network=actor_net,\n", " optimizer=optimizer,\n", " normalize_returns=True,\n", " train_step_counter=train_step_counter)\n", "tf_agent.initialize()" ] }, { "cell_type": "markdown", "metadata": { "id": "I0KLrEPwkn5x" }, "source": [ "## Policies\n", "\n", "In TF-Agents, policies represent the standard notion of policies in RL: given a `time_step` produce an action or a distribution over actions. The main method is `policy_step = policy.action(time_step)` where `policy_step` is a named tuple `PolicyStep(action, state, info)`. The `policy_step.action` is the `action` to be applied to the environment, `state` represents the state for stateful (RNN) policies and `info` may contain auxiliary information such as log probabilities of the actions.\n", "\n", "Agents contain two policies: the main policy that is used for evaluation/deployment (agent.policy) and another policy that is used for data collection (agent.collect_policy)." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:08.630846Z", "iopub.status.busy": "2023-12-22T14:05:08.630328Z", "iopub.status.idle": "2023-12-22T14:05:08.633596Z", "shell.execute_reply": "2023-12-22T14:05:08.632978Z" }, "id": "BwY7StuMkuV4" }, "outputs": [], "source": [ "eval_policy = tf_agent.policy\n", "collect_policy = tf_agent.collect_policy" ] }, { "cell_type": "markdown", "metadata": { "id": "94rCXQtbUbXv" }, "source": [ "## Metrics and Evaluation\n", "\n", "The most common metric used to evaluate a policy is the average return. The return is the sum of rewards obtained while running a policy in an environment for an episode, and we usually average this over a few episodes. We can compute the average return metric as follows.\n" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:08.636739Z", "iopub.status.busy": "2023-12-22T14:05:08.636497Z", "iopub.status.idle": "2023-12-22T14:05:08.641058Z", "shell.execute_reply": "2023-12-22T14:05:08.640376Z" }, "id": "bitzHo5_UbXy" }, "outputs": [], "source": [ "#@test {\"skip\": true}\n", "def compute_avg_return(environment, policy, num_episodes=10):\n", "\n", " total_return = 0.0\n", " for _ in range(num_episodes):\n", "\n", " time_step = environment.reset()\n", " episode_return = 0.0\n", "\n", " while not time_step.is_last():\n", " action_step = policy.action(time_step)\n", " time_step = environment.step(action_step.action)\n", " episode_return += time_step.reward\n", " total_return += episode_return\n", "\n", " avg_return = total_return / num_episodes\n", " return avg_return.numpy()[0]\n", "\n", "\n", "# Please also see the metrics module for standard implementations of different\n", "# metrics." ] }, { "cell_type": "markdown", "metadata": { "id": "NLva6g2jdWgr" }, "source": [ "## Replay Buffer\n", "\n", "In order to keep track of the data collected from the environment, we will use [Reverb](https://deepmind.com/research/open-source/Reverb), an efficient, extensible, and easy-to-use replay system by Deepmind. It stores experience data when we collect trajectories and is consumed during training.\n", "\n", "This replay buffer is constructed using specs describing the tensors that are to be stored, which can be obtained from the agent using `tf_agent.collect_data_spec`." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:08.644420Z", "iopub.status.busy": "2023-12-22T14:05:08.643928Z", "iopub.status.idle": "2023-12-22T14:05:08.660169Z", "shell.execute_reply": "2023-12-22T14:05:08.659543Z" }, "id": "vX2zGUWJGWAl" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[reverb/cc/platform/tfrecord_checkpointer.cc:162] Initializing TFRecordCheckpointer in /tmpfs/tmp/tmpkagdqs1n.\n", "[reverb/cc/platform/tfrecord_checkpointer.cc:565] Loading latest checkpoint from /tmpfs/tmp/tmpkagdqs1n\n", "[reverb/cc/platform/default/server.cc:71] Started replay server on port 41705\n" ] } ], "source": [ "table_name = 'uniform_table'\n", "replay_buffer_signature = tensor_spec.from_spec(\n", " tf_agent.collect_data_spec)\n", "replay_buffer_signature = tensor_spec.add_outer_dim(\n", " replay_buffer_signature)\n", "table = reverb.Table(\n", " table_name,\n", " max_size=replay_buffer_capacity,\n", " sampler=reverb.selectors.Uniform(),\n", " remover=reverb.selectors.Fifo(),\n", " rate_limiter=reverb.rate_limiters.MinSize(1),\n", " signature=replay_buffer_signature)\n", "\n", "reverb_server = reverb.Server([table])\n", "\n", "replay_buffer = reverb_replay_buffer.ReverbReplayBuffer(\n", " tf_agent.collect_data_spec,\n", " table_name=table_name,\n", " sequence_length=None,\n", " local_server=reverb_server)\n", "\n", "rb_observer = reverb_utils.ReverbAddEpisodeObserver(\n", " replay_buffer.py_client,\n", " table_name,\n", " replay_buffer_capacity\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "ZGNTDJpZs4NN" }, "source": [ "For most agents, the `collect_data_spec` is a `Trajectory` named tuple containing the observation, action, reward etc." ] }, { "cell_type": "markdown", "metadata": { "id": "rVD5nQ9ZGo8_" }, "source": [ "## Data Collection\n", "\n", "As REINFORCE learns from whole episodes, we define a function to collect an episode using the given data collection policy and save the data (observations, actions, rewards etc.) as trajectories in the replay buffer. Here we are using 'PyDriver' to run the experience collecting loop. You can learn more about TF Agents driver in our [drivers tutorial](https://www.tensorflow.org/agents/tutorials/4_drivers_tutorial)." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:08.663547Z", "iopub.status.busy": "2023-12-22T14:05:08.662908Z", "iopub.status.idle": "2023-12-22T14:05:08.666866Z", "shell.execute_reply": "2023-12-22T14:05:08.666320Z" }, "id": "wr1KSAEGG4h9" }, "outputs": [], "source": [ "#@test {\"skip\": true}\n", "\n", "def collect_episode(environment, policy, num_episodes):\n", "\n", " driver = py_driver.PyDriver(\n", " environment,\n", " py_tf_eager_policy.PyTFEagerPolicy(\n", " policy, use_tf_function=True),\n", " [rb_observer],\n", " max_episodes=num_episodes)\n", " initial_time_step = environment.reset()\n", " driver.run(initial_time_step)" ] }, { "cell_type": "markdown", "metadata": { "id": "hBc9lj9VWWtZ" }, "source": [ "## Training the agent\n", "\n", "The training loop involves both collecting data from the environment and optimizing the agent's networks. Along the way, we will occasionally evaluate the agent's policy to see how we are doing.\n", "\n", "The following will take ~3 minutes to run." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:05:08.670316Z", "iopub.status.busy": "2023-12-22T14:05:08.669689Z", "iopub.status.idle": "2023-12-22T14:07:49.310371Z", "shell.execute_reply": "2023-12-22T14:07:49.309563Z" }, "id": "0pTbJ3PeyF-u" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[reverb/cc/client.cc:165] Sampler and server are owned by the same process (47292) so Table uniform_table is accessed directly without gRPC.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", "I0000 00:00:1703253913.189247 48625 device_compiler.h:186] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[reverb/cc/client.cc:165] Sampler and server are owned by the same process (47292) so Table uniform_table is accessed directly without gRPC.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[reverb/cc/client.cc:165] Sampler and server are owned by the same process (47292) so Table uniform_table is accessed directly without gRPC.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[reverb/cc/client.cc:165] Sampler and server are owned by the same process (47292) so Table uniform_table is accessed directly without gRPC.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[reverb/cc/client.cc:165] Sampler and server are owned by the same process (47292) so Table uniform_table is accessed directly without gRPC.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 25: loss = 1.8318419456481934\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[reverb/cc/client.cc:165] Sampler and server are owned by the same process (47292) so Table uniform_table is accessed directly without gRPC.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 50: loss = 0.0070743560791015625\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 50: Average Return = 9.800000190734863\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[reverb/cc/client.cc:165] Sampler and server are owned by the same process (47292) so Table uniform_table is accessed directly without gRPC.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 75: loss = 1.1006038188934326\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 100: loss = 0.5719594955444336\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 100: Average Return = 50.29999923706055\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 125: loss = -1.2458715438842773\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[reverb/cc/client.cc:165] Sampler and server are owned by the same process (47292) so Table uniform_table is accessed directly without gRPC.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 150: loss = 1.9363441467285156\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 150: Average Return = 98.30000305175781\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 175: loss = 0.8784818649291992\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 200: loss = 1.9726766347885132\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 200: Average Return = 143.6999969482422\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 225: loss = 2.316105842590332\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 250: loss = 2.5175299644470215\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "step = 250: Average Return = 191.5\n" ] } ], "source": [ "#@test {\"skip\": true}\n", "try:\n", " %%time\n", "except:\n", " pass\n", "\n", "# (Optional) Optimize by wrapping some of the code in a graph using TF function.\n", "tf_agent.train = common.function(tf_agent.train)\n", "\n", "# Reset the train step\n", "tf_agent.train_step_counter.assign(0)\n", "\n", "# Evaluate the agent's policy once before training.\n", "avg_return = compute_avg_return(eval_env, tf_agent.policy, num_eval_episodes)\n", "returns = [avg_return]\n", "\n", "for _ in range(num_iterations):\n", "\n", " # Collect a few episodes using collect_policy and save to the replay buffer.\n", " collect_episode(\n", " train_py_env, tf_agent.collect_policy, collect_episodes_per_iteration)\n", "\n", " # Use data from the buffer and update the agent's network.\n", " iterator = iter(replay_buffer.as_dataset(sample_batch_size=1))\n", " trajectories, _ = next(iterator)\n", " train_loss = tf_agent.train(experience=trajectories)\n", "\n", " replay_buffer.clear()\n", "\n", " step = tf_agent.train_step_counter.numpy()\n", "\n", " if step % log_interval == 0:\n", " print('step = {0}: loss = {1}'.format(step, train_loss.loss))\n", "\n", " if step % eval_interval == 0:\n", " avg_return = compute_avg_return(eval_env, tf_agent.policy, num_eval_episodes)\n", " print('step = {0}: Average Return = {1}'.format(step, avg_return))\n", " returns.append(avg_return)" ] }, { "cell_type": "markdown", "metadata": { "id": "68jNcA_TiJDq" }, "source": [ "## Visualization\n" ] }, { "cell_type": "markdown", "metadata": { "id": "aO-LWCdbbOIC" }, "source": [ "### Plots\n", "\n", "We can plot return vs global steps to see the performance of our agent. In `Cartpole-v0`, the environment gives a reward of +1 for every time step the pole stays up, and since the maximum number of steps is 200, the maximum possible return is also 200." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:07:49.314255Z", "iopub.status.busy": "2023-12-22T14:07:49.313751Z", "iopub.status.idle": "2023-12-22T14:07:49.521328Z", "shell.execute_reply": "2023-12-22T14:07:49.520744Z" }, "id": "NxtL1mbOYCVO" }, "outputs": [ { "data": { "text/plain": [ "(0.7150002002716054, 250.0)" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAG2CAYAAACZEEfAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABGeklEQVR4nO3dd3RUdf7/8ddMGoEkEwKEJPRehESaEFEUCU2stKWosOuiYLChiKioKIpiWQUL/ly/4FJtFMWyi1SRABJChwgIhJICgTRC6tzfH8hoKJIJmUzJ83HOnMPMvXPzvpcw8+J9P/d+TIZhGAIAAPBQZmcXAAAA4EiEHQAA4NEIOwAAwKMRdgAAgEcj7AAAAI9G2AEAAB6NsAMAADwaYQcAAHg0wg4AAPBohB0AAODRnBp2pk6dqk6dOikwMFChoaG66667lJiYWGKdm2++WSaTqcRj9OjRJdZJSkpSv379VLVqVYWGhmr8+PEqKiqqyF0BAAAuytuZP3zNmjWKjY1Vp06dVFRUpGeeeUa9evXS7t27Va1aNdt6o0aN0ksvvWR7XrVqVdufi4uL1a9fP4WFhWn9+vVKTk7WfffdJx8fH7366qsVuj8AAMD1mFxpItATJ04oNDRUa9asUbdu3SSd6+xce+21eueddy75nu+//1633Xabjh8/rtq1a0uSZs6cqQkTJujEiRPy9fWtqPIBAIALcmpn50KZmZmSpJCQkBKvz5s3T3PnzlVYWJhuv/12TZo0ydbdiYuLU9u2bW1BR5J69+6tMWPGaNeuXWrXrt1FPyc/P1/5+fm251arVadOnVKNGjVkMpkcsWsAAKCcGYah7OxsRUREyGy+/Mgclwk7VqtVjz32mLp27ao2bdrYXh82bJgaNGigiIgIbd++XRMmTFBiYqIWLVokSUpJSSkRdCTZnqekpFzyZ02dOlWTJ0920J4AAICKdOTIEdWtW/eyy10m7MTGxmrnzp1at25didcfeOAB25/btm2r8PBw9ejRQwcOHFCTJk3K9LMmTpyocePG2Z5nZmaqfv36OnLkiIKCgsq2AwAAoEJlZWWpXr16CgwM/Mv1XCLsjB07VsuWLdPatWv/MplJUufOnSVJ+/fvV5MmTRQWFqZNmzaVWCc1NVWSFBYWdslt+Pn5yc/P76LXg4KCCDsAALiZKw1Bceql54ZhaOzYsVq8eLFWrlypRo0aXfE9W7dulSSFh4dLkqKjo7Vjxw6lpaXZ1lm+fLmCgoLUunVrh9QNAADch1M7O7GxsZo/f76WLl2qwMBA2xgbi8Uif39/HThwQPPnz9ett96qGjVqaPv27Xr88cfVrVs3RUZGSpJ69eql1q1b695779W0adOUkpKi5557TrGxsZfs3gAAgMrFqZeeX67tNGvWLI0cOVJHjhzRPffco507d+rMmTOqV6+e7r77bj333HMlTjcdPnxYY8aM0erVq1WtWjWNGDFCr732mry9S5flsrKyZLFYlJmZyWksAADcRGm/v13qPjvOQtgBAMD9lPb7m7mxAACARyPsAAAAj0bYAQAAHo2wAwAAPBphBwAAeDTCDgAA8GiEHQAA4NEIOwAAwKMRdgAAgEcj7AAAAI9G2AEAAB6NsAMAADwaYQcAAHg0wg4AAPBohB0AAODRCDsAAMCjEXYAAIBHI+wAAACPRtgBAAAejbADAAA8GmEHAAB4NMIOAADwaIQdAADg0Qg7AADAoxF2AACARyPsAAAAj0bYAQAAHo2wAwAAPBphBwAAeDTCDgAA8GiEHQAA4NEIOwAAwKMRdgAAgEcj7AAAAI9G2AEAAB6NsAMAADwaYQcAAHg0wg4AAPBohB0AAODRCDsAAMCjEXYAAIBHI+wAAACPRtgBAAAejbADAAA8GmEHAAB4NMIOAADwaIQdAADg0Qg7AADAoxF2AACARyPsAAAAj0bYAQAAHo2wAwAAPBphBwAAeDTCDgAA8GiEHQAA4NEIOwAAwKMRdgAAgEcj7AAAAI9G2AEAAB6NsAMAADwaYQcAAHg0wg4AAPBohB0AAODRnBp2pk6dqk6dOikwMFChoaG66667lJiYWGKdvLw8xcbGqkaNGgoICNCAAQOUmppaYp2kpCT169dPVatWVWhoqMaPH6+ioqKK3BUAAOCinBp21qxZo9jYWG3YsEHLly9XYWGhevXqpTNnztjWefzxx/XNN9/oiy++0Jo1a3T8+HH179/ftry4uFj9+vVTQUGB1q9fr08//VSzZ8/W888/74xdAgAALsZkGIbh7CLOO3HihEJDQ7VmzRp169ZNmZmZqlWrlubPn6+BAwdKkvbu3atWrVopLi5OXbp00ffff6/bbrtNx48fV+3atSVJM2fO1IQJE3TixAn5+vpe8edmZWXJYrEoMzNTQUFBDt1HAABQPkr7/e1SY3YyMzMlSSEhIZKk+Ph4FRYWKiYmxrZOy5YtVb9+fcXFxUmS4uLi1LZtW1vQkaTevXsrKytLu3btuuTPyc/PV1ZWVokHAADwTC4TdqxWqx577DF17dpVbdq0kSSlpKTI19dXwcHBJdatXbu2UlJSbOv8OeicX35+2aVMnTpVFovF9qhXr1457w0AAHAVLhN2YmNjtXPnTi1cuNDhP2vixInKzMy0PY4cOeLwnwkAAJzD29kFSNLYsWO1bNkyrV27VnXr1rW9HhYWpoKCAmVkZJTo7qSmpiosLMy2zqZNm0ps7/zVWufXuZCfn5/8/PzKeS8AAIArcmpnxzAMjR07VosXL9bKlSvVqFGjEss7dOggHx8frVixwvZaYmKikpKSFB0dLUmKjo7Wjh07lJaWZltn+fLlCgoKUuvWrStmRwAAgMtyamcnNjZW8+fP19KlSxUYGGgbY2OxWOTv7y+LxaL7779f48aNU0hIiIKCgvTwww8rOjpaXbp0kST16tVLrVu31r333qtp06YpJSVFzz33nGJjY+neAAAA5156bjKZLvn6rFmzNHLkSEnnbir4xBNPaMGCBcrPz1fv3r31wQcflDhFdfjwYY0ZM0arV69WtWrVNGLECL322mvy9i5dluPScwAA3E9pv79d6j47zkLYAQDA/bjlfXYAAADKG2EHAAB4NMIOAADwaIQdAADg0Qg7AADAYdKy8rQqMe3KKzoQYQcAADjE5kOn1G/GOo2eE6+dxzKdVodLTBcBAAA8h2EY+nT9IU35do+KrIaa1w5QNT/nRQ7CDgAAKDdnC4r1zOIdWpxwTJJ0W2S4Xh8QSdgBAADu73D6GT04J157U7LlZTZpYt+Wuv+GRpedMaGiEHYAAMBVW7U3TY8uTFBWXpFqBvjqvWHt1aVxDWeXJYmwAwAAroLVamj6yn16d8U+GYbUrn6wPhjeXuEWf2eXZkPYAQAAZZKZW6jHP9+qlXvPXVp+T5f6mnRba/l5ezm5spIIOwAAwG57krM0em68Dqfnys/brFfubquBHeo6u6xLIuwAAAC7LEk4pqcXbVdeoVV1q/tr5j0d1KaOxdllXRZhBwAAlEphsVWvfLtHs9cfkiTd2Kympg9pp+rVfJ1b2BUQdgAAwBWlZeUpdv4W/XLotCRpbPemerxnc3mZnXtZeWkQdgAAwF/afOiUHpq3RWnZ+Qr089Zbg6PU65owZ5dVaoQdAABwSZea9mHmPR3UuFaAs0uzC2EHAABc5MJpH/pFhmuak6d9KCv3qxgAADhUUnquHpwbrz3JWS417UNZEXYAAIDNhdM+zBjaXtFNXGPah7Ii7AAAALeY9qGsCDsAAFRymWcLNe6zrVrx+7QPwzvX1/O3u960D2VF2AEAoBL787QPvt5mvXJXGw3qWM/ZZZUrwg4AAJXU0q3HNOGrc9M+1An210f3uva0D2VF2AEAoJJx12kfyoqwAwBAJZKWnaex8xK06dApSVJs9yYa17OFW0z7UFaEHQAAKon4w6c0Zq77TvtQVoQdAAA8nGEY+k/cYb28bLeKrIaahQboo3vdb9qHsiLsAADgwTxp2oeyqjx7CgBAJXPhtA9P92mpf97ovtM+lBVhBwAAD7QqMU2PLdyqzLOFqlHNV+8Nc/9pH8qKsAMAgAexWg3NWLlf76z4VYYhXVsvWB/e4xnTPpQVYQcAAA/h6dM+lBVhBwAAD7A3JUsPzvlj2ocpd7XRYA+b9qGsCDsAALi5pVuP6emvduhsYbHqBPtr5j0d1Lau5037UFaEHQAA3FRhsVWvfrdHs34+JMnzp30oK8IOAABuqDJO+1BWhB0AANzMn6d9CPh92ofelWDah7Ii7AAA4CYMw9CcDYf10jd/TPsw894OalJJpn0oK8IOAABu4GxBsZ5dvEOLzk/70DZc0wZWrmkfyoojBACAi2Pah6tD2AEAwIVdOO3DjGHtdH2Tms4uy60QdgAAcEFWq6H3Vu3Xv348N+1DVL1gzazk0z6UFWEHAAAXk3m2UE98vlU/7jk37cOwzvX1AtM+lBlhBwAAF7I3JUuj58TrENM+lBvCDgAALoJpHxyDsAMAgJMVFls19bu9+r+fD0o6N+3Du0PaKYRpH8oFYQcAACdKy87T2PkJ2nSQaR8chbADAICTxB8+pYfmbVFqFtM+OBJhBwCACnZ+2oeXl+1WYTHTPjgaYQcAgAqUV1isZxbv0KItTPtQUcp0ZPft26dVq1YpLS1NVqu1xLLnn3++XAoDAMDTHDmVqwfnxGs30z5UKLvDzscff6wxY8aoZs2aCgsLK/EXZDKZCDsAAFzC6sQ0Pcq0D05hd9iZMmWKXnnlFU2YMMER9QAA4FEuNe3Dh8PbKyKYaR8qit1h5/Tp0xo0aJAjagEAwKMw7YNrMNv7hkGDBul///ufI2oBAMBjJKZk68731unHPWny9TZr2sBIvXp3W4KOE9jd2WnatKkmTZqkDRs2qG3btvLx8Smx/JFHHim34gAAcEdfbzuuCV9uZ9oHF2EyDMOw5w2NGjW6/MZMJv32229XXVRFy8rKksViUWZmpoKCgpxdDgDATV047cMNTWtq+lCmfXCU0n5/29XZMQxDq1evVmhoqPz9GVgFAMB5F0778NDNTfREL6Z9cAV2jdkxDEPNmjXT0aNHHVUPAABuJ/7wad0+Y502HTylAD9vzbyng57q05Kg4yLsCjtms1nNmjVTenq6o+oBAMBtGIahOXGHNOT/xSk1K1/NQgO0dGxX9WnD/FauxO6rsV577TWNHz9eO3fudEQ9AAC4hbzCYj3xxTZNWrpLhcWGbm0bpsWxXZnfygXZHXbuu+8+bdq0SVFRUfL391dISEiJhz3Wrl2r22+/XRERETKZTFqyZEmJ5SNHjpTJZCrx6NOnT4l1Tp06peHDhysoKEjBwcG6//77lZOTY+9uAQBQakdO5ar/B+u1aMsxmU3SM7e21PvD2iuA+a1ckt1/K++88065/fAzZ84oKipK//jHP9S/f/9LrtOnTx/NmjXL9tzPz6/E8uHDhys5OVnLly9XYWGh/v73v+uBBx7Q/Pnzy61OAADOW/PrCT2yIIFpH9yI3WFnxIgR5fbD+/btq759+/7lOn5+fgoLu/S5zz179uiHH37QL7/8oo4dO0qSZsyYoVtvvVVvvvmmIiIiyq1WAEDlZrUaen/Vfr3NtA9ux+6wk5SU9JfL69evX+ZiLuX8pe7Vq1fXLbfcoilTpqhGjRqSpLi4OAUHB9uCjiTFxMTIbDZr48aNuvvuuy+5zfz8fOXn59ueZ2VllWvNAADPwrQP7s3usNOwYcO/nIq+uLj4qgr6sz59+qh///5q1KiRDhw4oGeeeUZ9+/ZVXFycvLy8lJKSotDQ0BLv8fb2VkhIiFJSUi673alTp2ry5MnlVicAwHMlpmTrwTmbdSg9V77eZk25s40Gd6rn7LJgB7vDTkJCQonnhYWFSkhI0Ntvv61XXnml3AqTpCFDhtj+3LZtW0VGRqpJkyZavXq1evToUebtTpw4UePGjbM9z8rKUr16/OICAEq6cNqHD+9pr8i6wc4uC3ayO+xERUVd9FrHjh0VERGhN95447IDjctD48aNVbNmTe3fv189evRQWFiY0tLSSqxTVFSkU6dOXXacj3RuHNCFA50BADivsNiq177fq0/WMe2DJ7D70vPLadGihX755Zfy2twlHT16VOnp6QoPD5ckRUdHKyMjQ/Hx8bZ1Vq5cKavVqs6dOzu0FgCAZzqRna/h/95oCzoP3dxEn/7jOoKOG7O7s3PhYF7DMJScnKwXX3xRzZo1s2tbOTk52r9/v+35wYMHtXXrVts9eyZPnqwBAwYoLCxMBw4c0FNPPaWmTZuqd+/ekqRWrVqpT58+GjVqlGbOnKnCwkKNHTtWQ4YM4UosAIDd4g+f1kPz4pWala8AP2+9OSiKuyF7ALvDTnBw8EUDlA3DUL169bRw4UK7trV582Z1797d9vz8OJoRI0boww8/1Pbt2/Xpp58qIyNDERER6tWrl15++eUSp6DmzZunsWPHqkePHjKbzRowYICmT59u724BACoxwzA0d2OSXvrm3N2Qm4YGaOY9HdQ0lLshewKTYRiGPW9Ys2ZNiedms1m1atVS06ZN5e3tnneOLO0U8QAAz5NXWKxnF+/UV1vOTXJ9a9swTRsYxd2Q3UBpv7/t/ps0mUy6/vrrLwo2RUVFWrt2rbp162Z/tQAAOMGRU7kaPTdeu45nyWySnu7bUqNubPyXt1iB+7E77HTv3l3JyckX3d8mMzNT3bt3L9f77AAA4Ch/nvYhpJqv3hvaTtc3ZdoHT2R32DEM45KJNz09XdWqVSuXogAAcJSiYqve+XGf3l+9n2kfKolSh53z988xmUwaOXJkiUHCxcXF2r59u66//vryrxAAgHKSkpmnRxYmaNPBU5KY9qGyKHXYsVgsks51dgIDA+Xv/0cC9vX1VZcuXTRq1KjyrxAAgHKwOjFN4z7fplNnClTN10tTB0TqjihuU1IZlDrszJo1S9K5ubGefPJJTlkBANxCUbFVby3/VR+uPiBJah0epPeHt1ejmnyPVRZ2X3ounbvyavXq1Tpw4ICGDRumwMBAHT9+XEFBQQoIcL97EnDpOQB4puMZZ/XIggRtPnxaknRvlwZ6tl8rVfHhtJUncNil54cPH1afPn2UlJSk/Px89ezZU4GBgXr99deVn5+vmTNnXlXhAACUh5V7UzXu823KyC1UoJ+3XhsQqX6R4c4uC05g99xYjz76qDp27KjTp0+XGLdz9913a8WKFeVaHAAA9iosturV7/boH7M3KyO3UG3rWLTskRsIOpWY3Z2dn376SevXr5evb8kJ0Ro2bKhjx46VW2EAANjr6OlcPbwgQQlJGZKkkdc31MRbW3K1VSVnd9ixWq2XvHHg0aNHFRgYWC5FAQBgr//tStH4L7cr82yhAqt4642BkerThm4OynAaq1evXnrnnXdsz00mk3JycvTCCy/o1ltvLc/aAAC4ooIiq176ZrcemBOvzLOFiqpr0XeP3EjQgY3dV2MdPXpUvXv3lmEY2rdvnzp27Kh9+/apZs2aWrt27UXTSLgDrsYCAPd05FSuxs7fom1HMyVJ99/QSBP6tJSvt93/l4cbKu33d5kvPf/ss8+0bds25eTkqH379ho+fHiJAcvuhLADAO7nh53JGv/ldmXnFcni76M3B0WpZ+vazi4LFcihYedSkpOT9corr+i9994rj81VKMIOALiP/KJivfrtHn0ad1iS1K5+sGYMbae61as6uTJUNIfcZ2fXrl1atWqVfH19NXjwYAUHB+vkyZN65ZVXNHPmTDVu3PiqCwcA4HIOp5/R2PkJ2nHs3GmrB7s11pO9W8jHi9NWuLxSh52vv/5aAwcOVFFRkSRp2rRp+vjjjzV48GB16NBBixcvVp8+fRxWKACgcvt2e7Ke/mq7svOLVL2qj94aHKVbWnLaCldW6tNY1113nbp27aqXX35Z//73vzVu3Dhdc801+r//+z916tTJ0XU6FKexAMB15RUWa8q3uzV3Q5IkqWOD6poxrJ3CLe45ThTlp9zH7FgsFsXHx6tp06YqLi6Wn5+ffvjhB8XExJRb0c5C2AEA13Tw5BnFztui3clZkqSHbm6icT2by5vTVpADxuxkZ2fbNuTl5SV/f3/G6AAAHGbp1mN6ZtEOnSkoVkg1X/3rb9fqpua1nF0W3JBdA5T/+9//ymKxSDp3J+UVK1Zo586dJda54447yq86AEClk1dYrMnf7NKCTUckSdc1CtH0Ie0UZqni5Mrgrkp9GstsvnLL0GQyXXIqCVfHaSwAcA3703I0dv4W7U3Jlskkje3eVI/2aMZpK1xSuZ/Gslqt5VIYAACXsmjLUT23ZKdyC4pVM8BX7/ytnW5oVtPZZcED2D0RKAAA5elsQbGeX7pTX8QflSRFN66hd4dcq9AgTluhfBB2AABOsy81Ww/N26J9aTkymaRHezTTw7c0k5fZ5OzS4EEIOwAAp/hi8xFNWrpTeYVW1Qr007tDrtX1TThthfJH2AEAVKgz+UWatHSnFm05Jkm6sVlNvT34WtUK9HNyZfBUhB0AQIXZm5Kl2HlbdODEGZlN0riezfXQzU1l5rQVHKhMYScjI0NffvmlDhw4oPHjxyskJERbtmxR7dq1VadOnfKuEQDg5gzD0Ge/HNELX+9SfpFVtYP8NH1IO3VuXMPZpaESsDvsbN++XTExMbJYLDp06JBGjRqlkJAQLVq0SElJSfrPf/7jiDoBAG4qJ79Izy7eoaVbj0uSbmpeS28PjlKNAE5boWLYfZemcePGaeTIkdq3b5+qVPnjssBbb71Va9euLdfiAADubffxLN0xY52Wbj0uL7NJE/q01KyRnQg6qFB2d3Z++eUXffTRRxe9XqdOHaWkpJRLUQAA92YYhuZtTNJLy3aroMiqcEsVzRjaTh0bhji7NFRCdocdPz8/ZWVlXfT6r7/+qlq1mKANACq77LxCPb1oh77dnixJuqVlqN4aFKXq1XydXBkqK7tPY91xxx166aWXVFhYKOncfFhJSUmaMGGCBgwYUO4FAgDcx85jmbptxjp9uz1Z3maTnrm1pf59X0eCDpzK7rDz1ltvKScnR6GhoTp79qxuuukmNW3aVIGBgXrllVccUSMAwMUZhqFP1x9S/w/W63B6ruoE++vz0dF6oFsTLiuH09l9GstisWj58uVat26dtm/frpycHLVv314xMTGOqA8A4OIyzxbq6a+26/ud58ZtxrSqrTcHRSq4Kt0cuAaTYRiGs4twttJOEQ8AKGnbkQyNXbBFR06dlY+XSU/3baV/dG0ok4luDhyvtN/fdnd2pk+ffsnXTSaTqlSpoqZNm6pbt27y8vKyd9MAADdhGIZm/XxIU7/fo8JiQ3Wr++v9Ye0VVS/Y2aUBF7E77PzrX//SiRMnlJubq+rVq0uSTp8+rapVqyogIEBpaWlq3LixVq1apXr16pV7wQAA58rMLdT4L7fpf7tTJUl9rgnT6wMjZfH3cXJlwKXZPUD51VdfVadOnbRv3z6lp6crPT1dv/76qzp37qx3331XSUlJCgsL0+OPP+6IegEATpSQdFq3Tv9J/9udKl8vsybfcY0+vKc9QQcuze4xO02aNNFXX32la6+9tsTrCQkJGjBggH777TetX79eAwYMUHJycnnW6jCM2QGAv2YYhv7900G9/sNeFVkNNahRVe8Nba+2dS3OLg2VmMPG7CQnJ6uoqOii14uKimx3UI6IiFB2dra9mwYAuKDTZwr05BfbtGJvmiSpX2S4pvZvq6AqdHPgHuw+jdW9e3c9+OCDSkhIsL2WkJCgMWPG6JZbbpEk7dixQ40aNSq/KgEAThF/+JT6Tf9JK/amydfbrCl3tdF7Q9sRdOBW7A47n3zyiUJCQtShQwf5+fnJz89PHTt2VEhIiD755BNJUkBAgN56661yLxYAUDGsVkMfrj6gwR9t0PHMPDWqWU2LH7pe93RpwGXlcDtlvs/O3r179euvv0qSWrRooRYtWpRrYRWJMTsA8If0nHw98cU2rU48IUm6IypCr/ZvqwA/u0c+AA7lsDE757Vs2VItW7Ys69sBAC5o08FTenjBFqVm5cvP26wX77hGQzrVo5sDt1amsHP06FF9/fXXSkpKUkFBQYllb7/9drkUBgCoOFaroQ9W79fby3+V1ZAa16qm94e1V6twut1wf3aHnRUrVuiOO+5Q48aNtXfvXrVp00aHDh2SYRhq3769I2oEADjQyZx8Pf7ZVv2076QkqX+7Onr5rjaqxmkreAi7ByhPnDhRTz75pHbs2KEqVaroq6++0pEjR3TTTTdp0KBBjqgRAOAgcQfSdeu7P+mnfSdVxcesaQMj9dbgKIIOPIrdYWfPnj267777JEne3t46e/asAgIC9NJLL+n1118v9wIBAOWv2Gro3R/3afi/NygtO1/NQgP09dgbNLgj43PgeeyO7tWqVbON0wkPD9eBAwd0zTXXSJJOnjxZvtUBAMpdWnaeHv9sq37eny5JGtShribfeY2q+tLNgWey+ze7S5cuWrdunVq1aqVbb71VTzzxhHbs2KFFixapS5cujqgRAFBOft5/Uo8u3KqTOfny9/HSK3e3Uf/2dZ1dFuBQdoedt99+Wzk5OZKkyZMnKycnR5999pmaNWvGlVgA4KKKrYbeXbFPM1buk2FILWoH6v3h7dU0NMDZpQEOZ1fYKS4u1tGjRxUZGSnp3CmtmTNnOqQwAED5SM3K06MLE7Tht1OSpKHX1dMLt1+jKj5eTq4MqBh2hR0vLy/16tVLe/bsUXBwsINKAgCUl7W/ntDjn21V+pkCVfP10qv92+rOa+s4uyygQtl9GqtNmzb67bffmOgTAFxYUbFV//rxV32w+oAMQ2oVHqT3h7VT41qctkLlY3fYmTJlip588km9/PLL6tChg6pVq1ZiOXNLAYBzJWee1aMLtmrToXOnrYZ3rq9Jt7XmtBUqLbsnAjWb/7g1z5/vxWAYhkwmk4qLi8uvugrCRKAAPMWqxDSN+2yrTucWKsDPW68NaKvbIiOcXRbgEA6bCHTVqlVXVRgAoPwVFlv15v8S9dGa3yRJbeoE6b2h7dWwZrUrvBPwfHaHnZtuuskRdQAAyuhYxlk9siBB8YdPS5JGRDfQM/1ayc+b01aAVIbpIiTpp59+0j333KPrr79ex44dkyTNmTNH69atK9fiAAB/7cfdqeo3/SfFHz6twCre+nB4e02+sw1BB/gTu8POV199pd69e8vf319btmxRfn6+JCkzM1OvvvpquRcIALhYQZFVU5bt1j//s1kZuYWKqmvRtw/fqL5tw51dGuBy7A47U6ZM0cyZM/Xxxx/Lx8fH9nrXrl21ZcuWci0OAHCxI6dyNfijOP173UFJ0j+6NtIXo69X/RpVnVwZ4JrsDjuJiYnq1q3bRa9bLBZlZGTYta21a9fq9ttvV0REhEwmk5YsWVJiuWEYev755xUeHi5/f3/FxMRo3759JdY5deqUhg8frqCgIAUHB+v++++3TWcBAJ7mv7tS1G/6T9p6JENBVbz10b0d9PztreXrXaZRCUClYPe/jrCwMO3fv/+i19etW6fGjRvbta0zZ84oKipK77///iWXT5s2TdOnT9fMmTO1ceNGVatWTb1791ZeXp5tneHDh2vXrl1avny5li1bprVr1+qBBx6wb6cAwMUVFFk1+ZtdenBOvLLyinRtvWB9+8iN6n1NmLNLA1yfYadXX33VaN26tbFhwwYjMDDQ+Omnn4y5c+catWrVMqZPn27v5mwkGYsXL7Y9t1qtRlhYmPHGG2/YXsvIyDD8/PyMBQsWGIZhGLt37zYkGb/88ottne+//94wmUzGsWPHSv2zMzMzDUlGZmZmmesHAEc5fPKMcfuMn4wGE5YZDSYsM6Ys22XkFxY7uyzA6Ur7/W33pedPP/20rFarevToodzcXHXr1k1+fn568skn9fDDD5dbCDt48KBSUlIUExNje81isahz586Ki4vTkCFDFBcXp+DgYHXs2NG2TkxMjMxmszZu3Ki777673OoBAGf4fkeynvpyu7LzixRc1UdvDoxSTOvazi4LcCt2hx2TyaRnn31W48eP1/79+5WTk6PWrVsrIKB851tJSUmRJNWuXfIfde3atW3LUlJSFBoaWmK5t7e3QkJCbOtcSn5+vu0qMuncHRgBwJXkFRbr1e/26D9xhyVJHRpU1/Sh7VQn2N/JlQHux+4xO3PnzlVubq58fX3VunVrXXfddeUedBxt6tSpslgstke9evWcXRIA2Bw6eUYDPlxvCzoP3tRYCx/oQtABysjusPP4448rNDRUw4YN03fffeewubDCws4NuktNTS3xempqqm1ZWFiY0tLSSiwvKirSqVOnbOtcysSJE5WZmWl7HDlypJyrB4Cy+WFnim6bsU67jmepelUfzRrZSRP7tpKPF1dbAWVl97+e5ORkLVy4UCaTSYMHD1Z4eLhiY2O1fv36ci2sUaNGCgsL04oVK2yvZWVlaePGjYqOjpYkRUdHKyMjQ/Hx8bZ1Vq5cKavVqs6dO192235+fgoKCirxAABnKrYamvbDXo2eG6+c/CJ1alhd3z16o7q3DL3ymwH8JbvH7Hh7e+u2227TbbfdptzcXC1evFjz589X9+7dVbduXR04cKDU28rJySlxGfvBgwe1detWhYSEqH79+nrsscc0ZcoUNWvWTI0aNdKkSZMUERGhu+66S5LUqlUr9enTR6NGjdLMmTNVWFiosWPHasiQIYqIYJZfAO7h9JkCPbIwQT/tOylJuv+GRnq6b0u6OUA5sTvs/FnVqlXVu3dvnT59WocPH9aePXvsev/mzZvVvXt32/Nx48ZJkkaMGKHZs2frqaee0pkzZ/TAAw8oIyNDN9xwg3744QdVqVLF9p558+Zp7Nix6tGjh8xmswYMGKDp06dfzW4BQIXZeSxTo+fG6+jps/L38dJrA9rqzmvrOLsswKOYDMMw7H3T+Y7OvHnztGLFCtWrV09Dhw7V8OHD1bJlS0fU6VBZWVmyWCzKzMzklBaACvNV/FE9s3iH8ousalCjqmbe00GtwvkMAkqrtN/fdnd2hgwZomXLlqlq1aoaPHiwJk2aZBtDAwC4soIiq6Z8u9t2tVX3FrX0zt/ayVLV5wrvBFAWdocdLy8vff755+rdu7e8vLxKLNu5c6fatGlTbsUBgKdJzcrTQ/O2KP7waUnSoz2a6dEezWQ2m5xcGeC57A478+bNK/E8OztbCxYs0L///W/Fx8c77FJ0AHB3vxw6pYfmbdGJ7HwFVvHWO3+7Vj1acTdkwNHKPNR/7dq1GjFihMLDw/Xmm2/qlltu0YYNG8qzNgDwCIZh6NP1hzT0/23Qiex8tagdqK/H3kDQASqIXZ2dlJQUzZ49W5988omysrI0ePBg5efna8mSJWrdurWjagQAt3W2oFjPLt6hRQnHJEm3RYbr9QGRquZ3VRfDArBDqTs7t99+u1q0aKHt27frnXfe0fHjxzVjxgxH1gYAbu3IqVwN+HC9FiUck5fZpOf6tdKMoe0IOkAFK/W/uO+//16PPPKIxowZo2bNmjmyJgBwe2t/PaFHFiYoI7dQNar5asawdrq+SU1nlwVUSqXu7Kxbt07Z2dnq0KGDOnfurPfee08nT550ZG0A4HYMw9D7q/ZrxKxNysgtVFRdi755+AaCDuBEpQ47Xbp00ccff6zk5GQ9+OCDWrhwoSIiImS1WrV8+XJlZ2c7sk4AcHnZeYV6cE683vhvogxDGtKpnj57MFoRzFYOOFWZ7qB8XmJioj755BPNmTNHGRkZ6tmzp77++uvyrK9CcAdlAFdrf1q2HpgTr99OnJGvl1mT77xGQ6+r7+yyAI9W2u/vq5plrkWLFpo2bZqOHj2qBQsWXM2mAMBt/bAzWXe+97N+O3FG4ZYq+nx0NEEHcCFX1dnxFHR2AJRFsdXQm/9L1IerD0iSujQO0XvD2qtmgJ+TKwMqB4fNjQUAkE6dKdAjCxK0bv+5CzVG3dhIE/q0lLfXVTXMATgAYQcA7LTzWKYenBOvYxln5e/jpdcHRuqOqAhnlwXgMgg7AGCHL+OP6tnFO5RfZFXDGlU1894OahnG6W/AlRF2AKAUCoqsennZbs3ZcFiS1KNlqN7+27Wy+Ps4uTIAV0LYAYArSM3K05i58dqSlCGTSXqsR3M9fEtTmc0mZ5cGoBQIOwDwFzYdPKWH5m3RyZx8BVbx1rtDrtUtLZmtHHAnhB0AuATDMPTp+kOa8u0eFVkNtagdqI/u7aCGNas5uzQAdiLsAMAFzhYU65nFO7Q44Zgk6faoCL0+oK2q+vKRCbgj/uUCwJ8kpefqwbnx2pOcJS+zSRP7ttT9NzSSycT4HMBdEXYA4HerE9P06MKtyjxbqBrVfPXesPaKblLD2WUBuEqEHQCVntVq6IPV+/XW8l9lGFJUvWDNvKe9wi3MVg54AsIOgEotK69QT3y+Tct3p0qShl5XXy/e0Vp+3l5OrgxAeSHsAKi09qVm68E58frt5Bn5epn10p3XaAizlQMeh7ADoFL6bkeynvxim3ILihVuqaKZ93RQVL1gZ5cFwAEIOwAqlaJiq974X6I+WvObJCm6cQ3NGNZONQP8nFwZAEch7ACoNE6dKdDDC7bo5/3pkqQHujXWU71byNvL7OTKADgSYQdApbDjaKZGz43XsYyzqurrpWkDI3VbZISzywJQAQg7ADze55uP6LklO1VQZFXDGlX10b0d1SIs0NllAagghB0AHqugyKrJ3+zSvI1JkqSYVqF6a/C1svj7OLkyABWJsAPAI6Vk5mnMvHglJGXIZJIej2musd2bymxm2gegsiHsAPA4G39LV+z8BJ3MyVdQFW+9O6SdurcMdXZZAJyEsAPAYxiGodnrD+mVb/eoyGqoZVigPrq3gxrUqObs0gA4EWEHgEc4W1Cspxdt19KtxyVJd0RF6LUBbVXVl485oLLjUwCA20tKz9UDczZrb0q2vMwmPXNrK/2ja0OZTIzPAUDYAeDmViWm6dEFCcrKK1LNAF+9N6y9ujSu4eyyALgQwg4At2S1Gnp/1X69/eOvMgypXf1gfTC8vcIt/s4uDYCLIewAcDtZeYUa99k2/bgnVZI0vHN9PX97a/l5ezm5MgCuiLADwK38mpqtB+fE6+DJM/L1NmvKnW00uFM9Z5cFwIURdgC4jW+3J2v8l9uUW1CsCEsVzby3gyLrBju7LAAujrADwOUVFVv1xn8T9dHa3yRJ1zepoRlD26lGgJ+TKwPgDgg7AFxaek6+Hl6QoPUH0iVJD3ZrrPG9W8jby+zkygC4C8IOAJe1/WiGRs+J1/HMPFX19dIbA6PULzLc2WUBcDOEHQAu6fNfjui5pTtVUGRVo5rV9NG9HdS8dqCzywLghgg7AFxKflGxJn+zW/M3JkmSYlrV1tt/i1JQFR8nVwbAXRF2ALiM5MyzGjN3i7YeyZDJJI2Laa7Y7k1lNjPtA4CyI+wAcAkbfkvX2PlbdDKnQEFVvPXu0Hbq3iLU2WUB8ACEHQBOZRiG/u/nQ3r1uz0qthpqGRaoj+7toAY1qjm7NAAegrADwGlyC4o0cdEOLd16XJJ057UReq1/pPx9mfYBQPkh7ABwisPpZ/TgnHjtTcmWt9mkZ/u10sjrG8pkYnwOgPJF2AFQ4VbtTdOjCxOUlVekmgF+en9YO3VuXMPZZQHwUIQdABXGajU0Y+V+vbPiVxmG1K5+sD4c3kFhlirOLg2AByPsAKgQmWcLNe6zrVqxN02SdE+X+pp0W2v5eTM+B4BjEXYAOFxiSrZGz43XwZNn5Ott1pS72mhwx3rOLgtAJUHYAeBQy7Yf11NfblduQbHqBPtr5j0d1LauxdllAahECDsAHKKo2KrXf9irj386KEnq2rSGZgxtr5Bqvk6uDEBlQ9gBUO5O5uTr4fkJivstXZI0+qYmerJXc3l7mZ1cGYDKiLADoFxtPZKhMXPjlZyZp2q+XnpjUJRubRvu7LIAVGKEHQDl5rNfkjRpyS4VFFvVuGY1fXRvBzWrHejssgBUcoQdAFctv6hYL369Wws2JUmSeraurbcGRymoio+TKwMAwg6Aq5SceVaj527RtiMZMpmkJ3o210M3N5XZzLQPAFwDYQdAmcUdSNfY+VuUfqZAFn8fTR/aTjc1r+XssgCgBMIOALsZhqFP1h3U1O/3qthqqFV4kD66p4Pq16jq7NIA4CIufR3oiy++KJPJVOLRsmVL2/K8vDzFxsaqRo0aCggI0IABA5SamurEigHPl1tQpEcWbtWUb/eo2Gro7nZ1tGjM9QQdAC7L5Ts711xzjX788Ufbc2/vP0p+/PHH9e233+qLL76QxWLR2LFj1b9/f/3888/OKBXweIdOntGDc+KVmJotb7NJz/VrpRHXN5TJxPgcAK7L5cOOt7e3wsLCLno9MzNTn3zyiebPn69bbrlFkjRr1iy1atVKGzZsUJcuXSq6VMBjGYahZduT9cziHcrOK1KtQD99MLy9OjUMcXZpAHBFLn0aS5L27duniIgINW7cWMOHD1dS0rlLW+Pj41VYWKiYmBjbui1btlT9+vUVFxf3l9vMz89XVlZWiQeASzuRna8xc7fo4QUJys4rUocG1bXs4RsIOgDchkuHnc6dO2v27Nn64Ycf9OGHH+rgwYO68cYblZ2drZSUFPn6+io4OLjEe2rXrq2UlJS/3O7UqVNlsVhsj3r1mH0ZuJBhGPp623H1+tca/bArRd5mkx7t0UwLRnVR7aAqzi4PAErNpU9j9e3b1/bnyMhIde7cWQ0aNNDnn38uf3//Mm934sSJGjdunO15VlYWgQf4kxPZ+Zq0ZKd+2HXuPw6twoP05qBIXRPBbOUA3I9Lh50LBQcHq3nz5tq/f7969uypgoICZWRklOjupKamXnKMz5/5+fnJz8/PwdUC7ud8N+eFr3cpI7dQ3maTxt7SVA/d3FS+3i7dCAaAy3KrT6+cnBwdOHBA4eHh6tChg3x8fLRixQrb8sTERCUlJSk6OtqJVQLuKS07T6PnxuvRhVuVkVuoVuFBWjq2qx6LaU7QAeDWXLqz8+STT+r2229XgwYNdPz4cb3wwgvy8vLS0KFDZbFYdP/992vcuHEKCQlRUFCQHn74YUVHR3MlFmCHy3VzYrs3lY8XIQeA+3PpsHP06FENHTpU6enpqlWrlm644QZt2LBBtWqdux39v/71L5nNZg0YMED5+fnq3bu3PvjgAydXDbiPtOw8Pbd4p/63+9zNOFuHB+nNQVFqHRHk5MoAoPyYDMMwnF2Es2VlZclisSgzM1NBQXzIw/Ndqpvz8C3N9FD3JnRzALiN0n5/u3RnB0D5u7Cbc01EkN4YSDcHgOci7ACVhGEYWrr1XDcn82yhfLzOdXPG3Ew3B4BnI+wAlUBaVp6eXbJTy//UzXlzUJRahdPNAeD5CDuAB6ObAwCEHcBjpWXl6ZnFO/XjnnPdnDZ1zo3NoZsDoLIh7AAexjAMLdl6TC9+vdvWzXnklmYaTTcHQCVF2AE8yKW6OW8OilLLMLo5ACovwg7gAQzD0OKEY3rx613KyiuSj9e5GcofvIluDgAQdgA3l5qVp2cX79CPe9Ik0c0BgAsRdgA3RTcHAEqHsAO4odSsPD2zaIdW7D3XzWlbx6I3B0WpRVigkysDANdD2AHciGEYWrTlmCZ/80c357GY5nqwW2N5080BgEsi7ABu4sJuTmRdi94YSDcHAK6EsAO4OMMw9NWWY3rp926Or5dZj8Y0o5sDAKVE2AFcWEpmnp5ZvEMr6eYAQJkRdgAXdL6bM/mbXcqmmwMAV4WwA7iYlMw8TVy0XasST0iSoupa9MagKDWvTTcHAMqCsAO4CMMw9GX8Ub20bLetm/NYz2Z64Ea6OQBwNQg7gAu4VDfnzUFRakY3BwCuGmEHcCLDMPRF/FG9/KduzuM9m2vUjY3o5gBAOSHsAE6SnHlWExft0Gq6OQDgUIQdoILRzQGAikXYASpQcuZZPf3VDq359fduTr1gvTkwkm4OADgQYQeoAIZh6IvNv3dz8ovk623WuJ7N9c8b6OYAgKMRdgAHO55xbmzOn7s5bw2KVNNQujkAUBEIO4CDGIahzzcf0ZRle2zdnCd6Ntf9dHMAoEIRdgAHOJ5xVk8v2qG1v3dzrq0XrDfp5gCAUxB2gHJ0uW7OP29sLC+zydnlAUClRNgBysmxjLN6+qvt+mnfSUlSu/rBemNglJqGBji5MgCo3Ag7wFUyDEOf/XJEU77do5zfuzlP9mqu+2+gmwMAroCwA1wFujkA4PoIO0AZGIahhb8c0Su/d3P8vM16slcL/eOGRnRzAMDFEHYAO13YzWlfP1hvDIpSk1p0cwDAFRF2gFKimwMA7omwA5TC0dO5mrhoB90cAHBDhB3gLxiGoQWbjujV7/7o5ozv3UJ/70o3BwDcBWEHuIyjp3P19Fc7tG7/uW5OhwbVNW1gJN0cAHAzhB3gAoZhaP6mJL367R6dKSimmwMAbo6wA/zJpbo5bwyMVGO6OQDgtgg7gOjmAIAnI+yg0jtyKldPL9qun/enS5I6NqiuNwZFqVHNak6uDABQHgg7DnTw5Bn5+3ipZoCvvL3Mzi4HFzAMQ/M2Jmnqd+e6OVV8zBrfu6VGXt+Qbg4AeBDCjgPFztui3clZMpukmgF+qh1U5feHn8LO/9nyx3OLv49MJr5kK8KRU7ma8NV2rT9ANwcAPB1hx4FMJsnLbFKx1VBadr7SsvO141jmZdf38zbbwlDtoColA1Ggn8Is555X8fGqwL3wLFaroXmbznVzcn/v5jzVu6VG0M0BAI9lMgzDcHYRzpaVlSWLxaLMzEwFBQWV67aLrYbSz+QrLStfKZl5Ss3OU2pmnlKz8pWSlafU3x+ncwtLvU2Lv48tEP0Riv703FJFNapx6uxCF3ZzOjWsrmkD6eYAgLsq7fc3nR0H8zKbFBpYRaGBVdSmjuWy6+UVFutEdr5Ss/J+D0H5tiCUkpmntOxzYelsYbEyzxYq82yhfk3Nuez2zCapVuDFp85C/9QxCguqoiB/b48/dXa5bs7I6xvKTDcHADweYcdFVPHxUr2QqqoXUvWy6xiGoay8IqVdJhClZucrNTNPJ3LyVWw1fl+eL+nyp86q+Px+6iyw5Omy0As6Ru566uzIqVw99eV2xf12rptzXcMQTRsYqYZ0cwCg0iDsuBGTySSLv48s/j5qVjvwsusVWw2l5+RfdKrsXNco3xaWMnILlVdo1eH0XB1Oz/3Ln23x9/m9M+R32bFENQP8XGbci9VqaN7Gw5r6/V5bN2dCn5YaEU03BwAqG8KOB/IymxT6+ymrtvrrU2dpWflKzf69M2QLRX+EpJTMPOUXWW2nzhJTsy+7vfOnzkqeLis5lqh2oONPndHNAQD8GWGnEqvi46X6Naqqfo0rnDo7W3TZQHS+S3QiO19WQ/adOrONHfK74Pm5DpK9p86sVkNzNx7Wa3RzAAB/QtjBXzKZTLJU9ZGlqo+aX+HU2cmc/IvGD50fcJ32ezjKPFv6U2fBVX0uO5bofNeoxu+nzpLSc/XUV9u04bdTkqTrGoVo2gC6OQAAwg7KiZfZZOvMRNa9/Hp5hcUXd4b+FI7Onz7LL7IqI7dQGbl/ferMy2xSrQA/ZZwtUF6hVf4+XprQp4Xuo5sDAPgdYQcVqoqPlxrUqKYGNS7fcTEMQ5lnC0sOsP79HkUpmflK+/2U2snfrzpLycqTdK6b88bAyL/cNgCg8iHswOWYTCYFV/VVcFVftQi7/KmzomKrTuYUKDUrT0VWQ+3qBdPNAQBchLADt+XtZVaY5dxVXgAAXA7zCQAAAI9G2AEAAB6NsAMAADwaYQcAAHg0wg4AAPBohB0AAODRCDsAAMCjEXYAAIBHI+wAAACPRtgBAAAejekidG7iSUnKyspyciUAAKC0zn9vn/8evxzCjqTs7GxJUr169ZxcCQAAsFd2drYsFstll5uMK8WhSsBqter48eMKDAyUyVR+s2ZnZWWpXr16OnLkiIKCgsptuyiJ41xxONYVg+NcMTjOFcORx9kwDGVnZysiIkJm8+VH5tDZkWQ2m1W3bl2HbT8oKIh/SBWA41xxONYVg+NcMTjOFcNRx/mvOjrnMUAZAAB4NMIOAADwaIQdB/Lz89MLL7wgPz8/Z5fi0TjOFYdjXTE4zhWD41wxXOE4M0AZAAB4NDo7AADAoxF2AACARyPsAAAAj0bYAQAAHo2w40Dvv/++GjZsqCpVqqhz587atGmTs0tyay+++KJMJlOJR8uWLW3L8/LyFBsbqxo1aiggIEADBgxQamqqEyt2D2vXrtXtt9+uiIgImUwmLVmypMRywzD0/PPPKzw8XP7+/oqJidG+fftKrHPq1CkNHz5cQUFBCg4O1v3336+cnJwK3AvXd6XjPHLkyIt+v/v06VNiHY7zlU2dOlWdOnVSYGCgQkNDdddddykxMbHEOqX5rEhKSlK/fv1UtWpVhYaGavz48SoqKqrIXXFppTnON99880W/06NHjy6xTkUdZ8KOg3z22WcaN26cXnjhBW3ZskVRUVHq3bu30tLSnF2aW7vmmmuUnJxse6xbt8627PHHH9c333yjL774QmvWrNHx48fVv39/J1brHs6cOaOoqCi9//77l1w+bdo0TZ8+XTNnztTGjRtVrVo19e7dW3l5ebZ1hg8frl27dmn58uVatmyZ1q5dqwceeKCidsEtXOk4S1KfPn1K/H4vWLCgxHKO85WtWbNGsbGx2rBhg5YvX67CwkL16tVLZ86csa1zpc+K4uJi9evXTwUFBVq/fr0+/fRTzZ49W88//7wzdsklleY4S9KoUaNK/E5PmzbNtqxCj7MBh7juuuuM2NhY2/Pi4mIjIiLCmDp1qhOrcm8vvPCCERUVdcllGRkZho+Pj/HFF1/YXtuzZ48hyYiLi6ugCt2fJGPx4sW251ar1QgLCzPeeOMN22sZGRmGn5+fsWDBAsMwDGP37t2GJOOXX36xrfP9998bJpPJOHbsWIXV7k4uPM6GYRgjRoww7rzzzsu+h+NcNmlpaYYkY82aNYZhlO6z4rvvvjPMZrORkpJiW+fDDz80goKCjPz8/IrdATdx4XE2DMO46aabjEcfffSy76nI40xnxwEKCgoUHx+vmJgY22tms1kxMTGKi4tzYmXub9++fYqIiFDjxo01fPhwJSUlSZLi4+NVWFhY4pi3bNlS9evX55hfhYMHDyolJaXEcbVYLOrcubPtuMbFxSk4OFgdO3a0rRMTEyOz2ayNGzdWeM3ubPXq1QoNDVWLFi00ZswYpaen25ZxnMsmMzNTkhQSEiKpdJ8VcXFxatu2rWrXrm1bp3fv3srKytKuXbsqsHr3ceFxPm/evHmqWbOm2rRpo4kTJyo3N9e2rCKPMxOBOsDJkydVXFxc4i9QkmrXrq29e/c6qSr317lzZ82ePVstWrRQcnKyJk+erBtvvFE7d+5USkqKfH19FRwcXOI9tWvXVkpKinMK9gDnj92lfpfPL0tJSVFoaGiJ5d7e3goJCeHY26FPnz7q37+/GjVqpAMHDuiZZ55R3759FRcXJy8vL45zGVitVj322GPq2rWr2rRpI0ml+qxISUm55O/8+WUo6VLHWZKGDRumBg0aKCIiQtu3b9eECROUmJioRYsWSarY40zYgdvo27ev7c+RkZHq3LmzGjRooM8//1z+/v5OrAy4ekOGDLH9uW3btoqMjFSTJk20evVq9ejRw4mVua/Y2Fjt3LmzxNg+lL/LHec/jydr27atwsPD1aNHDx04cEBNmjSp0Bo5jeUANWvWlJeX10Wj+1NTUxUWFuakqjxPcHCwmjdvrv379yssLEwFBQXKyMgosQ7H/OqcP3Z/9bscFhZ20cD7oqIinTp1imN/FRo3bqyaNWtq//79kjjO9ho7dqyWLVumVatWqW7durbXS/NZERYWdsnf+fPL8IfLHedL6dy5sySV+J2uqONM2HEAX19fdejQQStWrLC9ZrVatWLFCkVHRzuxMs+Sk5OjAwcOKDw8XB06dJCPj0+JY56YmKikpCSO+VVo1KiRwsLCShzXrKwsbdy40XZco6OjlZGRofj4eNs6K1eulNVqtX24wX5Hjx5Venq6wsPDJXGcS8swDI0dO1aLFy/WypUr1ahRoxLLS/NZER0drR07dpQIl8uXL1dQUJBat25dMTvi4q50nC9l69atklTid7rCjnO5DneGzcKFCw0/Pz9j9uzZxu7du40HHnjACA4OLjHqHPZ54oknjNWrVxsHDx40fv75ZyMmJsaoWbOmkZaWZhiGYYwePdqoX7++sXLlSmPz5s1GdHS0ER0d7eSqXV92draRkJBgJCQkGJKMt99+20hISDAOHz5sGIZhvPbaa0ZwcLCxdOlSY/v27cadd95pNGrUyDh79qxtG3369DHatWtnbNy40Vi3bp3RrFkzY+jQoc7aJZf0V8c5OzvbePLJJ424uDjj4MGDxo8//mi0b9/eaNasmZGXl2fbBsf5ysaMGWNYLBZj9erVRnJysu2Rm5trW+dKnxVFRUVGmzZtjF69ehlbt241fvjhB6NWrVrGxIkTnbFLLulKx3n//v3GSy+9ZGzevNk4ePCgsXTpUqNx48ZGt27dbNuoyONM2HGgGTNmGPXr1zd8fX2N6667ztiwYYOzS3Jrf/vb34zw8HDD19fXqFOnjvG3v/3N2L9/v2352bNnjYceesioXr26UbVqVePuu+82kpOTnVixe1i1apUh6aLHiBEjDMM4d/n5pEmTjNq1axt+fn5Gjx49jMTExBLbSE9PN4YOHWoEBAQYQUFBxt///ncjOzvbCXvjuv7qOOfm5hq9evUyatWqZfj4+BgNGjQwRo0addF/jjjOV3apYyzJmDVrlm2d0nxWHDp0yOjbt6/h7+9v1KxZ03jiiSeMwsLCCt4b13Wl45yUlGR069bNCAkJMfz8/IymTZsa48ePNzIzM0tsp6KOs+n3ogEAADwSY3YAAIBHI+wAAACPRtgBAAAejbADAAA8GmEHAAB4NMIOAADwaIQdAADg0Qg7AADAoxF2ALiNEydOaMyYMapfv778/PwUFham3r176+eff5YkmUwmLVmyxLlFAnA53s4uAABKa8CAASooKNCnn36qxo0bKzU1VStWrFB6erqzSwPgwpguAoBbyMjIUPXq1bV69WrddNNNFy1v2LChDh8+bHveoEEDHTp0SJK0dOlSTZ48Wbt371ZERIRGjBihZ599Vt7e5/6/ZzKZ9MEHH+jrr7/W6tWrFR4ermnTpmngwIEVsm8AHIvTWADcQkBAgAICArRkyRLl5+dftPyXX36RJM2aNUvJycm25z/99JPuu+8+Pfroo9q9e7c++ugjzZ49W6+88kqJ90+aNEkDBgzQtm3bNHz4cA0ZMkR79uxx/I4BcDg6OwDcxldffaVRo0bp7Nmzat++vW666SYNGTJEkZGRks51aBYvXqy77rrL9p6YmBj16NFDEydOtL02d+5cPfXUUzp+/LjtfaNHj9aHH35oW6dLly5q3769Pvjgg4rZOQAOQ2cHgNsYMGCAjh8/rq+//lp9+vTR6tWr1b59e82ePfuy79m2bZteeuklW2coICBAo0aNUnJysnJzc23rRUdHl3hfdHQ0nR3AQzBAGYBbqVKlinr27KmePXtq0qRJ+uc//6kXXnhBI0eOvOT6OTk5mjx5svr373/JbQHwfHR2ALi11q1b68yZM5IkHx8fFRcXl1jevn17JSYmqmnTphc9zOY/PgI3bNhQ4n0bNmxQq1atHL8DAByOzg4At5Cenq5BgwbpH//4hyIjIxUYGKjNmzdr2rRpuvPOOyWduyJrxYoV6tq1q/z8/FS9enU9//zzuu2221S/fn0NHDhQZrNZ27Zt086dOzVlyhTb9r/44gt17NhRN9xwg+bNm6dNmzbpk08+cdbuAihHDFAG4Bby8/P14osv6n//+58OHDigwsJC1atXT4MGDdIzzzwjf39/ffPNNxo3bpwOHTqkOnXq2C49/+9//6uXXnpJCQkJ8vHxUcuWLfXPf/5To0aNknRugPL777+vJUuWaO3atQoPD9frr7+uwYMHO3GPAZQXwg6ASu9SV3EB8ByM2QEAAB6NsAMAADwaA5QBVHqczQc8G50dAADg0Qg7AADAoxF2AACARyPsAAAAj0bYAQAAHo2wAwAAPBphBwAAeDTCDgAA8GiEHQAA4NH+P+b7WbQImSwBAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "#@test {\"skip\": true}\n", "\n", "steps = range(0, num_iterations + 1, eval_interval)\n", "plt.plot(steps, returns)\n", "plt.ylabel('Average Return')\n", "plt.xlabel('Step')\n", "plt.ylim(top=250)" ] }, { "cell_type": "markdown", "metadata": { "id": "M7-XpPP99Cy7" }, "source": [ "### Videos" ] }, { "cell_type": "markdown", "metadata": { "id": "9pGfGxSH32gn" }, "source": [ "It is helpful to visualize the performance of an agent by rendering the environment at each step. Before we do that, let us first create a function to embed videos in this colab." ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:07:49.524818Z", "iopub.status.busy": "2023-12-22T14:07:49.524272Z", "iopub.status.idle": "2023-12-22T14:07:49.528329Z", "shell.execute_reply": "2023-12-22T14:07:49.527677Z" }, "id": "ULaGr8pvOKbl" }, "outputs": [], "source": [ "def embed_mp4(filename):\n", " \"\"\"Embeds an mp4 file in the notebook.\"\"\"\n", " video = open(filename,'rb').read()\n", " b64 = base64.b64encode(video)\n", " tag = '''\n", " '''.format(b64.decode())\n", "\n", " return IPython.display.HTML(tag)" ] }, { "cell_type": "markdown", "metadata": { "id": "9c_PH-pX4Pr5" }, "source": [ "The following code visualizes the agent's policy for a few episodes:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2023-12-22T14:07:49.531508Z", "iopub.status.busy": "2023-12-22T14:07:49.531193Z", "iopub.status.idle": "2023-12-22T14:07:56.632628Z", "shell.execute_reply": "2023-12-22T14:07:56.631727Z" }, "id": "owOVWB158NlF" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (400, 600) to (400, 608) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[swscaler @ 0x5563cf186880] Warning: data is not aligned! This can lead to a speed loss\n" ] }, { "data": { "text/html": [ "\n", " " ], "text/plain": [ "" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "num_episodes = 3\n", "video_filename = 'imageio.mp4'\n", "with imageio.get_writer(video_filename, fps=60) as video:\n", " for _ in range(num_episodes):\n", " time_step = eval_env.reset()\n", " video.append_data(eval_py_env.render())\n", " while not time_step.is_last():\n", " action_step = tf_agent.policy.action(time_step)\n", " time_step = eval_env.step(action_step.action)\n", " video.append_data(eval_py_env.render())\n", "\n", "embed_mp4(video_filename)" ] } ], "metadata": { "accelerator": "GPU", "colab": { "name": "6_reinforce_tutorial.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" } }, "nbformat": 4, "nbformat_minor": 0 }