{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CGyzr0tfeUTQ"
   },
   "source": [
    "**Copyright 2021 The TensorFlow Hub Authors.**\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-11T17:47:25.488489Z",
     "iopub.status.busy": "2024-01-11T17:47:25.487986Z",
     "iopub.status.idle": "2024-01-11T17:47:25.492049Z",
     "shell.execute_reply": "2024-01-11T17:47:25.491409Z"
    },
    "id": "zV1OQAGReaGQ"
   },
   "outputs": [],
   "source": [
    "# Copyright 2021 The TensorFlow Hub Authors. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# =============================================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L5bsDhkRfTpq"
   },
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>     <a target=\"_blank\" href=\"https://www.tensorflow.org/hub/tutorials/senteval_for_universal_sentence_encoder_cmlm\">     <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\">     TensorFlow.org で表示</a>\n",
    "</td>\n",
    "  <td>     <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ja/hub/tutorials/senteval_for_universal_sentence_encoder_cmlm.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Google Colabで実行</a>\n",
    "</td>\n",
    "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/ja/hub/tutorials/senteval_for_universal_sentence_encoder_cmlm.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">GitHub でソースを表示</a></td>\n",
    "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/ja/hub/tutorials/senteval_for_universal_sentence_encoder_cmlm.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\">ノートブックをダウンロード</a></td>\n",
    "  <td><a href=\"https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1\"><img src=\"https://www.tensorflow.org/images/hub_logo_32px.png\"> TF Hub モデルを参照</a></td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "owWqOcw1e-RZ"
   },
   "source": [
    "# ユニバーサルセンテンスエンコーダ SentEval の実演\n",
    "\n",
    "この Colab は、文の埋め込みの品質を測定するためのライブラリである [SentEval](https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1) ツールキットを使用した[ユニバーサルセンテンスエンコーダ CMLM モデル](https://github.com/facebookresearch/SentEval)を説明します。 SentEval ツールキットには、埋め込みモデルの一般化能力を評価し、エンコードされた言語プロパティを評価できる、さまざまな下流タスクのセットが含まれています。\n",
    "\n",
    "最初の 2 つのコードブロックを実行して環境をセットアップします。3 番目のコードブロックでは、SentEval タスクを選択してモデルを評価できます。この Colab を実行するには、GPU ランタイムをお勧めします。\n",
    "\n",
    "ユニバーサルセンテンスエンコーダ CMLM モデルの詳細については、https://openreview.net/forum?id=WDVD4lUCTzU を参照してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-11T17:47:25.495594Z",
     "iopub.status.busy": "2024-01-11T17:47:25.495158Z",
     "iopub.status.idle": "2024-01-11T17:49:22.730259Z",
     "shell.execute_reply": "2024-01-11T17:49:22.729276Z"
    },
    "id": "-CerULCLsjzV"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n",
      "tensorflow-datasets 4.9.3 requires protobuf>=3.20, but you have protobuf 3.19.6 which is incompatible.\r\n",
      "tensorflow-metadata 1.14.0 requires protobuf<4.21,>=3.20.3, but you have protobuf 3.19.6 which is incompatible.\u001b[0m\u001b[31m\r\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "#@title Install dependencies\n",
    "!pip install --quiet \"tensorflow-text==2.11.*\"\n",
    "!pip install --quiet torch==1.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LjqkqD6aiZGU"
   },
   "source": [
    "## SentEvalとタスクデータをダウンロードする\n",
    "\n",
    "このステップでは、github から SentEval をダウンロードし、データスクリプトを実行してタスクデータをダウンロードします。完了するまでに最大 5 分かかる場合があります。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-11T17:49:22.734740Z",
     "iopub.status.busy": "2024-01-11T17:49:22.734479Z",
     "iopub.status.idle": "2024-01-11T17:50:24.528395Z",
     "shell.execute_reply": "2024-01-11T17:50:24.527337Z"
    },
    "id": "3UwhHQiKJmSc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'SentEval'...\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "remote: Enumerating objects: 691, done.\u001b[K\r\n",
      "remote: Counting objects:  50% (1/2)\u001b[K\r",
      "remote: Counting objects: 100% (2/2)\u001b[K\r",
      "remote: Counting objects: 100% (2/2), done.\u001b[K\r\n",
      "remote: Compressing objects:  50% (1/2)\u001b[K\r",
      "remote: Compressing objects: 100% (2/2)\u001b[K\r",
      "remote: Compressing objects: 100% (2/2), done.\u001b[K\r\n",
      "Receiving objects:   0% (1/691)\r",
      "Receiving objects:   1% (7/691)\r",
      "Receiving objects:   2% (14/691)\r",
      "Receiving objects:   3% (21/691)\r",
      "Receiving objects:   4% (28/691)\r",
      "Receiving objects:   5% (35/691)\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Receiving objects:   6% (42/691)\r",
      "Receiving objects:   7% (49/691)\r",
      "Receiving objects:   8% (56/691)\r",
      "Receiving objects:   9% (63/691)\r",
      "Receiving objects:  10% (70/691)\r",
      "Receiving objects:  11% (77/691)\r",
      "Receiving objects:  12% (83/691)\r",
      "Receiving objects:  13% (90/691)\r",
      "Receiving objects:  14% (97/691)\r",
      "Receiving objects:  15% (104/691)\r",
      "Receiving objects:  16% (111/691)\r",
      "Receiving objects:  17% (118/691)\r",
      "Receiving objects:  18% (125/691)\r",
      "Receiving objects:  19% (132/691)\r",
      "Receiving objects:  20% (139/691)\r",
      "Receiving objects:  21% (146/691)\r",
      "Receiving objects:  22% (153/691)\r",
      "Receiving objects:  23% (159/691)\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Receiving objects:  24% (166/691), 11.57 MiB | 23.12 MiB/s\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Receiving objects:  24% (168/691), 26.68 MiB | 26.67 MiB/s\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Receiving objects:  25% (173/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  26% (180/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  27% (187/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  28% (194/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  29% (201/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  30% (208/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  31% (215/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  32% (222/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  33% (229/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  34% (235/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  35% (242/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  36% (249/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  37% (256/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  38% (263/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  39% (270/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  40% (277/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  41% (284/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  42% (291/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  43% (298/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  44% (305/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  45% (311/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  46% (318/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  47% (325/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  48% (332/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  49% (339/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  50% (346/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  51% (353/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  52% (360/691), 26.68 MiB | 26.67 MiB/s\r",
      "remote: Total 691 (delta 0), reused 2 (delta 0), pack-reused 689\u001b[K\r\n",
      "Receiving objects:  53% (367/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  54% (374/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  55% (381/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  56% (387/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  57% (394/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  58% (401/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  59% (408/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  60% (415/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  61% (422/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  62% (429/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  63% (436/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  64% (443/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  65% (450/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  66% (457/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  67% (463/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  68% (470/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  69% (477/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  70% (484/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  71% (491/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  72% (498/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  73% (505/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  74% (512/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  75% (519/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  76% (526/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  77% (533/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  78% (539/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  79% (546/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  80% (553/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  81% (560/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  82% (567/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  83% (574/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  84% (581/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  85% (588/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  86% (595/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  87% (602/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  88% (609/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  89% (615/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  90% (622/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  91% (629/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  92% (636/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  93% (643/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  94% (650/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  95% (657/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  96% (664/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  97% (671/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  98% (678/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects:  99% (685/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects: 100% (691/691), 26.68 MiB | 26.67 MiB/s\r",
      "Receiving objects: 100% (691/691), 33.25 MiB | 27.39 MiB/s, done.\r\n",
      "Resolving deltas:   0% (0/434)\r",
      "Resolving deltas:   1% (6/434)\r",
      "Resolving deltas:   2% (11/434)\r",
      "Resolving deltas:   3% (17/434)\r",
      "Resolving deltas:   7% (32/434)\r",
      "Resolving deltas:   9% (40/434)\r",
      "Resolving deltas:  11% (52/434)\r",
      "Resolving deltas:  12% (55/434)\r",
      "Resolving deltas:  15% (67/434)\r",
      "Resolving deltas:  17% (77/434)\r",
      "Resolving deltas:  18% (79/434)\r",
      "Resolving deltas:  19% (85/434)\r",
      "Resolving deltas:  21% (93/434)\r",
      "Resolving deltas:  23% (100/434)\r",
      "Resolving deltas:  27% (121/434)\r",
      "Resolving deltas:  35% (155/434)\r",
      "Resolving deltas:  36% (157/434)\r",
      "Resolving deltas:  38% (166/434)\r",
      "Resolving deltas:  42% (183/434)\r",
      "Resolving deltas:  43% (187/434)\r",
      "Resolving deltas:  44% (194/434)\r",
      "Resolving deltas:  45% (196/434)\r",
      "Resolving deltas:  47% (205/434)\r",
      "Resolving deltas:  51% (224/434)\r",
      "Resolving deltas:  53% (231/434)\r",
      "Resolving deltas:  54% (238/434)\r",
      "Resolving deltas:  55% (239/434)\r",
      "Resolving deltas:  57% (248/434)\r",
      "Resolving deltas:  59% (259/434)\r",
      "Resolving deltas:  60% (264/434)\r",
      "Resolving deltas:  61% (267/434)\r",
      "Resolving deltas:  62% (270/434)\r",
      "Resolving deltas:  63% (277/434)\r",
      "Resolving deltas:  64% (278/434)\r",
      "Resolving deltas:  65% (285/434)\r",
      "Resolving deltas:  67% (291/434)\r",
      "Resolving deltas:  68% (298/434)\r",
      "Resolving deltas:  69% (300/434)\r",
      "Resolving deltas:  70% (304/434)\r",
      "Resolving deltas:  73% (320/434)\r",
      "Resolving deltas:  77% (336/434)\r",
      "Resolving deltas:  79% (347/434)\r",
      "Resolving deltas:  80% (350/434)\r",
      "Resolving deltas:  86% (377/434)\r",
      "Resolving deltas:  87% (378/434)\r",
      "Resolving deltas:  88% (383/434)\r",
      "Resolving deltas:  90% (392/434)\r",
      "Resolving deltas:  91% (399/434)\r",
      "Resolving deltas:  92% (402/434)\r",
      "Resolving deltas:  94% (412/434)\r",
      "Resolving deltas:  97% (424/434)\r",
      "Resolving deltas:  98% (429/434)\r",
      "Resolving deltas: 100% (434/434)\r",
      "Resolving deltas: 100% (434/434), done.\r\n"
     ]
    }
   ],
   "source": [
    "#@title Install SentEval and download task data\n",
    "!rm -rf ./SentEval\n",
    "!git clone https://github.com/facebookresearch/SentEval.git\n",
    "!cd $PWD/SentEval/data/downstream && bash get_transfer_data.bash > /dev/null 2>&1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7a2ohPn8vMe2"
   },
   "source": [
    "＃SentEval 評価タスクの実行。次のコードブロックは SentEval タスクを実行して結果を出力し、次のタスクのいずれかを選択して USE CMLM モデルを評価します。\n",
    "\n",
    "```\n",
    "MR\tCR\tSUBJ\tMPQA\tSST\tTREC\tMRPC\tSICK-E\n",
    "```\n",
    "\n",
    "実行するモデル、パラメーター、およびタスクを選択します。ラピッドプロトタイピングパラメータは、計算時間を短縮するために使用でき、より速く結果を得ることができます。\n",
    "\n",
    "通常、**'rapid prototyping'** パラメータを使用してタスクを完了するには 5〜15 分かかり、**'slower, best performance'** パラメータでは最大 1 時間かかります。\n",
    "\n",
    "```\n",
    "params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}\n",
    "params['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,\n",
    "                                 'tenacity': 3, 'epoch_size': 2}\n",
    "```\n",
    "\n",
    "より良い結果を得るには、より遅い **'slower, best performance'** パラメータを使用してください。計算には最大 1 時間かかる場合があります。\n",
    "\n",
    "```\n",
    "params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}\n",
    "params['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,\n",
    "                                 'tenacity': 5, 'epoch_size': 6}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-11T17:50:24.533079Z",
     "iopub.status.busy": "2024-01-11T17:50:24.532791Z",
     "iopub.status.idle": "2024-01-11T17:54:25.079775Z",
     "shell.execute_reply": "2024-01-11T17:54:25.078965Z"
    },
    "id": "nenCcawjwowt"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.\n",
      "Instructions for updating:\n",
      "Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.\n",
      "Instructions for updating:\n",
      "Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating task CR with rapid prototyping parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 7s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 8s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 5s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 5s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 5s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 5s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 5s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 5s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 7s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/4 [======>.......................] - ETA: 4s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/4 [==============>...............] - ETA: 3s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "3/4 [=====================>........] - ETA: 1s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "4/4 [==============================] - 6s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/2 [==============>...............] - ETA: 2s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/2 [==============================] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/2 [==============================] - 4s 2s/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time took on task CR : 223.1. seconds\n",
      "{'devacc': 90.42, 'acc': 88.98, 'ndev': 3775, 'ntest': 3775}\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
    "\n",
    "import sys\n",
    "sys.path.append(f'{os.getcwd()}/SentEval')\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# Prevent TF from claiming all GPU memory so there is some left for pytorch.\n",
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "  # Memory growth needs to be the same across GPUs.\n",
    "  for gpu in gpus:\n",
    "    tf.config.experimental.set_memory_growth(gpu, True)\n",
    "\n",
    "import tensorflow_hub as hub\n",
    "import tensorflow_text\n",
    "import senteval\n",
    "import time\n",
    "\n",
    "PATH_TO_DATA = f'{os.getcwd()}/SentEval/data'\n",
    "MODEL = 'https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1' #@param ['https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1', 'https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-large/1']\n",
    "PARAMS = 'rapid prototyping' #@param ['slower, best performance', 'rapid prototyping']\n",
    "TASK = 'CR' #@param ['CR','MR', 'MPQA', 'MRPC', 'SICKEntailment', 'SNLI', 'SST2', 'SUBJ', 'TREC']\n",
    "\n",
    "params_prototyping = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}\n",
    "params_prototyping['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,\n",
    "                                 'tenacity': 3, 'epoch_size': 2}\n",
    "\n",
    "params_best = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}\n",
    "params_best['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,\n",
    "                                 'tenacity': 5, 'epoch_size': 6}\n",
    "\n",
    "params = params_best if PARAMS == 'slower, best performance' else params_prototyping\n",
    "\n",
    "preprocessor = hub.KerasLayer(\n",
    "    \"https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3\")\n",
    "encoder = hub.KerasLayer(\n",
    "    \"https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1\")\n",
    "\n",
    "inputs = tf.keras.Input(shape=tf.shape(''), dtype=tf.string)\n",
    "outputs = encoder(preprocessor(inputs))\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "\n",
    "def prepare(params, samples):\n",
    "    return\n",
    "\n",
    "def batcher(_, batch):\n",
    "    batch = [' '.join(sent) if sent else '.' for sent in batch]\n",
    "    return model.predict(tf.constant(batch))[\"default\"]\n",
    "\n",
    "\n",
    "se = senteval.engine.SE(params, batcher, prepare)\n",
    "print(\"Evaluating task %s with %s parameters\" % (TASK, PARAMS))\n",
    "start = time.time()\n",
    "results = se.eval(TASK)\n",
    "end = time.time()\n",
    "print('Time took on task %s : %.1f. seconds' % (TASK, end - start))\n",
    "print(results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SNvsY6Hsvs0_"
   },
   "source": [
    "詳細情報\n",
    "\n",
    "- その他のテキスト埋め込みモデルは、[TensorFlow Hub](https://tfhub.dev) をご覧ください。\n",
    "- [多言語ユニバーサルセンテンスエンコーダ CMLM モデル](https://tfhub.dev/google/universal-sentence-encoder-cmlm/multilingual-base-br/1)もご覧ください。\n",
    "- 他の[ユニバーサルセンテンスエンコーダモデル](https://tfhub.dev/google/collections/universal-sentence-encoder/1)を確認してください\n",
    "\n",
    "## リファレンス\n",
    "\n",
    "- Ziyi Yang、Yinfei Yang、Daniel Cer、Jax Law、Eric Darve。[Universal Sentence Representations Learning with Conditional Masked Language Model. November 2020](https://openreview.net/forum?id=WDVD4lUCTzU)\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "senteval_for_universal_sentence_encoder_cmlm.ipynb",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}