{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "hX4n9TsbGw-f" }, "source": [ "##### Copyright 2020 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2022-12-14T23:26:56.481851Z", "iopub.status.busy": "2022-12-14T23:26:56.481352Z", "iopub.status.idle": "2022-12-14T23:26:56.485333Z", "shell.execute_reply": "2022-12-14T23:26:56.484806Z" }, "id": "0nbI5DtDGw-i" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "AOpGoE2T-YXS" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
TensorFlow.org で表示\n", " Google Colab で実行\n", " GitHubでソースを表示ノートブックをダウンロード
" ] }, { "cell_type": "markdown", "metadata": { "id": "haJUNjSB60Kh" }, "source": [ "# word2vec" ] }, { "cell_type": "markdown", "metadata": { "id": "99d4ky2lWFvn" }, "source": [ "word2vec は単一のアルゴリズムではなく、大規模なデータセットから単語の埋め込みを学習するために使用できるモデルアーキテクチャと最適化のファミリです。word2vec により学習された埋め込みは、さまざまなダウンストリームの自然言語処理タスクで成功することが証明されています。\n", "\n", "注意: このチュートリアルは、[ベクトル空間での単語表現の効率的な推定](https://arxiv.org/pdf/1301.3781.pdf)と[単語とフレーズの分散表現とその構成](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)に基づいていますが、論文の正確な実装ではなく、重要なアイデアを説明することを目的としています。\n", "\n", "これらの論文では、単語の表現を学習するための 2 つの方法が提案されています。\n", "\n", "- **連続バッグオブワードモデル**では、周囲のコンテキストワードに基づいて中間の単語を予測します。コンテキストは、与えられた (中間) 単語の前後のいくつかの単語で構成されます。このアーキテクチャでは、コンテキスト内の単語の順序が重要ではないため、バッグオブワードモデルと呼ばれます。\n", "- **連続スキップグラムモデル**は、同じ文の与えられた単語の前後の特定の範囲内の単語を予測します。この例を以下に示します。\n", "\n", "このチュートリアルでは、スキップグラムアプローチを使用します。最初に、説明のために 1 つの文を使用して、スキップグラムとその他の概念について説明します。次に、小さなデータセットで独自の word2vec モデルをトレーニングします。このチュートリアルには、トレーニング済みの埋め込みをエクスポートして [TensorFlow Embedding Projector](http://projector.tensorflow.org/) で可視化するためのコードも含まれています。\n" ] }, { "cell_type": "markdown", "metadata": { "id": "xP00WlaMWBZC" }, "source": [ "## スキップグラムとネガティブサンプリング " ] }, { "cell_type": "markdown", "metadata": { "id": "Zr2wjv0bW236" }, "source": [ "バッグオブワードモデルは、与えられたコンテキスト (前後の単語) から単語を予測しますが、スキップグラムモデルは、与えられた単語自体から単語のコンテキスト (前後の単語) を予測します。モデルは、トークンをスキップできる n-gram であるスキップグラムでトレーニングされます (例については、下の図を参照してください)。単語のコンテキストは、`context_word` が `target_word` の前後のコンテキストに現れる `(target_word, context_word)` の一連のスキップグラムペアによって表すことができます。 " ] }, { "cell_type": "markdown", "metadata": { "id": "ICjc-McbaVTd" }, "source": [ "次の 8 つの単語の文を考えてみましょう。\n", "\n", "> The wide road shimmered in the hot sun.\n", "\n", "この文の 8 つの単語のそれぞれのコンテキストワードは、ウィンドウサイズによって定義されます。ウィンドウサイズは、`context word` と見なすことができる `target_word` の前後の単語の範囲を指定します。以下は、さまざまなウィンドウサイズに基づくターゲットワードのスキップグラムの表です。" ] }, { "cell_type": "markdown", "metadata": { "id": "YKE87IKT_YT8" }, "source": [ "注意: このチュートリアルでは、ウィンドウサイズ `n` は、前後に n 個の単語があり、合計ウィンドウ 範囲が 2*n+1 個の単語であるということを意味します。" ] }, { "cell_type": "markdown", "metadata": { "id": "RsCwQ07E8mqU" }, "source": [ "![word2vec_skipgrams](https://tensorflow.org/tutorials/text/images/word2vec_skipgram.png)" ] }, { "cell_type": "markdown", "metadata": { "id": "gK1gN1jwkMpU" }, "source": [ "スキップグラムモデルのトレーニングの目的は、与えられたターゲットワードからコンテキストワードを予測する確率を最大化することです。一連の単語 *w1、w2、... wT* の場合、目的は平均対数確率として記述できます。" ] }, { "cell_type": "markdown", "metadata": { "id": "pILO_iAc84e-" }, "source": [ "![word2vec_skipgram_objective](https://tensorflow.org/tutorials/text/images/word2vec_skipgram_objective.png)" ] }, { "cell_type": "markdown", "metadata": { "id": "Gsy6TUbtnz_K" }, "source": [ "ここで、`c` はトレーニングコンテキストのサイズです。基本的なスキップグラムの定式化では、ソフトマックス関数を使用してこの確率を定義します。" ] }, { "cell_type": "markdown", "metadata": { "id": "P81Qavbb9APd" }, "source": [ "![word2vec_full_softmax](https://tensorflow.org/tutorials/text/images/word2vec_full_softmax.png)" ] }, { "cell_type": "markdown", "metadata": { "id": "axZvd-hhotVB" }, "source": [ "ここで、*v* と *v'* は単語のターゲットとコンテキストのベクトル表現であり、*W* 語彙サイズです。 " ] }, { "cell_type": "markdown", "metadata": { "id": "SoLzxbqSpT6_" }, "source": [ "この定式化の分母を計算するには、語彙全体に対して完全なソフトマックスを実行する必要があります。これは、多くの場合、大きな項 (105-107) です。" ] }, { "cell_type": "markdown", "metadata": { "id": "Y5VWYtmFzHkU" }, "source": [ "[ノイズコントラスト推定](https://www.tensorflow.org/api_docs/python/tf/nn/nce_loss) (NCE) 損失関数は、完全なソフトマックスの効率的な近似値です。単語の分布をモデル化するのではなく、単語の埋め込みを学習することを目的として、NCE 損失を[単純化](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)してネガティブサンプリングを使用することができます。 " ] }, { "cell_type": "markdown", "metadata": { "id": "WTZBPf1RsOsg" }, "source": [ "ターゲットワードの単純化されたネガティブサンプリングの目的は、コンテキストワードをノイズ分布 *Pn(w)* ワードから抽出された `num_ns` のネガティブサンプルから区別することです。より正確には、語彙全体の完全なソフトマックスの効率的な近似は、スキップグラムペアの場合、コンテキストワードと `num_ns` ネガティブサンプル間の分類問題としてターゲットワードの損失を提示します。 " ] }, { "cell_type": "markdown", "metadata": { "id": "Cl0rSfHjt6Mf" }, "source": [ "ネガティブサンプルは、`context_word` が `target_word` の `window_size` の前後に現れないように、`(target_word, context_word)` ペアとして定義されます。この例の文の場合、以下はいくつかの潜在的なネガティブサンプルです (`window_size` が `2` の場合)。\n", "\n", "```\n", "(hot, shimmered)\n", "(wide, hot)\n", "(wide, sun)\n", "```" ] }, { "cell_type": "markdown", "metadata": { "id": "kq0q2uqbucFg" }, "source": [ "次のセクションでは、1 つの文に対してスキップグラムとネガティブサンプルを生成します。また、サブサンプリング手法についても学習し、チュートリアルの後半でポジティブトレーニングとネガティブトレーニングサンプルの分類モデルをトレーニングします。" ] }, { "cell_type": "markdown", "metadata": { "id": "mk4-Hpe1CH16" }, "source": [ "## セットアップ" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:56.490162Z", "iopub.status.busy": "2022-12-14T23:26:56.489695Z", "iopub.status.idle": "2022-12-14T23:26:58.437734Z", "shell.execute_reply": "2022-12-14T23:26:58.437016Z" }, "id": "RutaI-Tpev3T" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-12-14 23:26:57.448971: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n", "2022-12-14 23:26:57.449067: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n", "2022-12-14 23:26:57.449077: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n" ] } ], "source": [ "import io\n", "import re\n", "import string\n", "import tqdm\n", "\n", "import numpy as np\n", "\n", "import tensorflow as tf\n", "from tensorflow.keras import layers" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:58.441947Z", "iopub.status.busy": "2022-12-14T23:26:58.441539Z", "iopub.status.idle": "2022-12-14T23:26:58.446684Z", "shell.execute_reply": "2022-12-14T23:26:58.446118Z" }, "id": "10pyUMFkGKVQ" }, "outputs": [], "source": [ "# Load the TensorBoard notebook extension\n", "%load_ext tensorboard" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:58.449628Z", "iopub.status.busy": "2022-12-14T23:26:58.449380Z", "iopub.status.idle": "2022-12-14T23:26:58.452455Z", "shell.execute_reply": "2022-12-14T23:26:58.451804Z" }, "id": "XkJ5299Tek6B" }, "outputs": [], "source": [ "SEED = 42\n", "AUTOTUNE = tf.data.AUTOTUNE" ] }, { "cell_type": "markdown", "metadata": { "id": "RW-g5buCHwh3" }, "source": [ "### 例文をベクトル化する" ] }, { "cell_type": "markdown", "metadata": { "id": "y8TfZIgoQrcP" }, "source": [ "次の文を考えてみましょう。\n", "\n", "> The wide road shimmered in the hot sun.\n", "\n", "文をトークン化します。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:58.455729Z", "iopub.status.busy": "2022-12-14T23:26:58.455262Z", "iopub.status.idle": "2022-12-14T23:26:58.458757Z", "shell.execute_reply": "2022-12-14T23:26:58.458217Z" }, "id": "bsl7jBzV6_KK" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "8\n" ] } ], "source": [ "sentence = \"The wide road shimmered in the hot sun\"\n", "tokens = list(sentence.lower().split())\n", "print(len(tokens))" ] }, { "cell_type": "markdown", "metadata": { "id": "PU-bs1XtThEw" }, "source": [ "トークンから整数インデックスへのマッピングを保存する語彙を作成します。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:58.462206Z", "iopub.status.busy": "2022-12-14T23:26:58.461629Z", "iopub.status.idle": "2022-12-14T23:26:58.465718Z", "shell.execute_reply": "2022-12-14T23:26:58.465176Z" }, "id": "UdYv1HJUQ8XA" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'': 0, 'the': 1, 'wide': 2, 'road': 3, 'shimmered': 4, 'in': 5, 'hot': 6, 'sun': 7}\n" ] } ], "source": [ "vocab, index = {}, 1 # start indexing from 1\n", "vocab[''] = 0 # add a padding token\n", "for token in tokens:\n", " if token not in vocab:\n", " vocab[token] = index\n", " index += 1\n", "vocab_size = len(vocab)\n", "print(vocab)" ] }, { "cell_type": "markdown", "metadata": { "id": "ZpuP43Dddasr" }, "source": [ "整数インデックスからトークンへのマッピングを保存する逆語彙を作成します。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:58.469089Z", "iopub.status.busy": "2022-12-14T23:26:58.468446Z", "iopub.status.idle": "2022-12-14T23:26:58.472065Z", "shell.execute_reply": "2022-12-14T23:26:58.471480Z" }, "id": "o9ULAJYtEvKl" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{0: '', 1: 'the', 2: 'wide', 3: 'road', 4: 'shimmered', 5: 'in', 6: 'hot', 7: 'sun'}\n" ] } ], "source": [ "inverse_vocab = {index: token for token, index in vocab.items()}\n", "print(inverse_vocab)" ] }, { "cell_type": "markdown", "metadata": { "id": "n3qtuyxIRyii" }, "source": [ "文をベクトル化します。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:58.475373Z", "iopub.status.busy": "2022-12-14T23:26:58.474876Z", "iopub.status.idle": "2022-12-14T23:26:58.478238Z", "shell.execute_reply": "2022-12-14T23:26:58.477665Z" }, "id": "CsB3-9uQQYyl" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1, 2, 3, 4, 5, 1, 6, 7]\n" ] } ], "source": [ "example_sequence = [vocab[word] for word in tokens]\n", "print(example_sequence)" ] }, { "cell_type": "markdown", "metadata": { "id": "ox1I28JRIOdM" }, "source": [ "### 1 つの文からスキップグラムを生成する" ] }, { "cell_type": "markdown", "metadata": { "id": "t7NNKAmSiHvy" }, "source": [ "`tf.keras.preprocessing.sequence` モジュールは、word2vec のデータ準備を簡素化する便利な関数を提供します。 `tf.keras.preprocessing.sequence.skipgrams` を使用して、範囲 `[0, vocab_size)` のトークンから指定された `window_size` で `example_sequence` からスキップグラムペアを生成します。\n", "\n", "注意: `negative_samples` は、ここでは `0` に設定されています。これは、この関数によって生成されたネガティブサンプルのバッチ処理にコードが少し必要だからです。次のセクションでは、別の関数を使用してネガティブサンプリングを実行します。" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:58.481605Z", "iopub.status.busy": "2022-12-14T23:26:58.481049Z", "iopub.status.idle": "2022-12-14T23:26:58.485035Z", "shell.execute_reply": "2022-12-14T23:26:58.484452Z" }, "id": "USAJxW4RD7pn" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "26\n" ] } ], "source": [ "window_size = 2\n", "positive_skip_grams, _ = tf.keras.preprocessing.sequence.skipgrams(\n", " example_sequence,\n", " vocabulary_size=vocab_size,\n", " window_size=window_size,\n", " negative_samples=0)\n", "print(len(positive_skip_grams))" ] }, { "cell_type": "markdown", "metadata": { "id": "uc9uhiMwY-AQ" }, "source": [ "いくつかのポジティブのスキップグラムを出力します。" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:58.488479Z", "iopub.status.busy": "2022-12-14T23:26:58.487901Z", "iopub.status.idle": "2022-12-14T23:26:58.491643Z", "shell.execute_reply": "2022-12-14T23:26:58.491037Z" }, "id": "SCnqEukIE9pt" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(3, 5): (road, in)\n", "(3, 2): (road, wide)\n", "(2, 1): (wide, the)\n", "(4, 5): (shimmered, in)\n", "(4, 2): (shimmered, wide)\n" ] } ], "source": [ "for target, context in positive_skip_grams[:5]:\n", " print(f\"({target}, {context}): ({inverse_vocab[target]}, {inverse_vocab[context]})\")" ] }, { "cell_type": "markdown", "metadata": { "id": "_ua9PkMTISF0" }, "source": [ "### 1 つのスキップグラムのネガティブサンプリング " ] }, { "cell_type": "markdown", "metadata": { "id": "Esqn8WBfZnEK" }, "source": [ "`skipgrams` 関数は、指定されたウィンドウスパンをスライドすることにより、すべてのポジティブのスキップグラムのペアを返します。トレーニング用のネガティブサンプルとして機能する追加のスキップグラムのペアを生成するには、語彙からランダムな単語をサンプリングする必要があります。`tf.random.log_uniform_candidate_sampler` 関数を使用して、ウィンドウ内の特定のターゲットワードに対して `num_ns` のネガティブサンプルをサンプリングします。1 つのスキップグラムのターゲットワードで関数を呼び出し、コンテキストワードを真のクラスとして渡して、サンプリングから除外できます。\n" ] }, { "cell_type": "markdown", "metadata": { "id": "AgH3aSvw3xTD" }, "source": [ "重要点: `[5, 20]` 範囲の `num_ns`（ポジティブなコンテキストワードあたりのネガティブサンプルの数）は、小規模なデータセットで[機能することが示されています](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)。`[2, 5]` 範囲の `num_ns` は、より大きなデータセットの場合に十分です。" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:26:58.495176Z", "iopub.status.busy": "2022-12-14T23:26:58.494622Z", "iopub.status.idle": "2022-12-14T23:27:01.911781Z", "shell.execute_reply": "2022-12-14T23:27:01.911076Z" }, "id": "m_LmdzqIGr5L" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tf.Tensor([2 1 4 3], shape=(4,), dtype=int64)\n", "['wide', 'the', 'shimmered', 'road']\n" ] } ], "source": [ "# Get target and context words for one positive skip-gram.\n", "target_word, context_word = positive_skip_grams[0]\n", "\n", "# Set the number of negative samples per positive context.\n", "num_ns = 4\n", "\n", "context_class = tf.reshape(tf.constant(context_word, dtype=\"int64\"), (1, 1))\n", "negative_sampling_candidates, _, _ = tf.random.log_uniform_candidate_sampler(\n", " true_classes=context_class, # class that should be sampled as 'positive'\n", " num_true=1, # each positive skip-gram has 1 positive context class\n", " num_sampled=num_ns, # number of negative context words to sample\n", " unique=True, # all the negative samples should be unique\n", " range_max=vocab_size, # pick index of the samples from [0, vocab_size]\n", " seed=SEED, # seed for reproducibility\n", " name=\"negative_sampling\" # name of this operation\n", ")\n", "print(negative_sampling_candidates)\n", "print([inverse_vocab[index.numpy()] for index in negative_sampling_candidates])" ] }, { "cell_type": "markdown", "metadata": { "id": "8MSxWCrLIalp" }, "source": [ "### 1 つのトレーニングサンプルを作成する" ] }, { "cell_type": "markdown", "metadata": { "id": "Q6uEWdj8vKKv" }, "source": [ "与えられたポジティブの `(target_word, context_word)` スキップグラムに対して、`target_word` のウィンドウ サイズの前後に現れない `num_ns` のネガティブサンプルのコンテキストワードもあります。`1` のポジティブの `context_word` と `num_ns` のネガティブのコンテキストワードを 1 つのテンソルにバッチ処理します。これにより、ターゲットワードごとにポジティブのスキップグラム (`1` とラベル付ける) とネガティブのサンプル (`0` とラベル付ける) のセットが生成されます。" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:01.915502Z", "iopub.status.busy": "2022-12-14T23:27:01.914999Z", "iopub.status.idle": "2022-12-14T23:27:01.924164Z", "shell.execute_reply": "2022-12-14T23:27:01.923576Z" }, "id": "zSiZwifuLvHf" }, "outputs": [], "source": [ "# Add a dimension so you can use concatenation (in the next step).\n", "negative_sampling_candidates = tf.expand_dims(negative_sampling_candidates, 1)\n", "\n", "# Concatenate a positive context word with negative sampled words.\n", "context = tf.concat([context_class, negative_sampling_candidates], 0)\n", "\n", "# Label the first context word as `1` (positive) followed by `num_ns` `0`s (negative).\n", "label = tf.constant([1] + [0]*num_ns, dtype=\"int64\")\n", "\n", "# Reshape the target to shape `(1,)` and context and label to `(num_ns+1,)`.\n", "target = tf.squeeze(target_word)\n", "context = tf.squeeze(context)\n", "label = tf.squeeze(label)" ] }, { "cell_type": "markdown", "metadata": { "id": "OIJeoFCAwtXJ" }, "source": [ "上記のスキップグラムの例から、ターゲットワードのコンテキストと対応するラベルを確認してください。" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:01.927390Z", "iopub.status.busy": "2022-12-14T23:27:01.926899Z", "iopub.status.idle": "2022-12-14T23:27:01.933673Z", "shell.execute_reply": "2022-12-14T23:27:01.932981Z" }, "id": "tzyCPCuZwmdL" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "target_index : 3\n", "target_word : road\n", "context_indices : [5 2 1 4 3]\n", "context_words : ['in', 'wide', 'the', 'shimmered', 'road']\n", "label : [1 0 0 0 0]\n" ] } ], "source": [ "print(f\"target_index : {target}\")\n", "print(f\"target_word : {inverse_vocab[target_word]}\")\n", "print(f\"context_indices : {context}\")\n", "print(f\"context_words : {[inverse_vocab[c.numpy()] for c in context]}\")\n", "print(f\"label : {label}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "gBtTcUVQr8EO" }, "source": [ "`(target, context, label)` テンソルのタプルは、スキップグラム ネガティブサンプリング word2vec モデルをトレーニングするための 1 つのトレーニングサンプルを構成します。ターゲットの形状は `(1,)` であるのに対し、コンテキストとラベルの形状は `(1+num_ns,)` であることに注意してください。" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:01.936838Z", "iopub.status.busy": "2022-12-14T23:27:01.936354Z", "iopub.status.idle": "2022-12-14T23:27:01.940198Z", "shell.execute_reply": "2022-12-14T23:27:01.939607Z" }, "id": "x-FwkR8jx9-Z" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "target : tf.Tensor(3, shape=(), dtype=int32)\n", "context : tf.Tensor([5 2 1 4 3], shape=(5,), dtype=int64)\n", "label : tf.Tensor([1 0 0 0 0], shape=(5,), dtype=int64)\n" ] } ], "source": [ "print(\"target :\", target)\n", "print(\"context :\", context)\n", "print(\"label :\", label)" ] }, { "cell_type": "markdown", "metadata": { "id": "4bRJIlow4Dlv" }, "source": [ "### まとめ" ] }, { "cell_type": "markdown", "metadata": { "id": "pWkuha0oykG5" }, "source": [ "この図は、文からトレーニングサンプルを生成する手順をまとめたものです。\n" ] }, { "cell_type": "markdown", "metadata": { "id": "_KlwdiAa9crJ" }, "source": [ "![word2vec_negative_sampling](https://tensorflow.org/tutorials/text/images/word2vec_negative_sampling.png)" ] }, { "cell_type": "markdown", "metadata": { "id": "37e53f07f67c" }, "source": [ "`temperature` と `code` という単語は、入力文の一部ではないことに注意してください。これらは、上の図で使用されている他の特定のインデックスと同様の語彙に属しています。" ] }, { "cell_type": "markdown", "metadata": { "id": "9wmdO_MEIpaM" }, "source": [ "## すべてのステップを 1 つの関数にコンパイルする\n" ] }, { "cell_type": "markdown", "metadata": { "id": "iLKwNAczHsKg" }, "source": [ "### スキップグラム サンプリングテーブル " ] }, { "cell_type": "markdown", "metadata": { "id": "TUUK3uDtFNFE" }, "source": [ "大規模なデータセットでは語彙が多くなり、ストップワードなどのより頻繁に使用される単語の数も多くなります。一般的に出現する単語 (`the`、`is`、`on` など) のサンプリングから得られたトレーニングサンプルは、モデルの学習に役立つ情報をあまり提供しません。[Mikolov et al.](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) は、埋め込みの品質を改善するための有用な方法として、頻繁に使用される単語のサブサンプリングを提案しています。 " ] }, { "cell_type": "markdown", "metadata": { "id": "bPtbv7zNP7Dx" }, "source": [ "`tf.keras.preprocessing.sequence.skipgrams` 関数は、任意のトークンをサンプリングする確率をエンコードするためのサンプリングテーブル引数を受け入れます。`tf.keras.preprocessing.sequence.make_sampling_table` を使用して、単語頻度ランクに基づく確率的サンプリングテーブルを生成し、それを `skipgrams` 関数に渡します。`vocab_size` が 10 の場合のサンプリング確率を調べます。" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:01.943870Z", "iopub.status.busy": "2022-12-14T23:27:01.943620Z", "iopub.status.idle": "2022-12-14T23:27:01.947393Z", "shell.execute_reply": "2022-12-14T23:27:01.946798Z" }, "id": "Rn9zAnDccyRg" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0.00315225 0.00315225 0.00547597 0.00741556 0.00912817 0.01068435\n", " 0.01212381 0.01347162 0.01474487 0.0159558 ]\n" ] } ], "source": [ "sampling_table = tf.keras.preprocessing.sequence.make_sampling_table(size=10)\n", "print(sampling_table)" ] }, { "cell_type": "markdown", "metadata": { "id": "EHvSptcPk5fp" }, "source": [ "`sampling_table[i]` は、データセットで i 番目に最も一般的な単語をサンプリングする確率を示します。この関数は、サンプリングの単語頻度の [Zipf 分布](https://en.wikipedia.org/wiki/Zipf%27s_law)を想定しています。" ] }, { "cell_type": "markdown", "metadata": { "id": "mRHMssMmHgH-" }, "source": [ "重要点: `tf.random.log_uniform_candidate_sampler` は、語彙頻度が対数一様 (Zipf の) 分布に従うことを既に想定しています。これらの分布加重サンプリングを使用すると、ネガティブのサンプリング目標をトレーニングするための単純な損失関数を使用して、Noise Contrastive Estimation (NCE) 損失を概算するのにも役立ちます。" ] }, { "cell_type": "markdown", "metadata": { "id": "aj--8RFK6fgW" }, "source": [ "### トレーニングデータを生成する" ] }, { "cell_type": "markdown", "metadata": { "id": "dy5hl4lQ0B2M" }, "source": [ "上記のすべての手順を、任意のテキストデータセットから取得したベクトル化された文のリストに対して呼び出せる関数にコンパイルします。スキップグラムの単語ペアをサンプリングする前に、サンプリングテーブルが作成されることに注意してください。この関数は後のセクションで使用します。" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:01.951002Z", "iopub.status.busy": "2022-12-14T23:27:01.950437Z", "iopub.status.idle": "2022-12-14T23:27:01.956859Z", "shell.execute_reply": "2022-12-14T23:27:01.956324Z" }, "id": "63INISDEX1Hu" }, "outputs": [], "source": [ "# Generates skip-gram pairs with negative sampling for a list of sequences\n", "# (int-encoded sentences) based on window size, number of negative samples\n", "# and vocabulary size.\n", "def generate_training_data(sequences, window_size, num_ns, vocab_size, seed):\n", " # Elements of each training example are appended to these lists.\n", " targets, contexts, labels = [], [], []\n", "\n", " # Build the sampling table for `vocab_size` tokens.\n", " sampling_table = tf.keras.preprocessing.sequence.make_sampling_table(vocab_size)\n", "\n", " # Iterate over all sequences (sentences) in the dataset.\n", " for sequence in tqdm.tqdm(sequences):\n", "\n", " # Generate positive skip-gram pairs for a sequence (sentence).\n", " positive_skip_grams, _ = tf.keras.preprocessing.sequence.skipgrams(\n", " sequence,\n", " vocabulary_size=vocab_size,\n", " sampling_table=sampling_table,\n", " window_size=window_size,\n", " negative_samples=0)\n", "\n", " # Iterate over each positive skip-gram pair to produce training examples\n", " # with a positive context word and negative samples.\n", " for target_word, context_word in positive_skip_grams:\n", " context_class = tf.expand_dims(\n", " tf.constant([context_word], dtype=\"int64\"), 1)\n", " negative_sampling_candidates, _, _ = tf.random.log_uniform_candidate_sampler(\n", " true_classes=context_class,\n", " num_true=1,\n", " num_sampled=num_ns,\n", " unique=True,\n", " range_max=vocab_size,\n", " seed=seed,\n", " name=\"negative_sampling\")\n", "\n", " # Build context and label vectors (for one target word)\n", " negative_sampling_candidates = tf.expand_dims(\n", " negative_sampling_candidates, 1)\n", "\n", " context = tf.concat([context_class, negative_sampling_candidates], 0)\n", " label = tf.constant([1] + [0]*num_ns, dtype=\"int64\")\n", "\n", " # Append each element from the training example to global lists.\n", " targets.append(target_word)\n", " contexts.append(context)\n", " labels.append(label)\n", "\n", " return targets, contexts, labels" ] }, { "cell_type": "markdown", "metadata": { "id": "shvPC8Ji2cMK" }, "source": [ "## word2vec のトレーニングデータを準備する" ] }, { "cell_type": "markdown", "metadata": { "id": "j5mbZsZu6uKg" }, "source": [ "スキップグラム ネガティブ サンプリング ベースの word2vec モデルで 1 つの文を処理する方法を理解することにより、より大きな文のリストからトレーニングサンプルを生成できます。" ] }, { "cell_type": "markdown", "metadata": { "id": "OFlikI6L26nh" }, "source": [ "### テキストコーパスをダウンロードする\n" ] }, { "cell_type": "markdown", "metadata": { "id": "rEFavOgN98al" }, "source": [ "このチュートリアルでは、シェイクスピア著作のテキストファイルを使用します。次の行を変更して、このコードを独自のデータで実行します。" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:01.960105Z", "iopub.status.busy": "2022-12-14T23:27:01.959635Z", "iopub.status.idle": "2022-12-14T23:27:02.018799Z", "shell.execute_reply": "2022-12-14T23:27:02.018230Z" }, "id": "QFkitxzVVaAi" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 8192/1115394 [..............................] - ETA: 0s" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1115394/1115394 [==============================] - 0s 0us/step\n" ] } ], "source": [ "path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')" ] }, { "cell_type": "markdown", "metadata": { "id": "sOsbLq8a37dr" }, "source": [ "ファイルからテキストを読み取り、最初の数行を出力します。 " ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:02.022148Z", "iopub.status.busy": "2022-12-14T23:27:02.021653Z", "iopub.status.idle": "2022-12-14T23:27:02.029973Z", "shell.execute_reply": "2022-12-14T23:27:02.029383Z" }, "id": "lfgnsUw3ofMD" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "First Citizen:\n", "Before we proceed any further, hear me speak.\n", "\n", "All:\n", "Speak, speak.\n", "\n", "First Citizen:\n", "You are all resolved rather to die than to famish?\n", "\n", "All:\n", "Resolved. resolved.\n", "\n", "First Citizen:\n", "First, you know Caius Marcius is chief enemy to the people.\n", "\n", "All:\n", "We know't, we know't.\n", "\n", "First Citizen:\n", "Let us kill him, and we'll have corn at our own price.\n" ] } ], "source": [ "with open(path_to_file) as f:\n", " lines = f.read().splitlines()\n", "for line in lines[:20]:\n", " print(line)" ] }, { "cell_type": "markdown", "metadata": { "id": "gTNZYqUs5C2V" }, "source": [ "空でない行を使用して、次の手順として `tf.data.TextLineDataset` オブジェクトを作成します。" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:02.033238Z", "iopub.status.busy": "2022-12-14T23:27:02.032718Z", "iopub.status.idle": "2022-12-14T23:27:02.069830Z", "shell.execute_reply": "2022-12-14T23:27:02.069253Z" }, "id": "ViDrwy-HjAs9" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.\n", "Instructions for updating:\n", "Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089\n" ] } ], "source": [ "text_ds = tf.data.TextLineDataset(path_to_file).filter(lambda x: tf.cast(tf.strings.length(x), bool))" ] }, { "cell_type": "markdown", "metadata": { "id": "vfsc88zE9upk" }, "source": [ "### コーパスから文をベクトル化する" ] }, { "cell_type": "markdown", "metadata": { "id": "XfgZo8zR94KK" }, "source": [ "`TextVectorization` レイヤーを使用して、コーパスから文をベクトル化します。このレイヤの使用について詳しくは、[テキスト分類](https://www.tensorflow.org/tutorials/keras/text_classification)のチュートリアルを参照してください。上記の最初の数文から、テキストは大文字または小文字にする必要があり、句読点を削除する必要があることに注意してください。これを行うには、TextVectorization レイヤーで使用する `custom_standardization function` を定義します。" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:02.073598Z", "iopub.status.busy": "2022-12-14T23:27:02.073050Z", "iopub.status.idle": "2022-12-14T23:27:02.086581Z", "shell.execute_reply": "2022-12-14T23:27:02.086044Z" }, "id": "2MlsXzo-ZlfK" }, "outputs": [], "source": [ "# Now, create a custom standardization function to lowercase the text and\n", "# remove punctuation.\n", "def custom_standardization(input_data):\n", " lowercase = tf.strings.lower(input_data)\n", " return tf.strings.regex_replace(lowercase,\n", " '[%s]' % re.escape(string.punctuation), '')\n", "\n", "\n", "# Define the vocabulary size and the number of words in a sequence.\n", "vocab_size = 4096\n", "sequence_length = 10\n", "\n", "# Use the `TextVectorization` layer to normalize, split, and map strings to\n", "# integers. Set the `output_sequence_length` length to pad all samples to the\n", "# same length.\n", "vectorize_layer = layers.TextVectorization(\n", " standardize=custom_standardization,\n", " max_tokens=vocab_size,\n", " output_mode='int',\n", " output_sequence_length=sequence_length)" ] }, { "cell_type": "markdown", "metadata": { "id": "g92LuvnyBmz1" }, "source": [ "テキストデータセットで `TextVectorization.adapt` を呼び出して語彙を作成します。\n" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:02.090171Z", "iopub.status.busy": "2022-12-14T23:27:02.089647Z", "iopub.status.idle": "2022-12-14T23:27:03.304210Z", "shell.execute_reply": "2022-12-14T23:27:03.303374Z" }, "id": "seZau_iYMPFT" }, "outputs": [], "source": [ "vectorize_layer.adapt(text_ds.batch(1024))" ] }, { "cell_type": "markdown", "metadata": { "id": "jg2z7eeHMnH-" }, "source": [ "レイヤーの状態がテキストコーパスを表すように調整されると、`TextVectorization.get_vocabulary` を使用して語彙にアクセスできます。この関数は、頻度によって (降順で) 並べ替えられたすべての語彙トークンのリストを返します。" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:03.308508Z", "iopub.status.busy": "2022-12-14T23:27:03.307856Z", "iopub.status.idle": "2022-12-14T23:27:03.318283Z", "shell.execute_reply": "2022-12-14T23:27:03.317725Z" }, "id": "jgw9pTA7MRaU" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['', '[UNK]', 'the', 'and', 'to', 'i', 'of', 'you', 'my', 'a', 'that', 'in', 'is', 'not', 'for', 'with', 'me', 'it', 'be', 'your']\n" ] } ], "source": [ "# Save the created vocabulary for reference.\n", "inverse_vocab = vectorize_layer.get_vocabulary()\n", "print(inverse_vocab[:20])" ] }, { "cell_type": "markdown", "metadata": { "id": "DOQ30Tx6KA2G" }, "source": [ "`vectorize_layer` を使用して、`text_ds` (`tf.data.Dataset`) 内の各要素のベクトルを生成できるようになりました。`Dataset.batch`、`Dataset.prefetch`、`Dataset.map`、`Dataset.unbatch` を適用します。" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:03.321651Z", "iopub.status.busy": "2022-12-14T23:27:03.321113Z", "iopub.status.idle": "2022-12-14T23:27:03.364903Z", "shell.execute_reply": "2022-12-14T23:27:03.364311Z" }, "id": "yUVYrDp0araQ" }, "outputs": [], "source": [ "# Vectorize the data in text_ds.\n", "text_vector_ds = text_ds.batch(1024).prefetch(AUTOTUNE).map(vectorize_layer).unbatch()" ] }, { "cell_type": "markdown", "metadata": { "id": "7YyH_SYzB72p" }, "source": [ "### データセットから配列を取得する" ] }, { "cell_type": "markdown", "metadata": { "id": "NFUQLX0_KaRC" }, "source": [ "これで、整数でエンコードされた文の `tf.data.Dataset` ができました。word2vec モデルをトレーニングするためのデータセットを準備するには、データセットを文ベクトル シーケンスのリストにフラット化します。この手順は、データセット内の各文を繰り返し処理してポジティブなサンプルとネガティブなサンプルを生成するために必要です。\n", "\n", "注意: 前に定義した `generate_training_data()` は TensorFlow 以外の Python/NumPy 関数を使用するため、`tf.data.Dataset.map` で `tf.py_function` や `tf.numpy_function` を使用することもできます。" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:03.368513Z", "iopub.status.busy": "2022-12-14T23:27:03.368041Z", "iopub.status.idle": "2022-12-14T23:27:07.873309Z", "shell.execute_reply": "2022-12-14T23:27:07.872144Z" }, "id": "sGXoOh9y11pM" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "32777\n" ] } ], "source": [ "sequences = list(text_vector_ds.as_numpy_iterator())\n", "print(len(sequences))" ] }, { "cell_type": "markdown", "metadata": { "id": "tDc4riukLTqg" }, "source": [ "`sequences` からいくつかのサンプルを調べます。" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:07.877287Z", "iopub.status.busy": "2022-12-14T23:27:07.876667Z", "iopub.status.idle": "2022-12-14T23:27:07.882099Z", "shell.execute_reply": "2022-12-14T23:27:07.881246Z" }, "id": "WZf1RIbB2Dfb" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ 89 270 0 0 0 0 0 0 0 0] => ['first', 'citizen', '', '', '', '', '', '', '', '']\n", "[138 36 982 144 673 125 16 106 0 0] => ['before', 'we', 'proceed', 'any', 'further', 'hear', 'me', 'speak', '', '']\n", "[34 0 0 0 0 0 0 0 0 0] => ['all', '', '', '', '', '', '', '', '', '']\n", "[106 106 0 0 0 0 0 0 0 0] => ['speak', 'speak', '', '', '', '', '', '', '', '']\n", "[ 89 270 0 0 0 0 0 0 0 0] => ['first', 'citizen', '', '', '', '', '', '', '', '']\n" ] } ], "source": [ "for seq in sequences[:5]:\n", " print(f\"{seq} => {[inverse_vocab[i] for i in seq]}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "yDzSOjNwCWNh" }, "source": [ "### シーケンスからトレーニングサンプルを生成する" ] }, { "cell_type": "markdown", "metadata": { "id": "BehvYr-nEKyY" }, "source": [ "`sequences` は、int でエンコードされた文のリストになりました。前に定義した `generate_training_data` 関数を呼び出すだけで、word2vec モデルのトレーニングサンプルを生成できます。要約すると、関数は各シーケンスの各単語を反復処理して、ポジティブおよびネガティブなコンテキストワードを収集します。ターゲット、コンテキスト、およびラベルの長さは同じであり、トレーニングサンプルの総数を表す必要があります。" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:27:07.885756Z", "iopub.status.busy": "2022-12-14T23:27:07.885303Z", "iopub.status.idle": "2022-12-14T23:28:02.318530Z", "shell.execute_reply": "2022-12-14T23:28:02.317724Z" }, "id": "44DJ22M6nX5o" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\r", " 0%| | 0/32777 [00:00\n" ] } ], "source": [ "BATCH_SIZE = 1024\n", "BUFFER_SIZE = 10000\n", "dataset = tf.data.Dataset.from_tensor_slices(((targets, contexts), labels))\n", "dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)\n", "print(dataset)" ] }, { "cell_type": "markdown", "metadata": { "id": "tyrNX6Fs6K3F" }, "source": [ "`Dataset.cache` と `Dataset.prefetch` を適用してパフォーマンスを向上させます。" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:28:02.342868Z", "iopub.status.busy": "2022-12-14T23:28:02.342151Z", "iopub.status.idle": "2022-12-14T23:28:02.349329Z", "shell.execute_reply": "2022-12-14T23:28:02.348705Z" }, "id": "Y5Ueg6bcFPVL" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "dataset = dataset.cache().prefetch(buffer_size=AUTOTUNE)\n", "print(dataset)" ] }, { "cell_type": "markdown", "metadata": { "id": "1S-CmUMszyEf" }, "source": [ "## モデルとトレーニング" ] }, { "cell_type": "markdown", "metadata": { "id": "sQFqaBMPwBqC" }, "source": [ "word2vec モデルは、スキップグラムからの真のコンテキストワードと、ネガティブサンプリングによって取得された偽のコンテキストワードを区別する分類器として実装できます。ターゲットワードとコンテキストワードの埋め込みの間で内積乗算を実行して、ラベルの予測を取得し、データセット内の真のラベルに対する損失関数を計算できます。" ] }, { "cell_type": "markdown", "metadata": { "id": "oc7kTbiwD9sy" }, "source": [ "### サブクラス化された word2vec モデル" ] }, { "cell_type": "markdown", "metadata": { "id": "Jvr9pM1G1sQN" }, "source": [ "[Keras Subclassing API](https://www.tensorflow.org/guide/keras/custom_layers_and_models) を使用して、次のレイヤーで word2vec モデルを定義します。\n", "\n", "- `target_embedding`: `tf.keras.layers.Embedding` レイヤーは単語がターゲットワードとして表示されたときにその単語の埋め込みを検索します。このレイヤーのパラメータ数は `(vocab_size * embedded_dim)` です。\n", "- `context_embedding`: これはもう一つの `tf.keras.layers.Embedding` レイヤーで単語がコンテキストワードとして表示されたときに、その単語の埋め込みを検索します。このレイヤーのパラメータ数は、`target_embedding` のパラメータ数と同じです。つまり、`(vocab_size * embedded_dim)` です。\n", "- `dots`: これはトレーニングペアからターゲットとコンテキストの埋め込みの内積を計算する `tf.keras.layers.Dot` レイヤーです。\n", "- `flatten`: `tf.keras.layers.Flatten` レイヤーは、`dots` レイヤーの結果をロジットにフラット化します。\n", "\n", "サブクラス化されたモデルを使用すると、`(target, context)` ペアを受け入れる `call()` 関数を定義し、対応する埋め込みレイヤーに渡すことができる `context_embedding` の形状を変更して、`target_embedding` で内積を実行し、フラット化された結果を返します。" ] }, { "cell_type": "markdown", "metadata": { "id": "KiAwuIqqw7-7" }, "source": [ "重要点: `target_embedding` レイヤーと `context embedded` レイヤーも共有できます。また、両方の埋め込みを連結して、最終的な word2vec 埋め込みとして使用することもできます。" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:28:02.353101Z", "iopub.status.busy": "2022-12-14T23:28:02.352554Z", "iopub.status.idle": "2022-12-14T23:28:02.357721Z", "shell.execute_reply": "2022-12-14T23:28:02.357131Z" }, "id": "i9ec-sS6xd8Z" }, "outputs": [], "source": [ "class Word2Vec(tf.keras.Model):\n", " def __init__(self, vocab_size, embedding_dim):\n", " super(Word2Vec, self).__init__()\n", " self.target_embedding = layers.Embedding(vocab_size,\n", " embedding_dim,\n", " input_length=1,\n", " name=\"w2v_embedding\")\n", " self.context_embedding = layers.Embedding(vocab_size,\n", " embedding_dim,\n", " input_length=num_ns+1)\n", "\n", " def call(self, pair):\n", " target, context = pair\n", " # target: (batch, dummy?) # The dummy axis doesn't exist in TF2.7+\n", " # context: (batch, context)\n", " if len(target.shape) == 2:\n", " target = tf.squeeze(target, axis=1)\n", " # target: (batch,)\n", " word_emb = self.target_embedding(target)\n", " # word_emb: (batch, embed)\n", " context_emb = self.context_embedding(context)\n", " # context_emb: (batch, context, embed)\n", " dots = tf.einsum('be,bce->bc', word_emb, context_emb)\n", " # dots: (batch, context)\n", " return dots" ] }, { "cell_type": "markdown", "metadata": { "id": "-RLKz9LFECXu" }, "source": [ "### 損失関数の定義とモデルのコンパイル\n" ] }, { "cell_type": "markdown", "metadata": { "id": "I3Md-9QanqBM" }, "source": [ "簡単にするためには、ネガティブサンプリング損失の代わりに `tf.keras.losses.CategoricalCrossEntropy` を使用できます。独自のカスタム損失関数を記述する場合は、次のようにします。\n", "\n", "```python\n", "def custom_loss(x_logit, y_true):\n", " return tf.nn.sigmoid_cross_entropy_with_logits(logits=x_logit, labels=y_true)\n", "```\n", "\n", "モデルを構築します。128 の埋め込み次元で word2vec クラスをインスタンス化します (さまざまな値を試してみてください)。モデルを `tf.keras.optimizers.Adam` オプティマイザーでコンパイルします。 " ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:28:02.361102Z", "iopub.status.busy": "2022-12-14T23:28:02.360680Z", "iopub.status.idle": "2022-12-14T23:28:02.380547Z", "shell.execute_reply": "2022-12-14T23:28:02.379992Z" }, "id": "ekQg_KbWnnmQ" }, "outputs": [], "source": [ "embedding_dim = 128\n", "word2vec = Word2Vec(vocab_size, embedding_dim)\n", "word2vec.compile(optimizer='adam',\n", " loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n", " metrics=['accuracy'])" ] }, { "cell_type": "markdown", "metadata": { "id": "P3MUMrluqNX2" }, "source": [ "また、TensorBoard のトレーニング統計をログに記録するコールバックを定義します。" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:28:02.384100Z", "iopub.status.busy": "2022-12-14T23:28:02.383527Z", "iopub.status.idle": "2022-12-14T23:28:02.386789Z", "shell.execute_reply": "2022-12-14T23:28:02.386194Z" }, "id": "9d-ftBCeEZIR" }, "outputs": [], "source": [ "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=\"logs\")" ] }, { "cell_type": "markdown", "metadata": { "id": "h5wEBotlGZ7B" }, "source": [ "いくつかのエポックで、`dataset` でモデルをトレーニングします。" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:28:02.390167Z", "iopub.status.busy": "2022-12-14T23:28:02.389698Z", "iopub.status.idle": "2022-12-14T23:28:13.813524Z", "shell.execute_reply": "2022-12-14T23:28:13.812795Z" }, "id": "gmC1BJalEZIY" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 1:09 - loss: 1.6097 - accuracy: 0.2061" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 2/63 [..............................] - ETA: 9s - loss: 1.6094 - accuracy: 0.2100 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 3/63 [>.............................] - ETA: 9s - loss: 1.6095 - accuracy: 0.2008" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 4/63 [>.............................] - ETA: 8s - loss: 1.6096 - accuracy: 0.1956" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 5/63 [=>............................] - ETA: 8s - loss: 1.6094 - accuracy: 0.2000" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 6/63 [=>............................] - ETA: 8s - loss: 1.6094 - accuracy: 0.1987" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 7/63 [==>...........................] - ETA: 8s - loss: 1.6094 - accuracy: 0.2019" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 8/63 [==>...........................] - ETA: 8s - loss: 1.6094 - accuracy: 0.2023" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", " 9/63 [===>..........................] - ETA: 7s - loss: 1.6094 - accuracy: 0.2041" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "10/63 [===>..........................] - ETA: 7s - loss: 1.6093 - accuracy: 0.2050" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "11/63 [====>.........................] - ETA: 7s - loss: 1.6093 - accuracy: 0.2067" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "12/63 [====>.........................] - ETA: 7s - loss: 1.6093 - accuracy: 0.2086" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "13/63 [=====>........................] - ETA: 7s - loss: 1.6093 - accuracy: 0.2080" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "14/63 [=====>........................] - ETA: 7s - loss: 1.6093 - accuracy: 0.2087" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "16/63 [======>.......................] - ETA: 6s - loss: 1.6093 - accuracy: 0.2091" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "17/63 [=======>......................] - ETA: 6s - loss: 1.6092 - accuracy: 0.2098" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "18/63 [=======>......................] - ETA: 6s - loss: 1.6092 - accuracy: 0.2101" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "19/63 [========>.....................] - ETA: 6s - loss: 1.6092 - accuracy: 0.2104" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 6s - loss: 1.6092 - accuracy: 0.2110" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "21/63 [=========>....................] - ETA: 5s - loss: 1.6092 - accuracy: 0.2121" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "22/63 [=========>....................] - ETA: 5s - loss: 1.6092 - accuracy: 0.2124" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "23/63 [=========>....................] - ETA: 5s - loss: 1.6091 - accuracy: 0.2137" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "24/63 [==========>...................] - ETA: 5s - loss: 1.6091 - accuracy: 0.2153" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "25/63 [==========>...................] - ETA: 5s - loss: 1.6091 - accuracy: 0.2150" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "26/63 [===========>..................] - ETA: 5s - loss: 1.6090 - accuracy: 0.2160" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "27/63 [===========>..................] - ETA: 5s - loss: 1.6090 - accuracy: 0.2164" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "28/63 [============>.................] - ETA: 5s - loss: 1.6090 - accuracy: 0.2168" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "29/63 [============>.................] - ETA: 4s - loss: 1.6090 - accuracy: 0.2172" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "31/63 [=============>................] - ETA: 4s - loss: 1.6089 - accuracy: 0.2179" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "33/63 [==============>...............] - ETA: 4s - loss: 1.6089 - accuracy: 0.2187" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "35/63 [===============>..............] - ETA: 3s - loss: 1.6089 - accuracy: 0.2202" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "36/63 [================>.............] - ETA: 3s - loss: 1.6088 - accuracy: 0.2205" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "37/63 [================>.............] - ETA: 3s - loss: 1.6088 - accuracy: 0.2205" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 3s - loss: 1.6088 - accuracy: 0.2212" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "41/63 [==================>...........] - ETA: 2s - loss: 1.6088 - accuracy: 0.2219" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "44/63 [===================>..........] - ETA: 2s - loss: 1.6087 - accuracy: 0.2239" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "45/63 [====================>.........] - ETA: 2s - loss: 1.6087 - accuracy: 0.2241" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "47/63 [=====================>........] - ETA: 1s - loss: 1.6086 - accuracy: 0.2247" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "48/63 [=====================>........] - ETA: 1s - loss: 1.6086 - accuracy: 0.2250" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "49/63 [======================>.......] - ETA: 1s - loss: 1.6086 - accuracy: 0.2256" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "51/63 [=======================>......] - ETA: 1s - loss: 1.6086 - accuracy: 0.2261" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "54/63 [========================>.....] - ETA: 1s - loss: 1.6085 - accuracy: 0.2273" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "56/63 [=========================>....] - ETA: 0s - loss: 1.6084 - accuracy: 0.2279" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "57/63 [==========================>...] - ETA: 0s - loss: 1.6084 - accuracy: 0.2283" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "59/63 [===========================>..] - ETA: 0s - loss: 1.6084 - accuracy: 0.2290" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "60/63 [===========================>..] - ETA: 0s - loss: 1.6083 - accuracy: 0.2297" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 8s 109ms/step - loss: 1.6082 - accuracy: 0.2312\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 2/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 1.5902 - accuracy: 0.7568" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 1.5940 - accuracy: 0.5919" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 1.5923 - accuracy: 0.5680" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58/63 [==========================>...] - ETA: 0s - loss: 1.5899 - accuracy: 0.5545" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 1.5890 - accuracy: 0.5521\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 3/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 1.5595 - accuracy: 0.7451" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 1.5587 - accuracy: 0.6569" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 1.5517 - accuracy: 0.6277" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "57/63 [==========================>...] - ETA: 0s - loss: 1.5446 - accuracy: 0.6085" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 1.5419 - accuracy: 0.6035\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 4/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 1.4925 - accuracy: 0.6318" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 1.4848 - accuracy: 0.6014" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 1.4738 - accuracy: 0.5837" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58/63 [==========================>...] - ETA: 0s - loss: 1.4631 - accuracy: 0.5758" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 1.4601 - accuracy: 0.5742\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 5/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 1.4005 - accuracy: 0.5996" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 1.3876 - accuracy: 0.5886" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 1.3764 - accuracy: 0.5819" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "57/63 [==========================>...] - ETA: 0s - loss: 1.3664 - accuracy: 0.5806" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 1.3626 - accuracy: 0.5812\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 6/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 1.3016 - accuracy: 0.6152" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 1.2878 - accuracy: 0.6081" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 1.2781 - accuracy: 0.6048" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "57/63 [==========================>...] - ETA: 0s - loss: 1.2692 - accuracy: 0.6063" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 1.2656 - accuracy: 0.6077\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 7/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 1.2052 - accuracy: 0.6426" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "19/63 [========>.....................] - ETA: 0s - loss: 1.1928 - accuracy: 0.6406" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "37/63 [================>.............] - ETA: 0s - loss: 1.1860 - accuracy: 0.6387" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "56/63 [=========================>....] - ETA: 0s - loss: 1.1784 - accuracy: 0.6403" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 1.1744 - accuracy: 0.6421\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 8/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 1.1144 - accuracy: 0.6650" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 1.1059 - accuracy: 0.6734" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "38/63 [=================>............] - ETA: 0s - loss: 1.0994 - accuracy: 0.6739" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "56/63 [=========================>....] - ETA: 0s - loss: 1.0933 - accuracy: 0.6748" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 1.0895 - accuracy: 0.6768\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 9/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 1.0298 - accuracy: 0.7090" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 1.0249 - accuracy: 0.7074" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "38/63 [=================>............] - ETA: 0s - loss: 1.0193 - accuracy: 0.7087" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "56/63 [=========================>....] - ETA: 0s - loss: 1.0141 - accuracy: 0.7090" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 1.0105 - accuracy: 0.7108\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 10/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.9515 - accuracy: 0.7461" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 0.9500 - accuracy: 0.7401" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "38/63 [=================>............] - ETA: 0s - loss: 0.9452 - accuracy: 0.7400" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "56/63 [=========================>....] - ETA: 0s - loss: 0.9408 - accuracy: 0.7400" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.9374 - accuracy: 0.7411\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 11/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.8792 - accuracy: 0.7715" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 0.8810 - accuracy: 0.7648" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 0.8765 - accuracy: 0.7645" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58/63 [==========================>...] - ETA: 0s - loss: 0.8718 - accuracy: 0.7651" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.8698 - accuracy: 0.7659\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 12/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.8127 - accuracy: 0.7939" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 0.8174 - accuracy: 0.7857" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 0.8134 - accuracy: 0.7862" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58/63 [==========================>...] - ETA: 0s - loss: 0.8094 - accuracy: 0.7865" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.8075 - accuracy: 0.7873\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 13/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.7517 - accuracy: 0.8105" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 0.7591 - accuracy: 0.8062" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 0.7554 - accuracy: 0.8064" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "59/63 [===========================>..] - ETA: 0s - loss: 0.7516 - accuracy: 0.8071" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.7502 - accuracy: 0.8078\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 14/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.6959 - accuracy: 0.8262" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 0.7056 - accuracy: 0.8236" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 0.7022 - accuracy: 0.8240" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58/63 [==========================>...] - ETA: 0s - loss: 0.6995 - accuracy: 0.8242" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.6978 - accuracy: 0.8247\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 15/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.6451 - accuracy: 0.8379" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "19/63 [========>.....................] - ETA: 0s - loss: 0.6563 - accuracy: 0.8376" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "38/63 [=================>............] - ETA: 0s - loss: 0.6539 - accuracy: 0.8387" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "57/63 [==========================>...] - ETA: 0s - loss: 0.6518 - accuracy: 0.8386" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.6499 - accuracy: 0.8395\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 16/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.5990 - accuracy: 0.8535" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 0.6122 - accuracy: 0.8503" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 0.6094 - accuracy: 0.8523" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58/63 [==========================>...] - ETA: 0s - loss: 0.6077 - accuracy: 0.8526" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.6062 - accuracy: 0.8532\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 17/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.5571 - accuracy: 0.8613" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 0.5716 - accuracy: 0.8627" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 0.5690 - accuracy: 0.8643" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58/63 [==========================>...] - ETA: 0s - loss: 0.5678 - accuracy: 0.8646" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.5664 - accuracy: 0.8652\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 18/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.5193 - accuracy: 0.8750" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 0.5347 - accuracy: 0.8744" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "39/63 [=================>............] - ETA: 0s - loss: 0.5323 - accuracy: 0.8754" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "58/63 [==========================>...] - ETA: 0s - loss: 0.5315 - accuracy: 0.8754" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.5302 - accuracy: 0.8759\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 19/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.4850 - accuracy: 0.8877" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "19/63 [========>.....................] - ETA: 0s - loss: 0.5008 - accuracy: 0.8836" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "38/63 [=================>............] - ETA: 0s - loss: 0.4992 - accuracy: 0.8860" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "57/63 [==========================>...] - ETA: 0s - loss: 0.4985 - accuracy: 0.8853" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.4972 - accuracy: 0.8856\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 20/20\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " 1/63 [..............................] - ETA: 0s - loss: 0.4541 - accuracy: 0.8965" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "20/63 [========>.....................] - ETA: 0s - loss: 0.4705 - accuracy: 0.8934" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "38/63 [=================>............] - ETA: 0s - loss: 0.4689 - accuracy: 0.8952" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "57/63 [==========================>...] - ETA: 0s - loss: 0.4685 - accuracy: 0.8945" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "63/63 [==============================] - 0s 3ms/step - loss: 0.4673 - accuracy: 0.8948\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "word2vec.fit(dataset, epochs=20, callbacks=[tensorboard_callback])" ] }, { "cell_type": "markdown", "metadata": { "id": "wze38jG57XvZ" }, "source": [ "TensorBoard は、word2vec モデルの精度と損失を表示します。" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "22E9eqS55rgz" }, "outputs": [], "source": [ "#docs_infra: no_execute\n", "%tensorboard --logdir logs" ] }, { "cell_type": "markdown", "metadata": { "id": "awF3iRQCZOLj" }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "id": "TaDW2tIIz8fL" }, "source": [ "## 埋め込みのルックアップと分析" ] }, { "cell_type": "markdown", "metadata": { "id": "Zp5rv01WG2YA" }, "source": [ "`Model.get_layer` と `Layer.get_weights` を使用して、モデルから重みを取得します。`TextVectorization.get_vocabulary` 関数は、1 行に 1 つのトークンでメタデータファイルを作成するための語彙を提供します。" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:28:13.817710Z", "iopub.status.busy": "2022-12-14T23:28:13.817124Z", "iopub.status.idle": "2022-12-14T23:28:13.829305Z", "shell.execute_reply": "2022-12-14T23:28:13.828633Z" }, "id": "_Uamp1YH8RzU" }, "outputs": [], "source": [ "weights = word2vec.get_layer('w2v_embedding').get_weights()[0]\n", "vocab = vectorize_layer.get_vocabulary()" ] }, { "cell_type": "markdown", "metadata": { "id": "gWzdmUzS8Sl4" }, "source": [ "ベクトルとメタデータファイルを作成して保存します。" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:28:13.832894Z", "iopub.status.busy": "2022-12-14T23:28:13.832344Z", "iopub.status.idle": "2022-12-14T23:28:14.126238Z", "shell.execute_reply": "2022-12-14T23:28:14.125545Z" }, "id": "VLIahl9s53XT" }, "outputs": [], "source": [ "out_v = io.open('vectors.tsv', 'w', encoding='utf-8')\n", "out_m = io.open('metadata.tsv', 'w', encoding='utf-8')\n", "\n", "for index, word in enumerate(vocab):\n", " if index == 0:\n", " continue # skip 0, it's padding.\n", " vec = weights[index]\n", " out_v.write('\\t'.join([str(x) for x in vec]) + \"\\n\")\n", " out_m.write(word + \"\\n\")\n", "out_v.close()\n", "out_m.close()" ] }, { "cell_type": "markdown", "metadata": { "id": "1T8KcThhIU8-" }, "source": [ "`vectors.tsv` と `metadata.tsv` をダウンロードして、取得した埋め込みを[埋め込みプロジェクタ](https://projector.tensorflow.org/)で分析します。" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "execution": { "iopub.execute_input": "2022-12-14T23:28:14.130271Z", "iopub.status.busy": "2022-12-14T23:28:14.130022Z", "iopub.status.idle": "2022-12-14T23:28:14.133635Z", "shell.execute_reply": "2022-12-14T23:28:14.133076Z" }, "id": "lUsjQOKMIV2z" }, "outputs": [], "source": [ "try:\n", " from google.colab import files\n", " files.download('vectors.tsv')\n", " files.download('metadata.tsv')\n", "except Exception:\n", " pass" ] }, { "cell_type": "markdown", "metadata": { "id": "iS_uMeMw3Xpj" }, "source": [ "## 次のステップ\n" ] }, { "cell_type": "markdown", "metadata": { "id": "BSgAZpwF5xF_" }, "source": [ "このチュートリアルでは、ゼロからネガティブサンプリングを使用してスキップグラムの word2vec モデルを実装し、取得した単語の埋め込みを視覚化する方法を実演しました。\n", "\n", "- 単語ベクトルとその数学的表現についての詳細は、[こちら](https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes01-wordvecs1.pdf)を参照してください。\n", "\n", "- 高度なテキスト処理についての詳細は、[言語理解のための Transformer モデル](https://www.tensorflow.org/tutorials/text/transformer)チュートリアルを参照してください。\n", "\n", "- 事前トレーニング済みの埋め込みモデルに興味がある場合は、[TF-Hub CORD-19 Swivel Embeddings の探索](https://www.tensorflow.org/hub/tutorials/cord_19_embeddings_keras)や[多言語ユニバーサルセンテンス エンコーダー](https://www.tensorflow.org/hub/tutorials/cross_lingual_similarity_with_tf_hub_multilingual_universal_encoder)も参照してください。\n", "\n", "- また、新しいデータセットでモデルをトレーニングすることもできます（[TensorFlow データセット](https://www.tensorflow.org/datasets)には多くのデータセットがあります）。\n" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "word2vec.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" } }, "nbformat": 4, "nbformat_minor": 0 }