{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Tce3stUlHN0L" }, "source": [ "##### Copyright 2019 The TensorFlow Authors.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2024-01-11T18:09:26.321507Z", "iopub.status.busy": "2024-01-11T18:09:26.320780Z", "iopub.status.idle": "2024-01-11T18:09:26.324676Z", "shell.execute_reply": "2024-01-11T18:09:26.324114Z" }, "id": "tuOe1ymfHZPu" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "MfBg1C5NB3X0" }, "source": [ "# Estimator を使ったマルチワーカートレーニング\n", "\n", "\n", " \n", " \n", " \n", " \n", "
TensorFlow.org で表示 Google Colab で実行 GitHub でソースを表示 ノートブックをダウンロード
" ] }, { "cell_type": "markdown", "metadata": { "id": "-_ZO8y69hs-N" }, "source": [ "> 警告: 新しいコードには Estimators は推奨されません。Estimators は `v1.Session` スタイルのコードを実行しますが、これは正しく記述するのはより難しく、特に TF 2 コードと組み合わせると予期しない動作をする可能性があります。Estimators は、[互換性保証](https://tensorflow.org/guide/versions)の対象となりますが、セキュリティの脆弱性以外の修正は行われません。詳細については、[移行ガイド](https://tensorflow.org/guide/migrate)を参照してください。" ] }, { "cell_type": "markdown", "metadata": { "id": "xHxb-dlhMIzW" }, "source": [ "## 概要\n", "\n", "注意: `tf.distribute` API で Estimator を使用することもできますが、`tf.distribute`で Keras を使用することを推奨します。[Keras を使ったマルチワーカートレーニング](multi_worker_with_keras.ipynb)をご覧ください。`tf.distribute.Strategy` を使った Estimator のトレーニングのサポートは制限されています。\n", "\n", "このチュートリアルでは、`tf.estimator`を使った分散型マルチワーカートレーニングに `tf.distribute.Strategy` を使用する方法を実演します。`tf.estimator` を使って独自のコードを記述し、高性能のマシン一台以上のスケーリングを希望する場合は、このチュートリアルをご利用ください。\n", "\n", "始める前に、[分散ストラテジー](../../guide/distributed_training.ipynb)ガイドをお読みください。[マルチ GPU トレーニングのチュートリアル](./keras.ipynb)も関連しています。このチュートリアルでは同じモデルが使用されています。\n" ] }, { "cell_type": "markdown", "metadata": { "id": "MUXex9ctTuDB" }, "source": [ "## セットアップ\n", "\n", "最初に、TensorFlow と必要なインポートをセットアップします。" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2024-01-11T18:09:26.328641Z", "iopub.status.busy": "2024-01-11T18:09:26.328028Z", "iopub.status.idle": "2024-01-11T18:09:29.004242Z", "shell.execute_reply": "2024-01-11T18:09:29.003533Z" }, "id": "bnYxvfLD-LW-" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-01-11 18:09:27.183559: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", "2024-01-11 18:09:27.183611: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", "2024-01-11 18:09:27.185190: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n" ] } ], "source": [ "import tensorflow_datasets as tfds\n", "import tensorflow as tf\n", "\n", "import os, json" ] }, { "cell_type": "markdown", "metadata": { "id": "-xicK9byC7hi" }, "source": [ "注意: TF2.4 以降、マルチワーカーミラーリングストラテジーは、eager を有効にして実行すると (デフォルト)、Estimator で失敗します。TF2.4 のエラーは `TypeError: cannot pickle '_thread.lock' object` です。詳細については、[課題 #46556](https://github.com/tensorflow/tensorflow/issues/46556) を参照してください。回避策は、eager の実行を無効にすることです。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2024-01-11T18:09:29.008413Z", "iopub.status.busy": "2024-01-11T18:09:29.008006Z", "iopub.status.idle": "2024-01-11T18:09:29.011503Z", "shell.execute_reply": "2024-01-11T18:09:29.010930Z" }, "id": "5dJ6UYrGDsVs" }, "outputs": [], "source": [ "tf.compat.v1.disable_eager_execution()" ] }, { "cell_type": "markdown", "metadata": { "id": "hPBuZUNSZmrQ" }, "source": [ "## 入力関数\n", "\n", "このチュートリアルでは、[TensorFlow Datasets](https://www.tensorflow.org/datasets) の MNIST データセットを使用しています。このコードは[マルチ GPU トレーニングのチュートリアル](./keras.ipynb)のコードに似ていますが、大きな違いが 1 つあります。マルチワーカートレーニングに Estimator を使用する際は、モデルのコンバージェンスを可能にするように、ワーカーの数でデータセットをシャーディングする必要があります。入力データは、ワーカーインデックスでシャーディングされるため、各ワーカーは、データセットの各 `1/num_workers` の部分を処理します。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2024-01-11T18:09:29.014537Z", "iopub.status.busy": "2024-01-11T18:09:29.014306Z", "iopub.status.idle": "2024-01-11T18:09:29.019404Z", "shell.execute_reply": "2024-01-11T18:09:29.018812Z" }, "id": "dma_wUAxZqo2" }, "outputs": [], "source": [ "BUFFER_SIZE = 10000\n", "BATCH_SIZE = 64\n", "\n", "def input_fn(mode, input_context=None):\n", " datasets, info = tfds.load(name='mnist',\n", " with_info=True,\n", " as_supervised=True)\n", " mnist_dataset = (datasets['train'] if mode == tf.estimator.ModeKeys.TRAIN else\n", " datasets['test'])\n", "\n", " def scale(image, label):\n", " image = tf.cast(image, tf.float32)\n", " image /= 255\n", " return image, label\n", "\n", " if input_context:\n", " mnist_dataset = mnist_dataset.shard(input_context.num_input_pipelines,\n", " input_context.input_pipeline_id)\n", " return mnist_dataset.map(scale).cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE)" ] }, { "cell_type": "markdown", "metadata": { "id": "4BlcVXMhB59T" }, "source": [ "コンバージェンスを達成するためのもう 1 つの合理的なアプローチとして、各ワーカーで異なるシードを使ってデータベースをシャッフルする方法があります。" ] }, { "cell_type": "markdown", "metadata": { "id": "8YFpxrcsZ2xG" }, "source": [ "## マルチワーカー構成\n", "\n", "このチュートリアルの主な違いの 1 つに ([マルチ GPU トレーニングのチュートリアル](./keras.ipynb) と比較)、マルチワーカーのセットアップがあります。`TF_CONFIG` 環境変数は、クラスタの一部である各ワーカーにクラスタ構成を指定する標準的な方法です。\n", "\n", "`TF_CONFIG` には、`cluster` と `task` の 2 つのコンポーネントがあります。`cluster` は、クラスタのワーカーとパラメータサーバーを含むクラスタ全体に関する情報を提供するのに対し、`task` は、現在のタスクに関する情報を提供します。最初のコンポーネント `cluster` は、クラスタ内のすべてのワーカーとパラメータサーバーで同一であり、2 つ目のコンポーネント `task` は、各ワーカーとパラメータサーバー間で異なり、それぞれに `type` と `index` を指定します。この例では、タスクの `type` は `worker` で、タスクの `index` は `0` です。\n", "\n", "例示を目的として、このチュートリアルでは、`localhost` 上に 2 つのワーカーを持つ `TF_CONFIG` の設定方法を示しています。実際には、外部 IP アドレスとポートに複数のワーカーを作成し、各ワーカーに適切に `TF_CONFIG` を設定します (タスクの `index` を変更します)。\n", "\n", "警告: *Colab で次のコードを実行しないでください。*TensorFlow のランタイムは、指定された IP アドレスとポートに gRPC サーバーを作成しようとしますが、失敗する可能性があります。1 台のマシンで複数のワーカーをテスト実行する方法の例については、このチュートリアルの [keras バージョン](multi_worker_with_keras.ipynb)を参照してください。\n", "\n", "```\n", "os.environ['TF_CONFIG'] = json.dumps({\n", " 'cluster': {\n", " 'worker': [\"localhost:12345\", \"localhost:23456\"]\n", " },\n", " 'task': {'type': 'worker', 'index': 0}\n", "})\n", "```\n" ] }, { "cell_type": "markdown", "metadata": { "id": "qDreJzTffAP5" }, "source": [ "## モデルを定義する\n", "\n", "トレーニング用にレイヤー、オプティマイザ、および損失関数を記述します。このチュートリアルでは、[マルチ GPU トレーニングのチュートリアル](./keras.ipynb)と同様に、Keras レイヤーを使ったモデルを定義しています。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2024-01-11T18:09:29.022700Z", "iopub.status.busy": "2024-01-11T18:09:29.022481Z", "iopub.status.idle": "2024-01-11T18:09:29.028761Z", "shell.execute_reply": "2024-01-11T18:09:29.028181Z" }, "id": "WNvOn_OeiUYC" }, "outputs": [], "source": [ "LEARNING_RATE = 1e-4\n", "def model_fn(features, labels, mode):\n", " model = tf.keras.Sequential([\n", " tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)),\n", " tf.keras.layers.MaxPooling2D(),\n", " tf.keras.layers.Flatten(),\n", " tf.keras.layers.Dense(64, activation='relu'),\n", " tf.keras.layers.Dense(10)\n", " ])\n", " logits = model(features, training=False)\n", "\n", " if mode == tf.estimator.ModeKeys.PREDICT:\n", " predictions = {'logits': logits}\n", " return tf.estimator.EstimatorSpec(labels=labels, predictions=predictions)\n", "\n", " optimizer = tf.compat.v1.train.GradientDescentOptimizer(\n", " learning_rate=LEARNING_RATE)\n", " loss = tf.keras.losses.SparseCategoricalCrossentropy(\n", " from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(labels, logits)\n", " loss = tf.reduce_sum(loss) * (1. / BATCH_SIZE)\n", " if mode == tf.estimator.ModeKeys.EVAL:\n", " return tf.estimator.EstimatorSpec(mode, loss=loss)\n", "\n", " return tf.estimator.EstimatorSpec(\n", " mode=mode,\n", " loss=loss,\n", " train_op=optimizer.minimize(\n", " loss, tf.compat.v1.train.get_or_create_global_step()))" ] }, { "cell_type": "markdown", "metadata": { "id": "P94PrIW_kSCE" }, "source": [ "注意: この例の学習速度は固定されていますが、一般的に、グローバルバッチサイズに基づいて学習速度を調整する必要があります。" ] }, { "cell_type": "markdown", "metadata": { "id": "UhNtHfuxCGVy" }, "source": [ "## MultiWorkerMirroredStrategy\n", "\n", "モデルをトレーニングするために、`tf.distribute.experimental.MultiWorkerMirroredStrategy` のインスタンスを使用します。`MultiWorkerMirroredStrategy` は、すべてのワーカーの各デバイスにあるモデルのレイヤーにすべての変数のコピーを作成します。集合通信に使用する TensorFlow 演算子 `CollectiveOps` を使用して勾配を集め、変数の同期を維持します。このストラテジーの詳細は、[`tf.distribute.Strategy` ガイド](../../guide/distributed_training.ipynb)で説明されています。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2024-01-11T18:09:29.031727Z", "iopub.status.busy": "2024-01-11T18:09:29.031450Z", "iopub.status.idle": "2024-01-11T18:09:30.887381Z", "shell.execute_reply": "2024-01-11T18:09:30.886671Z" }, "id": "1uFSHCJXMrQ-" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_41770/349189047.py:1: _CollectiveAllReduceStrategyExperimental.__init__ (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "use distribute.MultiWorkerMirroredStrategy instead\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Using MirroredStrategy with devices ('/device:GPU:0', '/device:GPU:1', '/device:GPU:2', '/device:GPU:3')\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:GPU:0', '/device:GPU:1', '/device:GPU:2', '/device:GPU:3'), communication = CommunicationImplementation.AUTO\n" ] } ], "source": [ "strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()" ] }, { "cell_type": "markdown", "metadata": { "id": "H47DDcOgfzm7" }, "source": [ "## モデルをトレーニングして評価する\n", "\n", "次に、分散ストラテジーを Estimator の `RunConfig` に指定し、`tf.estimator.train_and_evaluate` を呼び出してトレーニングと評価を行います。このチュートリアルでは、`train_distribute` 経由でストラテジーを指定してトレーニングのみを分散しています。`eval_distribute` を使って評価を分散することもできます。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2024-01-11T18:09:30.890850Z", "iopub.status.busy": "2024-01-11T18:09:30.890584Z", "iopub.status.idle": "2024-01-11T18:09:43.792754Z", "shell.execute_reply": "2024-01-11T18:09:43.791807Z" }, "id": "BcsuBYrpgnlS" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_41770/2557501124.py:1: RunConfig.__init__ (from tensorflow_estimator.python.estimator.run_config) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Initializing RunConfig with distribution strategies.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Not using Distribute Coordinator.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_41770/2557501124.py:3: Estimator.__init__ (from tensorflow_estimator.python.estimator.estimator) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Using config: {'_model_dir': '/tmp/multiworker', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n", "graph_options {\n", " rewrite_options {\n", " meta_optimizer_iterations: ONE\n", " }\n", "}\n", ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': , '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_41770/2557501124.py:7: TrainSpec.__new__ (from tensorflow_estimator.python.estimator.training) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_41770/2557501124.py:8: EvalSpec.__new__ (from tensorflow_estimator.python.estimator.training) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/tmp/ipykernel_41770/2557501124.py:5: train_and_evaluate (from tensorflow_estimator.python.estimator.training) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Not using Distribute Coordinator.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_estimator/python/estimator/estimator.py:1246: StrategyBase.configure (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "use `update_config_proto` instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:The `input_fn` accepts an `input_context` which will be given by DistributionStrategy\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/data/ops/dataset_ops.py:462: UserWarning: To make it possible to preserve tf.data options across serialization boundaries, their implementation has moved to be part of the TensorFlow graph. As a consequence, the options value is in general no longer known at graph construction time. Invoking this method in graph mode retains the legacy behavior of the original implementation, but note that the returned value might not reflect the actual value of the options.\n", " warnings.warn(\"To make it possible to preserve tf.data options across \"\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Collective all_reduce tensors: 6 all_reduces, num_devices = 4, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Collective all_reduce tensors: 6 all_reduces, num_devices = 4, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py:459: EstimatorSpec.__new__ (from tensorflow_estimator.python.estimator.model_fn) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py:459: EstimatorSpec.__new__ (from tensorflow_estimator.python.estimator.model_fn) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 4, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 4, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_estimator/python/estimator/estimator.py:1416: NanTensorHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_estimator/python/estimator/estimator.py:1416: NanTensorHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_estimator/python/estimator/estimator.py:1419: LoggingTensorHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_estimator/python/estimator/estimator.py:1419: LoggingTensorHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/basic_session_run_hooks.py:232: SecondOrStepTimer.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/basic_session_run_hooks.py:232: SecondOrStepTimer.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_estimator/python/estimator/estimator.py:1456: CheckpointSaverHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_estimator/python/estimator/estimator.py:1456: CheckpointSaverHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Create CheckpointSaverHook.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Create CheckpointSaverHook.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:579: StepCounterHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:579: StepCounterHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:586: SummarySaverHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:586: SummarySaverHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_estimator/python/estimator/util.py:95: DistributedIteratorV1.initialize (from tensorflow.python.distribute.v1.input_lib) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use the iterator's `initializer` property instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_estimator/python/estimator/util.py:95: DistributedIteratorV1.initialize (from tensorflow.python.distribute.v1.input_lib) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use the iterator's `initializer` property instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Graph was finalized.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Graph was finalized.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Running local_init_op.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Running local_init_op.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done running local_init_op.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Done running local_init_op.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Saving checkpoints for 0 into /tmp/multiworker/model.ckpt.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Saving checkpoints for 0 into /tmp/multiworker/model.ckpt.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2024-01-11 18:09:37.808408: W tensorflow/core/grappler/utils/graph_view.cc:836] No registered 'MultiDeviceIteratorFromStringHandle' OpKernel for GPU devices compatible with node {{node MultiDeviceIteratorFromStringHandle}}\n", "\t. Registered: device='CPU'\n", "\n", "2024-01-11 18:09:37.809724: W tensorflow/core/grappler/utils/graph_view.cc:836] No registered 'MultiDeviceIteratorGetNextFromShard' OpKernel for GPU devices compatible with node {{node MultiDeviceIteratorGetNextFromShard}}\n", "\t. Registered: device='CPU'\n", "\n", "2024-01-11 18:09:37.816927: W tensorflow/core/grappler/utils/graph_view.cc:836] No registered 'MultiDeviceIteratorFromStringHandle' OpKernel for GPU devices compatible with node {{node MultiDeviceIteratorFromStringHandle}}\n", "\t. Registered: device='CPU'\n", "\n", "2024-01-11 18:09:37.817487: W tensorflow/core/grappler/utils/graph_view.cc:836] No registered 'MultiDeviceIteratorGetNextFromShard' OpKernel for GPU devices compatible with node {{node MultiDeviceIteratorGetNextFromShard}}\n", "\t. Registered: device='CPU'\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:1455: SessionRunArgs.__new__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2024-01-11 18:09:37.839310: W tensorflow/core/grappler/utils/graph_view.cc:836] No registered 'MultiDeviceIteratorFromStringHandle' OpKernel for GPU devices compatible with node {{node MultiDeviceIteratorFromStringHandle}}\n", "\t. Registered: device='CPU'\n", "\n", "2024-01-11 18:09:37.839826: W tensorflow/core/grappler/utils/graph_view.cc:836] No registered 'MultiDeviceIteratorGetNextFromShard' OpKernel for GPU devices compatible with node {{node MultiDeviceIteratorGetNextFromShard}}\n", "\t. Registered: device='CPU'\n", "\n", "2024-01-11 18:09:37.846436: W tensorflow/core/grappler/utils/graph_view.cc:836] No registered 'MultiDeviceIteratorFromStringHandle' OpKernel for GPU devices compatible with node {{node MultiDeviceIteratorFromStringHandle}}\n", "\t. Registered: device='CPU'\n", "\n", "2024-01-11 18:09:37.846962: W tensorflow/core/grappler/utils/graph_view.cc:836] No registered 'MultiDeviceIteratorGetNextFromShard' OpKernel for GPU devices compatible with node {{node MultiDeviceIteratorGetNextFromShard}}\n", "\t. Registered: device='CPU'\n", "\n", "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:1455: SessionRunArgs.__new__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:1454: SessionRunContext.__init__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:1454: SessionRunContext.__init__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:1474: SessionRunValues.__new__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/monitored_session.py:1474: SessionRunValues.__new__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:loss = 2.3079295, step = 0\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:loss = 2.3079295, step = 0\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:global_step/sec: 165.101\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:global_step/sec: 165.101\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:loss = 2.2927566, step = 100 (0.608 sec)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:loss = 2.2927566, step = 100 (0.608 sec)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:global_step/sec: 223.16\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:global_step/sec: 223.16\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:loss = 2.3051863, step = 200 (0.447 sec)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:loss = 2.3051863, step = 200 (0.447 sec)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 234...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 234...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Saving checkpoints for 234 into /tmp/multiworker/model.ckpt.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Saving checkpoints for 234 into /tmp/multiworker/model.ckpt.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 234...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 234...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Starting evaluation at 2024-01-11T18:09:42\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Starting evaluation at 2024-01-11T18:09:42\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/evaluation.py:260: FinalOpsHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/training/evaluation.py:260: FinalOpsHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.keras instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Graph was finalized.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Graph was finalized.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Restoring parameters from /tmp/multiworker/model.ckpt-234\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Restoring parameters from /tmp/multiworker/model.ckpt-234\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Running local_init_op.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Running local_init_op.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done running local_init_op.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Done running local_init_op.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [10/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [10/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [20/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [20/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [30/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [30/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [40/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [40/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [50/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [50/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [60/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [60/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [70/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [70/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [80/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [80/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [90/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [90/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [100/100]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [100/100]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Inference Time : 1.34798s\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Inference Time : 1.34798s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Finished evaluation at 2024-01-11-18:09:43\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Finished evaluation at 2024-01-11-18:09:43\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Saving dict for global step 234: global_step = 234, loss = 2.2969956\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Saving dict for global step 234: global_step = 234, loss = 2.2969956\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 234: /tmp/multiworker/model.ckpt-234\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 234: /tmp/multiworker/model.ckpt-234\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Loss for final step: 2.2864075.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Loss for final step: 2.2864075.\n" ] }, { "data": { "text/plain": [ "({'loss': 2.2969956, 'global_step': 234}, [])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "config = tf.estimator.RunConfig(train_distribute=strategy)\n", "\n", "classifier = tf.estimator.Estimator(\n", " model_fn=model_fn, model_dir='/tmp/multiworker', config=config)\n", "tf.estimator.train_and_evaluate(\n", " classifier,\n", " train_spec=tf.estimator.TrainSpec(input_fn=input_fn),\n", " eval_spec=tf.estimator.EvalSpec(input_fn=input_fn)\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "XVk4ftYx6JAO" }, "source": [ "## トレーニングのパフォーマンスを最適化する\n", "\n", "`tf.distribute.Strategy` により、モデルとマルチワーカー対応 Estimator の準備が整いました。次のテクニックに従って、マルチワーカートレーニングのパフォーマンスを最適化することができます。\n", "\n", "- *バッチサイズの増加:* ここで指定されるバッチサイズは、GPU 単位のサイズです。一般的に、GPU メモリに収まる最大バッチサイズの指定が推奨されます。\n", "\n", "- *変数のキャスト:* 可能であれば、`tf.float` に変数をキャストしてください。公式の ResNet モデルには、どのようにしてこれを行うかの[例](https://github.com/tensorflow/models/blob/8367cf6dabe11adf7628541706b660821f397dce/official/resnet/resnet_model.py#L466)が示されています。\n", "\n", "- *集合通信の使用:* `MultiWorkerMirroredStrategy` は、複数の[集合通信実装](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/distribute/cross_device_ops.py)を提供しています。\n", "\n", " - `RING` は、クロスホスト通信レイヤーとして、gRPC を使用したリング状の集合体を実装します。\n", " - `NCCL` は、[Nvidia の NCCL](https://developer.nvidia.com/nccl) を使用して集合体を実装します。\n", " - `AUTO` は、選択をランタイムに持ち越します。\n", "\n", " 最適な集合体実装の選択肢は、GPU 数と種類、および、クラスタ内の相互接続ネットワークにより異なります。自動選択をオーバーライドするには、`MultiWorkerMirroredStrategy` コンストラクタの `communication` パラメータに、 `communication=tf.distribute.experimental.CollectiveCommunication.NCCL` のように有効な値を指定します。\n", "\n", "ガイドの[パフォーマンスのセクション](../../guide/function.ipynb)に目を通し、独自の TensorFlow モデルのパフォーマンス最適化に使用できるほかのストラテジーや[ツール](../../guide/profiler.md)についてさらに詳しく学習しましょう。\n" ] }, { "cell_type": "markdown", "metadata": { "id": "AW0Hb2xM6EGX" }, "source": [ "## その他のコード例\n", "\n", "1. Kubernetes テンプレートを使った tensorflow/ecosystem でマルチワーカートレーニングを行うための[エンドツーエンドの例](https://github.com/tensorflow/ecosystem/tree/master/distribution_strategy)。この例は最初に Keras モデルを使用し、それを `tf.keras.estimator.model_to_estimator` API を使って Estimator に変換します。\n", "2. [公式モデル](https://github.com/tensorflow/models/tree/master/official)。この多くは、複数の分散ストラテジーで実行するように構成できます。\n" ] } ], "metadata": { "colab": { "collapsed_sections": [ "Tce3stUlHN0L" ], "name": "multi_worker_with_estimator.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" } }, "nbformat": 4, "nbformat_minor": 0 }