{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DjUA6S30k52h"
   },
   "source": [
    "##### Copyright 2021 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:11.314661Z",
     "iopub.status.busy": "2024-05-08T09:12:11.314074Z",
     "iopub.status.idle": "2024-05-08T09:12:11.318136Z",
     "shell.execute_reply": "2024-05-08T09:12:11.317488Z"
    },
    "id": "SpNWyqewk8fE"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6x1ypzczQCwy"
   },
   "source": [
    "# Model analysis using TFX Pipeline and TensorFlow Model Analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HU9YYythm0dx"
   },
   "source": [
    "Note: We recommend running this tutorial in a Colab notebook, with no setup required!  Just click \"Run in Google Colab\".\n",
    "\n",
    "<div class=\"devsite-table-wrapper\"><table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "<td><a target=\"_blank\" href=\"https://www.tensorflow.org/tfx/tutorials/tfx/penguin_tfma\">\n",
    "<img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\"/>View on TensorFlow.org</a></td>\n",
    "<td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/penguin_tfma.ipynb\">\n",
    "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Run in Google Colab</a></td>\n",
    "<td><a target=\"_blank\" href=\"https://github.com/tensorflow/tfx/tree/master/docs/tutorials/tfx/penguin_tfma.ipynb\">\n",
    "<img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">View source on GitHub</a></td>\n",
    "<td><a href=\"https://storage.googleapis.com/tensorflow_docs/tfx/docs/tutorials/tfx/penguin_tfma.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a></td>\n",
    "</table></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_VuwrlnvQJ5k"
   },
   "source": [
    "In this notebook-based tutorial, we will create and run a TFX pipeline\n",
    "which creates a simple classification model and analyzes its performance\n",
    "across multiple runs. This notebook is based on the TFX pipeline we built in\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "If you have not read that tutorial yet, you should read it before proceeding\n",
    "with this notebook.\n",
    "\n",
    "As you tweak your model or train it with a new dataset, you need to check\n",
    "whether your model has improved or become worse. Just checking top-level\n",
    "metrics like accuracy might not be enough. Every trained model should be\n",
    "evaluated before it is pushed to production.\n",
    "\n",
    "We will add an `Evaluator` component to the pipeline created in the previous\n",
    "tutorial. The Evaluator component performs deep analysis for your models and\n",
    "compare the new model against a baseline to determine they are \"good enough\".\n",
    "It is implemented using the\n",
    "[TensorFlow Model Analysis](https://www.tensorflow.org/tfx/guide/tfma) library.\n",
    "\n",
    "Please see\n",
    "[Understanding TFX Pipelines](https://www.tensorflow.org/tfx/guide/understanding_tfx_pipelines)\n",
    "to learn more about various concepts in TFX."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fmgi8ZvQkScg"
   },
   "source": [
    "## Set Up\n",
    "The Set up process is the same as the previous tutorial.\n",
    "\n",
    "We first need to install the TFX Python package and download\n",
    "the dataset which we will use for our model.\n",
    "\n",
    "### Upgrade Pip\n",
    "\n",
    "To avoid upgrading Pip in a system when running locally,\n",
    "check to make sure that we are running in Colab.\n",
    "Local systems can of course be upgraded separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:11.321596Z",
     "iopub.status.busy": "2024-05-08T09:12:11.321352Z",
     "iopub.status.idle": "2024-05-08T09:12:11.329629Z",
     "shell.execute_reply": "2024-05-08T09:12:11.328955Z"
    },
    "id": "as4OTe2ukSqm"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "  import colab\n",
    "  !pip install --upgrade pip\n",
    "except:\n",
    "  pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MZOYTt1RW4TK"
   },
   "source": [
    "### Install TFX\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:11.333089Z",
     "iopub.status.busy": "2024-05-08T09:12:11.332556Z",
     "iopub.status.idle": "2024-05-08T09:12:22.016549Z",
     "shell.execute_reply": "2024-05-08T09:12:22.015660Z"
    },
    "id": "iyQtljP-qPHY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tfx in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (1.15.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: ml-pipelines-sdk==1.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.15.0)\r\n",
      "Requirement already satisfied: absl-py<2.0.0,>=0.9 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.4.0)\r\n",
      "Requirement already satisfied: ml-metadata<1.16.0,>=1.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.15.0)\r\n",
      "Requirement already satisfied: packaging>=22 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (24.0)\r\n",
      "Requirement already satisfied: portpicker<2,>=1.3.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.6.0)\r\n",
      "Requirement already satisfied: protobuf<5,>=3.20.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (3.20.3)\r\n",
      "Requirement already satisfied: docker<5,>=4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (4.4.4)\r\n",
      "Requirement already satisfied: google-apitools<1,>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (0.5.31)\r\n",
      "Requirement already satisfied: google-api-python-client<2,>=1.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.12.11)\r\n",
      "Requirement already satisfied: jinja2<4,>=2.7.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (3.1.4)\r\n",
      "Requirement already satisfied: typing-extensions<5,>=3.10.0.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (4.11.0)\r\n",
      "Requirement already satisfied: apache-beam<3,>=2.47 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.56.0)\r\n",
      "Requirement already satisfied: attrs<24,>=19.3.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (23.2.0)\r\n",
      "Requirement already satisfied: click<9,>=7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (8.1.7)\r\n",
      "Requirement already satisfied: google-api-core<3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (2.19.0)\r\n",
      "Requirement already satisfied: google-cloud-aiplatform<2,>=1.6.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.50.0)\r\n",
      "Requirement already satisfied: google-cloud-bigquery<4,>=3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (3.22.0)\r\n",
      "Requirement already satisfied: grpcio<2,>=1.28.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.63.0)\r\n",
      "Requirement already satisfied: keras-tuner!=1.4.0,!=1.4.1,<2,>=1.0.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.4.7)\r\n",
      "Requirement already satisfied: kubernetes<13,>=10.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (12.0.1)\r\n",
      "Requirement already satisfied: numpy<2,>=1.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.26.4)\r\n",
      "Requirement already satisfied: pyarrow<11,>=10 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (10.0.1)\r\n",
      "Requirement already satisfied: scipy<1.13 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.12.0)\r\n",
      "Requirement already satisfied: pyyaml<7,>=6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (6.0.1)\r\n",
      "Requirement already satisfied: tensorflow<2.16,>=2.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (2.15.1)\r\n",
      "Requirement already satisfied: tensorflow-hub<0.16,>=0.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (0.15.0)\r\n",
      "Requirement already satisfied: tensorflow-data-validation<1.16.0,>=1.15.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.15.1)\r\n",
      "Requirement already satisfied: tensorflow-model-analysis<0.47.0,>=0.46.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (0.46.0)\r\n",
      "Requirement already satisfied: tensorflow-serving-api<2.16,>=2.15 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (2.15.1)\r\n",
      "Requirement already satisfied: tensorflow-transform<1.16.0,>=1.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.15.0)\r\n",
      "Requirement already satisfied: tfx-bsl<1.16.0,>=1.15.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.15.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: crcmod<2.0,>=1.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (1.7)\r\n",
      "Requirement already satisfied: orjson<4,>=3.9.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (3.10.3)\r\n",
      "Requirement already satisfied: dill<0.3.2,>=0.3.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.3.1.1)\r\n",
      "Requirement already satisfied: cloudpickle~=2.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (2.2.1)\r\n",
      "Requirement already satisfied: fastavro<2,>=0.23.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (1.9.4)\r\n",
      "Requirement already satisfied: fasteners<1.0,>=0.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.19)\r\n",
      "Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (2.7.3)\r\n",
      "Requirement already satisfied: httplib2<0.23.0,>=0.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.22.0)\r\n",
      "Requirement already satisfied: jsonschema<5.0.0,>=4.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (4.22.0)\r\n",
      "Requirement already satisfied: jsonpickle<4.0.0,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (3.0.4)\r\n",
      "Requirement already satisfied: objsize<0.8.0,>=0.6.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.7.0)\r\n",
      "Requirement already satisfied: pymongo<5.0.0,>=3.8.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (4.7.2)\r\n",
      "Requirement already satisfied: proto-plus<2,>=1.7.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (1.23.0)\r\n",
      "Requirement already satisfied: pydot<2,>=1.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (1.4.2)\r\n",
      "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (2.9.0.post0)\r\n",
      "Requirement already satisfied: pytz>=2018.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (2024.1)\r\n",
      "Requirement already satisfied: redis<6,>=5.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (5.0.4)\r\n",
      "Requirement already satisfied: regex>=2020.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (2024.4.28)\r\n",
      "Requirement already satisfied: requests<3.0.0,>=2.24.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (2.31.0)\r\n",
      "Requirement already satisfied: zstandard<1,>=0.18.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.22.0)\r\n",
      "Requirement already satisfied: pyarrow-hotfix<1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.6)\r\n",
      "Requirement already satisfied: js2py<1,>=0.74 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.74)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: cachetools<6,>=3.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (5.3.3)\r\n",
      "Requirement already satisfied: google-auth<3,>=1.18.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.29.0)\r\n",
      "Requirement already satisfied: google-auth-httplib2<0.3.0,>=0.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (0.2.0)\r\n",
      "Requirement already satisfied: google-cloud-datastore<3,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.19.0)\r\n",
      "Requirement already satisfied: google-cloud-pubsub<3,>=2.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.21.1)\r\n",
      "Requirement already satisfied: google-cloud-pubsublite<2,>=1.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (1.10.0)\r\n",
      "Requirement already satisfied: google-cloud-storage<3,>=2.14.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.16.0)\r\n",
      "Requirement already satisfied: google-cloud-bigquery-storage<3,>=2.6.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.25.0)\r\n",
      "Requirement already satisfied: google-cloud-core<3,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.4.1)\r\n",
      "Requirement already satisfied: google-cloud-bigtable<3,>=2.19.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.23.1)\r\n",
      "Requirement already satisfied: google-cloud-spanner<4,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (3.46.0)\r\n",
      "Requirement already satisfied: google-cloud-dlp<4,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (3.17.0)\r\n",
      "Requirement already satisfied: google-cloud-language<3,>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.13.3)\r\n",
      "Requirement already satisfied: google-cloud-videointelligence<3,>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.13.3)\r\n",
      "Requirement already satisfied: google-cloud-vision<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (3.7.2)\r\n",
      "Requirement already satisfied: google-cloud-recommendations-ai<0.11.0,>=0.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx) (0.10.10)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: six>=1.4.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from docker<5,>=4.1->tfx) (1.16.0)\r\n",
      "Requirement already satisfied: websocket-client>=0.32.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from docker<5,>=4.1->tfx) (1.8.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-api-core<3->tfx) (1.63.0)\r\n",
      "Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-api-python-client<2,>=1.8->tfx) (3.0.1)\r\n",
      "Requirement already satisfied: oauth2client>=1.4.12 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-apitools<1,>=0.5->tfx) (4.1.3)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx) (1.12.3)\r\n",
      "Requirement already satisfied: shapely<3.0.0dev in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx) (2.0.4)\r\n",
      "Requirement already satisfied: pydantic<3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx) (1.10.15)\r\n",
      "Requirement already satisfied: docstring-parser<1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx) (0.16)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-bigquery<4,>=3->tfx) (2.7.0)\r\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jinja2<4,>=2.7.3->tfx) (2.1.5)\r\n",
      "Requirement already satisfied: keras in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras-tuner!=1.4.0,!=1.4.1,<2,>=1.0.4->tfx) (2.15.0)\r\n",
      "Requirement already satisfied: kt-legacy in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras-tuner!=1.4.0,!=1.4.1,<2,>=1.0.4->tfx) (1.0.5)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: certifi>=14.05.14 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx) (2024.2.2)\r\n",
      "Requirement already satisfied: setuptools>=21.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx) (69.5.1)\r\n",
      "Requirement already satisfied: requests-oauthlib in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx) (2.0.0)\r\n",
      "Requirement already satisfied: urllib3>=1.24.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx) (1.26.18)\r\n",
      "Requirement already satisfied: psutil in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from portpicker<2,>=1.3.1->tfx) (5.9.8)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: astunparse>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (1.6.3)\r\n",
      "Requirement already satisfied: flatbuffers>=23.5.26 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (24.3.25)\r\n",
      "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (0.5.4)\r\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (0.2.0)\r\n",
      "Requirement already satisfied: h5py>=2.9.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (3.11.0)\r\n",
      "Requirement already satisfied: libclang>=13.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (18.1.1)\r\n",
      "Requirement already satisfied: ml-dtypes~=0.3.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (0.3.2)\r\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (3.3.0)\r\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (2.4.0)\r\n",
      "Requirement already satisfied: wrapt<1.15,>=1.11.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (1.14.1)\r\n",
      "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (0.37.0)\r\n",
      "Requirement already satisfied: tensorboard<2.16,>=2.15 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (2.15.2)\r\n",
      "Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx) (2.15.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: joblib>=1.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.16.0,>=1.15.1->tfx) (1.4.2)\r\n",
      "Requirement already satisfied: pandas<2,>=1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.16.0,>=1.15.1->tfx) (1.5.3)\r\n",
      "Requirement already satisfied: pyfarmhash<0.4,>=0.2.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.16.0,>=1.15.1->tfx) (0.3.2)\r\n",
      "Requirement already satisfied: tensorflow-metadata<1.16,>=1.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.16.0,>=1.15.1->tfx) (1.15.0)\r\n",
      "Requirement already satisfied: ipython<8,>=7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (7.34.0)\r\n",
      "Requirement already satisfied: ipywidgets<8,>=7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (7.8.1)\r\n",
      "Requirement already satisfied: pillow>=9.4.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (10.3.0)\r\n",
      "Requirement already satisfied: rouge-score<2,>=0.1.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.1.2)\r\n",
      "Requirement already satisfied: sacrebleu<4,>=2.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.4.2)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from astunparse>=1.6.0->tensorflow<2.16,>=2.15.0->tfx) (0.43.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: grpcio-status<2.0.dev0,>=1.33.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform<2,>=1.6.2->tfx) (1.48.2)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.47->tfx) (0.4.0)\r\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.47->tfx) (4.9)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-bigtable<3,>=2.19.0->apache-beam[gcp]<3,>=2.47->tfx) (0.13.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: overrides<8.0.0,>=6.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.47->tfx) (7.7.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sqlparse>=0.4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]<3,>=2.47->tfx) (0.5.0)\r\n",
      "Requirement already satisfied: grpc-interceptor>=0.15.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]<3,>=2.47->tfx) (0.15.4)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-storage<3,>=2.14.0->apache-beam[gcp]<3,>=2.47->tfx) (1.5.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: docopt in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.6.2)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from httplib2<0.23.0,>=0.8->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (3.1.2)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: jedi>=0.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.19.1)\r\n",
      "Requirement already satisfied: decorator in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (5.1.1)\r\n",
      "Requirement already satisfied: pickleshare in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.7.5)\r\n",
      "Requirement already satisfied: traitlets>=4.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (5.14.3)\r\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (3.0.43)\r\n",
      "Requirement already satisfied: pygments in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.18.0)\r\n",
      "Requirement already satisfied: backcall in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.2.0)\r\n",
      "Requirement already satisfied: matplotlib-inline in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.1.7)\r\n",
      "Requirement already satisfied: pexpect>4.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (4.9.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: comm>=0.1.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.2.2)\r\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.2.0)\r\n",
      "Requirement already satisfied: widgetsnbextension~=3.6.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (3.6.6)\r\n",
      "Requirement already satisfied: jupyterlab-widgets<3,>=1.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.1.7)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tzlocal>=1.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from js2py<1,>=0.74->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (5.2)\r\n",
      "Requirement already satisfied: pyjsparser>=2.5.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from js2py<1,>=0.74->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (2.7.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.0.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (2023.12.1)\r\n",
      "Requirement already satisfied: referencing>=0.28.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.0.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.35.1)\r\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.0.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (0.18.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pyasn1>=0.1.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from oauth2client>=1.4.12->google-apitools<1,>=0.5->tfx) (0.6.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: dnspython<3.0.0,>=1.16.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pymongo<5.0.0,>=3.8.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (2.6.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: async-timeout>=4.0.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from redis<6,>=5.0.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (4.0.3)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: charset-normalizer<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3.0.0,>=2.24.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (3.3.2)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3.0.0,>=2.24.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx) (3.7)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: nltk in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from rouge-score<2,>=0.1.2->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (3.8.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: portalocker in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from sacrebleu<4,>=2.3->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.8.2)\r\n",
      "Requirement already satisfied: tabulate>=0.8.9 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from sacrebleu<4,>=2.3->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.9.0)\r\n",
      "Requirement already satisfied: colorama in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from sacrebleu<4,>=2.3->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.4.6)\r\n",
      "Requirement already satisfied: lxml in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from sacrebleu<4,>=2.3->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (5.2.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx) (1.2.0)\r\n",
      "Requirement already satisfied: markdown>=2.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx) (3.6)\r\n",
      "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx) (0.7.2)\r\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx) (3.0.3)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: oauthlib>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx) (3.2.2)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jedi>=0.16->ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.8.4)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: importlib-metadata>=4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from markdown>=2.6.8->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx) (7.1.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: ptyprocess>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pexpect>4.3->ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.7.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: wcwidth in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.2.13)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: notebook>=4.4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (7.1.3)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tqdm in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nltk->rouge-score<2,>=0.1.2->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (4.66.4)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: zipp>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx) (3.18.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: jupyter-server<3,>=2.4.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.14.0)\r\n",
      "Requirement already satisfied: jupyterlab-server<3,>=2.22.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.27.1)\r\n",
      "Requirement already satisfied: jupyterlab<4.2,>=4.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (4.1.8)\r\n",
      "Requirement already satisfied: notebook-shim<0.3,>=0.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.2.4)\r\n",
      "Requirement already satisfied: tornado>=6.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (6.4)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: anyio>=3.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (4.3.0)\r\n",
      "Requirement already satisfied: argon2-cffi>=21.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (23.1.0)\r\n",
      "Requirement already satisfied: jupyter-client>=7.4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (8.6.1)\r\n",
      "Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (5.7.2)\r\n",
      "Requirement already satisfied: jupyter-events>=0.9.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.10.0)\r\n",
      "Requirement already satisfied: jupyter-server-terminals>=0.4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.5.3)\r\n",
      "Requirement already satisfied: nbconvert>=6.4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (7.16.4)\r\n",
      "Requirement already satisfied: nbformat>=5.3.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (5.10.4)\r\n",
      "Requirement already satisfied: prometheus-client>=0.9 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.20.0)\r\n",
      "Requirement already satisfied: pyzmq>=24 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (26.0.3)\r\n",
      "Requirement already satisfied: send2trash>=1.8.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.8.3)\r\n",
      "Requirement already satisfied: terminado>=0.8.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.18.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: async-lru>=1.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.0.4)\r\n",
      "Requirement already satisfied: httpx>=0.25.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.27.0)\r\n",
      "Requirement already satisfied: ipykernel>=6.5.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (6.29.4)\r\n",
      "Requirement already satisfied: jupyter-lsp>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.2.5)\r\n",
      "Requirement already satisfied: tomli>=1.2.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.0.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: babel>=2.10 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab-server<3,>=2.22.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.15.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: json5>=0.9.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab-server<3,>=2.22.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.9.25)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sniffio>=1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.3.1)\r\n",
      "Requirement already satisfied: exceptiongroup>=1.0.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.2.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: argon2-cffi-bindings in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (21.2.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: httpcore==1.* in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from httpx>=0.25.0->jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.0.5)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: h11<0.15,>=0.13 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from httpcore==1.*->httpx>=0.25.0->jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.14.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: debugpy>=1.6.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=6.5.0->jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.8.1)\r\n",
      "Requirement already satisfied: nest-asyncio in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=6.5.0->jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.6.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: platformdirs>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-core!=5.0.*,>=4.12->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (4.2.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: python-json-logger>=2.0.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.0.7)\r\n",
      "Requirement already satisfied: rfc3339-validator in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.1.4)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: rfc3986-validator>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.1.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: beautifulsoup4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (4.12.3)\r\n",
      "Requirement already satisfied: bleach!=5.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (6.1.0)\r\n",
      "Requirement already satisfied: defusedxml in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.7.1)\r\n",
      "Requirement already satisfied: jupyterlab-pygments in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.3.0)\r\n",
      "Requirement already satisfied: mistune<4,>=2.0.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (3.0.2)\r\n",
      "Requirement already satisfied: nbclient>=0.5.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.10.0)\r\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.5.1)\r\n",
      "Requirement already satisfied: tinycss2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.3.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: fastjsonschema>=2.15 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbformat>=5.3.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.19.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: webencodings in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from bleach!=5.0.0->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (0.5.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: fqdn in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.5.1)\r\n",
      "Requirement already satisfied: isoduration in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (20.11.0)\r\n",
      "Requirement already satisfied: jsonpointer>1.13 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.4)\r\n",
      "Requirement already satisfied: uri-template in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.3.0)\r\n",
      "Requirement already satisfied: webcolors>=1.11 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.13)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: cffi>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.16.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: soupsieve>1.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from beautifulsoup4->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.5)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pycparser in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.22)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: arrow>=0.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (1.3.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: types-python-dateutil>=2.8.10 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from arrow>=0.15.0->isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx) (2.9.0.20240316)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install -U tfx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EwT0nov5QO1M"
   },
   "source": [
    "### Did you restart the runtime?\n",
    "\n",
    "If you are using Google Colab, the first time that you run\n",
    "the cell above, you must restart the runtime by clicking\n",
    "above \"RESTART RUNTIME\" button or using \"Runtime > Restart\n",
    "runtime ...\" menu. This is because of the way that Colab\n",
    "loads packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BDnPgN8UJtzN"
   },
   "source": [
    "Check the TensorFlow and TFX versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:22.021057Z",
     "iopub.status.busy": "2024-05-08T09:12:22.020752Z",
     "iopub.status.idle": "2024-05-08T09:12:28.004269Z",
     "shell.execute_reply": "2024-05-08T09:12:28.003450Z"
    },
    "id": "6jh7vKSRqPHb"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-08 09:12:22.461208: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-05-08 09:12:22.461259: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-05-08 09:12:22.462861: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.15.1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TFX version: 1.15.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print('TensorFlow version: {}'.format(tf.__version__))\n",
    "from tfx import v1 as tfx\n",
    "print('TFX version: {}'.format(tfx.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aDtLdSkvqPHe"
   },
   "source": [
    "### Set up variables\n",
    "\n",
    "There are some variables used to define a pipeline. You can customize these\n",
    "variables as you want. By default all output from the pipeline will be\n",
    "generated under the current directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:28.008314Z",
     "iopub.status.busy": "2024-05-08T09:12:28.007878Z",
     "iopub.status.idle": "2024-05-08T09:12:28.012696Z",
     "shell.execute_reply": "2024-05-08T09:12:28.012049Z"
    },
    "id": "EcUseqJaE2XN"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "PIPELINE_NAME = \"penguin-tfma\"\n",
    "\n",
    "# Output directory to store artifacts generated from the pipeline.\n",
    "PIPELINE_ROOT = os.path.join('pipelines', PIPELINE_NAME)\n",
    "# Path to a SQLite DB file to use as an MLMD storage.\n",
    "METADATA_PATH = os.path.join('metadata', PIPELINE_NAME, 'metadata.db')\n",
    "# Output directory where created models from the pipeline will be exported.\n",
    "SERVING_MODEL_DIR = os.path.join('serving_model', PIPELINE_NAME)\n",
    "\n",
    "from absl import logging\n",
    "logging.set_verbosity(logging.INFO)  # Set default logging level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8F2SRwRLSYGa"
   },
   "source": [
    "### Prepare example data\n",
    "We will use the same\n",
    "[Palmer Penguins dataset](https://allisonhorst.github.io/palmerpenguins/articles/intro.html).\n",
    "\n",
    "There are four numeric features in this dataset which were already normalized\n",
    "to have range [0,1]. We will build a classification model which predicts the\n",
    "`species` of penguins."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "11J7XiCq6AFP"
   },
   "source": [
    "Because TFX ExampleGen reads inputs from a directory, we need to create a\n",
    "directory and copy dataset to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:28.016296Z",
     "iopub.status.busy": "2024-05-08T09:12:28.015735Z",
     "iopub.status.idle": "2024-05-08T09:12:28.161460Z",
     "shell.execute_reply": "2024-05-08T09:12:28.160794Z"
    },
    "id": "4fxMs6u86acP"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('/tmpfs/tmp/tfx-datakcma5ryu/data.csv',\n",
       " <http.client.HTTPMessage at 0x7fc5d80acb80>)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib.request\n",
    "import tempfile\n",
    "\n",
    "DATA_ROOT = tempfile.mkdtemp(prefix='tfx-data')  # Create a temporary directory.\n",
    "_data_url = 'https://raw.githubusercontent.com/tensorflow/tfx/master/tfx/examples/penguin/data/labelled/penguins_processed.csv'\n",
    "_data_filepath = os.path.join(DATA_ROOT, \"data.csv\")\n",
    "urllib.request.urlretrieve(_data_url, _data_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nH6gizcpSwWV"
   },
   "source": [
    "## Create a pipeline\n",
    "\n",
    "We will add an [`Evaluator`](https://www.tensorflow.org/tfx/guide/evaluator)\n",
    "component to the pipeline we created in the\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "\n",
    "An Evaluator component requires input data from an `ExampleGen` component and\n",
    "a model from a `Trainer` component and a\n",
    "[`tfma.EvalConfig`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/EvalConfig)\n",
    "object. We can optionally supply a baseline model which can be used to compare\n",
    "metrics with the newly trained model.\n",
    "\n",
    "An evaluator creates two kinds of output artifacts, `ModelEvaluation` and\n",
    "`ModelBlessing`. ModelEvaluation contains the detailed evaluation result which\n",
    "can be investigated and visualized further with TFMA library. ModelBlessing\n",
    "contains a boolean result whether the model passed given criteria and can be\n",
    "used in later components like a Pusher as a signal.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lOjDv93eS5xV"
   },
   "source": [
    "### Write model training code\n",
    "\n",
    "We will use the same model code as in the\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:28.165008Z",
     "iopub.status.busy": "2024-05-08T09:12:28.164576Z",
     "iopub.status.idle": "2024-05-08T09:12:28.167857Z",
     "shell.execute_reply": "2024-05-08T09:12:28.167169Z"
    },
    "id": "aES7Hv5QTDK3"
   },
   "outputs": [],
   "source": [
    "_trainer_module_file = 'penguin_trainer.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:28.171237Z",
     "iopub.status.busy": "2024-05-08T09:12:28.170627Z",
     "iopub.status.idle": "2024-05-08T09:12:28.177150Z",
     "shell.execute_reply": "2024-05-08T09:12:28.176495Z"
    },
    "id": "Gnc67uQNTDfW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing penguin_trainer.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {_trainer_module_file}\n",
    "\n",
    "# Copied from https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple\n",
    "\n",
    "from typing import List\n",
    "from absl import logging\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow_transform.tf_metadata import schema_utils\n",
    "\n",
    "from tfx.components.trainer.executor import TrainerFnArgs\n",
    "from tfx.components.trainer.fn_args_utils import DataAccessor\n",
    "from tfx_bsl.tfxio import dataset_options\n",
    "from tensorflow_metadata.proto.v0 import schema_pb2\n",
    "\n",
    "_FEATURE_KEYS = [\n",
    "    'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'\n",
    "]\n",
    "_LABEL_KEY = 'species'\n",
    "\n",
    "_TRAIN_BATCH_SIZE = 20\n",
    "_EVAL_BATCH_SIZE = 10\n",
    "\n",
    "# Since we're not generating or creating a schema, we will instead create\n",
    "# a feature spec.  Since there are a fairly small number of features this is\n",
    "# manageable for this dataset.\n",
    "_FEATURE_SPEC = {\n",
    "    **{\n",
    "        feature: tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)\n",
    "           for feature in _FEATURE_KEYS\n",
    "       },\n",
    "    _LABEL_KEY: tf.io.FixedLenFeature(shape=[1], dtype=tf.int64)\n",
    "}\n",
    "\n",
    "\n",
    "def _input_fn(file_pattern: List[str],\n",
    "              data_accessor: DataAccessor,\n",
    "              schema: schema_pb2.Schema,\n",
    "              batch_size: int = 200) -> tf.data.Dataset:\n",
    "  \"\"\"Generates features and label for training.\n",
    "\n",
    "  Args:\n",
    "    file_pattern: List of paths or patterns of input tfrecord files.\n",
    "    data_accessor: DataAccessor for converting input to RecordBatch.\n",
    "    schema: schema of the input data.\n",
    "    batch_size: representing the number of consecutive elements of returned\n",
    "      dataset to combine in a single batch\n",
    "\n",
    "  Returns:\n",
    "    A dataset that contains (features, indices) tuple where features is a\n",
    "      dictionary of Tensors, and indices is a single Tensor of label indices.\n",
    "  \"\"\"\n",
    "  return data_accessor.tf_dataset_factory(\n",
    "      file_pattern,\n",
    "      dataset_options.TensorFlowDatasetOptions(\n",
    "          batch_size=batch_size, label_key=_LABEL_KEY),\n",
    "      schema=schema).repeat()\n",
    "\n",
    "\n",
    "def _build_keras_model() -> tf.keras.Model:\n",
    "  \"\"\"Creates a DNN Keras model for classifying penguin data.\n",
    "\n",
    "  Returns:\n",
    "    A Keras Model.\n",
    "  \"\"\"\n",
    "  # The model below is built with Functional API, please refer to\n",
    "  # https://www.tensorflow.org/guide/keras/overview for all API options.\n",
    "  inputs = [keras.layers.Input(shape=(1,), name=f) for f in _FEATURE_KEYS]\n",
    "  d = keras.layers.concatenate(inputs)\n",
    "  for _ in range(2):\n",
    "    d = keras.layers.Dense(8, activation='relu')(d)\n",
    "  outputs = keras.layers.Dense(3)(d)\n",
    "\n",
    "  model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "  model.compile(\n",
    "      optimizer=keras.optimizers.Adam(1e-2),\n",
    "      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "      metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "\n",
    "  model.summary(print_fn=logging.info)\n",
    "  return model\n",
    "\n",
    "\n",
    "# TFX Trainer will call this function.\n",
    "def run_fn(fn_args: TrainerFnArgs):\n",
    "  \"\"\"Train the model based on given args.\n",
    "\n",
    "  Args:\n",
    "    fn_args: Holds args used to train the model as name/value pairs.\n",
    "  \"\"\"\n",
    "\n",
    "  # This schema is usually either an output of SchemaGen or a manually-curated\n",
    "  # version provided by pipeline author. A schema can also derived from TFT\n",
    "  # graph if a Transform component is used. In the case when either is missing,\n",
    "  # `schema_from_feature_spec` could be used to generate schema from very simple\n",
    "  # feature_spec, but the schema returned would be very primitive.\n",
    "  schema = schema_utils.schema_from_feature_spec(_FEATURE_SPEC)\n",
    "\n",
    "  train_dataset = _input_fn(\n",
    "      fn_args.train_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_TRAIN_BATCH_SIZE)\n",
    "  eval_dataset = _input_fn(\n",
    "      fn_args.eval_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_EVAL_BATCH_SIZE)\n",
    "\n",
    "  model = _build_keras_model()\n",
    "  model.fit(\n",
    "      train_dataset,\n",
    "      steps_per_epoch=fn_args.train_steps,\n",
    "      validation_data=eval_dataset,\n",
    "      validation_steps=fn_args.eval_steps)\n",
    "\n",
    "  # The result of the training should be saved in `fn_args.serving_model_dir`\n",
    "  # directory.\n",
    "  model.save(fn_args.serving_model_dir, save_format='tf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w3OkNz3gTLwM"
   },
   "source": [
    "### Write a pipeline definition\n",
    "\n",
    "We will define a function to create a TFX pipeline. In addition to the\n",
    "Evaluator component we mentioned above, we will add one more node called\n",
    "[`Resolver`](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/dsl/Resolver).\n",
    "To check a new model is getting better than previous model, we need to compare\n",
    "it against a previous published model, called baseline.\n",
    "[ML Metadata(MLMD)](https://www.tensorflow.org/tfx/guide/mlmd) tracks all\n",
    "previous artifacts of the pipeline and `Resolver` can find what was the latest\n",
    "*blessed* model -- a model passed Evaluator successfully -- from MLMD using a\n",
    "strategy class called `LatestBlessedModelStrategy`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:28.180241Z",
     "iopub.status.busy": "2024-05-08T09:12:28.179896Z",
     "iopub.status.idle": "2024-05-08T09:12:28.189741Z",
     "shell.execute_reply": "2024-05-08T09:12:28.189074Z"
    },
    "id": "M49yYVNBTPd4"
   },
   "outputs": [],
   "source": [
    "import tensorflow_model_analysis as tfma\n",
    "\n",
    "def _create_pipeline(pipeline_name: str, pipeline_root: str, data_root: str,\n",
    "                     module_file: str, serving_model_dir: str,\n",
    "                     metadata_path: str) -> tfx.dsl.Pipeline:\n",
    "  \"\"\"Creates a three component penguin pipeline with TFX.\"\"\"\n",
    "  # Brings data into the pipeline.\n",
    "  example_gen = tfx.components.CsvExampleGen(input_base=data_root)\n",
    "\n",
    "  # Uses user-provided Python function that trains a model.\n",
    "  trainer = tfx.components.Trainer(\n",
    "      module_file=module_file,\n",
    "      examples=example_gen.outputs['examples'],\n",
    "      train_args=tfx.proto.TrainArgs(num_steps=100),\n",
    "      eval_args=tfx.proto.EvalArgs(num_steps=5))\n",
    "\n",
    "  # NEW: Get the latest blessed model for Evaluator.\n",
    "  model_resolver = tfx.dsl.Resolver(\n",
    "      strategy_class=tfx.dsl.experimental.LatestBlessedModelStrategy,\n",
    "      model=tfx.dsl.Channel(type=tfx.types.standard_artifacts.Model),\n",
    "      model_blessing=tfx.dsl.Channel(\n",
    "          type=tfx.types.standard_artifacts.ModelBlessing)).with_id(\n",
    "              'latest_blessed_model_resolver')\n",
    "\n",
    "  # NEW: Uses TFMA to compute evaluation statistics over features of a model and\n",
    "  #   perform quality validation of a candidate model (compared to a baseline).\n",
    "\n",
    "  eval_config = tfma.EvalConfig(\n",
    "      model_specs=[tfma.ModelSpec(label_key='species')],\n",
    "      slicing_specs=[\n",
    "          # An empty slice spec means the overall slice, i.e. the whole dataset.\n",
    "          tfma.SlicingSpec(),\n",
    "          # Calculate metrics for each penguin species.\n",
    "          tfma.SlicingSpec(feature_keys=['species']),\n",
    "          ],\n",
    "      metrics_specs=[\n",
    "          tfma.MetricsSpec(per_slice_thresholds={\n",
    "              'sparse_categorical_accuracy':\n",
    "                  tfma.PerSliceMetricThresholds(thresholds=[\n",
    "                      tfma.PerSliceMetricThreshold(\n",
    "                          slicing_specs=[tfma.SlicingSpec()],\n",
    "                          threshold=tfma.MetricThreshold(\n",
    "                              value_threshold=tfma.GenericValueThreshold(\n",
    "                                   lower_bound={'value': 0.6}),\n",
    "                              # Change threshold will be ignored if there is no\n",
    "                              # baseline model resolved from MLMD (first run).\n",
    "                              change_threshold=tfma.GenericChangeThreshold(\n",
    "                                  direction=tfma.MetricDirection.HIGHER_IS_BETTER,\n",
    "                                  absolute={'value': -1e-10}))\n",
    "                       )]),\n",
    "          })],\n",
    "      )\n",
    "  evaluator = tfx.components.Evaluator(\n",
    "      examples=example_gen.outputs['examples'],\n",
    "      model=trainer.outputs['model'],\n",
    "      baseline_model=model_resolver.outputs['model'],\n",
    "      eval_config=eval_config)\n",
    "\n",
    "  # Checks whether the model passed the validation steps and pushes the model\n",
    "  # to a file destination if check passed.\n",
    "  pusher = tfx.components.Pusher(\n",
    "      model=trainer.outputs['model'],\n",
    "      model_blessing=evaluator.outputs['blessing'], # Pass an evaluation result.\n",
    "      push_destination=tfx.proto.PushDestination(\n",
    "          filesystem=tfx.proto.PushDestination.Filesystem(\n",
    "              base_directory=serving_model_dir)))\n",
    "\n",
    "  components = [\n",
    "      example_gen,\n",
    "      trainer,\n",
    "\n",
    "      # Following two components were added to the pipeline.\n",
    "      model_resolver,\n",
    "      evaluator,\n",
    "\n",
    "      pusher,\n",
    "  ]\n",
    "\n",
    "  return tfx.dsl.Pipeline(\n",
    "      pipeline_name=pipeline_name,\n",
    "      pipeline_root=pipeline_root,\n",
    "      metadata_connection_config=tfx.orchestration.metadata\n",
    "      .sqlite_metadata_connection_config(metadata_path),\n",
    "      components=components)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mIcu1LeeQbMt"
   },
   "source": [
    "We need to supply the following information to the Evaluator via `eval_config`:\n",
    "- Additional metrics to configure (if want more metrics than defined in model).\n",
    "- Slices to configure\n",
    "- Model validations thresholds to verify if validation to be included\n",
    "\n",
    "Because `SparseCategoricalAccuracy` was already included in the\n",
    "`model.compile()` call, it will be included in the analysis automatically. So\n",
    "we do not add any additional metrics here. `SparseCategoricalAccuracy` will be\n",
    "used to decide whether the model is good enough, too.\n",
    "\n",
    "We compute the metrics for the whole dataset and for each penguin species.\n",
    "`SlicingSpec` specifies how we aggregate the declared metrics.\n",
    "\n",
    "There are two thresholds that a new model should pass, one is an absolute\n",
    "threshold of 0.6 and the other is a relative threshold that it should\n",
    "be higher than the baseline model. When you run the pipeline for the first\n",
    "time, the `change_threshold` will be ignored and only the value_threshold will\n",
    "be checked. If you run the pipeline more than once, the `Resolver` will find a\n",
    "model from the previous run and it will be used as a baseline model for the\n",
    "comparison.\n",
    "\n",
    "See [Evaluator component guide](https://www.tensorflow.org/tfx/guide/evaluator#using_the_evaluator_component) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mJbq07THU2GV"
   },
   "source": [
    "## Run the pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7mp0AkmrPdUb"
   },
   "source": [
    "We will use `LocalDagRunner` as in the previous tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:28.193560Z",
     "iopub.status.busy": "2024-05-08T09:12:28.192946Z",
     "iopub.status.idle": "2024-05-08T09:12:43.535257Z",
     "shell.execute_reply": "2024-05-08T09:12:43.534561Z"
    },
    "id": "fAtfOZTYWJu-"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Generating ephemeral wheel package for '/tmpfs/src/temp/docs/tutorials/tfx/penguin_trainer.py' (including modules: ['penguin_trainer']).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:User module package has hash fingerprint version 1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Executing: ['/tmpfs/src/tf_docs_env/bin/python', '/tmpfs/tmp/tmprw4uskdx/_tfx_generated_setup.py', 'bdist_wheel', '--bdist-dir', '/tmpfs/tmp/tmp380pw4r5', '--dist-dir', '/tmpfs/tmp/tmp3ooau66m']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.\n",
      "!!\n",
      "\n",
      "        ********************************************************************************\n",
      "        Please avoid running ``setup.py`` directly.\n",
      "        Instead, use pypa/build, pypa/installer or other\n",
      "        standards-based tools.\n",
      "\n",
      "        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.\n",
      "        ********************************************************************************\n",
      "\n",
      "!!\n",
      "  self.initialize_options()\n",
      "INFO:absl:Successfully built user code wheel distribution at 'pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl'; target user module is 'penguin_trainer'.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Full user module path is 'penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Using deployment config:\n",
      " executor_specs {\n",
      "  key: \"CsvExampleGen\"\n",
      "  value {\n",
      "    beam_executable_spec {\n",
      "      python_executor_spec {\n",
      "        class_path: \"tfx.components.example_gen.csv_example_gen.executor.Executor\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "executor_specs {\n",
      "  key: \"Evaluator\"\n",
      "  value {\n",
      "    beam_executable_spec {\n",
      "      python_executor_spec {\n",
      "        class_path: \"tfx.components.evaluator.executor.Executor\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "executor_specs {\n",
      "  key: \"Pusher\"\n",
      "  value {\n",
      "    python_class_executable_spec {\n",
      "      class_path: \"tfx.components.pusher.executor.Executor\"\n",
      "    }\n",
      "  }\n",
      "}\n",
      "executor_specs {\n",
      "  key: \"Trainer\"\n",
      "  value {\n",
      "    python_class_executable_spec {\n",
      "      class_path: \"tfx.components.trainer.executor.GenericExecutor\"\n",
      "    }\n",
      "  }\n",
      "}\n",
      "custom_driver_specs {\n",
      "  key: \"CsvExampleGen\"\n",
      "  value {\n",
      "    python_class_executable_spec {\n",
      "      class_path: \"tfx.components.example_gen.driver.FileBasedDriver\"\n",
      "    }\n",
      "  }\n",
      "}\n",
      "metadata_connection_config {\n",
      "  database_connection_config {\n",
      "    sqlite {\n",
      "      filename_uri: \"metadata/penguin-tfma/metadata.db\"\n",
      "      connection_mode: READWRITE_OPENCREATE\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Using connection config:\n",
      " sqlite {\n",
      "  filename_uri: \"metadata/penguin-tfma/metadata.db\"\n",
      "  connection_mode: READWRITE_OPENCREATE\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component CsvExampleGen is running.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.example_gen.csv_example_gen.component.CsvExampleGen\"\n",
      "  }\n",
      "  id: \"CsvExampleGen\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2024-05-08T09:12:28.606391\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma.CsvExampleGen\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Examples\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          properties {\n",
      "            key: \"version\"\n",
      "            value: INT\n",
      "          }\n",
      "          base_type: DATASET\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"input_base\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"/tmpfs/tmp/tfx-datakcma5ryu\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"input_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"splits\\\": [\\n    {\\n      \\\"name\\\": \\\"single_split\\\",\\n      \\\"pattern\\\": \\\"*\\\"\\n    }\\n  ]\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"split_config\\\": {\\n    \\\"splits\\\": [\\n      {\\n        \\\"hash_buckets\\\": 2,\\n        \\\"name\\\": \\\"train\\\"\\n      },\\n      {\\n        \\\"hash_buckets\\\": 1,\\n        \\\"name\\\": \\\"eval\\\"\\n      }\\n    ]\\n  }\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_data_format\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 6\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_file_format\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 5\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "downstream_nodes: \"Evaluator\"\n",
      "downstream_nodes: \"Trainer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running bdist_wheel\n",
      "running build\n",
      "running build_py\n",
      "creating build\n",
      "creating build/lib\n",
      "copying penguin_trainer.py -> build/lib\n",
      "installing to /tmpfs/tmp/tmp380pw4r5\n",
      "running install\n",
      "running install_lib\n",
      "copying build/lib/penguin_trainer.py -> /tmpfs/tmp/tmp380pw4r5\n",
      "running install_egg_info\n",
      "running egg_info\n",
      "creating tfx_user_code_Trainer.egg-info\n",
      "writing tfx_user_code_Trainer.egg-info/PKG-INFO\n",
      "writing dependency_links to tfx_user_code_Trainer.egg-info/dependency_links.txt\n",
      "writing top-level names to tfx_user_code_Trainer.egg-info/top_level.txt\n",
      "writing manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'\n",
      "reading manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'\n",
      "writing manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'\n",
      "Copying tfx_user_code_Trainer.egg-info to /tmpfs/tmp/tmp380pw4r5/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3.9.egg-info\n",
      "running install_scripts\n",
      "creating /tmpfs/tmp/tmp380pw4r5/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/WHEEL\n",
      "creating '/tmpfs/tmp/tmp3ooau66m/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl' and adding '/tmpfs/tmp/tmp380pw4r5' to it\n",
      "adding 'penguin_trainer.py'\n",
      "adding 'tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/METADATA'\n",
      "adding 'tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/WHEEL'\n",
      "adding 'tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/top_level.txt'\n",
      "adding 'tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/RECORD'\n",
      "removing /tmpfs/tmp/tmp380pw4r5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:[CsvExampleGen] Resolved inputs: ({},)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:select span and version = (0, None)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:latest span and version = (0, None)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Going to run a new execution 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=1, input_dict={}, output_dict=defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: \"pipelines/penguin-tfma/CsvExampleGen/examples/1\"\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")]}), exec_properties={'output_file_format': 5, 'output_config': '{\\n  \"split_config\": {\\n    \"splits\": [\\n      {\\n        \"hash_buckets\": 2,\\n        \"name\": \"train\"\\n      },\\n      {\\n        \"hash_buckets\": 1,\\n        \"name\": \"eval\"\\n      }\\n    ]\\n  }\\n}', 'input_base': '/tmpfs/tmp/tfx-datakcma5ryu', 'input_config': '{\\n  \"splits\": [\\n    {\\n      \"name\": \"single_split\",\\n      \"pattern\": \"*\"\\n    }\\n  ]\\n}', 'output_data_format': 6, 'span': 0, 'version': None, 'input_fingerprint': 'split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548'}, execution_output_uri='pipelines/penguin-tfma/CsvExampleGen/.system/executor_execution/1/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/CsvExampleGen/.system/stateful_working_dir/71dcf2a2-5c57-4eda-b836-d76ab760acc0', tmp_dir='pipelines/penguin-tfma/CsvExampleGen/.system/executor_execution/1/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.example_gen.csv_example_gen.component.CsvExampleGen\"\n",
      "  }\n",
      "  id: \"CsvExampleGen\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2024-05-08T09:12:28.606391\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma.CsvExampleGen\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Examples\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          properties {\n",
      "            key: \"version\"\n",
      "            value: INT\n",
      "          }\n",
      "          base_type: DATASET\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"input_base\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"/tmpfs/tmp/tfx-datakcma5ryu\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"input_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"splits\\\": [\\n    {\\n      \\\"name\\\": \\\"single_split\\\",\\n      \\\"pattern\\\": \\\"*\\\"\\n    }\\n  ]\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"split_config\\\": {\\n    \\\"splits\\\": [\\n      {\\n        \\\"hash_buckets\\\": 2,\\n        \\\"name\\\": \\\"train\\\"\\n      },\\n      {\\n        \\\"hash_buckets\\\": 1,\\n        \\\"name\\\": \\\"eval\\\"\\n      }\\n    ]\\n  }\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_data_format\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 6\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_file_format\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 5\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "downstream_nodes: \"Evaluator\"\n",
      "downstream_nodes: \"Trainer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-tfma\"\n",
      ", pipeline_run_id='2024-05-08T09:12:28.606391', top_level_pipeline_run_id=None, frontend_url=None)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Generating examples.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "        if (typeof window.interactive_beam_jquery == 'undefined') {\n",
       "          var jqueryScript = document.createElement('script');\n",
       "          jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
       "          jqueryScript.type = 'text/javascript';\n",
       "          jqueryScript.onload = function() {\n",
       "            var datatableScript = document.createElement('script');\n",
       "            datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
       "            datatableScript.type = 'text/javascript';\n",
       "            datatableScript.onload = function() {\n",
       "              window.interactive_beam_jquery = jQuery.noConflict(true);\n",
       "              window.interactive_beam_jquery(document).ready(function($){\n",
       "                \n",
       "              });\n",
       "            }\n",
       "            document.head.appendChild(datatableScript);\n",
       "          };\n",
       "          document.head.appendChild(jqueryScript);\n",
       "        } else {\n",
       "          window.interactive_beam_jquery(document).ready(function($){\n",
       "            \n",
       "          });\n",
       "        }"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Processing input csv data /tmpfs/tmp/tfx-datakcma5ryu/* to TFExample.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Examples generated.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Value type <class 'NoneType'> of key version in exec_properties is not supported, going to drop it\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Value type <class 'list'> of key _beam_pipeline_args in exec_properties is not supported, going to drop it\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Cleaning up stateless execution info.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Execution 1 succeeded.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Cleaning up stateful execution info.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Deleted stateful_working_dir pipelines/penguin-tfma/CsvExampleGen/.system/stateful_working_dir/71dcf2a2-5c57-4eda-b836-d76ab760acc0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: \"pipelines/penguin-tfma/CsvExampleGen/examples/1\"\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")]}) for execution 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component CsvExampleGen is finished.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component latest_blessed_model_resolver is running.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.dsl.components.common.resolver.Resolver\"\n",
      "  }\n",
      "  id: \"latest_blessed_model_resolver\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2024-05-08T09:12:28.606391\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma.latest_blessed_model_resolver\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"_generated_model_3\"\n",
      "    value {\n",
      "      channels {\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Model\"\n",
      "            base_type: MODEL\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "      hidden: true\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"_generated_modelblessing_4\"\n",
      "    value {\n",
      "      channels {\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"ModelBlessing\"\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "      hidden: true\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      input_graph_ref {\n",
      "        graph_id: \"graph_1\"\n",
      "        key: \"model\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"model_blessing\"\n",
      "    value {\n",
      "      input_graph_ref {\n",
      "        graph_id: \"graph_1\"\n",
      "        key: \"model_blessing\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  input_graphs {\n",
      "    key: \"graph_1\"\n",
      "    value {\n",
      "      nodes {\n",
      "        key: \"dict_2\"\n",
      "        value {\n",
      "          output_data_type: ARTIFACT_MULTIMAP\n",
      "          dict_node {\n",
      "            node_ids {\n",
      "              key: \"model\"\n",
      "              value: \"input_3\"\n",
      "            }\n",
      "            node_ids {\n",
      "              key: \"model_blessing\"\n",
      "              value: \"input_4\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "      nodes {\n",
      "        key: \"input_3\"\n",
      "        value {\n",
      "          output_data_type: ARTIFACT_LIST\n",
      "          input_node {\n",
      "            input_key: \"_generated_model_3\"\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "      nodes {\n",
      "        key: \"input_4\"\n",
      "        value {\n",
      "          output_data_type: ARTIFACT_LIST\n",
      "          input_node {\n",
      "            input_key: \"_generated_modelblessing_4\"\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "      nodes {\n",
      "        key: \"op_1\"\n",
      "        value {\n",
      "          output_data_type: ARTIFACT_MULTIMAP\n",
      "          op_node {\n",
      "            op_type: \"tfx.dsl.input_resolution.strategies.latest_blessed_model_strategy.LatestBlessedModelStrategy\"\n",
      "            args {\n",
      "              node_id: \"dict_2\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "      result_node: \"op_1\"\n",
      "    }\n",
      "  }\n",
      "}\n",
      "downstream_nodes: \"Evaluator\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Running as an resolver node.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:[latest_blessed_model_resolver] Resolved inputs: ({'model_blessing': [], 'model': []},)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component latest_blessed_model_resolver is finished.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component Trainer is running.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.trainer.component.Trainer\"\n",
      "    base_type: TRAIN\n",
      "  }\n",
      "  id: \"Trainer\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2024-05-08T09:12:28.606391\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma.Trainer\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Model\"\n",
      "          base_type: MODEL\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"model_run\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ModelRun\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"eval_args\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"num_steps\\\": 5\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"module_path\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"train_args\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"num_steps\\\": 100\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "downstream_nodes: \"Evaluator\"\n",
      "downstream_nodes: \"Pusher\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:ArtifactQuery.property_predicate is not supported.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:[Trainer] Resolved inputs: ({'examples': [Artifact(artifact: id: 1\n",
      "type_id: 15\n",
      "uri: \"pipelines/penguin-tfma/CsvExampleGen/examples/1\"\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value {\n",
      "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"file_format\"\n",
      "  value {\n",
      "    string_value: \"tfrecords_gzip\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"payload_format\"\n",
      "  value {\n",
      "    string_value: \"FORMAT_TF_EXAMPLE\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"Examples\"\n",
      "create_time_since_epoch: 1715159549765\n",
      "last_update_time_since_epoch: 1715159549765\n",
      ", artifact_type: id: 15\n",
      "name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")]},)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Going to run a new execution 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=3, input_dict={'examples': [Artifact(artifact: id: 1\n",
      "type_id: 15\n",
      "uri: \"pipelines/penguin-tfma/CsvExampleGen/examples/1\"\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value {\n",
      "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"file_format\"\n",
      "  value {\n",
      "    string_value: \"tfrecords_gzip\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"payload_format\"\n",
      "  value {\n",
      "    string_value: \"FORMAT_TF_EXAMPLE\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"Examples\"\n",
      "create_time_since_epoch: 1715159549765\n",
      "last_update_time_since_epoch: 1715159549765\n",
      ", artifact_type: id: 15\n",
      "name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")]}, output_dict=defaultdict(<class 'list'>, {'model': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Trainer/model/3\"\n",
      ", artifact_type: name: \"Model\"\n",
      "base_type: MODEL\n",
      ")], 'model_run': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Trainer/model_run/3\"\n",
      ", artifact_type: name: \"ModelRun\"\n",
      ")]}), exec_properties={'module_path': 'penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl', 'custom_config': 'null', 'train_args': '{\\n  \"num_steps\": 100\\n}', 'eval_args': '{\\n  \"num_steps\": 5\\n}'}, execution_output_uri='pipelines/penguin-tfma/Trainer/.system/executor_execution/3/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/Trainer/.system/stateful_working_dir/cc01f07b-5e10-46fd-89d4-e0e7d78fb6fe', tmp_dir='pipelines/penguin-tfma/Trainer/.system/executor_execution/3/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.trainer.component.Trainer\"\n",
      "    base_type: TRAIN\n",
      "  }\n",
      "  id: \"Trainer\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2024-05-08T09:12:28.606391\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma.Trainer\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Model\"\n",
      "          base_type: MODEL\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"model_run\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ModelRun\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"eval_args\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"num_steps\\\": 5\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"module_path\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"train_args\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"num_steps\\\": 100\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "downstream_nodes: \"Evaluator\"\n",
      "downstream_nodes: \"Pusher\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-tfma\"\n",
      ", pipeline_run_id='2024-05-08T09:12:28.606391', top_level_pipeline_run_id=None, frontend_url=None)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Train on the 'train' split when train_args.splits is not set.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Evaluate on the 'eval' split when eval_args.splits is not set.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:udf_utils.get_fn {'module_path': 'penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl', 'custom_config': 'null', 'train_args': '{\\n  \"num_steps\": 100\\n}', 'eval_args': '{\\n  \"num_steps\": 5\\n}'} 'run_fn'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Installing 'pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl' to a temporary directory.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Executing: ['/tmpfs/src/tf_docs_env/bin/python', '-m', 'pip', 'install', '--target', '/tmpfs/tmp/tmpflpa4y_q', 'pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing ./pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Successfully installed 'pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl'.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Training model.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: tfx-user-code-Trainer\n",
      "Successfully installed tfx-user-code-Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tfx_bsl/tfxio/tf_example_record.py:343: parse_example_dataset (from tensorflow.python.data.experimental.ops.parsing_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(tf.io.parse_example(...))` instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tfx_bsl/tfxio/tf_example_record.py:343: parse_example_dataset (from tensorflow.python.data.experimental.ops.parsing_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(tf.io.parse_example(...))` instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Model: \"model\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:__________________________________________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: Layer (type)                Output Shape                 Param #   Connected to                  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:==================================================================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: culmen_length_mm (InputLay  [(None, 1)]                  0         []                            \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: er)                                                                                              \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                                                  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: culmen_depth_mm (InputLaye  [(None, 1)]                  0         []                            \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: r)                                                                                               \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                                                  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: flipper_length_mm (InputLa  [(None, 1)]                  0         []                            \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: yer)                                                                                             \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                                                  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: body_mass_g (InputLayer)    [(None, 1)]                  0         []                            \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                                                  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: concatenate (Concatenate)   (None, 4)                    0         ['culmen_length_mm[0][0]',    \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                     'culmen_depth_mm[0][0]',     \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                     'flipper_length_mm[0][0]',   \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                     'body_mass_g[0][0]']         \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                                                  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: dense (Dense)               (None, 8)                    40        ['concatenate[0][0]']         \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                                                  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: dense_1 (Dense)             (None, 8)                    72        ['dense[0][0]']               \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                                                  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: dense_2 (Dense)             (None, 3)                    27        ['dense_1[0][0]']             \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:                                                                                                  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:==================================================================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Total params: 139 (556.00 Byte)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Trainable params: 139 (556.00 Byte)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Non-trainable params: 0 (0.00 Byte)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:__________________________________________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
      "I0000 00:00:1715159557.217578   10799 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "  1/100 [..............................] - ETA: 3:00 - loss: 1.2572 - sparse_categorical_accuracy: 0.1500"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      " 20/100 [=====>........................] - ETA: 0s - loss: 0.9453 - sparse_categorical_accuracy: 0.6550  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      " 40/100 [===========>..................] - ETA: 0s - loss: 0.7846 - sparse_categorical_accuracy: 0.7237"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      " 60/100 [=================>............] - ETA: 0s - loss: 0.6434 - sparse_categorical_accuracy: 0.7683"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      " 80/100 [=======================>......] - ETA: 0s - loss: 0.5395 - sparse_categorical_accuracy: 0.8163"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "100/100 [==============================] - 2s 5ms/step - loss: 0.4620 - sparse_categorical_accuracy: 0.8455 - val_loss: 0.1694 - val_sparse_categorical_accuracy: 0.9400\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Function `_wrapped_model` contains input name(s) resource with unsupported characters which will be renamed to model_dense_2_biasadd_readvariableop_resource in the SavedModel.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: pipelines/penguin-tfma/Trainer/model/3/Format-Serving/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: pipelines/penguin-tfma/Trainer/model/3/Format-Serving/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Writing fingerprint to pipelines/penguin-tfma/Trainer/model/3/Format-Serving/fingerprint.pb\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Training complete. Model written to pipelines/penguin-tfma/Trainer/model/3/Format-Serving. ModelRun written to pipelines/penguin-tfma/Trainer/model_run/3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Cleaning up stateless execution info.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Execution 3 succeeded.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Cleaning up stateful execution info.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Deleted stateful_working_dir pipelines/penguin-tfma/Trainer/.system/stateful_working_dir/cc01f07b-5e10-46fd-89d4-e0e7d78fb6fe\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'model': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Trainer/model/3\"\n",
      ", artifact_type: name: \"Model\"\n",
      "base_type: MODEL\n",
      ")], 'model_run': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Trainer/model_run/3\"\n",
      ", artifact_type: name: \"ModelRun\"\n",
      ")]}) for execution 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component Trainer is finished.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component Evaluator is running.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.evaluator.component.Evaluator\"\n",
      "    base_type: EVALUATE\n",
      "  }\n",
      "  id: \"Evaluator\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2024-05-08T09:12:28.606391\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma.Evaluator\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"baseline_model\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"latest_blessed_model_resolver\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.latest_blessed_model_resolver\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Model\"\n",
      "            base_type: MODEL\n",
      "          }\n",
      "        }\n",
      "        output_key: \"model\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Trainer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.Trainer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Model\"\n",
      "            base_type: MODEL\n",
      "          }\n",
      "        }\n",
      "        output_key: \"model\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"blessing\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ModelBlessing\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"evaluation\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ModelEvaluation\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"eval_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"metrics_specs\\\": [\\n    {\\n      \\\"per_slice_thresholds\\\": {\\n        \\\"sparse_categorical_accuracy\\\": {\\n          \\\"thresholds\\\": [\\n            {\\n              \\\"slicing_specs\\\": [\\n                {}\\n              ],\\n              \\\"threshold\\\": {\\n                \\\"change_threshold\\\": {\\n                  \\\"absolute\\\": -1e-10,\\n                  \\\"direction\\\": \\\"HIGHER_IS_BETTER\\\"\\n                },\\n                \\\"value_threshold\\\": {\\n                  \\\"lower_bound\\\": 0.6\\n                }\\n              }\\n            }\\n          ]\\n        }\\n      }\\n    }\\n  ],\\n  \\\"model_specs\\\": [\\n    {\\n      \\\"label_key\\\": \\\"species\\\"\\n    }\\n  ],\\n  \\\"slicing_specs\\\": [\\n    {},\\n    {\\n      \\\"feature_keys\\\": [\\n        \\\"species\\\"\\n      ]\\n    }\\n  ]\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"example_splits\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"fairness_indicator_thresholds\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "upstream_nodes: \"Trainer\"\n",
      "upstream_nodes: \"latest_blessed_model_resolver\"\n",
      "downstream_nodes: \"Pusher\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:ArtifactQuery.property_predicate is not supported.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:ArtifactQuery.property_predicate is not supported.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:[Evaluator] Resolved inputs: ({'examples': [Artifact(artifact: id: 1\n",
      "type_id: 15\n",
      "uri: \"pipelines/penguin-tfma/CsvExampleGen/examples/1\"\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value {\n",
      "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"file_format\"\n",
      "  value {\n",
      "    string_value: \"tfrecords_gzip\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"payload_format\"\n",
      "  value {\n",
      "    string_value: \"FORMAT_TF_EXAMPLE\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"Examples\"\n",
      "create_time_since_epoch: 1715159549765\n",
      "last_update_time_since_epoch: 1715159549765\n",
      ", artifact_type: id: 15\n",
      "name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")], 'model': [Artifact(artifact: id: 2\n",
      "type_id: 18\n",
      "uri: \"pipelines/penguin-tfma/Trainer/model/3\"\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"Model\"\n",
      "create_time_since_epoch: 1715159558908\n",
      "last_update_time_since_epoch: 1715159558908\n",
      ", artifact_type: id: 18\n",
      "name: \"Model\"\n",
      "base_type: MODEL\n",
      ")], 'baseline_model': []},)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Going to run a new execution 4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=4, input_dict={'examples': [Artifact(artifact: id: 1\n",
      "type_id: 15\n",
      "uri: \"pipelines/penguin-tfma/CsvExampleGen/examples/1\"\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value {\n",
      "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"file_format\"\n",
      "  value {\n",
      "    string_value: \"tfrecords_gzip\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"payload_format\"\n",
      "  value {\n",
      "    string_value: \"FORMAT_TF_EXAMPLE\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"Examples\"\n",
      "create_time_since_epoch: 1715159549765\n",
      "last_update_time_since_epoch: 1715159549765\n",
      ", artifact_type: id: 15\n",
      "name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")], 'model': [Artifact(artifact: id: 2\n",
      "type_id: 18\n",
      "uri: \"pipelines/penguin-tfma/Trainer/model/3\"\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"Model\"\n",
      "create_time_since_epoch: 1715159558908\n",
      "last_update_time_since_epoch: 1715159558908\n",
      ", artifact_type: id: 18\n",
      "name: \"Model\"\n",
      "base_type: MODEL\n",
      ")], 'baseline_model': []}, output_dict=defaultdict(<class 'list'>, {'evaluation': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Evaluator/evaluation/4\"\n",
      ", artifact_type: name: \"ModelEvaluation\"\n",
      ")], 'blessing': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Evaluator/blessing/4\"\n",
      ", artifact_type: name: \"ModelBlessing\"\n",
      ")]}), exec_properties={'eval_config': '{\\n  \"metrics_specs\": [\\n    {\\n      \"per_slice_thresholds\": {\\n        \"sparse_categorical_accuracy\": {\\n          \"thresholds\": [\\n            {\\n              \"slicing_specs\": [\\n                {}\\n              ],\\n              \"threshold\": {\\n                \"change_threshold\": {\\n                  \"absolute\": -1e-10,\\n                  \"direction\": \"HIGHER_IS_BETTER\"\\n                },\\n                \"value_threshold\": {\\n                  \"lower_bound\": 0.6\\n                }\\n              }\\n            }\\n          ]\\n        }\\n      }\\n    }\\n  ],\\n  \"model_specs\": [\\n    {\\n      \"label_key\": \"species\"\\n    }\\n  ],\\n  \"slicing_specs\": [\\n    {},\\n    {\\n      \"feature_keys\": [\\n        \"species\"\\n      ]\\n    }\\n  ]\\n}', 'example_splits': 'null', 'fairness_indicator_thresholds': 'null'}, execution_output_uri='pipelines/penguin-tfma/Evaluator/.system/executor_execution/4/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/Evaluator/.system/stateful_working_dir/156cd629-f1d0-4e6d-8519-c9ad5128ceba', tmp_dir='pipelines/penguin-tfma/Evaluator/.system/executor_execution/4/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.evaluator.component.Evaluator\"\n",
      "    base_type: EVALUATE\n",
      "  }\n",
      "  id: \"Evaluator\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2024-05-08T09:12:28.606391\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma.Evaluator\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"baseline_model\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"latest_blessed_model_resolver\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.latest_blessed_model_resolver\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Model\"\n",
      "            base_type: MODEL\n",
      "          }\n",
      "        }\n",
      "        output_key: \"model\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Trainer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.Trainer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Model\"\n",
      "            base_type: MODEL\n",
      "          }\n",
      "        }\n",
      "        output_key: \"model\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"blessing\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ModelBlessing\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"evaluation\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ModelEvaluation\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"eval_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"metrics_specs\\\": [\\n    {\\n      \\\"per_slice_thresholds\\\": {\\n        \\\"sparse_categorical_accuracy\\\": {\\n          \\\"thresholds\\\": [\\n            {\\n              \\\"slicing_specs\\\": [\\n                {}\\n              ],\\n              \\\"threshold\\\": {\\n                \\\"change_threshold\\\": {\\n                  \\\"absolute\\\": -1e-10,\\n                  \\\"direction\\\": \\\"HIGHER_IS_BETTER\\\"\\n                },\\n                \\\"value_threshold\\\": {\\n                  \\\"lower_bound\\\": 0.6\\n                }\\n              }\\n            }\\n          ]\\n        }\\n      }\\n    }\\n  ],\\n  \\\"model_specs\\\": [\\n    {\\n      \\\"label_key\\\": \\\"species\\\"\\n    }\\n  ],\\n  \\\"slicing_specs\\\": [\\n    {},\\n    {\\n      \\\"feature_keys\\\": [\\n        \\\"species\\\"\\n      ]\\n    }\\n  ]\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"example_splits\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"fairness_indicator_thresholds\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "upstream_nodes: \"Trainer\"\n",
      "upstream_nodes: \"latest_blessed_model_resolver\"\n",
      "downstream_nodes: \"Pusher\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-tfma\"\n",
      ", pipeline_run_id='2024-05-08T09:12:28.606391', top_level_pipeline_run_id=None, frontend_url=None)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:udf_utils.get_fn {'eval_config': '{\\n  \"metrics_specs\": [\\n    {\\n      \"per_slice_thresholds\": {\\n        \"sparse_categorical_accuracy\": {\\n          \"thresholds\": [\\n            {\\n              \"slicing_specs\": [\\n                {}\\n              ],\\n              \"threshold\": {\\n                \"change_threshold\": {\\n                  \"absolute\": -1e-10,\\n                  \"direction\": \"HIGHER_IS_BETTER\"\\n                },\\n                \"value_threshold\": {\\n                  \"lower_bound\": 0.6\\n                }\\n              }\\n            }\\n          ]\\n        }\\n      }\\n    }\\n  ],\\n  \"model_specs\": [\\n    {\\n      \"label_key\": \"species\"\\n    }\\n  ],\\n  \"slicing_specs\": [\\n    {},\\n    {\\n      \"feature_keys\": [\\n        \"species\"\\n      ]\\n    }\\n  ]\\n}', 'example_splits': 'null', 'fairness_indicator_thresholds': 'null'} 'custom_eval_shared_model'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=\n",
      "model_specs {\n",
      "  label_key: \"species\"\n",
      "}\n",
      "slicing_specs {\n",
      "}\n",
      "slicing_specs {\n",
      "  feature_keys: \"species\"\n",
      "}\n",
      "metrics_specs {\n",
      "  per_slice_thresholds {\n",
      "    key: \"sparse_categorical_accuracy\"\n",
      "    value {\n",
      "      thresholds {\n",
      "        slicing_specs {\n",
      "        }\n",
      "        threshold {\n",
      "          value_threshold {\n",
      "            lower_bound {\n",
      "              value: 0.6\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Using pipelines/penguin-tfma/Trainer/model/3/Format-Serving as  model.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:The 'example_splits' parameter is not set, using 'eval' split.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Evaluating model.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:udf_utils.get_fn {'eval_config': '{\\n  \"metrics_specs\": [\\n    {\\n      \"per_slice_thresholds\": {\\n        \"sparse_categorical_accuracy\": {\\n          \"thresholds\": [\\n            {\\n              \"slicing_specs\": [\\n                {}\\n              ],\\n              \"threshold\": {\\n                \"change_threshold\": {\\n                  \"absolute\": -1e-10,\\n                  \"direction\": \"HIGHER_IS_BETTER\"\\n                },\\n                \"value_threshold\": {\\n                  \"lower_bound\": 0.6\\n                }\\n              }\\n            }\\n          ]\\n        }\\n      }\\n    }\\n  ],\\n  \"model_specs\": [\\n    {\\n      \"label_key\": \"species\"\\n    }\\n  ],\\n  \"slicing_specs\": [\\n    {},\\n    {\\n      \"feature_keys\": [\\n        \"species\"\\n      ]\\n    }\\n  ]\\n}', 'example_splits': 'null', 'fairness_indicator_thresholds': 'null'} 'custom_extractors'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=\n",
      "model_specs {\n",
      "  label_key: \"species\"\n",
      "}\n",
      "slicing_specs {\n",
      "}\n",
      "slicing_specs {\n",
      "  feature_keys: \"species\"\n",
      "}\n",
      "metrics_specs {\n",
      "  model_names: \"\"\n",
      "  per_slice_thresholds {\n",
      "    key: \"sparse_categorical_accuracy\"\n",
      "    value {\n",
      "      thresholds {\n",
      "        slicing_specs {\n",
      "        }\n",
      "        threshold {\n",
      "          value_threshold {\n",
      "            lower_bound {\n",
      "              value: 0.6\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=\n",
      "model_specs {\n",
      "  label_key: \"species\"\n",
      "}\n",
      "slicing_specs {\n",
      "}\n",
      "slicing_specs {\n",
      "  feature_keys: \"species\"\n",
      "}\n",
      "metrics_specs {\n",
      "  model_names: \"\"\n",
      "  per_slice_thresholds {\n",
      "    key: \"sparse_categorical_accuracy\"\n",
      "    value {\n",
      "      thresholds {\n",
      "        slicing_specs {\n",
      "        }\n",
      "        threshold {\n",
      "          value_threshold {\n",
      "            lower_bound {\n",
      "              value: 0.6\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:eval_shared_models have model_types: {'tf_keras'}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=\n",
      "model_specs {\n",
      "  label_key: \"species\"\n",
      "}\n",
      "slicing_specs {\n",
      "}\n",
      "slicing_specs {\n",
      "  feature_keys: \"species\"\n",
      "}\n",
      "metrics_specs {\n",
      "  model_names: \"\"\n",
      "  per_slice_thresholds {\n",
      "    key: \"sparse_categorical_accuracy\"\n",
      "    value {\n",
      "      thresholds {\n",
      "        slicing_specs {\n",
      "        }\n",
      "        threshold {\n",
      "          value_threshold {\n",
      "            lower_bound {\n",
      "              value: 0.6\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Evaluation complete. Results written to pipelines/penguin-tfma/Evaluator/evaluation/4.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Checking validation results.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:112: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use eager execution and: \n",
      "`tf.data.TFRecordDataset(path)`\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:112: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use eager execution and: \n",
      "`tf.data.TFRecordDataset(path)`\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Blessing result True written to pipelines/penguin-tfma/Evaluator/blessing/4.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Cleaning up stateless execution info.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Execution 4 succeeded.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Cleaning up stateful execution info.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Deleted stateful_working_dir pipelines/penguin-tfma/Evaluator/.system/stateful_working_dir/156cd629-f1d0-4e6d-8519-c9ad5128ceba\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'evaluation': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Evaluator/evaluation/4\"\n",
      ", artifact_type: name: \"ModelEvaluation\"\n",
      ")], 'blessing': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Evaluator/blessing/4\"\n",
      ", artifact_type: name: \"ModelBlessing\"\n",
      ")]}) for execution 4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component Evaluator is finished.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component Pusher is running.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.pusher.component.Pusher\"\n",
      "    base_type: DEPLOY\n",
      "  }\n",
      "  id: \"Pusher\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2024-05-08T09:12:28.606391\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma.Pusher\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Trainer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.Trainer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Model\"\n",
      "            base_type: MODEL\n",
      "          }\n",
      "        }\n",
      "        output_key: \"model\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"model_blessing\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Evaluator\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.Evaluator\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"ModelBlessing\"\n",
      "          }\n",
      "        }\n",
      "        output_key: \"blessing\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"pushed_model\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"PushedModel\"\n",
      "          base_type: MODEL\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"push_destination\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"filesystem\\\": {\\n    \\\"base_directory\\\": \\\"serving_model/penguin-tfma\\\"\\n  }\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"Evaluator\"\n",
      "upstream_nodes: \"Trainer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:ArtifactQuery.property_predicate is not supported.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:ArtifactQuery.property_predicate is not supported.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:[Pusher] Resolved inputs: ({'model_blessing': [Artifact(artifact: id: 5\n",
      "type_id: 22\n",
      "uri: \"pipelines/penguin-tfma/Evaluator/blessing/4\"\n",
      "custom_properties {\n",
      "  key: \"blessed\"\n",
      "  value {\n",
      "    int_value: 1\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"current_model\"\n",
      "  value {\n",
      "    string_value: \"pipelines/penguin-tfma/Trainer/model/3\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"current_model_id\"\n",
      "  value {\n",
      "    int_value: 2\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"ModelBlessing\"\n",
      "create_time_since_epoch: 1715159563475\n",
      "last_update_time_since_epoch: 1715159563475\n",
      ", artifact_type: id: 22\n",
      "name: \"ModelBlessing\"\n",
      ")], 'model': [Artifact(artifact: id: 2\n",
      "type_id: 18\n",
      "uri: \"pipelines/penguin-tfma/Trainer/model/3\"\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"Model\"\n",
      "create_time_since_epoch: 1715159558908\n",
      "last_update_time_since_epoch: 1715159558908\n",
      ", artifact_type: id: 18\n",
      "name: \"Model\"\n",
      "base_type: MODEL\n",
      ")]},)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Going to run a new execution 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=5, input_dict={'model_blessing': [Artifact(artifact: id: 5\n",
      "type_id: 22\n",
      "uri: \"pipelines/penguin-tfma/Evaluator/blessing/4\"\n",
      "custom_properties {\n",
      "  key: \"blessed\"\n",
      "  value {\n",
      "    int_value: 1\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"current_model\"\n",
      "  value {\n",
      "    string_value: \"pipelines/penguin-tfma/Trainer/model/3\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"current_model_id\"\n",
      "  value {\n",
      "    int_value: 2\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"ModelBlessing\"\n",
      "create_time_since_epoch: 1715159563475\n",
      "last_update_time_since_epoch: 1715159563475\n",
      ", artifact_type: id: 22\n",
      "name: \"ModelBlessing\"\n",
      ")], 'model': [Artifact(artifact: id: 2\n",
      "type_id: 18\n",
      "uri: \"pipelines/penguin-tfma/Trainer/model/3\"\n",
      "custom_properties {\n",
      "  key: \"is_external\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.15.0\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "type: \"Model\"\n",
      "create_time_since_epoch: 1715159558908\n",
      "last_update_time_since_epoch: 1715159558908\n",
      ", artifact_type: id: 18\n",
      "name: \"Model\"\n",
      "base_type: MODEL\n",
      ")]}, output_dict=defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Pusher/pushed_model/5\"\n",
      ", artifact_type: name: \"PushedModel\"\n",
      "base_type: MODEL\n",
      ")]}), exec_properties={'custom_config': 'null', 'push_destination': '{\\n  \"filesystem\": {\\n    \"base_directory\": \"serving_model/penguin-tfma\"\\n  }\\n}'}, execution_output_uri='pipelines/penguin-tfma/Pusher/.system/executor_execution/5/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/Pusher/.system/stateful_working_dir/12b47904-285c-43d9-bb2e-9b1b59dce2f0', tmp_dir='pipelines/penguin-tfma/Pusher/.system/executor_execution/5/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.pusher.component.Pusher\"\n",
      "    base_type: DEPLOY\n",
      "  }\n",
      "  id: \"Pusher\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2024-05-08T09:12:28.606391\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-tfma.Pusher\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Trainer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.Trainer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Model\"\n",
      "            base_type: MODEL\n",
      "          }\n",
      "        }\n",
      "        output_key: \"model\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"model_blessing\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Evaluator\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2024-05-08T09:12:28.606391\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-tfma.Evaluator\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"ModelBlessing\"\n",
      "          }\n",
      "        }\n",
      "        output_key: \"blessing\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"pushed_model\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"PushedModel\"\n",
      "          base_type: MODEL\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"push_destination\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"filesystem\\\": {\\n    \\\"base_directory\\\": \\\"serving_model/penguin-tfma\\\"\\n  }\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"Evaluator\"\n",
      "upstream_nodes: \"Trainer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-tfma\"\n",
      ", pipeline_run_id='2024-05-08T09:12:28.606391', top_level_pipeline_run_id=None, frontend_url=None)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Model version: 1715159563\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Model written to serving path serving_model/penguin-tfma/1715159563.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Model pushed to pipelines/penguin-tfma/Pusher/pushed_model/5.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Cleaning up stateless execution info.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Execution 5 succeeded.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Cleaning up stateful execution info.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Deleted stateful_working_dir pipelines/penguin-tfma/Pusher/.system/stateful_working_dir/12b47904-285c-43d9-bb2e-9b1b59dce2f0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: \"pipelines/penguin-tfma/Pusher/pushed_model/5\"\n",
      ", artifact_type: name: \"PushedModel\"\n",
      "base_type: MODEL\n",
      ")]}) for execution 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Component Pusher is finished.\n"
     ]
    }
   ],
   "source": [
    "tfx.orchestration.LocalDagRunner().run(\n",
    "  _create_pipeline(\n",
    "      pipeline_name=PIPELINE_NAME,\n",
    "      pipeline_root=PIPELINE_ROOT,\n",
    "      data_root=DATA_ROOT,\n",
    "      module_file=_trainer_module_file,\n",
    "      serving_model_dir=SERVING_MODEL_DIR,\n",
    "      metadata_path=METADATA_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ppERq0Mj6xvW"
   },
   "source": [
    "When the pipeline completed, you should be able to see something like following:\n",
    "```\n",
    "INFO:absl:Blessing result True written to pipelines/penguin-tfma/Evaluator/blessing/4.\n",
    "```\n",
    "Or you can also check manually the output directory where the generated\n",
    "artifacts are stored. If you visit\n",
    "`pipelines/penguin-tfma/Evaluator/blessing/` with a file broswer, you can see a\n",
    "file with a name `BLESSED` or `NOT_BLESSED` according to the evaluation result.\n",
    "\n",
    "If the blessing result is `False`, Pusher will refuse to push the model to the\n",
    "`serving_model_dir`, because the model is not good enough to be used in\n",
    "production."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zR9HcqMSTizW"
   },
   "source": [
    "You can run the pipeline again possibly with different evaluation configs. Even\n",
    "if you run the pipeline with the exact same config and dataset, the trained\n",
    "model might be slightly different due to the inherent randomness of the model\n",
    "training which can lead to a `NOT_BLESSED` model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cWcBI-bjoVTO"
   },
   "source": [
    "### Examine outputs of the pipeline\n",
    "\n",
    "You can use TFMA to investigate and visualize the evaluation result in\n",
    "ModelEvaluation artifact."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IXZ0N5GHm_tX"
   },
   "source": [
    "> **NOTE: If you are not on Colab, Install Jupyter Extensions.**\n",
    "You need an TensorFlow Model Analysis extension to see the visualization from\n",
    "TFMA. This extension is already installed on Google Colab, but you might need\n",
    "to install it if you are running this notebook on other environments.\n",
    "See installation direction of Jupyter extension in the\n",
    "[Install guide](https://github.com/tensorflow/model-analysis#installation).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9VIWOBq0opag"
   },
   "source": [
    "#### Get analysis result from output artifacts\n",
    "\n",
    "You can use MLMD APIs to locate these outputs programatically. First, we will\n",
    "define some utility functions to search for the output artifacts that were just\n",
    "produced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:43.539719Z",
     "iopub.status.busy": "2024-05-08T09:12:43.539041Z",
     "iopub.status.idle": "2024-05-08T09:12:43.544335Z",
     "shell.execute_reply": "2024-05-08T09:12:43.543604Z"
    },
    "id": "aiK6zbeAg3X5"
   },
   "outputs": [],
   "source": [
    "from ml_metadata.proto import metadata_store_pb2\n",
    "# Non-public APIs, just for showcase.\n",
    "from tfx.orchestration.portable.mlmd import execution_lib\n",
    "\n",
    "# TODO(b/171447278): Move these functions into the TFX library.\n",
    "\n",
    "def get_latest_artifacts(metadata, pipeline_name, component_id):\n",
    "  \"\"\"Output artifacts of the latest run of the component.\"\"\"\n",
    "  context = metadata.store.get_context_by_type_and_name(\n",
    "      'node', f'{pipeline_name}.{component_id}')\n",
    "  executions = metadata.store.get_executions_by_context(context.id)\n",
    "  latest_execution = max(executions,\n",
    "                         key=lambda e:e.last_update_time_since_epoch)\n",
    "  return execution_lib.get_output_artifacts(metadata, latest_execution.id)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tujLG8sTGZiv"
   },
   "source": [
    "We can find the latest execution of the `Evaluator` component and get output\n",
    "artifacts of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:43.547667Z",
     "iopub.status.busy": "2024-05-08T09:12:43.547199Z",
     "iopub.status.idle": "2024-05-08T09:12:43.557118Z",
     "shell.execute_reply": "2024-05-08T09:12:43.556502Z"
    },
    "id": "4FOo6PV5g5Mm"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    }
   ],
   "source": [
    "# Non-public APIs, just for showcase.\n",
    "from tfx.orchestration.metadata import Metadata\n",
    "from tfx.types import standard_component_specs\n",
    "\n",
    "metadata_connection_config = tfx.orchestration.metadata.sqlite_metadata_connection_config(\n",
    "    METADATA_PATH)\n",
    "\n",
    "with Metadata(metadata_connection_config) as metadata_handler:\n",
    "  # Find output artifacts from MLMD.\n",
    "  evaluator_output = get_latest_artifacts(metadata_handler, PIPELINE_NAME,\n",
    "                                          'Evaluator')\n",
    "  eval_artifact = evaluator_output[standard_component_specs.EVALUATION_KEY][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IXIJR840GpEq"
   },
   "source": [
    "`Evaluator` always returns one evaluation artifact, and we can visualize it\n",
    "using TensorFlow Model Analysis library. For example, following code will\n",
    "render the accuracy metrics for each penguin species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-08T09:12:43.560703Z",
     "iopub.status.busy": "2024-05-08T09:12:43.560124Z",
     "iopub.status.idle": "2024-05-08T09:12:43.571076Z",
     "shell.execute_reply": "2024-05-08T09:12:43.570389Z"
    },
    "id": "wTaKoEHrj0Gs"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d3750a2afc2a4936a4688bac61d287e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "SlicingMetricsViewer(config={'weightedExamplesColumn': 'example_count'}, data=[{'slice': 'species:0', 'metrics…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import tensorflow_model_analysis as tfma\n",
    "\n",
    "eval_result = tfma.load_eval_result(eval_artifact.uri)\n",
    "tfma.view.render_slicing_metrics(eval_result, slicing_column='species')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nSwaiRQ0JYMZ"
   },
   "source": [
    "If you choose 'sparse_categorical_accuracy' in `Show` drop-down list, you can\n",
    "see the accuracy values per species. You might want to add more slices and\n",
    "check whether your model is good for all distribution and if there is any\n",
    "possible bias."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "08R8qvweThRf"
   },
   "source": [
    "## Next steps\n",
    "\n",
    "Learn more on model analysis at\n",
    "[TensorFlow Model Analysis library tutorial](https://www.tensorflow.org/tfx/tutorials/model_analysis/tfma_basic).\n",
    "\n",
    "You can find more resources on https://www.tensorflow.org/tfx/tutorials.\n",
    "\n",
    "Please see\n",
    "[Understanding TFX Pipelines](https://www.tensorflow.org/tfx/guide/understanding_tfx_pipelines)\n",
    "to learn more about various concepts in TFX.\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "DjUA6S30k52h",
    "lOjDv93eS5xV"
   ],
   "name": "penguin_tfma.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "80e2143094bb421383c5fbbbf26100f6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "d3750a2afc2a4936a4688bac61d287e1": {
      "model_module": "tensorflow_model_analysis",
      "model_module_version": "0.46.0",
      "model_name": "SlicingMetricsModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "tensorflow_model_analysis",
       "_model_module_version": "0.46.0",
       "_model_name": "SlicingMetricsModel",
       "_view_count": null,
       "_view_module": "tensorflow_model_analysis",
       "_view_module_version": "0.46.0",
       "_view_name": "SlicingMetricsView",
       "config": {
        "weightedExamplesColumn": "example_count"
       },
       "data": [
        {
         "metrics": {
          "": {
           "": {
            "loss": {
             "doubleValue": 0.06464517116546631
            },
            "sparse_categorical_accuracy": {
             "doubleValue": 1.0
            }
           }
          }
         },
         "slice": "species:0"
        },
        {
         "metrics": {
          "": {
           "": {
            "loss": {
             "doubleValue": 0.4584958255290985
            },
            "sparse_categorical_accuracy": {
             "doubleValue": 0.8181818127632141
            }
           }
          }
         },
         "slice": "species:1"
        },
        {
         "metrics": {
          "": {
           "": {
            "loss": {
             "doubleValue": 0.020718639716506004
            },
            "sparse_categorical_accuracy": {
             "doubleValue": 1.0
            }
           }
          }
         },
         "slice": "species:2"
        }
       ],
       "js_events": [],
       "layout": "IPY_MODEL_80e2143094bb421383c5fbbbf26100f6"
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}