{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "pknVo1kM2wI2" }, "source": [ "##### Copyright 2021 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:32.120681Z", "iopub.status.busy": "2024-05-08T09:24:32.120023Z", "iopub.status.idle": "2024-05-08T09:24:32.124022Z", "shell.execute_reply": "2024-05-08T09:24:32.123392Z" }, "id": "SoFqANDE222Y" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "6x1ypzczQCwy" }, "source": [ "# Reading data from BigQuery with TFX and Vertex Pipelines\n" ] }, { "cell_type": "markdown", "metadata": { "id": "_445qeKq8e3-" }, "source": [ "
\n", "\n", "\n", "\n", "\n", "\n", "
\n", "View on TensorFlow.org\n", "Run in Google Colab\n", "View source on GitHubDownload notebookRun in Google Cloud Vertex AI Workbench
\n" ] }, { "cell_type": "markdown", "metadata": { "id": "_VuwrlnvQJ5k" }, "source": [ "This notebook-based tutorial will use\n", "[Google Cloud BigQuery](https://cloud.google.com/bigquery) as a data source to\n", "train an ML model. The ML pipeline will be constructed using TFX and run on\n", "Google Cloud Vertex Pipelines.\n", "\n", "This notebook is based on the TFX pipeline we built in\n", "[Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple).\n", "If you have not read that tutorial yet, you should read it before proceeding\n", "with this notebook.\n", "\n", "[BigQuery](https://cloud.google.com/bigquery) is serverless, highly scalable,\n", "and cost-effective multi-cloud data warehouse designed for business agility.\n", "TFX can be used to read training data from BigQuery and to\n", "[publish the trained model](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/extensions/google_cloud_big_query/Pusher)\n", "to BigQuery.\n", "\n", "In this tutorial, we will use the `BigQueryExampleGen` component which reads\n", "data from BigQuery to TFX pipelines.\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "cNA00n3irPgE" }, "source": [ "This notebook is intended to be run on\n", "[Google Colab](https://colab.research.google.com/notebooks/intro.ipynb) or on\n", "[AI Platform Notebooks](https://cloud.google.com/ai-platform-notebooks). If you\n", "are not using one of these, you can simply click \"Run in Google Colab\" button\n", "above.\n", "\n", "## Set up\n", "If you have completed\n", "[Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple),\n", "you will have a working GCP project and a GCS bucket and that is all we need\n", "for this tutorial. Please read the preliminary tutorial first if you missed it." ] }, { "cell_type": "markdown", "metadata": { "id": "WJbPaFzKrPgN" }, "source": [ "### Install python packages" ] }, { "cell_type": "markdown", "metadata": { "id": "QVWOEGgMrPgO" }, "source": [ "We will install required Python packages including TFX and KFP to author ML\n", "pipelines and submit jobs to Vertex Pipelines." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:32.127649Z", "iopub.status.busy": "2024-05-08T09:24:32.127290Z", "iopub.status.idle": "2024-05-08T09:24:47.085605Z", "shell.execute_reply": "2024-05-08T09:24:47.084738Z" }, "id": "osJJdvmIrPgP" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pip in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (24.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: tfx<2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx[kfp]<2) (1.15.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: ml-pipelines-sdk==1.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.15.0)\r\n", "Requirement already satisfied: absl-py<2.0.0,>=0.9 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.4.0)\r\n", "Requirement already satisfied: ml-metadata<1.16.0,>=1.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.15.0)\r\n", "Requirement already satisfied: packaging>=22 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (24.0)\r\n", "Requirement already satisfied: portpicker<2,>=1.3.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.6.0)\r\n", "Requirement already satisfied: protobuf<5,>=3.20.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (3.20.3)\r\n", "Requirement already satisfied: docker<5,>=4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (4.4.4)\r\n", "Requirement already satisfied: google-apitools<1,>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (0.5.31)\r\n", "Requirement already satisfied: google-api-python-client<2,>=1.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.12.11)\r\n", "Requirement already satisfied: jinja2<4,>=2.7.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (3.1.4)\r\n", "Requirement already satisfied: typing-extensions<5,>=3.10.0.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (4.11.0)\r\n", "Requirement already satisfied: apache-beam<3,>=2.47 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.56.0)\r\n", "Requirement already satisfied: attrs<24,>=19.3.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (23.2.0)\r\n", "Requirement already satisfied: click<9,>=7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (8.1.7)\r\n", "Requirement already satisfied: google-api-core<3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (2.19.0)\r\n", "Requirement already satisfied: google-cloud-aiplatform<2,>=1.6.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.50.0)\r\n", "Requirement already satisfied: google-cloud-bigquery<4,>=3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (3.22.0)\r\n", "Requirement already satisfied: grpcio<2,>=1.28.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.63.0)\r\n", "Requirement already satisfied: keras-tuner!=1.4.0,!=1.4.1,<2,>=1.0.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.4.7)\r\n", "Requirement already satisfied: kubernetes<13,>=10.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (12.0.1)\r\n", "Requirement already satisfied: numpy<2,>=1.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.26.4)\r\n", "Requirement already satisfied: pyarrow<11,>=10 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (10.0.1)\r\n", "Requirement already satisfied: scipy<1.13 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.12.0)\r\n", "Requirement already satisfied: pyyaml<7,>=6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (6.0.1)\r\n", "Requirement already satisfied: tensorflow<2.16,>=2.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (2.15.1)\r\n", "Requirement already satisfied: tensorflow-hub<0.16,>=0.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (0.15.0)\r\n", "Requirement already satisfied: tensorflow-data-validation<1.16.0,>=1.15.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.15.1)\r\n", "Requirement already satisfied: tensorflow-model-analysis<0.47.0,>=0.46.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (0.46.0)\r\n", "Requirement already satisfied: tensorflow-serving-api<2.16,>=2.15 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (2.15.1)\r\n", "Requirement already satisfied: tensorflow-transform<1.16.0,>=1.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.15.0)\r\n", "Requirement already satisfied: tfx-bsl<1.16.0,>=1.15.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx<2->tfx[kfp]<2) (1.15.1)\r\n", "Requirement already satisfied: kfp<2,>=1.8.14 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx[kfp]<2) (1.8.22)\r\n", "Requirement already satisfied: kfp-pipeline-spec<0.2,>=0.1.10 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx[kfp]<2) (0.1.16)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: crcmod<2.0,>=1.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (1.7)\r\n", "Requirement already satisfied: orjson<4,>=3.9.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (3.10.3)\r\n", "Requirement already satisfied: dill<0.3.2,>=0.3.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.3.1.1)\r\n", "Requirement already satisfied: cloudpickle~=2.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.2.1)\r\n", "Requirement already satisfied: fastavro<2,>=0.23.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (1.9.4)\r\n", "Requirement already satisfied: fasteners<1.0,>=0.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.19)\r\n", "Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.7.3)\r\n", "Requirement already satisfied: httplib2<0.23.0,>=0.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.22.0)\r\n", "Requirement already satisfied: jsonschema<5.0.0,>=4.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (4.22.0)\r\n", "Requirement already satisfied: jsonpickle<4.0.0,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (3.0.4)\r\n", "Requirement already satisfied: objsize<0.8.0,>=0.6.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.7.0)\r\n", "Requirement already satisfied: pymongo<5.0.0,>=3.8.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (4.7.2)\r\n", "Requirement already satisfied: proto-plus<2,>=1.7.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (1.23.0)\r\n", "Requirement already satisfied: pydot<2,>=1.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (1.4.2)\r\n", "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.9.0.post0)\r\n", "Requirement already satisfied: pytz>=2018.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2024.1)\r\n", "Requirement already satisfied: redis<6,>=5.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (5.0.4)\r\n", "Requirement already satisfied: regex>=2020.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2024.4.28)\r\n", "Requirement already satisfied: requests<3.0.0,>=2.24.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.31.0)\r\n", "Requirement already satisfied: zstandard<1,>=0.18.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.22.0)\r\n", "Requirement already satisfied: pyarrow-hotfix<1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.6)\r\n", "Requirement already satisfied: js2py<1,>=0.74 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.74)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: cachetools<6,>=3.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (5.3.3)\r\n", "Requirement already satisfied: google-auth<3,>=1.18.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.29.0)\r\n", "Requirement already satisfied: google-auth-httplib2<0.3.0,>=0.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.2.0)\r\n", "Requirement already satisfied: google-cloud-datastore<3,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.19.0)\r\n", "Requirement already satisfied: google-cloud-pubsub<3,>=2.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.21.1)\r\n", "Requirement already satisfied: google-cloud-pubsublite<2,>=1.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (1.10.0)\r\n", "Requirement already satisfied: google-cloud-storage<3,>=2.14.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.16.0)\r\n", "Requirement already satisfied: google-cloud-bigquery-storage<3,>=2.6.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.25.0)\r\n", "Requirement already satisfied: google-cloud-core<3,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.4.1)\r\n", "Requirement already satisfied: google-cloud-bigtable<3,>=2.19.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.23.1)\r\n", "Requirement already satisfied: google-cloud-spanner<4,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (3.46.0)\r\n", "Requirement already satisfied: google-cloud-dlp<4,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (3.17.0)\r\n", "Requirement already satisfied: google-cloud-language<3,>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.13.3)\r\n", "Requirement already satisfied: google-cloud-videointelligence<3,>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.13.3)\r\n", "Requirement already satisfied: google-cloud-vision<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (3.7.2)\r\n", "Requirement already satisfied: google-cloud-recommendations-ai<0.11.0,>=0.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.10.10)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: six>=1.4.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from docker<5,>=4.1->tfx<2->tfx[kfp]<2) (1.16.0)\r\n", "Requirement already satisfied: websocket-client>=0.32.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from docker<5,>=4.1->tfx<2->tfx[kfp]<2) (1.8.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-api-core<3->tfx<2->tfx[kfp]<2) (1.63.0)\r\n", "Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-api-python-client<2,>=1.8->tfx<2->tfx[kfp]<2) (3.0.1)\r\n", "Requirement already satisfied: oauth2client>=1.4.12 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-apitools<1,>=0.5->tfx<2->tfx[kfp]<2) (4.1.3)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx<2->tfx[kfp]<2) (1.12.3)\r\n", "Requirement already satisfied: shapely<3.0.0dev in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx<2->tfx[kfp]<2) (2.0.4)\r\n", "Requirement already satisfied: pydantic<3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx<2->tfx[kfp]<2) (1.10.15)\r\n", "Requirement already satisfied: docstring-parser<1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx<2->tfx[kfp]<2) (0.16)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-bigquery<4,>=3->tfx<2->tfx[kfp]<2) (2.7.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: MarkupSafe>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jinja2<4,>=2.7.3->tfx<2->tfx[kfp]<2) (2.1.5)\r\n", "Requirement already satisfied: keras in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras-tuner!=1.4.0,!=1.4.1,<2,>=1.0.4->tfx<2->tfx[kfp]<2) (2.15.0)\r\n", "Requirement already satisfied: kt-legacy in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras-tuner!=1.4.0,!=1.4.1,<2,>=1.0.4->tfx<2->tfx[kfp]<2) (1.0.5)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: requests-toolbelt<1,>=0.8.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kfp<2,>=1.8.14->tfx[kfp]<2) (0.10.1)\r\n", "Requirement already satisfied: kfp-server-api<2.0.0,>=1.1.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kfp<2,>=1.8.14->tfx[kfp]<2) (1.8.5)\r\n", "Requirement already satisfied: tabulate<1,>=0.8.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kfp<2,>=1.8.14->tfx[kfp]<2) (0.9.0)\r\n", "Requirement already satisfied: Deprecated<2,>=1.2.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kfp<2,>=1.8.14->tfx[kfp]<2) (1.2.14)\r\n", "Requirement already satisfied: strip-hints<1,>=0.1.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kfp<2,>=1.8.14->tfx[kfp]<2) (0.1.10)\r\n", "Requirement already satisfied: fire<1,>=0.3.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kfp<2,>=1.8.14->tfx[kfp]<2) (0.6.0)\r\n", "Requirement already satisfied: urllib3<2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kfp<2,>=1.8.14->tfx[kfp]<2) (1.26.18)\r\n", "Requirement already satisfied: typer<1.0,>=0.3.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kfp<2,>=1.8.14->tfx[kfp]<2) (0.12.3)\r\n", "Requirement already satisfied: certifi>=14.05.14 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx<2->tfx[kfp]<2) (2024.2.2)\r\n", "Requirement already satisfied: setuptools>=21.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx<2->tfx[kfp]<2) (69.5.1)\r\n", "Requirement already satisfied: requests-oauthlib in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx<2->tfx[kfp]<2) (2.0.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: psutil in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from portpicker<2,>=1.3.1->tfx<2->tfx[kfp]<2) (5.9.8)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: astunparse>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (1.6.3)\r\n", "Requirement already satisfied: flatbuffers>=23.5.26 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (24.3.25)\r\n", "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (0.5.4)\r\n", "Requirement already satisfied: google-pasta>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (0.2.0)\r\n", "Requirement already satisfied: h5py>=2.9.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (3.11.0)\r\n", "Requirement already satisfied: libclang>=13.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (18.1.1)\r\n", "Requirement already satisfied: ml-dtypes~=0.3.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (0.3.2)\r\n", "Requirement already satisfied: opt-einsum>=2.3.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (3.3.0)\r\n", "Requirement already satisfied: termcolor>=1.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (2.4.0)\r\n", "Requirement already satisfied: wrapt<1.15,>=1.11.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (1.14.1)\r\n", "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (0.37.0)\r\n", "Requirement already satisfied: tensorboard<2.16,>=2.15 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (2.15.2)\r\n", "Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (2.15.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: joblib>=1.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.16.0,>=1.15.1->tfx<2->tfx[kfp]<2) (1.4.2)\r\n", "Requirement already satisfied: pandas<2,>=1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.16.0,>=1.15.1->tfx<2->tfx[kfp]<2) (1.5.3)\r\n", "Requirement already satisfied: pyfarmhash<0.4,>=0.2.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.16.0,>=1.15.1->tfx<2->tfx[kfp]<2) (0.3.2)\r\n", "Requirement already satisfied: tensorflow-metadata<1.16,>=1.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.16.0,>=1.15.1->tfx<2->tfx[kfp]<2) (1.15.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: ipython<8,>=7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (7.34.0)\r\n", "Requirement already satisfied: ipywidgets<8,>=7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (7.8.1)\r\n", "Requirement already satisfied: pillow>=9.4.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (10.3.0)\r\n", "Requirement already satisfied: rouge-score<2,>=0.1.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.1.2)\r\n", "Requirement already satisfied: sacrebleu<4,>=2.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.4.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: wheel<1.0,>=0.23.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from astunparse>=1.6.0->tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (0.43.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: grpcio-status<2.0.dev0,>=1.33.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform<2,>=1.6.2->tfx<2->tfx[kfp]<2) (1.48.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pyasn1-modules>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.4.0)\r\n", "Requirement already satisfied: rsa<5,>=3.1.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (4.9)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-bigtable<3,>=2.19.0->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.13.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: overrides<8.0.0,>=6.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (7.7.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: sqlparse>=0.4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.5.0)\r\n", "Requirement already satisfied: grpc-interceptor>=0.15.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.15.4)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-storage<3,>=2.14.0->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (1.5.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: docopt in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.6.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from httplib2<0.23.0,>=0.8->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (3.1.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: jedi>=0.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.19.1)\r\n", "Requirement already satisfied: decorator in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (5.1.1)\r\n", "Requirement already satisfied: pickleshare in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.7.5)\r\n", "Requirement already satisfied: traitlets>=4.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (5.14.3)\r\n", "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (3.0.43)\r\n", "Requirement already satisfied: pygments in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.18.0)\r\n", "Requirement already satisfied: backcall in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.2.0)\r\n", "Requirement already satisfied: matplotlib-inline in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.1.7)\r\n", "Requirement already satisfied: pexpect>4.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (4.9.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: comm>=0.1.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.2.2)\r\n", "Requirement already satisfied: ipython-genutils~=0.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.2.0)\r\n", "Requirement already satisfied: widgetsnbextension~=3.6.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (3.6.6)\r\n", "Requirement already satisfied: jupyterlab-widgets<3,>=1.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.1.7)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: tzlocal>=1.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from js2py<1,>=0.74->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (5.2)\r\n", "Requirement already satisfied: pyjsparser>=2.5.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from js2py<1,>=0.74->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.7.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.0.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2023.12.1)\r\n", "Requirement already satisfied: referencing>=0.28.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.0.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.35.1)\r\n", "Requirement already satisfied: rpds-py>=0.7.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.0.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (0.18.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pyasn1>=0.1.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from oauth2client>=1.4.12->google-apitools<1,>=0.5->tfx<2->tfx[kfp]<2) (0.6.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: dnspython<3.0.0,>=1.16.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pymongo<5.0.0,>=3.8.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (2.6.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: async-timeout>=4.0.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from redis<6,>=5.0.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (4.0.3)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: charset-normalizer<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3.0.0,>=2.24.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (3.3.2)\r\n", "Requirement already satisfied: idna<4,>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3.0.0,>=2.24.0->apache-beam<3,>=2.47->apache-beam[gcp]<3,>=2.47->tfx<2->tfx[kfp]<2) (3.7)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: nltk in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from rouge-score<2,>=0.1.2->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (3.8.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: portalocker in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from sacrebleu<4,>=2.3->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.8.2)\r\n", "Requirement already satisfied: colorama in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from sacrebleu<4,>=2.3->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.4.6)\r\n", "Requirement already satisfied: lxml in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from sacrebleu<4,>=2.3->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (5.2.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (1.2.0)\r\n", "Requirement already satisfied: markdown>=2.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (3.6)\r\n", "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (0.7.2)\r\n", "Requirement already satisfied: werkzeug>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (3.0.3)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: shellingham>=1.3.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from typer<1.0,>=0.3.2->kfp<2,>=1.8.14->tfx[kfp]<2) (1.5.4)\r\n", "Requirement already satisfied: rich>=10.11.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from typer<1.0,>=0.3.2->kfp<2,>=1.8.14->tfx[kfp]<2) (13.7.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: oauthlib>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx<2->tfx[kfp]<2) (3.2.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jedi>=0.16->ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.8.4)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: importlib-metadata>=4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from markdown>=2.6.8->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (7.1.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: ptyprocess>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pexpect>4.3->ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.7.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: wcwidth in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.2.13)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: markdown-it-py>=2.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from rich>=10.11.0->typer<1.0,>=0.3.2->kfp<2,>=1.8.14->tfx[kfp]<2) (3.0.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: notebook>=4.4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (7.1.3)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: tqdm in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nltk->rouge-score<2,>=0.1.2->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (4.66.4)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: zipp>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tfx<2->tfx[kfp]<2) (3.18.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: mdurl~=0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0,>=0.3.2->kfp<2,>=1.8.14->tfx[kfp]<2) (0.1.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: jupyter-server<3,>=2.4.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.14.0)\r\n", "Requirement already satisfied: jupyterlab-server<3,>=2.22.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.27.1)\r\n", "Requirement already satisfied: jupyterlab<4.2,>=4.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (4.1.8)\r\n", "Requirement already satisfied: notebook-shim<0.3,>=0.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.2.4)\r\n", "Requirement already satisfied: tornado>=6.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (6.4)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: anyio>=3.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (4.3.0)\r\n", "Requirement already satisfied: argon2-cffi>=21.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (23.1.0)\r\n", "Requirement already satisfied: jupyter-client>=7.4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (8.6.1)\r\n", "Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (5.7.2)\r\n", "Requirement already satisfied: jupyter-events>=0.9.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.10.0)\r\n", "Requirement already satisfied: jupyter-server-terminals>=0.4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.5.3)\r\n", "Requirement already satisfied: nbconvert>=6.4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (7.16.4)\r\n", "Requirement already satisfied: nbformat>=5.3.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (5.10.4)\r\n", "Requirement already satisfied: prometheus-client>=0.9 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.20.0)\r\n", "Requirement already satisfied: pyzmq>=24 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (26.0.3)\r\n", "Requirement already satisfied: send2trash>=1.8.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.8.3)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: terminado>=0.8.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.18.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: async-lru>=1.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.0.4)\r\n", "Requirement already satisfied: httpx>=0.25.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.27.0)\r\n", "Requirement already satisfied: ipykernel>=6.5.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (6.29.4)\r\n", "Requirement already satisfied: jupyter-lsp>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.2.5)\r\n", "Requirement already satisfied: tomli>=1.2.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.0.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: babel>=2.10 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab-server<3,>=2.22.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.15.0)\r\n", "Requirement already satisfied: json5>=0.9.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyterlab-server<3,>=2.22.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.9.25)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: sniffio>=1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.3.1)\r\n", "Requirement already satisfied: exceptiongroup>=1.0.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.2.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: argon2-cffi-bindings in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (21.2.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: httpcore==1.* in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from httpx>=0.25.0->jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.0.5)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: h11<0.15,>=0.13 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from httpcore==1.*->httpx>=0.25.0->jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.14.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: debugpy>=1.6.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=6.5.0->jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.8.1)\r\n", "Requirement already satisfied: nest-asyncio in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=6.5.0->jupyterlab<4.2,>=4.1.1->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.6.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: platformdirs>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-core!=5.0.*,>=4.12->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (4.2.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: python-json-logger>=2.0.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.0.7)\r\n", "Requirement already satisfied: rfc3339-validator in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.1.4)\r\n", "Requirement already satisfied: rfc3986-validator>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.1.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: beautifulsoup4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (4.12.3)\r\n", "Requirement already satisfied: bleach!=5.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (6.1.0)\r\n", "Requirement already satisfied: defusedxml in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.7.1)\r\n", "Requirement already satisfied: jupyterlab-pygments in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.3.0)\r\n", "Requirement already satisfied: mistune<4,>=2.0.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (3.0.2)\r\n", "Requirement already satisfied: nbclient>=0.5.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.10.0)\r\n", "Requirement already satisfied: pandocfilters>=1.4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.5.1)\r\n", "Requirement already satisfied: tinycss2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.3.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: fastjsonschema>=2.15 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbformat>=5.3.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.19.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: webencodings in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from bleach!=5.0.0->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (0.5.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: fqdn in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.5.1)\r\n", "Requirement already satisfied: isoduration in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (20.11.0)\r\n", "Requirement already satisfied: jsonpointer>1.13 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.4)\r\n", "Requirement already satisfied: uri-template in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.3.0)\r\n", "Requirement already satisfied: webcolors>=1.11 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.13)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: cffi>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.16.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: soupsieve>1.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from beautifulsoup4->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.5)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pycparser in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.22)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: arrow>=0.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (1.3.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: types-python-dateutil>=2.8.10 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from arrow>=0.15.0->isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7->tensorflow-model-analysis<0.47.0,>=0.46.0->tfx<2->tfx[kfp]<2) (2.9.0.20240316)\r\n" ] } ], "source": [ "# Use the latest version of pip.\n", "!pip install --upgrade pip\n", "!pip install --upgrade \"tfx[kfp]<2\"" ] }, { "cell_type": "markdown", "metadata": { "id": "X5GiQFjprPgP" }, "source": [ "#### Did you restart the runtime?\n", "\n", "If you are using Google Colab, the first time that you run\n", "the cell above, you must restart the runtime by clicking\n", "above \"RESTART RUNTIME\" button or using \"Runtime > Restart\n", "runtime ...\" menu. This is because of the way that Colab\n", "loads packages." ] }, { "cell_type": "markdown", "metadata": { "id": "y3TRhlDvrPgQ" }, "source": [ "If you are not on Colab, you can restart runtime with following cell." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "JYKpuhamrPgQ" }, "outputs": [], "source": [ "# docs_infra: no_execute\n", "import sys\n", "if not 'google.colab' in sys.modules:\n", " # Automatically restart kernel after installs\n", " import IPython\n", " app = IPython.Application.instance()\n", " app.kernel.do_shutdown(True)" ] }, { "cell_type": "markdown", "metadata": { "id": "NjGI8rzFrPgQ" }, "source": [ "### Login in to Google for this notebook\n", "If you are running this notebook on Colab, authenticate with your user account:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:47.090320Z", "iopub.status.busy": "2024-05-08T09:24:47.090035Z", "iopub.status.idle": "2024-05-08T09:24:47.094365Z", "shell.execute_reply": "2024-05-08T09:24:47.093716Z" }, "id": "FY8IqqnmrPgQ" }, "outputs": [], "source": [ "import sys\n", "if 'google.colab' in sys.modules:\n", " from google.colab import auth\n", " auth.authenticate_user()" ] }, { "cell_type": "markdown", "metadata": { "id": "KI3AVxMPrPgQ" }, "source": [ "**If you are on AI Platform Notebooks**, authenticate with Google Cloud before\n", "running the next section, by running\n", "```sh\n", "gcloud auth login\n", "```\n", "**in the Terminal window** (which you can open via **File** > **New** in the\n", "menu). You only need to do this once per notebook instance." ] }, { "cell_type": "markdown", "metadata": { "id": "g3pkMt6zrPgQ" }, "source": [ "Check the package versions." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:47.097562Z", "iopub.status.busy": "2024-05-08T09:24:47.097325Z", "iopub.status.idle": "2024-05-08T09:24:53.140043Z", "shell.execute_reply": "2024-05-08T09:24:53.139237Z" }, "id": "mvZS3XW2rPgR" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-05-08 09:24:47.541894: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", "2024-05-08 09:24:47.541946: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", "2024-05-08 09:24:47.543649: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "TensorFlow version: 2.15.1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "TFX version: 1.15.0\n", "KFP version: 1.8.22\n" ] } ], "source": [ "import tensorflow as tf\n", "print('TensorFlow version: {}'.format(tf.__version__))\n", "from tfx import v1 as tfx\n", "print('TFX version: {}'.format(tfx.__version__))\n", "import kfp\n", "print('KFP version: {}'.format(kfp.__version__))" ] }, { "cell_type": "markdown", "metadata": { "id": "aDtLdSkvqPHe" }, "source": [ "### Set up variables\n", "\n", "We will set up some variables used to customize the pipelines below. Following\n", "information is required:\n", "\n", "* GCP Project id and number. See\n", "[Identifying your project id and number](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects).\n", "* GCP Region to run pipelines. For more information about the regions that\n", "Vertex Pipelines is available in, see the\n", "[Vertex AI locations guide](https://cloud.google.com/vertex-ai/docs/general/locations#feature-availability).\n", "* Google Cloud Storage Bucket to store pipeline outputs.\n", "\n", "**Enter required values in the cell below before running it**.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:53.143875Z", "iopub.status.busy": "2024-05-08T09:24:53.143406Z", "iopub.status.idle": "2024-05-08T09:24:53.148810Z", "shell.execute_reply": "2024-05-08T09:24:53.148106Z" }, "id": "EcUseqJaE2XN" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "ERROR:absl:Please set all required parameters.\n" ] } ], "source": [ "GOOGLE_CLOUD_PROJECT = '' # <--- ENTER THIS\n", "GOOGLE_CLOUD_PROJECT_NUMBER = '' # <--- ENTER THIS\n", "GOOGLE_CLOUD_REGION = '' # <--- ENTER THIS\n", "GCS_BUCKET_NAME = '' # <--- ENTER THIS\n", "\n", "if not (GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_PROJECT_NUMBER and GOOGLE_CLOUD_REGION and GCS_BUCKET_NAME):\n", " from absl import logging\n", " logging.error('Please set all required parameters.')" ] }, { "cell_type": "markdown", "metadata": { "id": "GAaCPLjgiJrO" }, "source": [ "Set `gcloud` to use your project." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:53.152132Z", "iopub.status.busy": "2024-05-08T09:24:53.151538Z", "iopub.status.idle": "2024-05-08T09:24:54.043085Z", "shell.execute_reply": "2024-05-08T09:24:54.042099Z" }, "id": "VkWdxe4TXRHk" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1;31mERROR:\u001b[0m (gcloud.config.set) argument VALUE: Must be specified.\r\n", "Usage: gcloud config set SECTION/PROPERTY VALUE [optional flags]\r\n", " optional flags may be --help | --installation\r\n", "\r\n", "For detailed information on this command and its flags, run:\r\n", " gcloud config set --help\r\n" ] } ], "source": [ "!gcloud config set project {GOOGLE_CLOUD_PROJECT}" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:54.047379Z", "iopub.status.busy": "2024-05-08T09:24:54.047093Z", "iopub.status.idle": "2024-05-08T09:24:54.053336Z", "shell.execute_reply": "2024-05-08T09:24:54.052604Z" }, "id": "CPN6UL5CazNy" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "PIPELINE_ROOT: gs:///pipeline_root/penguin-bigquery\n" ] } ], "source": [ "PIPELINE_NAME = 'penguin-bigquery'\n", "\n", "# Path to various pipeline artifact.\n", "PIPELINE_ROOT = 'gs://{}/pipeline_root/{}'.format(\n", " GCS_BUCKET_NAME, PIPELINE_NAME)\n", "\n", "# Paths for users' Python module.\n", "MODULE_ROOT = 'gs://{}/pipeline_module/{}'.format(\n", " GCS_BUCKET_NAME, PIPELINE_NAME)\n", "\n", "# Paths for users' data.\n", "DATA_ROOT = 'gs://{}/data/{}'.format(GCS_BUCKET_NAME, PIPELINE_NAME)\n", "\n", "# This is the path where your model will be pushed for serving.\n", "SERVING_MODEL_DIR = 'gs://{}/serving_model/{}'.format(\n", " GCS_BUCKET_NAME, PIPELINE_NAME)\n", "\n", "print('PIPELINE_ROOT: {}'.format(PIPELINE_ROOT))" ] }, { "cell_type": "markdown", "metadata": { "id": "szKDDoD_KipW" }, "source": [ "By default the Vertex Pipelines uses the default GCE VM service account of\n", "format `[project-number]-compute@developer.gserviceaccount.com`. We need to\n", "give a permission to use BigQuery to this account to access BigQuery in the\n", "pipeline. We will add 'BigQuery User' role to the account." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:54.057190Z", "iopub.status.busy": "2024-05-08T09:24:54.056537Z", "iopub.status.idle": "2024-05-08T09:24:54.938852Z", "shell.execute_reply": "2024-05-08T09:24:54.937863Z" }, "id": "4aii8K3dJEyj" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1;31mERROR:\u001b[0m (gcloud.projects.add-iam-policy-binding) argument PROJECT_ID: Must be specified.\r\n", "Usage: gcloud projects add-iam-policy-binding PROJECT_ID --member=PRINCIPAL --role=ROLE [optional flags]\r\n", " optional flags may be --condition | --condition-from-file | --help\r\n", "\r\n", "For detailed information on this command and its flags, run:\r\n", " gcloud projects add-iam-policy-binding --help\r\n" ] } ], "source": [ "!gcloud projects add-iam-policy-binding {GOOGLE_CLOUD_PROJECT} \\\n", " --member=serviceAccount:{GOOGLE_CLOUD_PROJECT_NUMBER}-compute@developer.gserviceaccount.com \\\n", " --role=roles/bigquery.user" ] }, { "cell_type": "markdown", "metadata": { "id": "v3ktk1j9s1PP" }, "source": [ "Please see\n", "[Vertex documentation](https://cloud.google.com/vertex-ai/docs/pipelines/configure-project)\n", "to learn more about service accounts and IAM configuration." ] }, { "cell_type": "markdown", "metadata": { "id": "nH6gizcpSwWV" }, "source": [ "## Create a pipeline\n", "\n", "TFX pipelines are defined using Python APIs as we did in\n", "[Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple).\n", "We previously used `CsvExampleGen` which reads data from a CSV file. In this\n", "tutorial, we will use\n", "[`BigQueryExampleGen`](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/extensions/google_cloud_big_query/BigQueryExampleGen)\n", "component which reads data from BigQuery.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "hNg73Slwn8nq" }, "source": [ "### Prepare BigQuery query\n", "\n", "We will use the same\n", "[Palmer Penguins dataset](https://allisonhorst.github.io/palmerpenguins/articles/intro.html). However, we will read it from a BigQuery table\n", "`tfx-oss-public.palmer_penguins.palmer_penguins` which is populated using the\n", "same CSV file.\n", "\n", "If you are using Google Colab, you can examine the content of the BigQuery\n", "table directly." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Mb_Kj1U8pBhZ" }, "outputs": [], "source": [ "# docs_infra: no_execute\n", "%%bigquery --project {GOOGLE_CLOUD_PROJECT}\n", "SELECT *\n", "FROM `tfx-oss-public.palmer_penguins.palmer_penguins`\n", "LIMIT 5" ] }, { "cell_type": "markdown", "metadata": { "id": "arvdbM5jpjNm" }, "source": [ "All features were already normalized to 0~1 except `species` which is the\n", "label. We will build a classification model which predicts the `species` of\n", "penguins.\n", "\n", "`BigQueryExampleGen` requires a query to specify which data to fetch. Because\n", "we will use all the fields of all rows in the table, the query is quite simple.\n", "You can also specify field names and add `WHERE` conditions as needed according\n", "to the\n", "[BigQuery Standard SQL syntax](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax)." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:54.944231Z", "iopub.status.busy": "2024-05-08T09:24:54.943452Z", "iopub.status.idle": "2024-05-08T09:24:54.947623Z", "shell.execute_reply": "2024-05-08T09:24:54.946911Z" }, "id": "7AwysGAVnfJA" }, "outputs": [], "source": [ "QUERY = \"SELECT * FROM `tfx-oss-public.palmer_penguins.palmer_penguins`\"" ] }, { "cell_type": "markdown", "metadata": { "id": "lOjDv93eS5xV" }, "source": [ "### Write model code.\n", "\n", "We will use the same model code as in the\n", "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple)." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:54.950848Z", "iopub.status.busy": "2024-05-08T09:24:54.950594Z", "iopub.status.idle": "2024-05-08T09:24:54.953808Z", "shell.execute_reply": "2024-05-08T09:24:54.953115Z" }, "id": "aES7Hv5QTDK3" }, "outputs": [], "source": [ "_trainer_module_file = 'penguin_trainer.py'" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:54.957574Z", "iopub.status.busy": "2024-05-08T09:24:54.957337Z", "iopub.status.idle": "2024-05-08T09:24:54.964235Z", "shell.execute_reply": "2024-05-08T09:24:54.963460Z" }, "id": "Gnc67uQNTDfW" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Writing penguin_trainer.py\n" ] } ], "source": [ "%%writefile {_trainer_module_file}\n", "\n", "# Copied from https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple\n", "\n", "from typing import List\n", "from absl import logging\n", "import tensorflow as tf\n", "from tensorflow import keras\n", "from tensorflow_transform.tf_metadata import schema_utils\n", "\n", "from tfx import v1 as tfx\n", "from tfx_bsl.public import tfxio\n", "\n", "from tensorflow_metadata.proto.v0 import schema_pb2\n", "\n", "_FEATURE_KEYS = [\n", " 'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'\n", "]\n", "_LABEL_KEY = 'species'\n", "\n", "_TRAIN_BATCH_SIZE = 20\n", "_EVAL_BATCH_SIZE = 10\n", "\n", "# Since we're not generating or creating a schema, we will instead create\n", "# a feature spec. Since there are a fairly small number of features this is\n", "# manageable for this dataset.\n", "_FEATURE_SPEC = {\n", " **{\n", " feature: tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)\n", " for feature in _FEATURE_KEYS\n", " },\n", " _LABEL_KEY: tf.io.FixedLenFeature(shape=[1], dtype=tf.int64)\n", "}\n", "\n", "\n", "def _input_fn(file_pattern: List[str],\n", " data_accessor: tfx.components.DataAccessor,\n", " schema: schema_pb2.Schema,\n", " batch_size: int) -> tf.data.Dataset:\n", " \"\"\"Generates features and label for training.\n", "\n", " Args:\n", " file_pattern: List of paths or patterns of input tfrecord files.\n", " data_accessor: DataAccessor for converting input to RecordBatch.\n", " schema: schema of the input data.\n", " batch_size: representing the number of consecutive elements of returned\n", " dataset to combine in a single batch\n", "\n", " Returns:\n", " A dataset that contains (features, indices) tuple where features is a\n", " dictionary of Tensors, and indices is a single Tensor of label indices.\n", " \"\"\"\n", " return data_accessor.tf_dataset_factory(\n", " file_pattern,\n", " tfxio.TensorFlowDatasetOptions(\n", " batch_size=batch_size, label_key=_LABEL_KEY),\n", " schema=schema).repeat()\n", "\n", "\n", "def _make_keras_model() -> tf.keras.Model:\n", " \"\"\"Creates a DNN Keras model for classifying penguin data.\n", "\n", " Returns:\n", " A Keras Model.\n", " \"\"\"\n", " # The model below is built with Functional API, please refer to\n", " # https://www.tensorflow.org/guide/keras/overview for all API options.\n", " inputs = [keras.layers.Input(shape=(1,), name=f) for f in _FEATURE_KEYS]\n", " d = keras.layers.concatenate(inputs)\n", " for _ in range(2):\n", " d = keras.layers.Dense(8, activation='relu')(d)\n", " outputs = keras.layers.Dense(3)(d)\n", "\n", " model = keras.Model(inputs=inputs, outputs=outputs)\n", " model.compile(\n", " optimizer=keras.optimizers.Adam(1e-2),\n", " loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n", " metrics=[keras.metrics.SparseCategoricalAccuracy()])\n", "\n", " model.summary(print_fn=logging.info)\n", " return model\n", "\n", "\n", "# TFX Trainer will call this function.\n", "def run_fn(fn_args: tfx.components.FnArgs):\n", " \"\"\"Train the model based on given args.\n", "\n", " Args:\n", " fn_args: Holds args used to train the model as name/value pairs.\n", " \"\"\"\n", "\n", " # This schema is usually either an output of SchemaGen or a manually-curated\n", " # version provided by pipeline author. A schema can also derived from TFT\n", " # graph if a Transform component is used. In the case when either is missing,\n", " # `schema_from_feature_spec` could be used to generate schema from very simple\n", " # feature_spec, but the schema returned would be very primitive.\n", " schema = schema_utils.schema_from_feature_spec(_FEATURE_SPEC)\n", "\n", " train_dataset = _input_fn(\n", " fn_args.train_files,\n", " fn_args.data_accessor,\n", " schema,\n", " batch_size=_TRAIN_BATCH_SIZE)\n", " eval_dataset = _input_fn(\n", " fn_args.eval_files,\n", " fn_args.data_accessor,\n", " schema,\n", " batch_size=_EVAL_BATCH_SIZE)\n", "\n", " model = _make_keras_model()\n", " model.fit(\n", " train_dataset,\n", " steps_per_epoch=fn_args.train_steps,\n", " validation_data=eval_dataset,\n", " validation_steps=fn_args.eval_steps)\n", "\n", " # The result of the training should be saved in `fn_args.serving_model_dir`\n", " # directory.\n", " model.save(fn_args.serving_model_dir, save_format='tf')" ] }, { "cell_type": "markdown", "metadata": { "id": "-LsYx8MpYvPv" }, "source": [ "Copy the module file to GCS which can be accessed from the pipeline components.\n", "Because model training happens on GCP, we need to upload this model definition.\n", "\n", "Otherwise, you might want to build a container image including the module file\n", "and use the image to run the pipeline." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:54.967711Z", "iopub.status.busy": "2024-05-08T09:24:54.967255Z", "iopub.status.idle": "2024-05-08T09:24:56.494857Z", "shell.execute_reply": "2024-05-08T09:24:56.493890Z" }, "id": "rMMs5wuNYAbc" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "InvalidUrlError: Cloud URL scheme should be followed by colon and two slashes: \"://\". Found: \"gs:///pipeline_module/penguin-bigquery/\".\r\n" ] } ], "source": [ "!gsutil cp {_trainer_module_file} {MODULE_ROOT}/" ] }, { "cell_type": "markdown", "metadata": { "id": "w3OkNz3gTLwM" }, "source": [ "### Write a pipeline definition\n", "\n", "We will define a function to create a TFX pipeline. We need to use\n", "`BigQueryExampleGen` which takes `query` as an argument. One more change from\n", "the previous tutorial is that we need to pass `beam_pipeline_args` which is\n", "passed to components when they are executed. We will use `beam_pipeline_args`\n", "to pass additional parameters to BigQuery.\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2024-05-08T09:24:56.500001Z", "iopub.status.busy": "2024-05-08T09:24:56.499248Z", "iopub.status.idle": "2024-05-08T09:24:56.506804Z", "shell.execute_reply": "2024-05-08T09:24:56.506184Z" }, "id": "M49yYVNBTPd4" }, "outputs": [], "source": [ "from typing import List, Optional\n", "\n", "def _create_pipeline(pipeline_name: str, pipeline_root: str, query: str,\n", " module_file: str, serving_model_dir: str,\n", " beam_pipeline_args: Optional[List[str]],\n", " ) -> tfx.dsl.Pipeline:\n", " \"\"\"Creates a TFX pipeline using BigQuery.\"\"\"\n", "\n", " # NEW: Query data in BigQuery as a data source.\n", " example_gen = tfx.extensions.google_cloud_big_query.BigQueryExampleGen(\n", " query=query)\n", "\n", " # Uses user-provided Python function that trains a model.\n", " trainer = tfx.components.Trainer(\n", " module_file=module_file,\n", " examples=example_gen.outputs['examples'],\n", " train_args=tfx.proto.TrainArgs(num_steps=100),\n", " eval_args=tfx.proto.EvalArgs(num_steps=5))\n", "\n", " # Pushes the model to a file destination.\n", " pusher = tfx.components.Pusher(\n", " model=trainer.outputs['model'],\n", " push_destination=tfx.proto.PushDestination(\n", " filesystem=tfx.proto.PushDestination.Filesystem(\n", " base_directory=serving_model_dir)))\n", "\n", " components = [\n", " example_gen,\n", " trainer,\n", " pusher,\n", " ]\n", "\n", " return tfx.dsl.Pipeline(\n", " pipeline_name=pipeline_name,\n", " pipeline_root=pipeline_root,\n", " components=components,\n", " # NEW: `beam_pipeline_args` is required to use BigQueryExampleGen.\n", " beam_pipeline_args=beam_pipeline_args)" ] }, { "cell_type": "markdown", "metadata": { "id": "mJbq07THU2GV" }, "source": [ "## Run the pipeline on Vertex Pipelines.\n", "\n", "We will use Vertex Pipelines to run the pipeline as we did in\n", "[Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple).\n" ] }, { "cell_type": "markdown", "metadata": { "id": "7mp0AkmrPdUb" }, "source": [ "We also need to pass `beam_pipeline_args` for the BigQueryExampleGen. It\n", "includes configs like the name of the GCP project and the temporary storage for\n", "the BigQuery execution." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fAtfOZTYWJu-" }, "outputs": [], "source": [ "# docs_infra: no_execute\n", "import os\n", "\n", "# We need to pass some GCP related configs to BigQuery. This is currently done\n", "# using `beam_pipeline_args` parameter.\n", "BIG_QUERY_WITH_DIRECT_RUNNER_BEAM_PIPELINE_ARGS = [\n", " '--project=' + GOOGLE_CLOUD_PROJECT,\n", " '--temp_location=' + os.path.join('gs://', GCS_BUCKET_NAME, 'tmp'),\n", " ]\n", "\n", "PIPELINE_DEFINITION_FILE = PIPELINE_NAME + '_pipeline.json'\n", "\n", "runner = tfx.orchestration.experimental.KubeflowV2DagRunner(\n", " config=tfx.orchestration.experimental.KubeflowV2DagRunnerConfig(),\n", " output_filename=PIPELINE_DEFINITION_FILE)\n", "_ = runner.run(\n", " _create_pipeline(\n", " pipeline_name=PIPELINE_NAME,\n", " pipeline_root=PIPELINE_ROOT,\n", " query=QUERY,\n", " module_file=os.path.join(MODULE_ROOT, _trainer_module_file),\n", " serving_model_dir=SERVING_MODEL_DIR,\n", " beam_pipeline_args=BIG_QUERY_WITH_DIRECT_RUNNER_BEAM_PIPELINE_ARGS))" ] }, { "cell_type": "markdown", "metadata": { "id": "fWyITYSDd8w4" }, "source": [ "The generated definition file can be submitted using kfp client." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tI71jlEvWMV7" }, "outputs": [], "source": [ "# docs_infra: no_execute\n", "from google.cloud import aiplatform\n", "from google.cloud.aiplatform import pipeline_jobs\n", "import logging\n", "logging.getLogger().setLevel(logging.INFO)\n", "\n", "aiplatform.init(project=GOOGLE_CLOUD_PROJECT, location=GOOGLE_CLOUD_REGION)\n", "\n", "job = pipeline_jobs.PipelineJob(template_path=PIPELINE_DEFINITION_FILE,\n", " display_name=PIPELINE_NAME)\n", "job.submit()" ] }, { "cell_type": "markdown", "metadata": { "id": "L3k9f5IVQXcQ" }, "source": [ "Now you can visit the link in the output above or visit 'Vertex AI > Pipelines'\n", "in [Google Cloud Console](https://console.cloud.google.com/) to see the\n", "progress." ] } ], "metadata": { "colab": { "collapsed_sections": [ "pknVo1kM2wI2" ], "name": "Reading data from BigQuery with TFX and Vertex Pipelines", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.19" } }, "nbformat": 4, "nbformat_minor": 0 }