{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "wdeKOEkv1Fe8"
},
"source": [
"##### Copyright 2021 The TensorFlow Authors."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:24:50.657621Z",
"iopub.status.busy": "2023-07-28T10:24:50.657326Z",
"iopub.status.idle": "2023-07-28T10:24:50.662348Z",
"shell.execute_reply": "2023-07-28T10:24:50.661531Z"
},
"id": "SpNWyqewk8fE"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hPxSVqcvqPHO"
},
"source": [
"# TFX Python function component tutorial\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WdRDkO2wQHUw"
},
"source": [
"Note: We recommend running this tutorial in a Colab notebook, with no setup\n",
"required! Just click \"Run in Google Colab\".\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GkwEwr0FQKJW"
},
"source": [
"\n",
"This notebook contains an examples on how to author and run Python function\n",
"components within the TFX InteractiveContext and in a locally-orchestrated TFX\n",
"pipeline.\n",
"\n",
"For more context and information, see the [Custom Python function components](https://www.tensorflow.org/tfx/guide/custom_function_component)\n",
"page on the TFX documentation site."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d-Av6cm0oBFV"
},
"source": [
"## Setup\n",
"\n",
"We will first install TFX and import necessary modules. TFX requires Python 3."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fwZ0aXisoBFW"
},
"source": [
"### Check the system Python version\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:24:50.666901Z",
"iopub.status.busy": "2023-07-28T10:24:50.666668Z",
"iopub.status.idle": "2023-07-28T10:24:50.677670Z",
"shell.execute_reply": "2023-07-28T10:24:50.676690Z"
},
"id": "PQ-QwavmqPHP",
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"'3.9.17 (main, Jun 6 2023, 20:11:04) \\n[GCC 9.4.0]'"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import sys\n",
"sys.version"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "88XXV3_DQ_Nx"
},
"source": [
"### Upgrade Pip\n",
"\n",
"To avoid upgrading Pip in a system when running locally, check to make sure\n",
"that we're running in Colab. Local systems can of course be upgraded\n",
"separately."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:24:50.723876Z",
"iopub.status.busy": "2023-07-28T10:24:50.723580Z",
"iopub.status.idle": "2023-07-28T10:24:50.728148Z",
"shell.execute_reply": "2023-07-28T10:24:50.727305Z"
},
"id": "UHvIH-wORCuV"
},
"outputs": [],
"source": [
"try:\n",
" import colab\n",
" !pip install --upgrade pip\n",
"except:\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FDFSXMLLRGJC"
},
"source": [
"### Install TFX\n",
"\n",
"**Note: In Google Colab, because of package updates, the first time you run\n",
"this cell you must restart the runtime (Runtime > Restart runtime ...).**"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:24:50.731837Z",
"iopub.status.busy": "2023-07-28T10:24:50.731508Z",
"iopub.status.idle": "2023-07-28T10:24:59.790846Z",
"shell.execute_reply": "2023-07-28T10:24:59.789791Z"
},
"id": "wGpQOmYIVlSV"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: tfx in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (1.13.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: ml-pipelines-sdk==1.13.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.13.0)\r\n",
"Requirement already satisfied: absl-py<2.0.0,>=0.9 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.4.0)\r\n",
"Requirement already satisfied: ml-metadata<1.14.0,>=1.13.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.13.1)\r\n",
"Requirement already satisfied: packaging<21,>=20 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (20.9)\r\n",
"Requirement already satisfied: portpicker<2,>=1.3.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.5.2)\r\n",
"Requirement already satisfied: protobuf<5,>=3.20.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (3.20.3)\r\n",
"Requirement already satisfied: docker<5,>=4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (4.4.4)\r\n",
"Requirement already satisfied: google-apitools<1,>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (0.5.31)\r\n",
"Requirement already satisfied: google-api-python-client<2,>=1.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.12.11)\r\n",
"Requirement already satisfied: jinja2<4,>=2.7.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (3.1.2)\r\n",
"Requirement already satisfied: typing-extensions<5,>=3.10.0.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (4.5.0)\r\n",
"Requirement already satisfied: apache-beam[gcp]<3,>=2.40 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (2.48.0)\r\n",
"Requirement already satisfied: attrs<22,>=19.3.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (21.4.0)\r\n",
"Requirement already satisfied: click<9,>=7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (8.1.6)\r\n",
"Requirement already satisfied: google-api-core<1.33 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.32.0)\r\n",
"Requirement already satisfied: google-cloud-aiplatform<1.18,>=1.6.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.17.1)\r\n",
"Requirement already satisfied: google-cloud-bigquery<3,>=2.26.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (2.34.4)\r\n",
"Requirement already satisfied: grpcio<2,>=1.28.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.57.0rc1)\r\n",
"Requirement already satisfied: keras-tuner<2,>=1.0.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.3.5)\r\n",
"Requirement already satisfied: kubernetes<13,>=10.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (12.0.1)\r\n",
"Requirement already satisfied: numpy<2,>=1.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.22.4)\r\n",
"Requirement already satisfied: pyarrow<7,>=6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (6.0.1)\r\n",
"Requirement already satisfied: pyyaml<6,>=3.12 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (5.4.1)\r\n",
"Requirement already satisfied: tensorflow<2.13,>=2.12.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (2.12.1)\r\n",
"Requirement already satisfied: tensorflow-hub<0.13,>=0.9.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (0.12.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: tensorflow-data-validation<1.14.0,>=1.13.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.13.0)\r\n",
"Requirement already satisfied: tensorflow-model-analysis<0.45.0,>=0.44.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (0.44.0)\r\n",
"Requirement already satisfied: tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (2.12.2)\r\n",
"Requirement already satisfied: tensorflow-transform<1.14.0,>=1.13.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.13.0)\r\n",
"Requirement already satisfied: tfx-bsl<1.14.0,>=1.13.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tfx) (1.13.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: crcmod<2.0,>=1.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (1.7)\r\n",
"Requirement already satisfied: orjson<4.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (3.9.2)\r\n",
"Requirement already satisfied: dill<0.3.2,>=0.3.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (0.3.1.1)\r\n",
"Requirement already satisfied: cloudpickle~=2.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.2.1)\r\n",
"Requirement already satisfied: fastavro<2,>=0.23.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (1.8.2)\r\n",
"Requirement already satisfied: fasteners<1.0,>=0.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (0.18)\r\n",
"Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.7.0)\r\n",
"Requirement already satisfied: httplib2<0.23.0,>=0.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (0.22.0)\r\n",
"Requirement already satisfied: objsize<0.7.0,>=0.6.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (0.6.1)\r\n",
"Requirement already satisfied: pymongo<5.0.0,>=3.8.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (4.4.1)\r\n",
"Requirement already satisfied: proto-plus<2,>=1.7.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (1.22.3)\r\n",
"Requirement already satisfied: pydot<2,>=1.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (1.4.2)\r\n",
"Requirement already satisfied: python-dateutil<3,>=2.8.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.8.2)\r\n",
"Requirement already satisfied: pytz>=2018.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2023.3)\r\n",
"Requirement already satisfied: regex>=2020.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2023.6.3)\r\n",
"Requirement already satisfied: requests<3.0.0,>=2.24.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.31.0)\r\n",
"Requirement already satisfied: zstandard<1,>=0.18.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (0.21.0)\r\n",
"Requirement already satisfied: cachetools<6,>=3.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (4.2.4)\r\n",
"Requirement already satisfied: google-auth<3,>=1.18.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (1.35.0)\r\n",
"Requirement already satisfied: google-auth-httplib2<0.2.0,>=0.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (0.1.0)\r\n",
"Requirement already satisfied: google-cloud-datastore<3,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.11.0)\r\n",
"Requirement already satisfied: google-cloud-pubsub<3,>=2.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.13.11)\r\n",
"Requirement already satisfied: google-cloud-pubsublite<2,>=1.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (1.6.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: google-cloud-bigquery-storage<3,>=2.6.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.16.2)\r\n",
"Requirement already satisfied: google-cloud-core<3,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.3.3)\r\n",
"Requirement already satisfied: google-cloud-bigtable<2.18.0,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.15.0)\r\n",
"Requirement already satisfied: google-cloud-spanner<4,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (3.26.0)\r\n",
"Requirement already satisfied: google-cloud-dlp<4,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (3.9.2)\r\n",
"Requirement already satisfied: google-cloud-language<3,>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.6.1)\r\n",
"Requirement already satisfied: google-cloud-videointelligence<3,>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (2.8.3)\r\n",
"Requirement already satisfied: google-cloud-vision<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (3.1.4)\r\n",
"Requirement already satisfied: google-cloud-recommendations-ai<0.11.0,>=0.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from apache-beam[gcp]<3,>=2.40->tfx) (0.7.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: six>=1.4.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from docker<5,>=4.1->tfx) (1.16.0)\r\n",
"Requirement already satisfied: websocket-client>=0.32.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from docker<5,>=4.1->tfx) (1.6.1)\r\n",
"Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-api-core<1.33->tfx) (1.59.1)\r\n",
"Requirement already satisfied: setuptools>=40.3.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-api-core<1.33->tfx) (68.0.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-api-python-client<2,>=1.8->tfx) (3.0.1)\r\n",
"Requirement already satisfied: oauth2client>=1.4.12 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-apitools<1,>=0.5->tfx) (4.1.3)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<1.18,>=1.6.2->tfx) (2.10.0)\r\n",
"Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-aiplatform<1.18,>=1.6.2->tfx) (1.6.3)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-bigquery<3,>=2.26.0->tfx) (2.5.0)\r\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jinja2<4,>=2.7.3->tfx) (2.1.3)\r\n",
"Requirement already satisfied: kt-legacy in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras-tuner<2,>=1.0.4->tfx) (1.0.5)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: certifi>=14.05.14 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx) (2023.7.22)\r\n",
"Requirement already satisfied: requests-oauthlib in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx) (1.3.1)\r\n",
"Requirement already satisfied: urllib3>=1.24.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from kubernetes<13,>=10.0.1->tfx) (1.26.16)\r\n",
"Requirement already satisfied: pyparsing>=2.0.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from packaging<21,>=20->tfx) (3.1.0)\r\n",
"Requirement already satisfied: psutil in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from portpicker<2,>=1.3.1->tfx) (5.9.5)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: astunparse>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (1.6.3)\r\n",
"Requirement already satisfied: flatbuffers>=2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (23.5.26)\r\n",
"Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (0.4.0)\r\n",
"Requirement already satisfied: google-pasta>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (0.2.0)\r\n",
"Requirement already satisfied: h5py>=2.9.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (3.9.0)\r\n",
"Requirement already satisfied: jax>=0.3.15 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (0.4.14)\r\n",
"Requirement already satisfied: keras<2.13,>=2.12.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (2.12.0)\r\n",
"Requirement already satisfied: libclang>=13.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (16.0.6)\r\n",
"Requirement already satisfied: opt-einsum>=2.3.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (3.3.0)\r\n",
"Requirement already satisfied: tensorboard<2.13,>=2.12 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (2.12.3)\r\n",
"Requirement already satisfied: tensorflow-estimator<2.13,>=2.12.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (2.12.0)\r\n",
"Requirement already satisfied: termcolor>=1.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (2.3.0)\r\n",
"Requirement already satisfied: wrapt<1.15,>=1.11.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (1.14.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow<2.13,>=2.12.0->tfx) (0.32.0)\r\n",
"Requirement already satisfied: joblib>=1.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.14.0,>=1.13.0->tfx) (1.3.1)\r\n",
"Requirement already satisfied: pandas<2,>=1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.14.0,>=1.13.0->tfx) (1.5.3)\r\n",
"Requirement already satisfied: pyfarmhash<0.4,>=0.2.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.14.0,>=1.13.0->tfx) (0.3.2)\r\n",
"Requirement already satisfied: tensorflow-metadata<1.14,>=1.13.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-data-validation<1.14.0,>=1.13.0->tfx) (1.13.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: ipython<8,>=7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (7.34.0)\r\n",
"Requirement already satisfied: ipywidgets<8,>=7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (7.7.5)\r\n",
"Requirement already satisfied: scipy<2,>=1.4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.11.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: wheel<1.0,>=0.23.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from astunparse>=1.6.0->tensorflow<2.13,>=2.12.0->tfx) (0.41.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: pyasn1-modules>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.40->tfx) (0.3.0)\r\n",
"Requirement already satisfied: rsa<5,>=3.1.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.40->tfx) (4.9)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-bigtable<2.18.0,>=2.0.0->apache-beam[gcp]<3,>=2.40->tfx) (0.12.6)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: grpcio-status>=1.16.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-pubsub<3,>=2.1.0->apache-beam[gcp]<3,>=2.40->tfx) (1.48.2)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: overrides<7.0.0,>=6.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.40->tfx) (6.5.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: sqlparse>=0.3.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]<3,>=2.40->tfx) (0.4.4)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx) (1.5.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: docopt in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]<3,>=2.40->tfx) (0.6.2)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: jedi>=0.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.18.2)\r\n",
"Requirement already satisfied: decorator in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (5.1.1)\r\n",
"Requirement already satisfied: pickleshare in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.7.5)\r\n",
"Requirement already satisfied: traitlets>=4.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (5.9.0)\r\n",
"Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (3.0.39)\r\n",
"Requirement already satisfied: pygments in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (2.15.1)\r\n",
"Requirement already satisfied: backcall in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.2.0)\r\n",
"Requirement already satisfied: matplotlib-inline in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.1.6)\r\n",
"Requirement already satisfied: pexpect>4.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (4.8.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: ipykernel>=4.5.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (6.25.0)\r\n",
"Requirement already satisfied: ipython-genutils~=0.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.2.0)\r\n",
"Requirement already satisfied: widgetsnbextension~=3.6.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (3.6.4)\r\n",
"Requirement already satisfied: jupyterlab-widgets<3,>=1.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.1.4)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: ml-dtypes>=0.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jax>=0.3.15->tensorflow<2.13,>=2.12.0->tfx) (0.2.0)\r\n",
"Requirement already satisfied: importlib-metadata>=4.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jax>=0.3.15->tensorflow<2.13,>=2.12.0->tfx) (6.8.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: pyasn1>=0.1.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from oauth2client>=1.4.12->google-apitools<1,>=0.5->tfx) (0.5.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: dnspython<3.0.0,>=1.16.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pymongo<5.0.0,>=3.8.0->apache-beam[gcp]<3,>=2.40->tfx) (2.4.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: charset-normalizer<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.40->tfx) (3.2.0)\r\n",
"Requirement already satisfied: idna<4,>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.40->tfx) (3.4)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.13,>=2.12->tensorflow<2.13,>=2.12.0->tfx) (0.5.3)\r\n",
"Requirement already satisfied: markdown>=2.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.13,>=2.12->tensorflow<2.13,>=2.12.0->tfx) (3.4.4)\r\n",
"Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.13,>=2.12->tensorflow<2.13,>=2.12.0->tfx) (0.7.1)\r\n",
"Requirement already satisfied: werkzeug>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.13,>=2.12->tensorflow<2.13,>=2.12.0->tfx) (2.3.6)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: oauthlib>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx) (3.2.2)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: zipp>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from importlib-metadata>=4.6->jax>=0.3.15->tensorflow<2.13,>=2.12.0->tfx) (3.16.2)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: comm>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.1.3)\r\n",
"Requirement already satisfied: debugpy>=1.6.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.6.7)\r\n",
"Requirement already satisfied: jupyter-client>=6.1.12 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (7.4.9)\r\n",
"Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (5.3.1)\r\n",
"Requirement already satisfied: nest-asyncio in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.5.7)\r\n",
"Requirement already satisfied: pyzmq>=20 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (24.0.1)\r\n",
"Requirement already satisfied: tornado>=6.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (6.3.2)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: parso<0.9.0,>=0.8.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jedi>=0.16->ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.8.3)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: ptyprocess>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pexpect>4.3->ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.7.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: wcwidth in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.2.6)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: notebook>=4.4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (6.5.5)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: entrypoints in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.4)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: platformdirs>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (3.9.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: argon2-cffi in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (21.3.0)\r\n",
"Requirement already satisfied: nbformat in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (5.9.1)\r\n",
"Requirement already satisfied: nbconvert>=5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (7.7.3)\r\n",
"Requirement already satisfied: Send2Trash>=1.8.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.8.2)\r\n",
"Requirement already satisfied: terminado>=0.8.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.17.1)\r\n",
"Requirement already satisfied: prometheus-client in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.17.1)\r\n",
"Requirement already satisfied: nbclassic>=0.4.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.0.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: jupyter-server>=1.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (2.7.0)\r\n",
"Requirement already satisfied: notebook-shim>=0.2.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.2.3)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: beautifulsoup4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (4.12.2)\r\n",
"Requirement already satisfied: bleach!=5.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (6.0.0)\r\n",
"Requirement already satisfied: defusedxml in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.7.1)\r\n",
"Requirement already satisfied: jupyterlab-pygments in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.2.2)\r\n",
"Requirement already satisfied: mistune<4,>=2.0.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (3.0.1)\r\n",
"Requirement already satisfied: nbclient>=0.5.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.8.0)\r\n",
"Requirement already satisfied: pandocfilters>=1.4.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.5.0)\r\n",
"Requirement already satisfied: tinycss2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.2.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: fastjsonschema in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (2.18.0)\r\n",
"Requirement already satisfied: jsonschema>=2.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (4.17.3)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: argon2-cffi-bindings in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (21.2.0)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: webencodings in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from bleach!=5.0.0->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.5.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.19.3)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: anyio>=3.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (3.7.1)\r\n",
"Requirement already satisfied: jupyter-events>=0.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.6.3)\r\n",
"Requirement already satisfied: jupyter-server-terminals in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.4.4)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: cffi>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.15.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: soupsieve>1.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from beautifulsoup4->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (2.4.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: sniffio>=1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.3.0)\r\n",
"Requirement already satisfied: exceptiongroup in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.1.2)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: pycparser in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (2.21)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: python-json-logger>=2.0.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-events>=0.6.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (2.0.7)\r\n",
"Requirement already satisfied: rfc3339-validator in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-events>=0.6.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.1.4)\r\n",
"Requirement already satisfied: rfc3986-validator>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jupyter-events>=0.6.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (0.1.1)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: fqdn in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.5.1)\r\n",
"Requirement already satisfied: isoduration in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (20.11.0)\r\n",
"Requirement already satisfied: jsonpointer>1.13 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (2.4)\r\n",
"Requirement already satisfied: uri-template in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.3.0)\r\n",
"Requirement already satisfied: webcolors>=1.11 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.13)\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: arrow>=0.15.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from isoduration->jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.4->ipywidgets<8,>=7->tensorflow-model-analysis<0.45.0,>=0.44.0->tfx) (1.2.3)\r\n"
]
}
],
"source": [
"!pip install tfx"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RxQ89gnRijuc"
},
"source": [
"### Uninstall shapely\n",
"\n",
"TODO(b/263441833) This is a temporal solution to avoid an\n",
"ImportError. Ultimately, it should be handled by supporting a\n",
"recent version of Bigquery, instead of uninstalling other extra\n",
"dependencies."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:24:59.795898Z",
"iopub.status.busy": "2023-07-28T10:24:59.795487Z",
"iopub.status.idle": "2023-07-28T10:25:00.598230Z",
"shell.execute_reply": "2023-07-28T10:25:00.597077Z"
},
"id": "akSWlt-Bij9w"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[33mWARNING: Skipping shapely as it is not installed.\u001b[0m\u001b[33m\r\n",
"\u001b[0m"
]
}
],
"source": [
"!pip uninstall shapely -y"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hvXNj9a4VlSV"
},
"source": [
"## Did you restart the runtime?\n",
"\n",
"If you are using Google Colab, the first time that you run the cell above, you\n",
"must restart the runtime (Runtime > Restart runtime ...). This is because of\n",
"the way that Colab loads packages."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0v45llP5ReKx"
},
"source": [
"### Import packages\n",
"We import TFX and check its version."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:00.602960Z",
"iopub.status.busy": "2023-07-28T10:25:00.602662Z",
"iopub.status.idle": "2023-07-28T10:25:06.969447Z",
"shell.execute_reply": "2023-07-28T10:25:06.968485Z"
},
"id": "bRY0RFJ0VlSV"
},
"outputs": [
{
"data": {
"text/plain": [
"'1.13.0'"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Check version\n",
"from tfx import v1 as tfx\n",
"tfx.__version__"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hvb0SspyqPH4"
},
"source": [
"## Custom Python function components\n",
"\n",
"In this section, we will create components from Python functions. We will not be\n",
"doing any real ML problem — these simple functions are just used to illustrate\n",
"the Python function component development process.\n",
"\n",
"See [Python function based component\n",
"guide](https://www.tensorflow.org/tfx/guide/custom_function_component)\n",
"for more documentation."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mhYjn9Fj6mdo"
},
"source": [
"### Create Python custom components\n",
"\n",
"We begin by writing a function that generate some dummy data. This is written\n",
"to its own Python module file."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:06.973881Z",
"iopub.status.busy": "2023-07-28T10:25:06.973456Z",
"iopub.status.idle": "2023-07-28T10:25:06.979656Z",
"shell.execute_reply": "2023-07-28T10:25:06.978757Z"
},
"id": "cHNtKTuiqPH4"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Writing my_generator.py\n"
]
}
],
"source": [
"%%writefile my_generator.py\n",
"\n",
"import os\n",
"import tensorflow as tf # Used for writing files.\n",
"\n",
"from tfx import v1 as tfx\n",
"\n",
"# Non-public APIs, just for showcase.\n",
"from tfx.types.experimental.simple_artifacts import Dataset\n",
"\n",
"@tfx.dsl.components.component\n",
"def MyGenerator(data: tfx.dsl.components.OutputArtifact[Dataset]):\n",
" \"\"\"Create a file with dummy data in the output artifact.\"\"\"\n",
" with tf.io.gfile.GFile(os.path.join(data.uri, 'data_file.txt'), 'w') as f:\n",
" f.write('Dummy data')\n",
"\n",
" # Set metadata and ensure that it gets passed to downstream components.\n",
" data.set_string_custom_property('my_custom_field', 'my_custom_value')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MtapXcbSqPH6"
},
"source": [
"Next, we write a second component that uses the dummy data produced.\n",
"We will just calculate hash of the data and return it."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:06.983392Z",
"iopub.status.busy": "2023-07-28T10:25:06.983060Z",
"iopub.status.idle": "2023-07-28T10:25:06.988428Z",
"shell.execute_reply": "2023-07-28T10:25:06.987611Z"
},
"id": "27ZEf2xQqPH7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Writing my_consumer.py\n"
]
}
],
"source": [
"%%writefile my_consumer.py\n",
"\n",
"import hashlib\n",
"import os\n",
"import tensorflow as tf\n",
"\n",
"from tfx import v1 as tfx\n",
"\n",
"# Non-public APIs, just for showcase.\n",
"from tfx.types.experimental.simple_artifacts import Dataset\n",
"from tfx.types.standard_artifacts import String\n",
"\n",
"@tfx.dsl.components.component\n",
"def MyConsumer(data: tfx.dsl.components.InputArtifact[Dataset],\n",
" hash: tfx.dsl.components.OutputArtifact[String],\n",
" algorithm: tfx.dsl.components.Parameter[str] = 'sha256'):\n",
" \"\"\"Reads the contents of data and calculate.\"\"\"\n",
" with tf.io.gfile.GFile(\n",
" os.path.join(data.uri, 'data_file.txt'), 'r') as f:\n",
" contents = f.read()\n",
" h = hashlib.new(algorithm)\n",
" h.update(tf.compat.as_bytes(contents))\n",
" hash.value = h.hexdigest()\n",
"\n",
" # Read a custom property from the input artifact and set to the output.\n",
" custom_value = data.get_string_custom_property('my_custom_field')\n",
" hash.set_string_custom_property('input_custom_field', custom_value)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lIrGHQzFqPII"
},
"source": [
"### Run in-notebook with the InteractiveContext\n",
"Now, we will demonstrate usage of our new components in the TFX\n",
"InteractiveContext.\n",
"\n",
"For more information on what you can do with the TFX notebook\n",
"InteractiveContext, see the in-notebook [TFX Keras Component Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/components_keras)."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:06.992006Z",
"iopub.status.busy": "2023-07-28T10:25:06.991752Z",
"iopub.status.idle": "2023-07-28T10:25:06.997531Z",
"shell.execute_reply": "2023-07-28T10:25:06.996751Z"
},
"id": "j43snQpRqPII"
},
"outputs": [],
"source": [
"from my_generator import MyGenerator\n",
"from my_consumer import MyConsumer"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_1Rf6FCMSnbM"
},
"source": [
"#### Construct the InteractiveContext"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:07.001139Z",
"iopub.status.busy": "2023-07-28T10:25:07.000887Z",
"iopub.status.idle": "2023-07-28T10:25:07.023975Z",
"shell.execute_reply": "2023-07-28T10:25:07.022944Z"
},
"id": "dEXGvLLmKviI"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING:absl:InteractiveContext pipeline_root argument not provided: using temporary directory /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_ as root for pipeline outputs.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING:absl:InteractiveContext metadata_connection_config not provided: using SQLite ML Metadata database at /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/metadata.sqlite.\n"
]
}
],
"source": [
"# Here, we create an InteractiveContext using default parameters. This will\n",
"# use a temporary directory with an ephemeral ML Metadata database instance.\n",
"# To use your own pipeline root or database, the optional properties\n",
"# `pipeline_root` and `metadata_connection_config` may be passed to\n",
"# InteractiveContext. Calls to InteractiveContext are no-ops outside of the\n",
"# notebook.\n",
"from tfx.orchestration.experimental.interactive.interactive_context import InteractiveContext\n",
"context = InteractiveContext()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XxXVtZzdTWg_"
},
"source": [
"#### Run your component interactively with `context.run()`\n",
"Next, we run our components interactively within the notebook with\n",
"`context.run()`. Our consumer component uses the outputs of the generator\n",
"component."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:07.027889Z",
"iopub.status.busy": "2023-07-28T10:25:07.027512Z",
"iopub.status.idle": "2023-07-28T10:25:07.360920Z",
"shell.execute_reply": "2023-07-28T10:25:07.359993Z"
},
"id": "kfNmI5qULlSA"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"ExecutionResult at 0x7fb9341c2a30
.execution_id | 1 |
.component | \n",
"\n",
"MyGenerator at 0x7fb7c48d0430 .inputs | {} | .outputs | ['data'] | \n",
"\n",
"Channel of type 'Dataset' (1 artifact) at 0x7fb7c48d0370 .type_name | Dataset | ._artifacts | [0] | \n",
"\n",
"Artifact of type 'Dataset' (uri: /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyGenerator/data/1) at 0x7fb930151370 .type | <class 'tfx.types.experimental.simple_artifacts.Dataset'> | .uri | /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyGenerator/data/1 |
|
|
|
| .exec_properties | {} |
|
.component.inputs | {} |
.component.outputs | ['data'] | \n",
"\n",
"Channel of type 'Dataset' (1 artifact) at 0x7fb7c48d0370 .type_name | Dataset | ._artifacts | [0] | \n",
"\n",
"Artifact of type 'Dataset' (uri: /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyGenerator/data/1) at 0x7fb930151370 .type | <class 'tfx.types.experimental.simple_artifacts.Dataset'> | .uri | /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyGenerator/data/1 |
|
|
|
|
"
],
"text/plain": [
"ExecutionResult(\n",
" component_id: MyGenerator\n",
" execution_id: 1\n",
" outputs:\n",
" data: OutputChannel(artifact_type=Dataset, producer_component_id=MyGenerator, output_key=data, additional_properties={}, additional_custom_properties={}, _input_trigger=None)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"generator = MyGenerator()\n",
"context.run(generator)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:07.364676Z",
"iopub.status.busy": "2023-07-28T10:25:07.364356Z",
"iopub.status.idle": "2023-07-28T10:25:07.544409Z",
"shell.execute_reply": "2023-07-28T10:25:07.543525Z"
},
"id": "cRxVZIfFLsL4"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"ExecutionResult at 0x7fb9341c2bb0
.execution_id | 2 |
.component | \n",
"\n",
"MyConsumer at 0x7fb7c48d0280 .inputs | ['data'] | \n",
"\n",
"Channel of type 'Dataset' (1 artifact) at 0x7fb7c48d0370 .type_name | Dataset | ._artifacts | [0] | \n",
"\n",
"Artifact of type 'Dataset' (uri: /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyGenerator/data/1) at 0x7fb930151370 .type | <class 'tfx.types.experimental.simple_artifacts.Dataset'> | .uri | /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyGenerator/data/1 |
|
|
|
| .outputs | ['hash'] | \n",
"\n",
"Channel of type 'String' (1 artifact) at 0x7fb7c48d0a60 .type_name | String | ._artifacts | [0] | \n",
"\n",
"Artifact of type 'String' (uri: /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyConsumer/hash/2/value) at 0x7fb930151d00 .type | <class 'tfx.types.standard_artifacts.String'> | .uri | /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyConsumer/hash/2/value |
|
|
|
| .exec_properties | |
|
.component.inputs | ['data'] | \n",
"\n",
"Channel of type 'Dataset' (1 artifact) at 0x7fb7c48d0370 .type_name | Dataset | ._artifacts | [0] | \n",
"\n",
"Artifact of type 'Dataset' (uri: /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyGenerator/data/1) at 0x7fb930151370 .type | <class 'tfx.types.experimental.simple_artifacts.Dataset'> | .uri | /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyGenerator/data/1 |
|
|
|
|
.component.outputs | ['hash'] | \n",
"\n",
"Channel of type 'String' (1 artifact) at 0x7fb7c48d0a60 .type_name | String | ._artifacts | [0] | \n",
"\n",
"Artifact of type 'String' (uri: /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyConsumer/hash/2/value) at 0x7fb930151d00 .type | <class 'tfx.types.standard_artifacts.String'> | .uri | /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyConsumer/hash/2/value |
|
|
|
|
"
],
"text/plain": [
"ExecutionResult(\n",
" component_id: MyConsumer\n",
" execution_id: 2\n",
" outputs:\n",
" hash: OutputChannel(artifact_type=String, producer_component_id=MyConsumer, output_key=hash, additional_properties={}, additional_custom_properties={}, _input_trigger=None)"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"consumer = MyConsumer(\n",
" data=generator.outputs['data'],\n",
" algorithm='md5')\n",
"context.run(consumer)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pO_Ggc4QTo0B"
},
"source": [
"After execution, we can inspect the contents of the \"hash\" output artifact of\n",
"the consumer component on disk."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:07.548064Z",
"iopub.status.busy": "2023-07-28T10:25:07.547784Z",
"iopub.status.idle": "2023-07-28T10:25:07.706227Z",
"shell.execute_reply": "2023-07-28T10:25:07.705088Z"
},
"id": "h4P3Mx_CT0mP"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"==> /tmpfs/tmp/tfx-interactive-2023-07-28T10_25_07.019368-ljo_tv2_/MyConsumer/hash/2/value <==\r\n",
"0015fe7975d1a2794b59aa12635703f1"
]
}
],
"source": [
"!tail -v {consumer.outputs['hash'].get()[0].uri}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RRosvALKSWe0"
},
"source": [
"That's it, and you've now written and executed your own custom components!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uvXKtg17O6mF"
},
"source": [
"### Write a pipeline definition\n",
"\n",
"Next, we will author a pipeline using these same components. While using the\n",
"`InteractiveContext` within a notebook works well for experimentation, defining\n",
"a pipeline lets you deploy your pipeline on local or remote runners for\n",
"production usage.\n",
"\n",
"Here, we will demonstrate usage of the LocalDagRunner running locally on your\n",
"machine. For production execution, the Airflow or Kubeflow runners may\n",
"be more suitable."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ft9fbSpnU7C6"
},
"source": [
"#### Construct a pipeline"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:07.711205Z",
"iopub.status.busy": "2023-07-28T10:25:07.710928Z",
"iopub.status.idle": "2023-07-28T10:25:07.719563Z",
"shell.execute_reply": "2023-07-28T10:25:07.718741Z"
},
"id": "NpkQ805-LyJu"
},
"outputs": [],
"source": [
"import os\n",
"import tempfile\n",
"from tfx import v1 as tfx\n",
"\n",
"# Select a persistent TFX root directory to store your output artifacts.\n",
"# For demonstration purposes only, we use a temporary directory.\n",
"PIPELINE_ROOT = tempfile.mkdtemp()\n",
"# Select a pipeline name so that multiple runs of the same logical pipeline\n",
"# can be grouped.\n",
"PIPELINE_NAME = \"function-based-pipeline\"\n",
"# We use a ML Metadata configuration that uses a local SQLite database in\n",
"# the pipeline root directory. Other backends for ML Metadata are available\n",
"# for production usage.\n",
"METADATA_CONNECTION_CONFIG = tfx.orchestration.metadata.sqlite_metadata_connection_config(\n",
" os.path.join(PIPELINE_ROOT, 'metadata.sqlite'))\n",
"\n",
"def function_based_pipeline():\n",
" # Here, we construct our generator and consumer components in the same way.\n",
" generator = MyGenerator()\n",
" consumer = MyConsumer(\n",
" data=generator.outputs['data'],\n",
" algorithm='md5')\n",
"\n",
" return tfx.dsl.Pipeline(\n",
" pipeline_name=PIPELINE_NAME,\n",
" pipeline_root=PIPELINE_ROOT,\n",
" components=[generator, consumer],\n",
" metadata_connection_config=METADATA_CONNECTION_CONFIG)\n",
"\n",
"my_pipeline = function_based_pipeline()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Mj-Z3cbFWPbK"
},
"source": [
"#### Run your pipeline with the `LocalDagRunner`"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:07.723118Z",
"iopub.status.busy": "2023-07-28T10:25:07.722845Z",
"iopub.status.idle": "2023-07-28T10:25:08.265598Z",
"shell.execute_reply": "2023-07-28T10:25:08.264725Z"
},
"id": "PLtGO2PkMQbO"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING:absl:ArtifactQuery.property_predicate is not supported.\n"
]
}
],
"source": [
"tfx.orchestration.LocalDagRunner().run(my_pipeline)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ry4vU3mOWeN1"
},
"source": [
"We can inspect the output artifacts generated by this pipeline execution."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"execution": {
"iopub.execute_input": "2023-07-28T10:25:08.269984Z",
"iopub.status.busy": "2023-07-28T10:25:08.269618Z",
"iopub.status.idle": "2023-07-28T10:25:08.426245Z",
"shell.execute_reply": "2023-07-28T10:25:08.425042Z"
},
"id": "fyvYTsx8Mp1N"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/tmpfs/tmp/tmp7icf5apc\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyGenerator\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyGenerator/data\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyGenerator/data/1\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyGenerator/data/1/data_file.txt\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyGenerator/.system\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyGenerator/.system/executor_execution\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyGenerator/.system/executor_execution/1\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyConsumer\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyConsumer/hash\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyConsumer/hash/2\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyConsumer/hash/2/value\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyConsumer/.system\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyConsumer/.system/executor_execution\r\n",
"/tmpfs/tmp/tmp7icf5apc/MyConsumer/.system/executor_execution/2\r\n",
"/tmpfs/tmp/tmp7icf5apc/metadata.sqlite\r\n"
]
}
],
"source": [
"!find {PIPELINE_ROOT}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M4CsceadWqHp"
},
"source": [
"You have now written your own custom components and orchestrated their\n",
"execution on the LocalDagRunner! For next steps, check out additional tutorials\n",
"and guides on the [TFX website](https://www.tensorflow.org/tfx)."
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [
"wdeKOEkv1Fe8"
],
"name": "TFX Python function component tutorial",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.17"
}
},
"nbformat": 4,
"nbformat_minor": 0
}