{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "4JlLTP1Y-WHg" }, "source": [ "##### Copyright 2020 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2026-01-15T12:19:39.997127Z", "iopub.status.busy": "2026-01-15T12:19:39.996887Z", "iopub.status.idle": "2026-01-15T12:19:40.000410Z", "shell.execute_reply": "2026-01-15T12:19:39.999786Z" }, "id": "if-ujOZN-Par" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "Uq9kCbELjzgJ" }, "source": [ "# Listwise ranking\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "r4BKWZyB_Hmf" }, "source": [ "In [the basic ranking tutorial](basic_ranking), we trained a model that can predict ratings for user/movie pairs. The model was trained to minimize the mean squared error of predicted ratings.\n", "\n", "However, optimizing the model's predictions on individual movies is not necessarily the best method for training ranking models. We do not need ranking models to predict scores with great accuracy. Instead, we care more about the ability of the model to generate an ordered list of items that matches the user's preference ordering.\n", "\n", "Instead of optimizing the model's predictions on individual query/item pairs, we can optimize the model's ranking of a list as a whole. This method is called _listwise ranking_.\n", "\n", "In this tutorial, we will use TensorFlow Recommenders to build listwise ranking models. To do so, we will make use of ranking losses and metrics provided by [TensorFlow Ranking](https://github.com/tensorflow/ranking), a TensorFlow package that focuses on [learning to rank](https://www.microsoft.com/en-us/research/publication/learning-to-rank-from-pairwise-approach-to-listwise-approach/)." ] }, { "cell_type": "markdown", "metadata": { "id": "3XS680n2n0rL" }, "source": [ "## Preliminaries\n", "\n", "If TensorFlow Ranking is not available in your runtime environment, you can install it using `pip`:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:19:40.003352Z", "iopub.status.busy": "2026-01-15T12:19:40.003109Z", "iopub.status.idle": "2026-01-15T12:20:13.194594Z", "shell.execute_reply": "2026-01-15T12:20:13.193680Z" }, "id": "gr_BrcNMKji6" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\r\n", "tf-keras 2.20.1 requires tensorflow<2.21,>=2.20, but you have tensorflow 2.15.1 which is incompatible.\u001b[0m\u001b[31m\r\n", "\u001b[0m" ] } ], "source": [ "!pip install -q tensorflow-recommenders\n", "!pip install -q --upgrade tensorflow-datasets\n", "!pip install -q tensorflow-ranking" ] }, { "cell_type": "markdown", "metadata": { "id": "rPQqa1uYKrw2" }, "source": [ "We can then import all the necessary packages:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:20:13.198478Z", "iopub.status.busy": "2026-01-15T12:20:13.198210Z", "iopub.status.idle": "2026-01-15T12:20:15.865958Z", "shell.execute_reply": "2026-01-15T12:20:15.865068Z" }, "id": "6ekaJkcuHsiY" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2026-01-15 12:20:13.674337: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", "2026-01-15 12:20:13.674382: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", "2026-01-15 12:20:13.676073: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2026-01-15 12:20:14.518543: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" ] } ], "source": [ "import pprint\n", "\n", "import numpy as np\n", "import tensorflow as tf\n", "import tensorflow_datasets as tfds" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:20:15.869803Z", "iopub.status.busy": "2026-01-15T12:20:15.869396Z", "iopub.status.idle": "2026-01-15T12:20:15.986171Z", "shell.execute_reply": "2026-01-15T12:20:15.985465Z" }, "id": "WdTPCz136mvc" }, "outputs": [], "source": [ "import tensorflow_ranking as tfr\n", "import tensorflow_recommenders as tfrs" ] }, { "cell_type": "markdown", "metadata": { "id": "fNEB3VRs3bOS" }, "source": [ "We will continue to use the MovieLens 100K dataset. As before, we load the datasets and keep only the user id, movie title, and user rating features for this tutorial. We also do some houskeeping to prepare our vocabularies." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:20:15.989492Z", "iopub.status.busy": "2026-01-15T12:20:15.989208Z", "iopub.status.idle": "2026-01-15T12:20:21.008120Z", "shell.execute_reply": "2026-01-15T12:20:21.007363Z" }, "id": "k-VF30hJn5-3" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2026-01-15 12:20:17.549717: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2256] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.\n", "Skipping registering GPU devices...\n" ] } ], "source": [ "ratings = tfds.load(\"movielens/100k-ratings\", split=\"train\")\n", "movies = tfds.load(\"movielens/100k-movies\", split=\"train\")\n", "\n", "ratings = ratings.map(lambda x: {\n", " \"movie_title\": x[\"movie_title\"],\n", " \"user_id\": x[\"user_id\"],\n", " \"user_rating\": x[\"user_rating\"],\n", "})\n", "movies = movies.map(lambda x: x[\"movie_title\"])\n", "\n", "unique_movie_titles = np.unique(np.concatenate(list(movies.batch(1000))))\n", "unique_user_ids = np.unique(np.concatenate(list(ratings.batch(1_000).map(\n", " lambda x: x[\"user_id\"]))))" ] }, { "cell_type": "markdown", "metadata": { "id": "xIBH-Axc7oqB" }, "source": [ "## Data preprocessing\n", "\n", "However, we cannot use the MovieLens dataset for list optimization directly. To perform listwise optimization, we need to have access to a list of movies each user has rated, but each example in the MovieLens 100K dataset contains only the rating of a single movie.\n", "\n", "To get around this we transform the dataset so that each example contains a user id and a list of movies rated by that user. Some movies in the list will be ranked higher than others; the goal of our model will be to make predictions that match this ordering.\n", "\n", "To do this, we use the `tfrs.examples.movielens.movielens_to_listwise` helper function. It takes the MovieLens 100K dataset and generates a dataset containing list examples as discussed above. The implementation details can be found in the [source code](https://github.com/tensorflow/recommenders/blob/main/tensorflow_recommenders/examples/movielens.py)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:20:21.011537Z", "iopub.status.busy": "2026-01-15T12:20:21.011284Z", "iopub.status.idle": "2026-01-15T12:20:56.974121Z", "shell.execute_reply": "2026-01-15T12:20:56.973343Z" }, "id": "X99torl5z4Iu" }, "outputs": [], "source": [ "tf.random.set_seed(42)\n", "\n", "# Split between train and tests sets, as before.\n", "shuffled = ratings.shuffle(100_000, seed=42, reshuffle_each_iteration=False)\n", "\n", "train = shuffled.take(80_000)\n", "test = shuffled.skip(80_000).take(20_000)\n", "\n", "# We sample 50 lists for each user for the training data. For each list we\n", "# sample 5 movies from the movies the user rated.\n", "train = tfrs.examples.movielens.sample_listwise(\n", " train,\n", " num_list_per_user=50,\n", " num_examples_per_list=5,\n", " seed=42\n", ")\n", "test = tfrs.examples.movielens.sample_listwise(\n", " test,\n", " num_list_per_user=1,\n", " num_examples_per_list=5,\n", " seed=42\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "8zuAfGrgCBJP" }, "source": [ "We can inspect an example from the training data. The example includes a user id, a list of 10 movie ids, and their ratings by the user." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:20:56.980248Z", "iopub.status.busy": "2026-01-15T12:20:56.979990Z", "iopub.status.idle": "2026-01-15T12:20:57.002097Z", "shell.execute_reply": "2026-01-15T12:20:57.001269Z" }, "id": "AO52eZqOzOUV" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'movie_title': ,\n", " 'user_id': ,\n", " 'user_rating': }\n" ] } ], "source": [ "for example in train.take(1):\n", " pprint.pprint(example)" ] }, { "cell_type": "markdown", "metadata": { "id": "aM3uu5hgN4-v" }, "source": [ "## Model definition\n", "\n", "We will train the same model with three different losses: \n", "\n", "- mean squared error,\n", "- pairwise hinge loss, and\n", "- a listwise ListMLE loss. \n", "\n", "These three losses correspond to pointwise, pairwise, and listwise optimization." ] }, { "cell_type": "markdown", "metadata": { "id": "BXmqkuAShHO7" }, "source": [ "To evaluate the model we use [normalized discounted cumulative gain (NDCG)](https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG). NDCG measures a predicted ranking by taking a weighted sum of the actual rating of each candidate. The ratings of movies that are ranked lower by the model would be discounted more. As a result, a good model that ranks highly-rated movies on top would have a high NDCG result. Since this metric takes the ranked position of each candidate into account, it is a listwise metric." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:20:57.005157Z", "iopub.status.busy": "2026-01-15T12:20:57.004899Z", "iopub.status.idle": "2026-01-15T12:20:57.014077Z", "shell.execute_reply": "2026-01-15T12:20:57.013362Z" }, "id": "A1RTt67fhR52" }, "outputs": [], "source": [ "class RankingModel(tfrs.Model):\n", "\n", " def __init__(self, loss):\n", " super().__init__()\n", " embedding_dimension = 32\n", "\n", " # Compute embeddings for users.\n", " self.user_embeddings = tf.keras.Sequential([\n", " tf.keras.layers.StringLookup(\n", " vocabulary=unique_user_ids),\n", " tf.keras.layers.Embedding(len(unique_user_ids) + 2, embedding_dimension)\n", " ])\n", "\n", " # Compute embeddings for movies.\n", " self.movie_embeddings = tf.keras.Sequential([\n", " tf.keras.layers.StringLookup(\n", " vocabulary=unique_movie_titles),\n", " tf.keras.layers.Embedding(len(unique_movie_titles) + 2, embedding_dimension)\n", " ])\n", "\n", " # Compute predictions.\n", " self.score_model = tf.keras.Sequential([\n", " # Learn multiple dense layers.\n", " tf.keras.layers.Dense(256, activation=\"relu\"),\n", " tf.keras.layers.Dense(64, activation=\"relu\"),\n", " # Make rating predictions in the final layer.\n", " tf.keras.layers.Dense(1)\n", " ])\n", "\n", " self.task = tfrs.tasks.Ranking(\n", " loss=loss,\n", " metrics=[\n", " tfr.keras.metrics.NDCGMetric(name=\"ndcg_metric\"),\n", " tf.keras.metrics.RootMeanSquaredError()\n", " ]\n", " )\n", "\n", " def call(self, features):\n", " # We first convert the id features into embeddings.\n", " # User embeddings are a [batch_size, embedding_dim] tensor.\n", " user_embeddings = self.user_embeddings(features[\"user_id\"])\n", "\n", " # Movie embeddings are a [batch_size, num_movies_in_list, embedding_dim]\n", " # tensor.\n", " movie_embeddings = self.movie_embeddings(features[\"movie_title\"])\n", " \n", " # We want to concatenate user embeddings with movie emebeddings to pass\n", " # them into the ranking model. To do so, we need to reshape the user\n", " # embeddings to match the shape of movie embeddings.\n", " list_length = features[\"movie_title\"].shape[1]\n", " user_embedding_repeated = tf.repeat(\n", " tf.expand_dims(user_embeddings, 1), [list_length], axis=1)\n", "\n", " # Once reshaped, we concatenate and pass into the dense layers to generate\n", " # predictions.\n", " concatenated_embeddings = tf.concat(\n", " [user_embedding_repeated, movie_embeddings], 2)\n", " \n", " return self.score_model(concatenated_embeddings)\n", "\n", " def compute_loss(self, features, training=False):\n", " labels = features.pop(\"user_rating\")\n", "\n", " scores = self(features)\n", "\n", " return self.task(\n", " labels=labels,\n", " predictions=tf.squeeze(scores, axis=-1),\n", " )" ] }, { "cell_type": "markdown", "metadata": { "id": "HcTElTWbOImt" }, "source": [ "## Training the models\n", "\n", "We can now train each of the three models." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:20:57.016656Z", "iopub.status.busy": "2026-01-15T12:20:57.016405Z", "iopub.status.idle": "2026-01-15T12:20:57.027478Z", "shell.execute_reply": "2026-01-15T12:20:57.026765Z" }, "id": "7U530Yk-s-g9" }, "outputs": [], "source": [ "epochs = 30\n", "\n", "cached_train = train.shuffle(100_000).batch(8192).cache()\n", "cached_test = test.batch(4096).cache()" ] }, { "cell_type": "markdown", "metadata": { "id": "OQNnD7DNTkYC" }, "source": [ "### Mean squared error model\n", "\n", "This model is very similar to the model in [the basic ranking tutorial](basic_ranking). We train the model to minimize the mean squared error between the actual ratings and predicted ratings. Therefore, this loss is computed individually for each movie and the training is pointwise." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:20:57.030339Z", "iopub.status.busy": "2026-01-15T12:20:57.030102Z", "iopub.status.idle": "2026-01-15T12:20:57.085245Z", "shell.execute_reply": "2026-01-15T12:20:57.084653Z" }, "id": "F0lq0Nq7_xTW" }, "outputs": [], "source": [ "mse_model = RankingModel(tf.keras.losses.MeanSquaredError())\n", "mse_model.compile(optimizer=tf.keras.optimizers.Adagrad(0.1))" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:20:57.087921Z", "iopub.status.busy": "2026-01-15T12:20:57.087685Z", "iopub.status.idle": "2026-01-15T12:21:04.782945Z", "shell.execute_reply": "2026-01-15T12:21:04.782147Z" }, "id": "6NBl543nRtIo" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mse_model.fit(cached_train, epochs=epochs, verbose=False)" ] }, { "cell_type": "markdown", "metadata": { "id": "2DHphbdxUS7l" }, "source": [ "### Pairwise hinge loss model\n", "\n", "By minimizing the pairwise hinge loss, the model tries to maximize the difference between the model's predictions for a highly rated item and a low rated item: the bigger that difference is, the lower the model loss. However, once the difference is large enough, the loss becomes zero, stopping the model from further optimizing this particular pair and letting it focus on other pairs that are incorrectly ranked\n", "\n", "This loss is not computed for individual movies, but rather for pairs of movies. Hence the training using this loss is pairwise." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:21:04.785809Z", "iopub.status.busy": "2026-01-15T12:21:04.785516Z", "iopub.status.idle": "2026-01-15T12:21:04.819945Z", "shell.execute_reply": "2026-01-15T12:21:04.819302Z" }, "id": "px_wZPxCOrBt" }, "outputs": [], "source": [ "hinge_model = RankingModel(tfr.keras.losses.PairwiseHingeLoss())\n", "hinge_model.compile(optimizer=tf.keras.optimizers.Adagrad(0.1))" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:21:04.822637Z", "iopub.status.busy": "2026-01-15T12:21:04.822415Z", "iopub.status.idle": "2026-01-15T12:21:12.433690Z", "shell.execute_reply": "2026-01-15T12:21:12.432875Z" }, "id": "rqbd9aDXO6mP" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "hinge_model.fit(cached_train, epochs=epochs, verbose=False)" ] }, { "cell_type": "markdown", "metadata": { "id": "d79_Y2cuUal-" }, "source": [ "### Listwise model\n", "\n", "The `ListMLE` loss from TensorFlow Ranking expresses list maximum likelihood estimation. To calculate the ListMLE loss, we first use the user ratings to generate an optimal ranking. We then calculate the likelihood of each candidate being out-ranked by any item below it in the optimal ranking using the predicted scores. The model tries to minimize such likelihood to ensure highly rated candidates are not out-ranked by low rated candidates. You can learn more about the details of ListMLE in section 2.2 of the paper [Position-aware ListMLE: A Sequential Learning Process](http://auai.org/uai2014/proceedings/individuals/164.pdf).\n", "\n", "Note that since the likelihood is computed with respect to a candidate and all candidates below it in the optimal ranking, the loss is not pairwise but listwise. Hence the training uses list optimization." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:21:12.436593Z", "iopub.status.busy": "2026-01-15T12:21:12.436314Z", "iopub.status.idle": "2026-01-15T12:21:12.470621Z", "shell.execute_reply": "2026-01-15T12:21:12.469970Z" }, "id": "_IO8N2JQvASN" }, "outputs": [], "source": [ "listwise_model = RankingModel(tfr.keras.losses.ListMLELoss())\n", "listwise_model.compile(optimizer=tf.keras.optimizers.Adagrad(0.1))" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:21:12.473165Z", "iopub.status.busy": "2026-01-15T12:21:12.472938Z", "iopub.status.idle": "2026-01-15T12:21:19.757507Z", "shell.execute_reply": "2026-01-15T12:21:19.756393Z" }, "id": "_aA0lqGovDrw" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "listwise_model.fit(cached_train, epochs=epochs, verbose=False)" ] }, { "cell_type": "markdown", "metadata": { "id": "U5y0ucbFSEZi" }, "source": [ "## Comparing the models" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:21:19.760621Z", "iopub.status.busy": "2026-01-15T12:21:19.760340Z", "iopub.status.idle": "2026-01-15T12:21:20.179406Z", "shell.execute_reply": "2026-01-15T12:21:20.178370Z" }, "id": "hfPgRvnXSJwm" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s - ndcg_metric: 0.9053 - root_mean_squared_error: 0.9655 - loss: 0.9323 - regularization_loss: 0.0000e+00 - total_loss: 0.9323" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 408ms/step - ndcg_metric: 0.9053 - root_mean_squared_error: 0.9655 - loss: 0.9323 - regularization_loss: 0.0000e+00 - total_loss: 0.9323\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "NDCG of the MSE Model: 0.9053\n" ] } ], "source": [ "mse_model_result = mse_model.evaluate(cached_test, return_dict=True)\n", "print(\"NDCG of the MSE Model: {:.4f}\".format(mse_model_result[\"ndcg_metric\"]))" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:21:20.182286Z", "iopub.status.busy": "2026-01-15T12:21:20.181979Z", "iopub.status.idle": "2026-01-15T12:21:20.677326Z", "shell.execute_reply": "2026-01-15T12:21:20.676264Z" }, "id": "EwTaEJ6JPF9k" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s - ndcg_metric: 0.9069 - root_mean_squared_error: 3.4001 - loss: 1.0223 - regularization_loss: 0.0000e+00 - total_loss: 1.0223" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 485ms/step - ndcg_metric: 0.9069 - root_mean_squared_error: 3.4001 - loss: 1.0223 - regularization_loss: 0.0000e+00 - total_loss: 1.0223\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "NDCG of the pairwise hinge loss model: 0.9069\n" ] } ], "source": [ "hinge_model_result = hinge_model.evaluate(cached_test, return_dict=True)\n", "print(\"NDCG of the pairwise hinge loss model: {:.4f}\".format(hinge_model_result[\"ndcg_metric\"]))" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2026-01-15T12:21:20.680316Z", "iopub.status.busy": "2026-01-15T12:21:20.680043Z", "iopub.status.idle": "2026-01-15T12:21:21.114112Z", "shell.execute_reply": "2026-01-15T12:21:21.113342Z" }, "id": "qR8xQs6BSO0X" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s - ndcg_metric: 0.9076 - root_mean_squared_error: 3.0739 - loss: 4.5369 - regularization_loss: 0.0000e+00 - total_loss: 4.5369" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 0s 424ms/step - ndcg_metric: 0.9076 - root_mean_squared_error: 3.0739 - loss: 4.5369 - regularization_loss: 0.0000e+00 - total_loss: 4.5369\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "NDCG of the ListMLE model: 0.9076\n" ] } ], "source": [ "listwise_model_result = listwise_model.evaluate(cached_test, return_dict=True)\n", "print(\"NDCG of the ListMLE model: {:.4f}\".format(listwise_model_result[\"ndcg_metric\"]))" ] }, { "cell_type": "markdown", "metadata": { "id": "2XeWqIq4O1xo" }, "source": [ "Of the three models, the model trained using ListMLE has the highest NDCG metric. This result shows how listwise optimization can be used to train ranking models and can potentially produce models that perform better than models optimized in a pointwise or pairwise fashion." ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "listwise_ranking.ipynb", "private_outputs": true, "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.5" } }, "nbformat": 4, "nbformat_minor": 0 }