Predicting Board Game Collections

rahdo’s Collection

About

This report details the results of training and evaluating a classification model for predicting games for a user’s boardgame collection.

Note

To view games predicted by the model, go to Section 5.

Collection

The data in this project comes from BoardGameGeek.com. The data used is at the game level, where an individual observation contains features about a game, such as its publisher, categories, and playing time, among many others.

I train a classification model at the user level to learn the relationship between game features and games that a user owns - what predicts a user’s collection?

username	status	games
rahdo	ever_owned	1687
rahdo	own	502
rahdo	rated	452

I evaluate the model’s performance on a training set of historical games via resampling, then validate the model’s performance on a set aside set of newer relases. I then refit the model on the training and validation in order and predict upcoming releases in order to find new games that the user is most likely to add to their collection.

username	years	type	Own
			no	yes
rahdo	-3500-2020	train	24173	345
rahdo	2021-2022	valid	9809	88
rahdo	2023-2028	test	9030	69

Types of Games

What types of game does the user own? The following plot displays the most frequent publishers, mechanics, designers, artists, etc that appear in a user’s collection.

collection |>
        filter(own == 1) |>
        collection_by_category(
                games = games_raw
        ) |>
        plot_collection_by_category()+
        ylab("feature")

The following plot shows the years in which games in the user’s collection were published. This can usually indicate when someone first entered the hobby.

Games in Collection

What games does the user currently have in their collection? The following table can be used to examine games the user owns, along with some helpful information for selecting the right game for a game night!

Use the filters above the table to sort/filter based on information about the game, such as year published, recommended player counts, or playing time.

collection |>
        filter(own == 1) |>
        prep_collection_datatable(
                games = games_raw
        ) |>
        filter(!is.na(image)) |>
        collection_datatable()

Modeling

I’ll now the examine predictive models trained on the user’s collection.

For an individual user, I train a predictive model on their collection in order to predict whether a user owns a game. The outcome, in this case, is binary: does the user have a game listed in their collection or not? This is the setting for training a classification model, where the model aims to learn the probability that a user will add a game to their collection based on its observable features.

How does a model learn what a user is likely to own? The training process is a matter of examining historical games and finding patterns that exist between game features (designers, mechanics, playing time, etc) and games in the user’s collection.

I make use of many potential features for games, the vast majority of which are dummies indicating the presence or absence of the presence or absence of things such as a publisher/artist/designer. The “standard” BGG features for every game contain information that is typically listed on the box its playing time, player counts, or its recommended minimum age.

Note

I train models to predict whether a user owns a game based only on information that could be observed about the game at its release: playing time, player count, mechanics, categories, genres, and selected designers, artists, and publishers. I do not make use of BGG community information, such as its average rating, weight, or number of user ratings. This is to ensure the model can predict newly released games without relying on information from the BGG community.

What Predicts A Collection?

A predictive model gives us more than just predictions. We can also ask, what did the model learn from the data? What predicts the outcome? In the case of predicting a boardgame collection, what did the model find to be predictive of games a user has in their collection?

To answer this, I examine the coefficients from a model logistic regression with ridge regularization (which I will refer to as a penalized logistic regression).

Positive values indicate that a feature increases a user’s probability of owning/rating a game, while negative values indicate a feature decreases the probability. To be precise, the coefficients indicate the effect of a particular feature on the log-odds of a user owning a game.

The following visualization shows the path of each feature as it enters the model, with highly influential features tending to enter the model early with large positive or negative effects. The dotted line indicates the level of regularization that was selected during tuning.

model_glmnet |> 
        pluck("wflow", 1) |>
        trace_plot.glmnet(max.overlaps = 30)+
        facet_wrap(~params$username)

Partial Effects

What are the effects of individual features?

Use the buttons below to examine the effects different types of predictors had in predicting the user’s collection.

Categories

Mechanics

Themes

Families

Publishers

Designers

Artists

Components

Mechanisms

Assessment

How well did the model do in predicting the user’s collection?

This section contains a variety of visualizations and metrics for assessing the performance of the model(s). If you’re not particularly interested in predictive modeling, skip down further to the predictions from the model.

The following displays the model’s performance in resampling on a training set, a validation set, and a holdout set of upcoming games.

metrics |>
        mutate_if(is.numeric, round, 3) |>
        pivot_wider(
                names_from = c(".metric"),
                values_from = c(".estimate")) |>
        gt::gt() |>
        gt::sub_missing() |>
        gt_options()

username	wflow_id	type	.estimator	mn_log_loss	roc_auc	pr_auc
rahdo	glmnet	resamples	binary	0.049	0.942	0.258
rahdo	glmnet	test	binary	0.040	0.887	0.079
rahdo	glmnet	valid	binary	0.036	0.939	0.189

An easy way to visually examine the performance of classification model is to view a separation plot.

I plot the predicted probabilities from the model for every game (during resampling) from lowest to highest. I then overlay a blue line for any game that the user does own. A good classifier is one that is able to separate the blue (games owned by the user) from the white (games not owned by the user), with most of the blue occurring at the highest probabilities (left side of the chart).

preds |>
        filter(type %in% c('resamples', 'valid')) |>
        plot_separation(outcome = params$outcome)

I can more formally assess how well each model did in resampling by looking at the area under the ROC curve (roc_auc). A perfect model would receive a score of 1, while a model that cannot predict the outcome will default to a score of 0.5. The extent to which something is a good score depends on the setting, but generally anything in the .8 to .9 range is very good while the .7 to .8 range is perfectly acceptable.

preds |>
        nest(data = -c(username, wflow_id, type)) |>
        mutate(roc_curve = map(data, safely( ~ .x |> safe_roc_curve(truth = params$outcome)))) |>
        mutate(result = map(roc_curve, ~ .x |> pluck("result"))) |>
        select(username, wflow_id, type, result) |>
        unnest(result) |>
        plot_roc_curve()

Top Games in Training

What were the model’s top games in the training set?

preds |>
        filter(type == 'resamples') |>
        prep_predictions_datatable(
                games = games, 
                outcome = params$outcome
        ) |>
        predictions_datatable(outcome = params$outcome,
                remove_description = T, 
                remove_image = T, 
                pagelength = 15)

Top Games in Validation

What were the model’s top games in the validation set?

preds |>
        filter(type %in% c("valid")) |>
        prep_predictions_datatable(
                games = games,
                outcome = params$outcome
        ) |>
        predictions_datatable(
                outcome = params$outcome,
                remove_description = T, 
                remove_image = T, 
                pagelength = 15)

Top Games by Year

Displaying the model’s top games for individual years in recent years.

preds |>
        filter(type %in% c('resamples', 'valid')) |>
        top_n_preds(
                games = games,
                outcome = params$outcome,
                top_n = 15,
                n_years = 15
        ) |>
        gt_top_n(collection = collection |> prep_collection())

Rank	2008	2009	2010	2011	2012	2013	2014	2015	2016	2017	2018	2019	2020	2021	2022
1	Le Havre	At the Gates of Loyang	Glen More	Tournay	Robinson Crusoe: Adventures on the Cursed Island	Lewis & Clark: The Expedition	Imperial Settlers	Viticulture Essential Edition	Honshū	The Castles of Burgundy: The Dice Game	Newton	Chocolate Factory	Monster Expedition	Boonlake	Wayfarers of the South Tigris
2	Bloom	Shipyard	Troyes	The Castles of Burgundy	Keyflower	Bruges	Port Royal	Oh My Goods!	Lorenzo il Magnifico	Heaven & Ale	Sprawlopolis	Tapestry	Gloomhaven: Jaws of the Lion	Lorenzo il Magnifico: Big Box	Woodcraft: Roll and Write
3	The Hanging Gardens	Dominion: Intrigue	Luna	Ora et Labora	The Manhattan Project	Amerigo	La Granja	Minerva	Star Wars: Rebellion	Gloomhaven	Underwater Cities	The Castles of Burgundy	Hallertau	Agropolis	The Lord of the Rings: The Card Game – Revised Core Set
4	Space Alert	Macao	7 Wonders	The City	Terra Mystica	Glass Road	Orléans	Between Two Cities	The Castles of Burgundy: The Card Game	Sagrada	Space Park	The Magnificent	Glasgow	Sleeping Gods	Endless Winter: Paleoamericans
5	Nefertiti	Alea Iacta Est	The Speicherstadt	Eminent Domain	Suburbia	Russian Railroads	Nations: The Dice Game	Tiny Epic Galaxies	Key to the City: London	Santa Maria	Between Two Castles of Mad King Ludwig	Herbaceous Sprouts	Tekhenu: Obelisk of the Sun	Llamaland	Wingspan Asia
6	Byzanz	Hansa Teutonica	Merkator	Walnut Grove	Ginkgopolis	Caverna: The Cave Farmers	Roll for the Galaxy	Raiders of the North Sea	Agricola (Revised Edition)	Circle the Wagons	Sunset Over Water	Paladins of the West Kingdom	On Mars	Corrosion	Tribes of the Wind
7	Stone Age	Homesteaders	Poseidon	Puerto Rico	Galaxy Trucker: Anniversary Edition	Patchistory	Patchwork	Through the Ages: A New Story of Civilization	A Feast for Odin	Ex Libris	Hokkaido	Barrage	Bonfire	Imperium: Legends	Hamburg
8	Cavum	Fzzzt!	Mousquetaires du Roy	Singapore	OddVille	City of Iron	Isle of Trains	Discoveries: The Journals of Lewis & Clark	Scythe	Anachrony	Cosmic Run: Regeneration	Maracaibo	Pendulum	Imperium: Classics	Agricola 15
9	Battlestar Galactica: The Board Game	Finca	Famiglia	Rune Age	Snowdonia	Rococo	AquaSphere	Signorie	The Oracle of Delphi	Altiplano	Fleet: The Dice Game	Ragusa	Fallout Shelter: The Board Game	Great Plains	Frosthaven
10	Roll Through the Ages: The Bronze Age	Egizia	51st State	Pergamemnon	Eight-Minute Empire	Bora Bora	Viticulture: Complete Collector's Edition	Lancaster: Big Box	Last Will	Tybor the Builder	Everdell	Copenhagen: Roll & Write	The Castles of Tuscany	Rolling Realms	Smitten
11	The Rich and the Good	Cyclades	Forbidden Island	Drako: Dragon & Dwarves	Tzolk'in: The Mayan Calendar	Madeira	Fields of Arle	The Voyages of Marco Polo	Inis	Charterstone	Concordia Venus	Black Angel	My City	Dinosaur Island: Rawr 'n Write	Marrakesh
12	Ice Flow	Carson City	Rattus	Belfort	Among the Stars	City of the Living	Artifacts, Inc.	Kraftwagen	Fields of Green	Lisboa	Finca	PARKS	Raiders of Scythia	Hungry Bunnies on Parade	Amsterdam
13	Dominion	Dungeon Lords	Dominion: Big Box	Pergamon	Il Vecchio	Citrus	La Isla	Grand Austria Hotel	Cottage Garden	Indian Summer	Architects of the West Kingdom	Aeon's End: Legacy	Lost Ruins of Arnak	Biblios: Quill and Parchment	Three Sisters
14	Charioteer	Arena: Roma II	Funfair	Mage Knight Board Game	The Palaces of Carrara	Eight-Minute Empire: Legends	Istanbul	The Gallerist	51st State: Master Set	Keyper	Founders of Gloomhaven	The Taverns of Tiefenthal	Puerto Rico	Ark Nova	Carnegie
15	Toledo	Jaipur	Magnum Sal	Principato	Asgard	Bruxelles 1893	San Juan (Second Edition)	Mombasa (Limited Edition)	Chariot Race	In the Year of the Dragon: 10th Anniversary	The Rise of Queensdale	Roam	Merlin: Big Box	Stroganov	Planet Unknown

Predictions

New and Upcoming Games

What were the model’s top predictions for new and upcoming board game releases?

new_preds |>
        filter(type == 'upcoming') |>
        # imposing a minimum threshold to filter out games with no info
        filter(usersrated >= 1) |>
        # removing this goddamn boxing game that has every mechanic listed
        filter(game_id != 420629) |>
        prep_predictions_datatable(
                games = games_new,
                outcome = params$outcome
        ) |>
        predictions_datatable(outcome = params$outcome)

Older Games

What were the model’s top predictions for older games?