Predicting Board Game Collections

aboardgamebarrage’s Collection

About

This report details the results of training and evaluating a classification model for predicting games for a user’s boardgame collection.

Note

To view games predicted by the model, go to Section 5.

Collection

The data in this project comes from BoardGameGeek.com. The data used is at the game level, where an individual observation contains features about a game, such as its publisher, categories, and playing time, among many others.

I train a classification model at the user level to learn the relationship between game features and games that a user owns - what predicts a user’s collection?

username	status	games
aboardgamebarrage	ever_owned	456
aboardgamebarrage	own	236
aboardgamebarrage	rated	303

I evaluate the model’s performance on a training set of historical games via resampling, then validate the model’s performance on a set aside set of newer relases. I then refit the model on the training and validation in order and predict upcoming releases in order to find new games that the user is most likely to add to their collection.

username	years	type	Own
			no	yes
aboardgamebarrage	-3500-2020	train	24339	179
aboardgamebarrage	2021-2022	valid	9867	30
aboardgamebarrage	2023-2028	test	9072	27

Types of Games

What types of game does the user own? The following plot displays the most frequent publishers, mechanics, designers, artists, etc that appear in a user’s collection.

collection |>
        filter(own == 1) |>
        collection_by_category(
                games = games_raw
        ) |>
        plot_collection_by_category()+
        ylab("feature")

The following plot shows the years in which games in the user’s collection were published. This can usually indicate when someone first entered the hobby.

Games in Collection

What games does the user currently have in their collection? The following table can be used to examine games the user owns, along with some helpful information for selecting the right game for a game night!

Use the filters above the table to sort/filter based on information about the game, such as year published, recommended player counts, or playing time.

collection |>
        filter(own == 1) |>
        prep_collection_datatable(
                games = games_raw
        ) |>
        filter(!is.na(image)) |>
        collection_datatable()

Modeling

I’ll now the examine predictive models trained on the user’s collection.

For an individual user, I train a predictive model on their collection in order to predict whether a user owns a game. The outcome, in this case, is binary: does the user have a game listed in their collection or not? This is the setting for training a classification model, where the model aims to learn the probability that a user will add a game to their collection based on its observable features.

How does a model learn what a user is likely to own? The training process is a matter of examining historical games and finding patterns that exist between game features (designers, mechanics, playing time, etc) and games in the user’s collection.

I make use of many potential features for games, the vast majority of which are dummies indicating the presence or absence of the presence or absence of things such as a publisher/artist/designer. The “standard” BGG features for every game contain information that is typically listed on the box its playing time, player counts, or its recommended minimum age.

Note

I train models to predict whether a user owns a game based only on information that could be observed about the game at its release: playing time, player count, mechanics, categories, genres, and selected designers, artists, and publishers. I do not make use of BGG community information, such as its average rating, weight, or number of user ratings. This is to ensure the model can predict newly released games without relying on information from the BGG community.

What Predicts A Collection?

A predictive model gives us more than just predictions. We can also ask, what did the model learn from the data? What predicts the outcome? In the case of predicting a boardgame collection, what did the model find to be predictive of games a user has in their collection?

To answer this, I examine the coefficients from a model logistic regression with ridge regularization (which I will refer to as a penalized logistic regression).

Positive values indicate that a feature increases a user’s probability of owning/rating a game, while negative values indicate a feature decreases the probability. To be precise, the coefficients indicate the effect of a particular feature on the log-odds of a user owning a game.

The following visualization shows the path of each feature as it enters the model, with highly influential features tending to enter the model early with large positive or negative effects. The dotted line indicates the level of regularization that was selected during tuning.

model_glmnet |> 
        pluck("wflow", 1) |>
        trace_plot.glmnet(max.overlaps = 30)+
        facet_wrap(~params$username)

Partial Effects

What are the effects of individual features?

Use the buttons below to examine the effects different types of predictors had in predicting the user’s collection.

Categories

Mechanics

Themes

Families

Publishers

Designers

Artists

Components

Mechanisms

Assessment

How well did the model do in predicting the user’s collection?

This section contains a variety of visualizations and metrics for assessing the performance of the model(s). If you’re not particularly interested in predictive modeling, skip down further to the predictions from the model.

The following displays the model’s performance in resampling on a training set, a validation set, and a holdout set of upcoming games.

metrics |>
        mutate_if(is.numeric, round, 3) |>
        pivot_wider(
                names_from = c(".metric"),
                values_from = c(".estimate")) |>
        gt::gt() |>
        gt::sub_missing() |>
        gt_options()

username	wflow_id	type	.estimator	mn_log_loss	roc_auc	pr_auc
aboardgamebarrage	glmnet	resamples	binary	0.035	0.887	0.091
aboardgamebarrage	glmnet	test	binary	0.020	0.857	0.019
aboardgamebarrage	glmnet	valid	binary	0.020	0.771	0.016

An easy way to visually examine the performance of classification model is to view a separation plot.

I plot the predicted probabilities from the model for every game (during resampling) from lowest to highest. I then overlay a blue line for any game that the user does own. A good classifier is one that is able to separate the blue (games owned by the user) from the white (games not owned by the user), with most of the blue occurring at the highest probabilities (left side of the chart).

preds |>
        filter(type %in% c('resamples', 'valid')) |>
        plot_separation(outcome = params$outcome)

I can more formally assess how well each model did in resampling by looking at the area under the ROC curve (roc_auc). A perfect model would receive a score of 1, while a model that cannot predict the outcome will default to a score of 0.5. The extent to which something is a good score depends on the setting, but generally anything in the .8 to .9 range is very good while the .7 to .8 range is perfectly acceptable.

preds |>
        nest(data = -c(username, wflow_id, type)) |>
        mutate(roc_curve = map(data, safely( ~ .x |> safe_roc_curve(truth = params$outcome)))) |>
        mutate(result = map(roc_curve, ~ .x |> pluck("result"))) |>
        select(username, wflow_id, type, result) |>
        unnest(result) |>
        plot_roc_curve()

Top Games in Training

What were the model’s top games in the training set?

preds |>
        filter(type == 'resamples') |>
        prep_predictions_datatable(
                games = games, 
                outcome = params$outcome
        ) |>
        predictions_datatable(outcome = params$outcome,
                remove_description = T, 
                remove_image = T, 
                pagelength = 15)

Top Games in Validation

What were the model’s top games in the validation set?

preds |>
        filter(type %in% c("valid")) |>
        prep_predictions_datatable(
                games = games,
                outcome = params$outcome
        ) |>
        predictions_datatable(
                outcome = params$outcome,
                remove_description = T, 
                remove_image = T, 
                pagelength = 15)

Top Games by Year

Displaying the model’s top games for individual years in recent years.

preds |>
        filter(type %in% c('resamples', 'valid')) |>
        top_n_preds(
                games = games,
                outcome = params$outcome,
                top_n = 15,
                n_years = 15
        ) |>
        gt_top_n(collection = collection |> prep_collection())

Rank	2008	2009	2010	2011	2012	2013	2014	2015	2016	2017	2018	2019	2020	2021	2022
1	Byzanz	Jaipur	Innovation	Discworld: Ankh-Morpork	The Resistance: Avalon	Hemloch: Vault of Darkness	Spyfall	GEM	Omen: Edge of the Aegean	Dungeon of Mandom VIII	Cosmic Encounter: 42nd Anniversary Edition	Dune	Nidavellir	For Sale Autorama	SPYBAM
2	Modern Art Card Game	Hansa Teutonica	Politico: The Fall of Caesar	Omen: A Reign of War	Archipelago	Kobayakawa	Maskmen	7 Wonders Duel	Honshū	Azul	Mr. Face	Omen: Fires in the East	Cat in the box	Oath	Cat in the Box: Deluxe Edition
3	Ultimate Werewolf: Ultimate Edition	Samurai: The Card Game	Dominion: Big Box	A Fake Artist Goes to New York	Fleet	Barrel Dice	Chimera	The Game + The Game on Fire	Insider	Merchants of Muziris	Brass: Birmingham	Detective: City of Angels	Omen: Heir to the Dunes	Kingdom Come	Kongkang: The Wild Party
4	Seii Taishogun	Greed Incorporated	Earth Reborn	Vanuatu	Terra Mystica	NFL Game Day	Tricks & Deserts	Keep	Neolithic	Breaking Bad: The Board Game	Pax Emancipation	One Night Ultimate Super Heroes	Insider Black	Import / Export: Definitive Edition	Harry Potter: Kampf gegen die dunklen Mächte
5	Steel Driver	Kuhhandel Master	London	Artus	Libertalia	Stone & Relic	The Nile Ran Red	Elysium	Vampire Queen	Cartouche Dynasties	The Quacks of Quedlinburg	One Night Ultimate Super Villains	Mezo	Kemet: Blood and Sand – Kickstarter Edition	White Elephant: A Gift Exchange
6	Red November	A Brief History of the World	Time's Up! Family	A Game of Thrones: The Board Game (Second Edition)	Tooth & Nail: Factions	A Study in Emerald	Port Royal	Hemloch: Midnight Edition	Junk Art	Jump Drive	TOKYO METRO	Marco Polo II: In the Service of the Khan	Florenza: X Anniversary Edition	The Diamond Swap	Make the Difference
7	Time's Up! Edición Amarilla	Telestrations	Bhazum	Takenoko	Uchronia	Skull King	La Isla	Watson & Holmes	Hit Z Road	Hemloch: Dark Promenade	Decrypto	SCOUT	Hello Neighbor: The Secret Neighbor Party Game	Moon Adventure	A Game of Thrones: B'Twixt
8	Le Havre	Steam	De Vulgari Eloquentia	Village	Yedo	Blueprints	Power Grid Deluxe: Europe/North America	Hordes of Grimoor	Scythe	Startups	Coimbra	Obscurio	Guild Master	Kemet: Blood and Sand	1877: Stockholm Tramways
9	Cosmic Encounter	The Resistance	Age of Industry	Tournay	Rex: Final Days of an Empire	Habe fertig	Irish Gauge	Het Koninkrijk Dominion	Welcome Back to the Dungeon	The Game: Face to Face	Rising Sun	Century: A New World	Sacred Rites	Rome: Rising Empires	Desamparados: Stalingrado
10	Kheops	Imperial 2030	The Hobbit	Mundus Novus	The Great Zimbabwe	Glass Road	Pandemic: Contagion	Soulfall	One Night Ultimate Vampire	Custom Heroes	Underwater Cities	Subtext	The Cost	Mint Bid	Blood on the Clocktower
11	Strozzi	Endeavor	In a Grove	The Castles of Burgundy	Urbania	Cinque Terre	Deception: Murder in Hong Kong	T.I.M.E Stories	SYNOD	Werewords	Century: Eastern Wonders	TOKYO COIN LAUNDRY	Hues and Cues	Facility 07	GridL
12	Córdoba	Warhammer: Invasion	Irondale	Hemloch	Keyflower	Carcassonne: South Seas	Orongo	Grand Austria Hotel	GearSeed	One Night Ultimate Alien	The Mind	Ohanami	The Red Cathedral	Quest: Avalon Big Box Edition	Hunch!
13	Toledo	American Rails	Hotel Samoa	Friday	Kemet	Relic	Imperial Settlers	TROLL	Codenames: Deep Undercover	Sidereal Confluence	Yellow & Yangtze	Rurik: Dawn of Kiev	Enigma: Beyond Code	Ankh: Gods of Egypt	Revive
14	Giants	Pocket Rockets	Mystery Express	Dungeon Petz	The Palaces of Carrara	Bora Bora	Onirim (Second Edition)	One Night Ultimate Werewolf: Daybreak	Game of Thrones: The Iron Throne	A Game of Thrones: Catan – Brotherhood of the Watch	Newton	TOKYO GAME SHOW	Ninja Catfoot and the Covert Action	Bad Company	The Middle Ages
15	Nefertiti	Dominion: Intrigue	Asara	The City	卑怯なコウモリ (Cowardly Bat)	Tash-Kalar: Arena of Legends	Sons of Anarchy: Men of Mayhem	Stockpile	Mino Dice	Claim	The Binding of Isaac: Four Souls	KOMBIO	13 Monsters	Vitamors Conspiro	Ultimate Werewolf: Extreme – Super Collector's Edition

Predictions

New and Upcoming Games

What were the model’s top predictions for new and upcoming board game releases?

new_preds |>
        filter(type == 'upcoming') |>
        # imposing a minimum threshold to filter out games with no info
        filter(usersrated >= 1) |>
        # removing this goddamn boxing game that has every mechanic listed
        filter(game_id != 420629) |>
        prep_predictions_datatable(
                games = games_new,
                outcome = params$outcome
        ) |>
        predictions_datatable(outcome = params$outcome)

Older Games

What were the model’s top predictions for older games?