Predicting Board Game Collections

VWValker’s Collection

About

This report details the results of training and evaluating a classification model for predicting games for a user’s boardgame collection.

Note

To view games predicted by the model, go to Section 5.

Collection

The data in this project comes from BoardGameGeek.com. The data used is at the game level, where an individual observation contains features about a game, such as its publisher, categories, and playing time, among many others.

I train a classification model at the user level to learn the relationship between game features and games that a user owns - what predicts a user’s collection?

username	status	games
VWValker	ever_owned	183
VWValker	own	183
VWValker	rated	104

I evaluate the model’s performance on a training set of historical games via resampling, then validate the model’s performance on a set aside set of newer relases. I then refit the model on the training and validation in order and predict upcoming releases in order to find new games that the user is most likely to add to their collection.

username	years	type	Own
			no	yes
VWValker	-3500-2020	train	24420	98
VWValker	2021-2022	valid	9824	73
VWValker	2023-2028	test	9087	12

Types of Games

What types of game does the user own? The following plot displays the most frequent publishers, mechanics, designers, artists, etc that appear in a user’s collection.

collection |>
        filter(own == 1) |>
        collection_by_category(
                games = games_raw
        ) |>
        plot_collection_by_category()+
        ylab("feature")

The following plot shows the years in which games in the user’s collection were published. This can usually indicate when someone first entered the hobby.

Games in Collection

What games does the user currently have in their collection? The following table can be used to examine games the user owns, along with some helpful information for selecting the right game for a game night!

Use the filters above the table to sort/filter based on information about the game, such as year published, recommended player counts, or playing time.

collection |>
        filter(own == 1) |>
        prep_collection_datatable(
                games = games_raw
        ) |>
        filter(!is.na(image)) |>
        collection_datatable()

Modeling

I’ll now the examine predictive models trained on the user’s collection.

For an individual user, I train a predictive model on their collection in order to predict whether a user owns a game. The outcome, in this case, is binary: does the user have a game listed in their collection or not? This is the setting for training a classification model, where the model aims to learn the probability that a user will add a game to their collection based on its observable features.

How does a model learn what a user is likely to own? The training process is a matter of examining historical games and finding patterns that exist between game features (designers, mechanics, playing time, etc) and games in the user’s collection.

I make use of many potential features for games, the vast majority of which are dummies indicating the presence or absence of the presence or absence of things such as a publisher/artist/designer. The “standard” BGG features for every game contain information that is typically listed on the box its playing time, player counts, or its recommended minimum age.

Note

I train models to predict whether a user owns a game based only on information that could be observed about the game at its release: playing time, player count, mechanics, categories, genres, and selected designers, artists, and publishers. I do not make use of BGG community information, such as its average rating, weight, or number of user ratings. This is to ensure the model can predict newly released games without relying on information from the BGG community.

What Predicts A Collection?

A predictive model gives us more than just predictions. We can also ask, what did the model learn from the data? What predicts the outcome? In the case of predicting a boardgame collection, what did the model find to be predictive of games a user has in their collection?

To answer this, I examine the coefficients from a model logistic regression with ridge regularization (which I will refer to as a penalized logistic regression).

Positive values indicate that a feature increases a user’s probability of owning/rating a game, while negative values indicate a feature decreases the probability. To be precise, the coefficients indicate the effect of a particular feature on the log-odds of a user owning a game.

The following visualization shows the path of each feature as it enters the model, with highly influential features tending to enter the model early with large positive or negative effects. The dotted line indicates the level of regularization that was selected during tuning.

model_glmnet |> 
        pluck("wflow", 1) |>
        trace_plot.glmnet(max.overlaps = 30)+
        facet_wrap(~params$username)

Partial Effects

What are the effects of individual features?

Use the buttons below to examine the effects different types of predictors had in predicting the user’s collection.

Categories

Mechanics

Themes

Families

Publishers

Designers

Artists

Components

Mechanisms

Assessment

How well did the model do in predicting the user’s collection?

This section contains a variety of visualizations and metrics for assessing the performance of the model(s). If you’re not particularly interested in predictive modeling, skip down further to the predictions from the model.

The following displays the model’s performance in resampling on a training set, a validation set, and a holdout set of upcoming games.

metrics |>
        mutate_if(is.numeric, round, 3) |>
        pivot_wider(
                names_from = c(".metric"),
                values_from = c(".estimate")) |>
        gt::gt() |>
        gt::sub_missing() |>
        gt_options()

username	wflow_id	type	.estimator	mn_log_loss	roc_auc	pr_auc
VWValker	glmnet	resamples	binary	0.022	0.869	0.048
VWValker	glmnet	test	binary	0.017	0.829	0.019
VWValker	glmnet	valid	binary	0.036	0.885	0.100

An easy way to visually examine the performance of classification model is to view a separation plot.

I plot the predicted probabilities from the model for every game (during resampling) from lowest to highest. I then overlay a blue line for any game that the user does own. A good classifier is one that is able to separate the blue (games owned by the user) from the white (games not owned by the user), with most of the blue occurring at the highest probabilities (left side of the chart).

preds |>
        filter(type %in% c('resamples', 'valid')) |>
        plot_separation(outcome = params$outcome)

I can more formally assess how well each model did in resampling by looking at the area under the ROC curve (roc_auc). A perfect model would receive a score of 1, while a model that cannot predict the outcome will default to a score of 0.5. The extent to which something is a good score depends on the setting, but generally anything in the .8 to .9 range is very good while the .7 to .8 range is perfectly acceptable.

preds |>
        nest(data = -c(username, wflow_id, type)) |>
        mutate(roc_curve = map(data, safely( ~ .x |> safe_roc_curve(truth = params$outcome)))) |>
        mutate(result = map(roc_curve, ~ .x |> pluck("result"))) |>
        select(username, wflow_id, type, result) |>
        unnest(result) |>
        plot_roc_curve()

Top Games in Training

What were the model’s top games in the training set?

preds |>
        filter(type == 'resamples') |>
        prep_predictions_datatable(
                games = games, 
                outcome = params$outcome
        ) |>
        predictions_datatable(outcome = params$outcome,
                remove_description = T, 
                remove_image = T, 
                pagelength = 15)

Top Games in Validation

What were the model’s top games in the validation set?

preds |>
        filter(type %in% c("valid")) |>
        prep_predictions_datatable(
                games = games,
                outcome = params$outcome
        ) |>
        predictions_datatable(
                outcome = params$outcome,
                remove_description = T, 
                remove_image = T, 
                pagelength = 15)

Top Games by Year

Displaying the model’s top games for individual years in recent years.

preds |>
        filter(type %in% c('resamples', 'valid')) |>
        top_n_preds(
                games = games,
                outcome = params$outcome,
                top_n = 15,
                n_years = 15
        ) |>
        gt_top_n(collection = collection |> prep_collection())

Rank	2008	2009	2010	2011	2012	2013	2014	2015	2016	2017	2018	2019	2020	2021	2022
1	Le Havre	Axis & Allies: Pacific 1940	Prêt-à-Porter	Mage Knight Board Game	Terra Mystica	Tash-Kalar: Arena of Legends	Orléans	Codenames	Star Wars: Rebellion	Charterstone	Nemesis	Tainted Grail: The Fall of Avalon	Hansa Teutonica: Big Box	Bloodborne: The Board Game	Gateway Island
2	Byzanz	Shipyard	Warhammer: The Island of Blood	Mansions of Madness	Descent: Journeys in the Dark (Second Edition)	Sushi Go!	Imperial Settlers	Signorie	Codenames: Deep Undercover	Pandemic Legacy: Season 2	Rising Sun	Siege of the Citadel	Gùgōng: Deluxe Big Box	Kingdom Come	Endless Winter: Paleoamericans
3	Axis & Allies Anniversary Edition	Dungeon Lords	Sneaks & Snitches	Pictomania	Axis & Allies: 1941	Madeira	Star Wars: Imperial Assault	M.U.L.E. The Board Game	Perdition's Mouth: Abyssal Rift	Gloomhaven	Everdell	Zombicide: Invader	Eclipse: Second Dawn for the Galaxy	Ankh: Gods of Egypt	Marvel Zombies: Heroes' Resistance
4	Space Alert	Space Pirates	Firenze	The New Era	Star Wars: The Card Game	Hanamikoji	Nations: The Dice Game	Blood Rage	SeaFall	Anachrony	The Quacks of Quedlinburg	Clank! Legacy: Acquisitions Incorporated	CloudAge	Tabannusi: Builders of Ur	Frostpunk: The Board Game
5	Senji	Skyline 3000	Troyes	Ora et Labora	Rex: Final Days of an Empire	Glass Road	Three Kingdoms Redux	Through the Ages: A New Story of Civilization	Terraforming Mars	This War of Mine: The Board Game	Mage Knight: Ultimate Edition	PARKS	Rush M.D.	Llamaland	Nemesis: Lockdown
6	Android	War of the Ring	Battles of Westeros	Gears of War: The Board Game	Cockroach Poker Royal	Lewis & Clark: The Expedition	Praetor	Heroes	Scythe	Gaia Project	Architects of the West Kingdom	Crystal Palace	The Quacks of Quedlinburg: Big Box	Key Pai Sho	Mosaic: A Story of Civilization
7	Sorry! Sliders	Axis & Allies: 1942	Flicochet	Ascending Empires	Warhammer 40,000 (Sixth Edition)	1775: Rebellion	Arcadia Quest	Mombasa	Great Western Trail	Massive Darkness	Underwater Cities	Aftermath	Gloomhaven: Jaws of the Lion	Rome: Rising Empires	Tindaya
8	Giants	Steam	Warhammer: The Game of Fantasy Battles (8th Edition)	Rune Age	卑怯なコウモリ (Cowardly Bat)	Terror in Meeple City	Fields of Arle	Viticulture Essential Edition	Codenames: Pictures	Twilight Imperium: Fourth Edition	Crown of Emara	Tiny Towns	Pandemic Legacy: Season 0	Boonlake	Carnegie
9	Cosmic Encounter	Middle-Earth Quest	Space Hulk: Death Angel – The Card Game	Tournay	Robinson Crusoe: Adventures on the Cursed Island	Francis Drake	Castles of Mad King Ludwig	Risk: Europe	Santorini	Dungeon of Mandom VIII	The Estates	Detective: City of Angels	On Mars	Marvel United: X-Men	Frosthaven
10	Red November	The Adventurers: The Temple of Chac	Vinhos	A Fake Artist Goes to New York	Suburbia	The Builders: Middle Ages	Tiny Epic Kingdoms	Oh My Goods!	Imhotep	My Little Scythe	Gizmos	HATE	Tawantinsuyu: The Inca Empire	Steamwatchers	Skymines
11	Mutant Chronicles Collectible Miniatures Game	Age of Conan: The Strategy Board Game	51st State	Dungeon Raiders	Keyflower	Theseus: The Dark Orbit	luz	Steampunk Rally	Arkham Horror: The Card Game	Calimala	Hokkaido	The Hunters A.D. 2114	Lost Ruins of Arnak	The Great Wall	Quacks & Co.: Quedlinburg Dash
12	Kakerlakensuppe	Vasco da Gama	Runewars	Eminent Domain	Tzolk'in: The Mayan Calendar	Concept	Dungeon Lords: Happy Anniversary	Cacao	Aeon's End	Mythic Battles: Pantheon	Newton	Judge Dredd: Helter Skelter	Starcadia Quest	Ark Nova	Wayfarers of the South Tigris
13	Battlestar Galactica: The Board Game	Cyclades	Glen More	The Lord of the Rings: The Card Game	OddVille	Canterbury	Sons of Anarchy: Men of Mayhem	Dungeon Saga: Dwarf King's Quest	Fields of Green	Flick 'em Up!: Dead of Winter	Blackout: Hong Kong	Zona: The Secret of Chernobyl	Altar Quest	Clash of Cultures: Monumental Edition	Hunch!
14	Cannonball Colony	Warhammer: Invasion	Norenberc	War of the Ring: Second Edition	Ginkgopolis	Deadzone	Camel Up	Forbidden Stars	Star Trek: Frontiers	Unfair	Container: 10th Anniversary Jumbo Edition!	Hellboy: The Board Game – Deluxe Edition	Glasgow	Batman: The Animated Series Adventures – Shadow of the Bat	Warhammer: The Horus Heresy – Age of Darkness
15	World of Warcraft: The Adventure Game	Tarantel Tango	GOSU	A Game of Thrones: The Board Game (Second Edition)	Galaxy Trucker: Anniversary Edition	Romolo o Remo?	Johari	Minerva	Clank!: A Deck-Building Adventure	Smile	The World of SMOG: Rise of Moloch	Star Wars: Outer Rim	Florenza: X Anniversary Edition	Corrosion	Agricola 15

Predictions

New and Upcoming Games

What were the model’s top predictions for new and upcoming board game releases?

new_preds |>
        filter(type == 'upcoming') |>
        # imposing a minimum threshold to filter out games with no info
        filter(usersrated >= 1) |>
        # removing this goddamn boxing game that has every mechanic listed
        filter(game_id != 420629) |>
        prep_predictions_datatable(
                games = games_new,
                outcome = params$outcome
        ) |>
        predictions_datatable(outcome = params$outcome)

Older Games

What were the model’s top predictions for older games?