username | status | games |
---|---|---|
phenrickson | ever_owned | 146 |
phenrickson | own | 141 |
phenrickson | rated | 96 |
Predicting Board Game Collections
phenrickson’s Collection
About
This report details the results of training and evaluating a classification model for predicting games for a user’s boardgame collection.
To view games predicted by the model, go to Section 5.
Collection
The data in this project comes from BoardGameGeek.com. The data used is at the game level, where an individual observation contains features about a game, such as its publisher, categories, and playing time, among many others.
I train a classification model at the user level to learn the relationship between game features and games that a user owns - what predicts a user’s collection?
I evaluate the model’s performance on a training set of historical games via resampling, then validate the model’s performance on a set aside set of newer relases. I then refit the model on the training and validation in order and predict upcoming releases in order to find new games that the user is most likely to add to their collection.
username | years | type | Own | |
---|---|---|---|---|
no | yes | |||
phenrickson | -3500-2020 | train | 24405 | 113 |
phenrickson | 2021-2022 | valid | 9873 | 24 |
phenrickson | 2023-2028 | test | 9095 | 4 |
Types of Games
What types of game does the user own? The following plot displays the most frequent publishers, mechanics, designers, artists, etc that appear in a user’s collection.
Show the code
|>
collection filter(own == 1) |>
collection_by_category(
games = games_raw
|>
) plot_collection_by_category()+
ylab("feature")
The following plot shows the years in which games in the user’s collection were published. This can usually indicate when someone first entered the hobby.
Games in Collection
What games does the user currently have in their collection? The following table can be used to examine games the user owns, along with some helpful information for selecting the right game for a game night!
Use the filters above the table to sort/filter based on information about the game, such as year published, recommended player counts, or playing time.
Show the code
|>
collection filter(own == 1) |>
prep_collection_datatable(
games = games_raw
|>
) filter(!is.na(image)) |>
collection_datatable()
Modeling
I’ll now the examine predictive models trained on the user’s collection.
For an individual user, I train a predictive model on their collection in order to predict whether a user owns a game. The outcome, in this case, is binary: does the user have a game listed in their collection or not? This is the setting for training a classification model, where the model aims to learn the probability that a user will add a game to their collection based on its observable features.
How does a model learn what a user is likely to own? The training process is a matter of examining historical games and finding patterns that exist between game features (designers, mechanics, playing time, etc) and games in the user’s collection.
I make use of many potential features for games, the vast majority of which are dummies indicating the presence or absence of the presence or absence of things such as a publisher/artist/designer. The “standard” BGG features for every game contain information that is typically listed on the box its playing time, player counts, or its recommended minimum age.
I train models to predict whether a user owns a game based only on information that could be observed about the game at its release: playing time, player count, mechanics, categories, genres, and selected designers, artists, and publishers. I do not make use of BGG community information, such as its average rating, weight, or number of user ratings. This is to ensure the model can predict newly released games without relying on information from the BGG community.
What Predicts A Collection?
A predictive model gives us more than just predictions. We can also ask, what did the model learn from the data? What predicts the outcome? In the case of predicting a boardgame collection, what did the model find to be predictive of games a user has in their collection?
To answer this, I examine the coefficients from a model logistic regression with ridge regularization (which I will refer to as a penalized logistic regression).
Positive values indicate that a feature increases a user’s probability of owning/rating a game, while negative values indicate a feature decreases the probability. To be precise, the coefficients indicate the effect of a particular feature on the log-odds of a user owning a game.
The following visualization shows the path of each feature as it enters the model, with highly influential features tending to enter the model early with large positive or negative effects. The dotted line indicates the level of regularization that was selected during tuning.
Show the code
|>
model_glmnet pluck("wflow", 1) |>
trace_plot.glmnet(max.overlaps = 30)+
facet_wrap(~params$username)
Partial Effects
What are the effects of individual features?
Use the buttons below to examine the effects different types of predictors had in predicting the user’s collection.
Assessment
How well did the model do in predicting the user’s collection?
This section contains a variety of visualizations and metrics for assessing the performance of the model(s). If you’re not particularly interested in predictive modeling, skip down further to the predictions from the model.
The following displays the model’s performance in resampling on a training set, a validation set, and a holdout set of upcoming games.
Show the code
|>
metrics mutate_if(is.numeric, round, 3) |>
pivot_wider(
names_from = c(".metric"),
values_from = c(".estimate")) |>
::gt() |>
gt::sub_missing() |>
gtgt_options()
username | wflow_id | type | .estimator | mn_log_loss | roc_auc | pr_auc |
---|---|---|---|---|---|---|
phenrickson | glmnet | resamples | binary | 0.020 | 0.959 | 0.162 |
phenrickson | glmnet | test | binary | 0.006 | 0.976 | 0.042 |
phenrickson | glmnet | valid | binary | 0.011 | 0.977 | 0.234 |
An easy way to visually examine the performance of classification model is to view a separation plot.
I plot the predicted probabilities from the model for every game (during resampling) from lowest to highest. I then overlay a blue line for any game that the user does own. A good classifier is one that is able to separate the blue (games owned by the user) from the white (games not owned by the user), with most of the blue occurring at the highest probabilities (left side of the chart).
Show the code
|>
preds filter(type %in% c('resamples', 'valid')) |>
plot_separation(outcome = params$outcome)
I can more formally assess how well each model did in resampling by looking at the area under the ROC curve (roc_auc). A perfect model would receive a score of 1, while a model that cannot predict the outcome will default to a score of 0.5. The extent to which something is a good score depends on the setting, but generally anything in the .8 to .9 range is very good while the .7 to .8 range is perfectly acceptable.
Show the code
|>
preds nest(data = -c(username, wflow_id, type)) |>
mutate(roc_curve = map(data, safely( ~ .x |> safe_roc_curve(truth = params$outcome)))) |>
mutate(result = map(roc_curve, ~ .x |> pluck("result"))) |>
select(username, wflow_id, type, result) |>
unnest(result) |>
plot_roc_curve()
Top Games in Training
What were the model’s top games in the training set?
Show the code
|>
preds filter(type == 'resamples') |>
prep_predictions_datatable(
games = games,
outcome = params$outcome
|>
) predictions_datatable(outcome = params$outcome,
remove_description = T,
remove_image = T,
pagelength = 15)
Top Games in Validation
What were the model’s top games in the validation set?
Show the code
|>
preds filter(type %in% c("valid")) |>
prep_predictions_datatable(
games = games,
outcome = params$outcome
|>
) predictions_datatable(
outcome = params$outcome,
remove_description = T,
remove_image = T,
pagelength = 15)
Top Games by Year
Displaying the model’s top games for individual years in recent years.
Show the code
|>
preds filter(type %in% c('resamples', 'valid')) |>
top_n_preds(
games = games,
outcome = params$outcome,
top_n = 15,
n_years = 15
|>
) gt_top_n(collection = collection |> prep_collection())
Rank | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Cosmic Encounter | Hansa Teutonica | Troyes | Mage Knight Board Game | Descent: Journeys in the Dark (Second Edition) | Rococo | Alchemists | 7 Wonders Duel | Star Wars: Rebellion | Sherlock Holmes Consulting Detective: Vanishing from Hyde Park | Century: Eastern Wonders | Maracaibo | Unmatched: Little Red Riding Hood vs. Beowulf | Boonlake | Frosthaven |
2 | Le Havre | Jaipur | Earth Reborn | A Game of Thrones: The Board Game (Second Edition) | Wiz-War (Eighth Edition) | Lewis & Clark: The Expedition | Pandemic: The Cure | Food Chain Magnate | Scythe | Century: Spice Road | Newton | Clank! Legacy: Acquisitions Incorporated | Unmatched: Cobble & Fog | Railroad Ink Challenge: Shining Yellow Edition | Unmatched: Redemption Row |
3 | Space Alert | American Rails | Merchants & Marauders | Puerto Rico | Keyflower | Glass Road | AquaSphere | Mombasa | Junk Art | Gloomhaven | Rising Sun | Century: A New World | On Mars | Ark Nova | Unmatched: Hell's Kitchen |
4 | Byzanz | Age of Conan: The Strategy Board Game | Mousquetaires du Roy | Mansions of Madness | Archipelago | 1775: Rebellion | Orléans | The Gallerist | Hit Z Road | Pandemic Legacy: Season 2 | Cosmic Encounter: 42nd Anniversary Edition | Era: Medieval Age | Hallertau | Arkham Horror: The Card Game (Revised Edition) | Planet Unknown |
5 | Battlestar Galactica: The Board Game | Greed Incorporated | Labyrinth: The War on Terror, 2001 – ? | Eminent Domain | Terra Mystica | Gearworld: The Borderlands | Port Royal | Trambahn | Sherlock Holmes Consulting Detective: Jack the Ripper & West End Adventures | Stop Thief! | Azul: Stained Glass of Sintra | Dune | Gloomhaven: Jaws of the Lion | Unmatched: Battle of Legends, Volume Two | Return to Dark Tower |
6 | Giants | The Adventurers: The Temple of Chac | Tikal II: The Lost Temple | Gears of War: The Board Game | Galaxy Trucker: Anniversary Edition | Caverna: The Cave Farmers | La Granja | Elysium | Agricola (Revised Edition) | Azul | Concordia Venus | Silver & Gold | Pandemic Legacy: Season 0 | Cartographers Heroes | The Great Split |
7 | Roll Through the Ages: The Bronze Age | Middle-Earth Quest | Glen More | A Few Acres of Snow | Il Vecchio | Rise of Augustus | Splendor | Empires: Age of Discovery | Terraforming Mars | My Little Scythe | Coimbra | Cartographers | Undaunted: North Africa | Welcome to the Moon | Endless Winter: Paleoamericans |
8 | Pandemic | Shipyard | Sid Meier's Civilization: The Board Game | BraveRats | Clash of Cultures | Blueprints | Akrotiri | Oh My Goods! | Mansions of Madness: Second Edition | Century: Golem Edition | War Chest | Tainted Grail: The Fall of Avalon | Unmatched: Buffy the Vampire Slayer | Railroad Ink Challenge: Lush Green Edition | Carnegie |
9 | Toledo | At the Gates of Loyang | Dominant Species | Ora et Labora | The Great Zimbabwe | Eldritch Horror | The Battle at Kemble's Cascade | Arboretum | Reign of Cthulhu | Gaia Project | Lords of Hellas | The Castles of Burgundy | New York Zoo | Imperial Steam | Undaunted: Stalingrad |
10 | Diamonds Club | Dice Town | Firenze | The Lord of the Rings: The Card Game | Kemet | NFL Game Day | Isle of Trains | Karuba | Welcome Back to the Dungeon | Sagrada | Root | Blitzkrieg!: World War Two in 20 Minutes | Welcome to New Las Vegas | Kemet: Blood and Sand | Warhammer: The Horus Heresy – Age of Darkness |
11 | Nefertiti | The Resistance | Paris Connection | Letters from Whitechapel | Mage Wars Arena | Forbidden Desert | Three Kingdoms Redux | Watson & Holmes | Great Western Trail | Folklore: The Affliction | Istanbul: Big Box | Unmatched Game System | Rococo: Deluxe Edition | Galaxy Trucker (Second Edition) | アンドーンテッド:ノルマンディー・プラス (Undaunted: Normandy Plus) |
12 | Duck Dealer | Macao | Wars of the Roses: Lancaster vs. York | Tournay | The Manhattan Project | Cube Quest | Castles of Mad King Ludwig | Blood Rage | Iberia | Iberian Gauge | Yellow & Yangtze | Unmatched: Robin Hood vs. Bigfoot | Via Magica | Llamaland | Nemesis: Lockdown |
13 | Planet Steam | FITS | BattleCON Fighting System | Sekigahara: The Unification of Japan | Robinson Crusoe: Adventures on the Cursed Island | Spyrium | Power Grid Deluxe: Europe/North America | GEM | When I Dream | Sherlock Holmes Consulting Detective: Carlton House & Queen's Park | Blackout: Hong Kong | Paladins of the West Kingdom | Merv: The Heart of the Silk Road | Kemet: Blood and Sand – Kickstarter Edition | Foundations of Rome |
14 | Stone Age | Cyclades | Innovation | Takenoko | Libertalia | Room 25 | Camel Up | Runebound (Third Edition) | Honshū | Lisboa | The Estates | Black Angel | Florenza: X Anniversary Edition | Corrosion | Foundations of Rome (Emperor Edition) |
15 | Wasabi! | Small World | Rattus | Singapore | Agricola: All Creatures Big and Small | Russian Railroads | La Isla | Codenames | Sakura Arms | Bunny Kingdom | Western Legends | Tapestry | Sherlock Holmes Consulting Detective: The Baker Street Irregulars | Rome: Rising Empires | Gateway Island |
Predictions
New and Upcoming Games
What were the model’s top predictions for new and upcoming board game releases?
Show the code
|>
new_preds filter(type == 'upcoming') |>
# imposing a minimum threshold to filter out games with no info
filter(usersrated >= 1) |>
# removing this goddamn boxing game that has every mechanic listed
filter(game_id != 420629) |>
prep_predictions_datatable(
games = games_new,
outcome = params$outcome
|>
) predictions_datatable(outcome = params$outcome)
Older Games
What were the model’s top predictions for older games?