^{1}

^{*}

^{2}

^{3}

^{1}

^{4}

^{5}

The authors have declared that no competing interests exist.

Analyzed the data: JF. Contributed reagents/materials/analysis tools: JF RPM MB. Wrote the paper: JF RPM CL MB. Designed and collected data: MB. Conceived the idea: JF CL.

The approach presented in this article represents a generalizable and adaptable methodology for identifying complex interactions in educational systems and for investigating how manipulation of these systems may affect educational outcomes of interest. Multilayer Minimum Spanning Tree and Monte-Carlo methods are used. A virtual Sandbox University is created in order to facilitate effective identification of successful and stable initiatives within higher education, which can affect students' credits and student retention – something that has been lacking up until now. The results highlight the importance of teacher feedback and teacher-student rapport, which is congruent with current educational findings, illustrating the methodology's potential to provide a new basis for further empirical studies of issues in higher education from a complex systems perspective.

Interest in modelling higher education as a complex system has grown rapidly during the last decades. Although relevant across the whole spectrum of higher education, the fields of physics, mathematics, and engineering are presently in the forefront of research in this area

Previous research into higher education as a complex system lacks, as Sabelli

There are generally two ways of constructing a skeleton for system simulations of processes within higher education; one theoretical, and the other empirical. The approach proposed by Sabelli

We have chosen to focus on the credits students achieved, which is an integral part of student retention, as the target of our analysis. This is because a critical first step for students continuing towards graduation is for them to complete their courses, thus getting the credits needed to continue their studies, also called

Researchers, building on central models of student retention - which academic withdrawal is a part of

In an effort to address this challenge, we report on the creation of a virtual ‘Sandbox University’ (SU), where changes in institutional practice can be simulated, estimated, and compared. The SU is empirically estimated based on questionnaire data consisting of first-year study experiences obtained from engineering students who have physics as a part of their curriculum at the highly regarded Technical University of Delft. We do this in order to: 1) create a localized model which can inform local institutional practice; and, 2) create a system in which it is possible to circumvent the problem that proposed changes can be hindered by exogenous processes of the real-world system. For example, the changes forced on the SU will be “noiseless” – that is free of influence from a changing external environment outside the system being studied, which is of course impossible in the real-world system

Our Sandbox University is composed from 78 previously identified critical aspects of student retention – aspects of students' experience of studying at a university that have been found to have a positive impact on students' abilities to persist through their higher education studies - which also includes students' credits achieved. The data was collected in three-year bachelor programmes from a wide variety of engineering and engineering science programmes in the fall of 2010 at the Technical University of Delft in the Netherlands. The cohort studied consisted of first-year students and the data collection was carried out by using an online questionnaire. The response rate was 25% (573 of 2292). The questionnaire was designed to obtain students' first-year study experiences

The University (TU Delft) where the data was collected required no specific ethics submission, had no ethics board in place, and had no formal procedures to be followed in human subjects' research. Even though this was the case, an informal committee of university researchers and administrators was gathered before data collection to approve the design of the study. This committee consisted out of the Director of Student and Teacher Services and two research professors. Moreover, the data collection followed the ethical guidelines as described by Cohen, Manion, and Morrison

The magenta node is where effect of changes is sought. The black nodes are nodes which are held constant. The blue and grey nodes represent First- and Second-order nodes as per the grouping in

The relationship between the 78 aspects was estimated through an implementation of MMST analysis

The MMST analysis was chosen because, in contrast to a correlation network where everything tends to be connected to everything else, the edges are not a result of choosing a cut-off of the strength of the correlation but through the reproducibility of edges (as shown in

In this study, we used an implementation of MMST analysis

Edge weights (strength of links) in the MMST represent the frequency of that correlation found in each bootstrapped sample. In our implementation, both positive and negative correlations were present in the MST and thus positive and negative relations within the network were identified and colour coded in the visualization as grey (for positive relationship) and red (as negative relationships). In the visualization produced, the 15% weakest (non-frequent) edges are removed. Before this manipulation was done almost every node had weak edges to all other nodes, which resulted in a very noisy visualisation.

The elements of the created network are the measured aspects as per the questionnaire. In each iteration of MMST analysis correlations between questionnaire items are calculated for subsets of the raw data, which are, in turn, recalculated to a distance matrix. Then a minimum spanning tree

In order to estimate the influence and uncertainty that a change in an aspect would have on the target aspect, Gibbs sampling

For example, in our network, students' previous grade in mathematics, students feeling that they have done sufficient preparatory study, and students' who only want to pass and not care about the grades are adjacent to the number of credits achieved. Following

The Gibbs sampling drew from a normal distribution where the mean of this distribution is the weighted mean of the adjacent nodes (

The standard deviation used for the Gibbs sampling was estimated by the unbiased estimator for the weighted sample variance (

Where

Each iteration of Gibbs sampling estimated all interrelated aspects in a random order. The Gibbs sampling ran for 60 000 iterations, with a burn-in period of 1000 to allow for convergence, and with a thinning of 100 to increase the statistical independence of generated values. The estimations are the results of what would have happened to the target aspect when proposing that you could “improve” an aspect from 20% below to 20% above the average of the measured aspect.

The SU was estimated from the observed correlations. However, there are multiple ways of building such networks, such as from a theoretical starting point

As not all aspects can be easily changed, the 78 aspects measured by the questionnaire were then divided into three groups (see

Constant | First-order | Second-order |

Students' age | Teacher expectations (2 Expec) | Students' re-enrolment expectations |

Stem profile combination |
University facilities (5 Uf) | Students' experiences of university facilities (2 Ufs) |

Students' parents' education | Scheduling (6 N) | Degree importance (2 Important) |

Students' biological gender | Course materials (4 Cm) | Language skills (2 Language) |

Students' housing situation | Teacher behaviours (7 Tb) | Fraternity membership |

Students' impairments | Travel time to campus | Students' experience of course materials (2 Cms) |

Students' exposure to university PR | Assessment and feedback (9 Af) | Students' study behaviour (20 Sb) |

Students' prior education | Students' self-evaluated skills (3 Skill) | |

Previous achievement in mathematics | ||

Previous achievement in physics |

The relationships, as estimated by MMST analysis, between these aspects resulted in a network map of how the aspects interrelate (see

Black nodes are the constant nodes, blue are the First-order grouped nodes and grey are the Second-order grouped nodes, the red node is the target node for the proposed changes to institutional practice. The widths of the edges indicate the strength of the estimated links, and the colour represents positive (grey) and negative (red) relationships.

In order to estimate influence and uncertainty of a change in one aspect on the target aspect, Gibbs sampling

The resulting estimations were compared with the estimated standard deviation of each aspect (shown in

The estimated change in student credits achieved is compared in

First-order Aspects | Estimated Change (%) | Estimated Standard Deviation (%) | Hattie Rank | Hattie Theme |

(5) Teacher expectations - Expec_difficulties | 11 | 30 | 10 | Teacher - Feedback |

(32) Course materials - Cm_material | 9 | 32 | - | |

(64) Teacher behaviours - Tb_empathize | 8 | 30 | 11 | Teacher - Teacher-Student Relationships |

(63) Teacher behaviours - Tb_content | 8 | 30 | 11 | Teacher - Teacher-Student Relationships |

(30) Course materials - Cm_feedback | 8 | 30 | 10 | Teacher - Feedback |

(31) Course materials - Cm_late | 7 | 30 | 10 | Teacher - Feedback |

(65) Teacher behaviours - Tb_enthusiasm | 6 | 29 | 11 | Teacher - Teacher-Student Relationships |

(66) Teacher behaviours - Tb_explain | 6 | 30 | 11 | Teacher - Teacher-Student Relationships |

(74) Assessment & feedback - Af_level | 6 | 30 | 10 | Teacher - Feedback |

(71) Assessment & feedback - Af_constr | 6 | 30 | 10 | Teacher - Feedback |

(62) Teacher behaviours - Tb_available | 5 | 30 | 11 | Teacher - Teacher-Student Relationships |

(6) Teacher expectations - Expec_interest | 5 | 28 | 10 | Teacher - Feedback - |

(25) Scheduling - N_lectures |
5 | 80 | - | - |

The largest estimated effect comes from improving teachers' ability to deal with students' expectations, which relates to students' experience of teachers' feedback on how students are doing with the courses. Teacher feedback (especially dealing with students' expectations) has long been recognised as an important factor for student learning within the field of educational research

Other aspects showed lower estimated effects and are thus not reported here. This is because from this simplified model it is highly uncertain that these would have any desirable effects on credits achieved. However, lower estimations could, when introducing more complexity, have more substantial effects, but not consistent ones.

We built a virtual Sandbox University by using empirical data from student questionnaires to identify aspects of the student experience that are most strongly linked. These links were then used to construct a network of interrelated aspects. Based on this network we simulated the effect of changing aspects of the student experience that can plausibly be directly manipulated, investigating the expected impact of each such intervention on student credits achieved. We thus identified the areas where interventions would be most likely to substantially improve student outcomes – such as students' credits achieved, and student retention.

The limitations described previously when using a theoretically driven skeleton for simulations are mirrored in this study, as our results are only as good as the methodology used for creating the network. However, our methodology can be used as an exemplar of how such skeleton networks can be fruitfully estimated. The network created also only covers first-year engineering students. How the network might change over time is beyond the scope of this article.

Our simulation resulted in two important broad and common themes: Teacher feedback and Teacher-student relationships, which have been found to be at the top end of effectiveness when their impact on student achievements has been studied

Within the resulting common themes, an unexpected finding is that the aspect corresponding to students obtaining, and being informed about, the required materials for the courses ((32) Cm_material) has a mean effect size above 5%. This is surprising since this has neither been recognized as the top influence on student credits achieved

The approach presented in this article represents a generalizable and adaptable methodology for identifying complex interactions in educational systems and for investigating how manipulation of these systems may affect outcomes of interest. This approach enables the effective identification of successful and stable initiatives within higher education that can affect students' credits achieved and student retention – something that has been lacking up until now

(DOCX)

(DOCX)

The authors would like to thank Anne Linder and John Airey for their comments on earlier drafts of the article, and Anne Linder for her very helpful editing suggestions.