Exploring the Development of an Interprofessional Situational Judgment Test

INTRODUCTION Although the prevalence of interprofessional education (IPE) is increasing, research measuring competency in IPE is underdeveloped (MacDonald et al., 2010; Shrader et al., 2017). Interprofessional assessment tools are plentiful, but most rely on self-report rather than performance data. Thus, the purpose of this study was to develop a computer-based situational judgment test to evaluate health professional student preparedness for interprofessional practice. METHODS The investigators developed the Creighton Interprofessional Situational Judgment Test (C-IPSJT) which includes nine interprofessional clinical case scenarios, each with multiple questions reflecting IPEC competencies. Subsequently, focus groups representing both IPE and health education experts were convened to assess the C-IPSJT. RESULTS Qualitative analysis revealed several themes. Participants found that the C-IPSJT was a novel assessment of interprofessional skills and case scenarios presented were reflective of current clinical practice. Challenges were found in understanding the role of the test taker, discerning correct responses, and the need for further validity testing. Additional themes included the impact team dynamics have on learner responses and how student responses could be affected by their place on the learning continuum. DISCUSSION Findings from the initial testing of the C-IPSJT demonstrate positive content validity through two independent content expert reviews. Content experts identified that the C-IPSJT could be a viable mechanism to differentiate between novice and competent health professional students. Future research includes item revision, testing over a larger sample in which psychometric


Introduction
Interprofessional education (IPE) is vital to the development of communication and teamwork skills necessary for health care personnel (Health Professions Accreditors Collaborative, 2019). IPE provides insight into other team members' roles and facilitates an understanding of the unique contributions of each profession and how they intersect (Health Professions Accreditors Collaborative, 2019;Jacobs & McCormack, 2019).The IPEC core competencies highlight four domains to prepare students to engage with other healthcare professionals: interprofessional values and ethics, knowledge of health professional roles and responsibilities, healthcare communication among the team, and interprofessional teamwork to provide client-centered care (Interprofessional Education Collaborative, 2022).
As educational institutions set benchmarks for interprofessional skill development at identified times along the curricula, there arises a need to assess student performance along the learning continuum. Dreyfus and Dreyfus (1980) describe the necessity of students passing through multiple levels of learning in acquisition and development of skills. Using the Dreyfus Model of Skill acquisition as a theoretical framework, the Creighton Collaborator IPE Learning Blueprint was developed after review of several key documents including IPEC core competencies, accreditation standards, and key literature in the learning sciences and in IPE. The blueprint was internally developed through a multi-phase collaborative process and vetted with numerous stakeholders, including students, faculty, and clinicians. Learning objectives were created to specify expectations for interprofessional skill development and facilitate the assessment of learners. For each of the IPEC core competencies, there are one to three learning objectives specific to the level of the learner. These learning objectives can be found in Table 1.
In order to capitalize on the benefits of collaborative practice and work toward the Quintuple Aim, IPE programming requires robust assessment and evaluation of IPEC competencies in student learners. Numerous IPE assessment measures exist but many lack appropriate psychometric criteria, and most depend on selfreports. Thannhauser et al. (2010) found 23 various IPE assessment instruments that evaluated attitudes, readiness or communication factors related to IPE with eight of them being considered by the authors as formal assessments: 1) the Index of Interdisciplinary Collaboration (IIC), 2) Multidisciplinary Collaboration Instrument (MDC), 3) Interprofessional Perceptions Scale (IPS), 4) Role Perceptions Questionnaire (RPQ), 5) University of Western England Interprofessional Questionnaire (UWE IQ), 6) Modified Index of Interdisciplinary Collaboration (MIIC), 7) Readiness for Interprofessional Learning Scale (RIPLS), and 8) the Interdisciplinary Education Perception Scale (IEPS). All these instruments are based on self-reported data and have variable validity and reliability. Shrader et al. (2017) found 36 IPE assessments tools that could be used to measure quantitative IPE outcomes in their systematic review; however, none of these instruments fulfilled IPE assessment needs for pharmacy education (Shrader, 2017). In addition, the Interprofessional Collaborative Competencies Attainment Survey (IC-CAS) is currently utilized by the author's institution for evaluation of interprofessional skill attainment and includes all domains of the IPEC core competencies. This assessment utilizes a retrospective pretest/posttest methodology which has the potential to eliminate previously identified IPE survey concerns with response shift bias but may also result in social desirability bias (Tona et al., 2021).

Implications for Interprofessional Practice
• The C-IPSJT assessment has value as a method of evaluation of interprofessional core competencies, including roles and responsibilities.
• The C-IPSJT presents itself as a cost-effective, low resource assessment tool that may be successful in evaluating interprofessional teaching strategies.
• Further research is necessary for validating student responses with clinical performance. The format of a situational judgment test (SJT) lends a comprehensive assessment of student interprofessional skills that can be aligned with the IPEC core competencies. The SJT is a standardized assessment approach containing questions that require test takers to select their most likely action in various circumstances (Smith et al., 2020;Tiffin et al., 2019). SJTs present a realistic, hypothetical situation and ask respondents to select or rank their judgement or knowledge of appropriate behavior in a variety of settings. SJTs range in their modalities which can be delivered with video vignettes, audio recordings, or narrative case scenarios (Yudkowsky et al.). Response options are designed to tap into a single trait or variety of 'non-academic' traits such as professionalism, compassion, and integrity as a means of predicting future behavior. Scoring is typically achieved based on the similarity of the response to subject matter experts (Tiffin et al., 2019).
Traditionally used for professional school admission, SJT's have been found as reliable and valid tools to measure students' abilities in the affective domain (Smith et al., 2020). They provide a consistent, costeffective method to assess individual student communication and performance in a clinical setting that ideally will correlate with future workplace performance (Black et al., 2021;Tiffin et al.,  1. Describe the roles and responsibilities of other providers and how the team works together to provide care, promote health and prevent disease. 1.Distinguish each member's responsibility in executing components of a treatment plan or public health intervention. the project reported herein chose to adapt a SJT to evaluate the Interprofessional Education Collaborative (IPEC) core competencies in health professional students. A SJT was proposed to address the limitations in assessment of non-cognitive attributes and move toward a competency-based evaluation of student interprofessional skills, and at the time of its construction, had not been previously attempted in interprofessional literature. The question raised by the authors at this stage of the C-IPSJT is what is the feasibility of using a situational judgement test to assess interprofessional core competencies?

Methods
The development of the C-IPSJT began with a thorough review of situational judgment tests in health profession education from published literature (Holmbe, 2018;Yudkowsky et.al, 2020). Creation of the items followed the traditional route of developing a situational judgement test through generating case scenarios relevant to the concepts under study. Two of the researchers developed scenarios collaboratively based on clinical expertise and experience in interprofessional collaborative. The backgrounds of the collaborating researchers include health professional education in nursing and physical therapy, with specialization in IPE. Nine interprofessional case scenarios were constructed with the intent of representing all the health science professional programs at the institution. These professions include medicine, nursing (graduate and undergraduate), emergency medical services, physician assistant, physical therapy, occupational therapy, pharmacy, public health, and dentistry. Responses to each scenario were leveled using the IPE Learning Blueprint to reflect learner responses at the novice, advanced beginner, and competent levels. Each question was written using the SJT format what "would" you do, rather than what "should" you do, to focus learners on responses that align with their developmental level. Responses were formed from desirable traits agreed upon by subject matter experts in health professional education in the represented fields. Additionally, researchers intentionally created cases that included a variety of settings across the care continuum to increase the breadth of the exam by assessing team dynamics in a variety of practice settings. For each scenario, there are four SJT questions that reflect the IPEC competencies, resulting in 36 test items. Questions re-lated to values/ethics, communication, and teamwork were phrased in a way that participants could respond regardless of their professional roles. When focusing on the Roles/Responsibilities competency, instructions directed students to respond from the perspective of a specific profession. Throughout the exam, each of the nine health professions at this university are represented once in the Roles/Responsibilities section of each scenario. Figure 1 provides a sample question from one of the C-IPSJT clinical scenarios that assesses learner knowledge of the core competency: interpersonal communication. In the case scenario (not listed), the provider is attempting to delay an urgent patient issue that requires communication between health care providers. The possible response options are identified as either novice, advanced beginner, or competent. The rationale for these responses is mapped to the corresponding learning objectives in the Creighton Collaborator IPE Learner Blueprint (Table 1). The list of possible responses were all written as correct answers with each choice correlating either to the novice, advanced beginner, or competent levels on the Creighton Collaborator IPE Learning Blueprint (Table 1).
Once the initial draft of the C-IPSJT was established, researchers received an exemption from institutional review board review, as this project was a quality improvement project, not meeting the federal regulatory definition of research. Data collection through targeted focus groups occurred in two phases. The first phase began with purposeful sampling of university faculty members engaged in didactic and/or clinical teaching in their health professional programs. Nine faculty were invited via email to participate in the study. The faculty representatives of the first focus group were invited and prioritized with respect to their clinical teaching expertise in their field of study and participation in interprofessional learning activities or involvement in interprofessional curriculum (Figure 2). Prior to the focus group, each participant was asked to complete the 36 item C-IPSJT and respond to a Qualtrics survey that recorded initial impressions or questions. The focus group was designed to explore participant perceptions of item construction, relevance, and accuracy related to current teaching practices. Questions on the initial survey and during the focus group were structured using Kane's validity framework (Kane, 2006).
The second phase was structured similarly, but with a different focus group purposefully sampled from national experts in interprofessional education and/or practice. The six participants were selected based on recommendations from interprofessional collaborators and recruited via email invitation from differing locations nationally (Figure 2). Their backgrounds include professions in medicine, athletic training, physical therapy, higher education research and evaluation, and lead interprofessional centers or hold leadership roles in interprofessional organizations. Similar to phase one and prior to the focus group, the participants were asked to complete the C-IPSJT and respond to a Qualtrics survey with initial impressions. This semi-structured focus group were asked similar questions related to item construction and relevance, but also were asked probing questions related to item validity and generalizability. Participants were incentivized with a $10 gift card. Both focus groups were conducted over an hour and a half via Zoom, audio and video recorded, and transcribed using professional transcription services (Figure 2).

Figure 2. Focus Group Composition
Researchers performed a phenomenological qualitative inquiry to identify clusters of meaning based on the framework of Castleberry & Nolen (2018). To establish credibility of the data, the investigators completing the coding had prior experience in qualitative research design and implementation. This process begins with compiling collected data and then instructing researchers to disassemble the data by looking for important statements and placing statements with connections in meaningful groupings. Reassembling includes putting data into a context with each other and beginning theme identification. Data are then analyzed in the interpretation phase before conclusions are reached (Castleberry & Nolen, 2018). Data from the focus group transcriptions, responses on the C-IPSJT, and feedback on the Qualtrics survey were compiled for analysis so that researchers could work from common transcripts and highlight important phrases and sentences. These data were then analyzed in two steps: 1) Individual thematic analysis using a coding instrument to align the coders by supporting themes with excerpts from the data and 2) second-pass coding and thematic analysis within the research group that allowed the researchers to draw conclusions. Data saturation was reached between the focus group transcriptions and qualitative responses from the survey. Additionally, a detailed track record of the data collection process was maintained through an audit trail, and triangulation techniques were applied in the data analysis. The research team came together regularly to provide inter-coder reliability of the themes before coming to consensus. This process was continued until data saturation was reached.

Results
Several themes emerged from the focus group coding in both focus groups and are described here with supporting participant statements.

Plausibility of the scenarios
Plausibility of the scenarios was a theme that represented participant feedback indicating that overall the quality and realism of the scenarios and potential responses within the C-IPSJT were satisfactory. One participant stated "they seemed realistic. I didn't notice particular questions that seemed way out there."

Role confusion when responding
Several participants reported experiencing role confusion when they were responding to questions on the C-IPSJT. The questions on the draft form of the C-IPSJT are constructed in such a way that many of the questions ask the respondent to identify the best course of action from the perspective of a role which may not be the participant's actual profession. Many focus group participants noted that this was difficult and confusing. One participant noted: "...trying to understand who I was supposed to be and answering these questions was really hard for me".

Learner continuum
Focus group feedback emphasized the importance of identifying where within the learner continuum the assessment should be placed and the impact that it would have on the respondent's answers. This theme emphasized that the level of the student within the curriculum will be different among each of the nine professions represented on the C-IPSJT. For example, one participant stated "…if you have an early learner, those questions (roles/responsibilities) I think are going to be very frustrating because they may not know that part."

Impact of team dynamics
This theme represented feedback on how team dynamics and the learner's experience working within teams may influence how they respond to the C-IPSJT items. For example, the health of the team and the level of comfort the other team members have with each other may impact how students respond to the items on the C-IPSJT. One respondent noted: "...people may act differently in a team where they feel really comfortable with each other…", while another stated "…I think it'd be different if you have good team dynamics versus poor team dynamics…"

"Would vs. Should"
Members of the focus groups gave considerable feedback about how respondents on the C-IPSJT may try to answer according to the ideal (What should I respond?) versus the real response (How would I really respond?). Participants indicated that students may try to answer in a way that they think would be pleasing to the faculty. One participant stated: "I wonder about how authentic the responses would be because I felt myself… to be like should I say what I really think or what I think they want me to say?" This tension between the "would vs. should" caused participants to question the authenticity of student responses.

Challenges distinguishing between competencies
This theme represents the difficulties that participants experienced while taking the C-IPSJT related to their perceived level of competency. As all participants were either experts in their respective profession or experts in interprofessional education, most expected themselves to be able to identify and select the "competent" response (as opposed to the novice and/or advanced beginner responses). However, many found themselves struggling with determining the level of competency that the answers were representing. One respondent noted: "…it was clear to me which one was the novice response, but often I had trouble with which one might have been the advanced beginner versus which one was competent…"

Necessity of further validity testing
This theme represents the overall feedback on the need to continue to do further validity testing of the C-IP-SJT. Many participants stated that it would have been helpful to have understood how items were developed. "I didn't know which questions were tied to what com-petency…" Participants support and recommended further validity testing.

Consider goals of IP assessment
Finally, focus group members voiced the need to clearly identify the goals of the C-IPSJT specific to the assessment of interprofessional skill development in learners. Participants remarked, "Maybe our goal should really just be to identify those students who are truly defi-cient…" and another participant stated "…no scoring is probably ever going to get you to the complexity of the behavior and nuances and the layers." This discussion was inclusive of the need to further reflect on how interprofessional skill assessment will inform curriculum.

Discussion
Feedback from multiple focus groups supports the continued development, refinement, and exploration of the C-IPSJT to assess interprofessional programing and student knowledge of the IPEC Core Competencies. Researchers acknowledge that a need exists to conduct efficient evaluation of interprofessional programming, with responsible utilization of resources, to be supportive of health professional program curriculum and accreditation efforts (Institute of Medicine, 2015). The SJT methodology was chosen to assess these "nonacademic abilities" because SJTs have been successfully used to identify medical school applicants (Sahota 2020; Webster 2020) and assessment of affective skills in pharmacy students (Patterson, 2019;Smith et al., 2020;Wolcott, 2019). The C-IPSJT was found to include scenarios and response options that appeared to represent contemporary healthcare scenarios; however, focus groups revealed that learners would likely have difficulty determining the perspective from which to answer the questions. Researchers plan to revise SJT instructions for learners to clarify the appropriate perspective of the learner for each question. STJ exam clarification regarding team dynamics was also recommended in focus groups to help assess learners' ability to effectively select response options that are specific to social factors that are at play (healthy or dysfunctional team dynamics). Future instructions will emphasize that students taking the exam should focus on the immediate conflict at hand and assume that healthy team dynamics exist among the team members in each scenario.
Additionally, focus groups revealed that although response options were realistic, it was difficult to clearly identify competency leveling between response options. Researchers plan to revise response options to clearly represent the learner continuum by increasing leveling between response options in hopes of being able to distinguish between novice, advanced beginner, and competent learners. Similar to feedback collected in our focus groups, Black et al. (2021) found that it was slightly easier for respondents to identify the least likely response (novice response) than the most likely/ effective behavior (competent response). Therefore, by increasing the leveling differences between novice, advanced beginner, and competent responses, the learners can be more clearly assessed. Rich dialogue occurred during focus groups regarding whether educators should focus on assessment of students along the learner continuum or should be prioritizing which students are demonstrating competence. The latter might be more feasible with a C-IPSJT, but the former has the potential to provide useful information about the impact of IPE activities on student progression toward competency. A benefit to the separation would be to better track and assess student progress with the intent of guiding them toward activities that will help develop skills that are not progressing as expected. Answers to these questions might come out with greater sample sizes of students as they progress through health profession programs. With further testing, the authors' intent is to develop a valid and reliable C-IPSJT to collect information that will help assess the effectiveness of a variety of IPE teaching strategies, and therefore, make evidence-based decisions to improve the quality of these IPE activities.
Health professional social and behavioral competency is necessary for effective collaborative interprofessional healthcare delivery (Gunaldo et al., 2017;Interprofessional Education Collaborative, 2022;Jacobs & McCormack, 2019 HIP E & following the completion of IPE offerings to assess test item validity. Focus group discussion also encouraged the triangulation of data between the C-IPSJT and competence assessed by faculty members prior to and following experiential IPE learning opportunities to assess learner knowledge across the learner continuum and investigate if the C-IPSJT is sensitive enough to identify learner needs specific to the IPEC core competencies. Researchers plan to revise and distribute the C-IPSJT to health professional students to assess validity and determine next best steps in the creation of an exam to assess IPE skills because a need exists to develop a valid and reliable assessment tool that is specific to the IPEC Core Competencies. Although multiple IPE assessment tools for health professional students exist, the vast majority of these tools are either selfreport measures (Holmboe et al. 2018;Thannhauser et al., 2010) or limit the ability to assess individual performance (Blue et al., 2015).

Limitations
Limitations to the findings of this study include the purposive sampling method used for recruitment to both focus groups. While the first focus group, comprised of clinical and teaching faculty, represented all professions included in the C-IPSJT, no recruitment of faculty outside of the institution occurred creating opportunities for bias. Also, the subject matter experts represented on our second focus group lead the nation in expertise on interprofessional education but not on SJT exams and establishing their validity. The focus on creating the assessment was in response to a need in a single university made up of nine health professional programs, and therefore, all health professions are not represented in the resulting C-IPSJT which limits its usability in the larger academic setting.
Additionally, the content experts from the focus groups were concerned about social desirability bias associated with future validity testing. Potential exists for students to answer questions on a C-IPSJT based on how they think faculty want them to respond (how they "should'' answer questions) rather than choosing an answer that represents how they are likely to act (how they "would'' respond) in real life. Ideally the C-IP-SJT would be used as a low stakes exam that allowed students to answer in a minimally stressful scenario in hopes of facilitating honest responses regarding how they "would" act in each scenario. Researchers found that emphasizing that there were no incorrect answers was a strength of the exam and allowed test takers to answer more freely with emotional stress related to the "should" response. This provided students more latitude to explore and hopefully respond with how they "would" act.

Future Recommendations
SJTs have been proven to be valid and reliable assessment tools to measure affective skills such as professionalism and leadership in pharmacy and medicine (Goss 2017, Patterson 2016, Patterson 2012Smith et al 2020). Additionally, studies have found a low correlation between SJT scores and knowledge tests (cognitive ability) and personality tests (Lievens 2012, Cousans et al 2017Patterson, 2015;Wolcott, 2019). More research is needed to determine the best recommendations for using SJTs in health professions students.
To the author's knowledge, only one 18 item C-IPSJT has previously been developed and piloted to assess health professional students IPE competence (Black et al., 2021). This exam used a three-factor model (ethics/ values, teams/teamwork, communication), and unlike the C-IPSJT, did not assess the IPEC competency of roles and responsibilities. Black et al. (2021) also assessed students' ability to identify the most and least likely/effective behavior response. This is in contrast to the C-IPSJT which is designed to assess learners on a continuum. Strengths of both exams include cost effectiveness with the ability to distribute the exam electronically and low requirements for faculty resources. Black et al. (2021), provided early evidence that "multiple IPEC competencies can be assessed in early learners via an SJT" from their pilot data (Black et al., 2021).
Future development, revision, and refinement of the C-IPSJT will focus on further validating the instrument with additional content experts after revisions have been made. Additionally, exploring concurrent and convergent validity of the C-IPSJT could be beneficial to identify if the C-IPSJT is effective at measuring similar constructs as other like assessment methods. For example, the C-IPJT could be administered to measure health professions students' performance in each scenario while simultaneously administering alternative instruments designed to measure IPEC competencies, such as the Creighton -Interprofessional Competency Evaluation (C-ICE) (Iverson et al., 2018). Establish-ing that the students who are able to score highly on the C-IPSJT while simultaneously scoring highly on the C-ICE could add to the body of literature available on assessing IPEC competency in health professions students. Future studies could also include examination of interrater reliability among evaluators when multiple evaluators use the C-IPSJT to assess student attainment of competency achievement.
Additional studies will include exploring whether a correlation exists between C-IPSJT performance and experiential performance through observation of student skills in simulated lab environments. Following validity testing, authors plan to utilize scores on the C-IPSJT to assess how students are progressing and identify those at risk of not reaching IPE competency. For those students who do not meet competency expectations, remediation strategies can be put in place for competence attainment based on the deficiencies identified by the C-IPSJT. The intent of the exam is to discern which interprofessional skills need development, and also, to assess the quality of IPE activities. Low SJT scores have been correlated with "an increased risk of professionalism concerns" in undergraduate medical students (Sahota, 2020). To prepare students to graduate with necessary interprofessional skills, further validity testing of the C-IPSJT can assist with earlier identification and intervention to ensure students are not at risk of entering the workforce without proper skills necessary for working in interprofessional teams.

Conclusion
Findings from the initial testing of the C-IPSJT demonstrate realistic health professional scenarios through two independent content expert reviews. Content experts identified that the C-IPSJT could be a viable mechanism to differentiate between novice, advanced beginner, and competent health professional students. This assessment could provide a valuable way to assess student performance and design tailored learning experiences for health professions students to further develop interprofessional skills in the core competencies of values/ethics, roles and responsibilities, communication, and teamwork. Further validity and reliability testing is needed across a larger sample of health professions educators and students once revisions to the C-IPSJT are constructed in response to focus group feedback Although the nature of interpro-fessional competency development is challenging, the C-IPSJT can provide one mechanism for educators to assess student competence and gauge student readiness for practice in interprofessional teams.