426 - Interactive Survey Based Assessment as tool for Assessment of the Simulation Curriculum “One Pill Can Kill – The Beta Blocker Ingestion: A Simulation Case for Pediatric Residents”

Poster Number: 426
Publication Number: 426.3341

Background: Traditional post-simulation assessments involve learner reaction and multiple-choice questions (MCQ), which lack the ability to evaluate higher levels of learning on Kirkpatrick’s Model. In-person simulation-based assessments can be challenging due to logistical constraints. “Choose your own adventure” (CYOA) gaming has not been studied as a method to assess learning following simulation.

Objective: To create an assessment that evaluates resident learning of a simulation (at Kirkpatrick’s level 2 (learner understanding) and level 3 (behavior change or application of knowledge)) and evaluate the interactive survey as an asynchronous assessment tool that can be reproduced for various simulation curricula.

Design/Methods: Residents rotating through the pediatric emergency department participated in a beta-blocker ingestion simulation. Following the simulation, residents received a text message with one board-style MCQ. One month after the simulation, learners received an email with a CYOA-style survey, through which learners engaged with a patient case, making choices that would direct subsequent questions or feedback they received (Fig 1). Metrics included completion rates, engagement, and content knowledge.

Results: Out of 54 participants, 37 (69%) completed the text-based MCQ and 27 (50% of total residents, 73% of those that completed the MCQ) completed the CYOA assessment (Table 1). Median completion time was 5 minutes 41 seconds. Residents completed the survey in 12 or 13 questions with fewer total questions needed for completion signaling more correct answers. 18 (67%) chose the correct diagnosis of beta-blocker ingestion and 5 (19%) were concerned for an unknown ingestion (Table 2). Participants strongly agreed (63%) or agreed (33%) that the simulation prepared them for the assessment and strongly agreed (44%) or agreed (52%) that this was an engaging way to assess content knowledge. The majority of participants (89%) felt the amount of questions/length of the assessment tool was appropriate.

Conclusion(s): The interactive CYOA assessment tool proved feasible for evaluating Kirkpatrick’s higher levels of learning post-simulation. The assessment was completed asynchronously and was completed in a feasible amount of time by most residents. The novel CYOA assessment method, provides insight into the learner’s understanding of content and decision-making during the assessment, enhancing evaluation beyond mere knowledge retention which could allow for higher-level assessment (Kirkpatrick’s level 2 (learning) and 3 (knowledge application)), instead of reaction and knowledge alone.