398 - Social Determinants of Health Competency in Undergraduate Medical Students: Development and Initial Validity Evidence for a Novel Evaluation Instrument

Poster Number: 398
Publication Number: 398.1397

Background: Social determinants of health (SDH) are a critical factor in patient care outcomes. Incorporation of SDH curricula is growing in pre-clinical and clinical education, yet no formal instrument exists to evaluate learner competency. To address this gap, we previously developed and investigated content, response process, and consequences of testing validity evidence of a novel SDH competency assessment for medical students in a pediatric elective.

Objective: We aimed to add to our validity evidence by examining internal structure (reliability) of our assessment.

Design/Methods: We used the SDH competency assessment at our institution in a pediatric outpatient advocacy elective for fourth-year medical students. Clinical settings in which evaluations occurred included 1:1 didactics and case-based sessions, acute and well pediatric and adolescent visits. Table 1 defines 10 SDH competencies assessed. Students were rated in each competency on a Likert scale ranging from 1 (absolute beginner) to 5 (expert). A Multifaceted Rasch analysis was conducted to investigate evaluator consistency (0.70=acceptable, 0.80=good, and 0.90=excellent) and scale use when employing the assessment.

Results: Seven fourth-year medical students were evaluated by 6 supervising pediatricians (evaluators). Four students were evaluated by a total of 4 evaluators, and 3 were evaluated by a total of 3 evaluators for a total of 25 evaluations completed. Of the 6 evaluators that completed the instrument, 5 evaluators used the instrument completely. These 5 evaluators who participated each used the tool in a productive and internally consistent manner. Rasch reliability for evaluators was 0.97, which is considered “excellent” for internal consistency. There was “good” reliability (0.85) across raters, indicating the evaluators interpreted the components in a consistent fashion. Figure 1 demonstrates that raters mostly used ratings 3 (intermediate), 4 (proficient/practiced/skilled), and 5 (expert) and no evaluators used the 1 (absolute beginner) or 2 (beginner) rating scale points.

Conclusion(s): The evaluation instrument described is performing well with strong reliability and shows promise for future use in undergraduate medical education. Due to evaluator feedback and our results, the instrument was shortened to have only 4 rating scale points: 1 (beginner), 2 (intermediate), 3 (proficient), 4 (expert). We have expanded instrument use in additional institutions, levels and types of learner, with a goal of determining its ability to discriminate between lower levels of SDH competency.