Correlations among evaluation systems of US internal medicine residents

1. Milestone-based rating correlates with resident annual evaluation summary ratings but with a greater difference across training years that may account for graded developmental progression across residency training.

2. Milestone-based rating may better detect professionalism-related problems. Medical knowledge ratings in both systems correlate with certification examination scores.

Evidence Rating Level: 2 (Good)

Study Rundown: Residency programs must often evaluate residents to measure their competency through the course of their training program. The resident annual evaluation summary (RAES) was utilized historically to measure residents’ proficiency across different metrics. This rating was largely replaced with milestone-based ratings in 2014 to better guide feedback to residents and provide programs with data across various competencies and sub-competencies. The current cross-sectional study was initiated to assess the validity of the newer assessment. A cohort consisting of 21 284 US internal medicine residents was studied during the 2013-2014 academic year, when programs were required to submit both RAES and milestone ratings. The different rating systems progressively increased in their correlation at later stages of training. For each resident, the difference in ratings between PGY-1 and PGY-3 years was greater with milestone ratings, possibly reflective of a graded increase in competency over time. Residents that failed their certification exam had lower ratings for medical knowledge on both the RAES and milestone ratings. Study results also indicate that milestone ratings may be better suited to detect problems with professionalism.

Overall, the study suggests that milestone ratings provide programs a valid tool to assess the competency of residents through their training period. The study is limited by the lack of generalizability of the findings to other specialties, and because the cross-sectional study utilizes a cohort from the first year of implementation, it is difficult to comment on the trajectory of cohorts through the various competencies over time. Future studies should explore patterns of milestone ratings found in rural and community based programs, whether correlations between sub-competency milestones persist or change over time, and whether low milestone professionalism ratings can be correlated to other evidence of professionalism concerns within a training program.

Click to read the study in JAMA

Relevant Reading: The internal medicine reporting milestones: cross-sectional description of initial implementation in US residency programs

In-Depth [cross-sectional study]: A cross-sectional study of all internal medicine residents from the 2013-2014 academic year was performed during which RAES and milestone ratings were used to evaluate resident performance. RAES ratings were obtained from a form that contained 9 items while milestone ratings included proficiency across 6 competencies and 22 sub-competencies. Residents with matched data from both evaluation systems were analyzed. Spearman correlation was utilized to compare mean RAES and milestone ratings by program year. A stacked linear regression model was used to better understand if the RAES and milestone evaluation systems had different slopes of ratings across program years. Professionalism ratings with RAES and milestones evaluation were descriptively analyzed. Overall, 21 774 residents with were analyzed across all program years: 7048 PGY-1, 7233 PGY-2, and 7003 PGY-3. The 618 residents that failed the certification examination on their first attempt had lower ratings on both the RAES and milestones (RAES difference -0.9, 95% CI, -1.0 to -0.8, p < 0.001; milestone medical knowledge 1 difference -0.3, 95% CI, -0.3 to -0.3, p < 0.001; milestone medical knowledge 2 difference -0.2, 95% CI, -0.3 to -0.2, p < 0.001). Overall, correlation between RAES ratings for professionalism with lowest professionalism milestone ratings was 0.44 (95% CI, 0.43 to 0.45, p < 0.001).

Image: PD

©2016 2 Minute Medicine, Inc. All rights reserved. No works may be reproduced without expressed written consent from 2 Minute Medicine, Inc. Inquire about licensing here. No article should be construed as medical advice and is not intended as such by the authors or by 2 Minute Medicine, Inc.