Assistant Professor of Neurology and Pediatrics Children's Hospital of Philadelphia / University of Pennsylvania
Rationale: The American Epilepsy Society self-assessment committee developed a national in-service examination, the Epilepsy Fellowship In-Training Exam (EpiFITE). We present our program evaluation findings, including evidence of reliability, validity, acceptability and potential impact on further learning and instruction. Methods: A panel of expert epilepsy educators was convened in the summer of 2019, and an examination blueprint was constructed based on the content outline of the ABPN Initial Certification Examination in Epilepsy. After receiving formal instruction on item writing, each expert wrote five-15 multiple choice questions (MCQ). Questions were reviewed and edited by at least one other committee member (content expert), the committee chair, and an expert in professional self-assessment tools. After this initial review, a consensus review of all MCQ items was performed by the entire panel. The panel also generated a consensus prediction of the difficulty of each item on a four-point scale (1 = easiest; 4 = most difficult), using a shared mental model of the average epilepsy fellow. The result was a 110-item exam that was administered on a web-based platform during a testing window from March 11-18, 2020. Items were analyzed using discrimination index, difficulty index, and the relationship between predicted and actual difficulty. Internal consistency of the exam was tested using Cronbach’s alpha. Fellowship directors (FDs) and individual fellows were surveyed regarding the quality and potential effects of the EpiFITE exam. Results: A total of 173 fellows from 67 programs completed the test in March 2020. The mean score was 66% correct (SD 8.2, range 38-82%). Cronbach’s alpha was 0.776, suggesting acceptable internal consistency. The mean difficulty index for the items was 66% (SD 21.8, range 6-99%), suggesting a moderately difficult examination overall. The mean discrimination index was 0.11 (SD 0.6, range ) suggesting relatively weak discrimination, but only one invalid item (discrimination index < 0) was identified. There was a statistically significant correlation between predicted and actual difficulty (Pearson Correlation -0.58, p < 0.0001; Figure 1). Fifty-eight fellows (33%) and 24 FDs (36%) responded to the post-test survey (Table 1). Most fellows agreed that the EpiFITE identified areas for self-guided improvement and for preparation for the epilepsy certification exam. One hundred percent of FDs agreed that they would want to continue using the EpiFITE yearly. There was strong agreement that the EpiFITE could inform medical knowledge assessment for the Epilepsy Milestones, guide preparation for the board examinations, and identify areas for program improvement. Conclusions: The 2020 EpiFITE was a valid, reliable, and acceptable assessment tool for Epilepsy Fellows. Both fellows and FDs found it to be useful to guide self-directed learning, certification exam preparation and program improvement. Individual test items had relatively weak discrimination, and future iterations of the examination could focus on improving this discrimination index, in order to better inform assessment for individual fellows and programs. Funding: Please list any funding that was received in support of this abstract.: none
