46 - Quality Assessment of Systematic Reviews Published in a Flagship Journal of AAMC and an Open Access Journal

Objectives: Systematic reviews (SR) are a form of investigation that synthesize existing evidence to provide statements of conclusion and help guide decision making in evidence-based medicine. A crucial component when starting a SR consists of forming the right multidisciplinary team, creating a protocol, and utilizing appropriate search strategies and data abstraction tools to minimize bias. The purpose of this study was to evaluate the quality of published SRs in medical education between a “flagship” journal of AAMC, Academic Medicine (AM), and an open access journal, BioMed Central Medical Education (BMC).

Methods: Three authors searched PubMed for SRs published in BMC and AM between 2020-21 for a total of 39 articles. A quality assessment tool drawing on items from the AMSTAR2 and PRISMA was created to assess the SRs. Each article was analyzed independently by two students and disagreements were resolved via discussion and the 3rd reviewer.

Results: While SRs published in both journals reported their study design, inclusion and exclusion criteria, risk of bias assessment tools, both journals had discrepancies in reporting of other criteria. 21% of the SRs in AM (n=4) and 40% (n=8) in BMC reported and registered their protocol, 53% (n=10) in AM and 55% (n=11) in BMC consulted an informational specialist, 48% in AM (n=9) and 60% in BMC (n=12) mentioned use of MeSH terms or indexed terms, 84% (n=16) in AM and 65% (n=13) in BMC provided a sample search strategy, and two SRs in each journal published with only two authors.

Conclusions: Assuring the quality of SRs can be challenging and measuring the quality is a prudent step in evaluating an SR’s impact in guiding decision making. Evaluation of the two peer-reviewed journals focusing on medical education reveals that the quality of SRs published is not always consistent in following the SR guidelines or standards.