Symposia
Dissemination & Implementation Science
Nicole B. Gumport, Ph.D.
Postdoctoral Fellow
Stanford University
Stanford, California
Samantha Hernandez, PhD
Project Coordinator
National Center for PTSD; Stanford University
Menlo Park, California
Alayna Park, Ph.D.
University of Oregon
Eugene, Oregon
Jiyoung Song, PhD
PhD Student
University of California, Berkeley
Berkeley, California
Amber Calloway, Ph.D.
Research Associate
The Penn Collaborative for CBT and Implementation Science, Perelman School of Medicine, University of Pennsylvania
Philadelphia, Pennsylvania
Kimberlye E. Dean, Ph.D.
Clinical Research Fellow
Massachusetts General Hospital/Harvard Medical School
Boston, Massachusetts
Dawne Vogt, PhD
Health Science Specialist
National Center for PTSD
Boston, Massachusetts
Soo Young, Ph.D.
Assistant Professor
Harvard Medical School
Cambridge, Massachusetts
Clara Johnson, PhD
Graduate Student
University of Washington, Seattle
Seattle, Washington
Robert DeRubeis, PhD
Professor
University of Pennsylvania
Philadelphia, PA
Luana Marques, PhD
Associate Professor
Harvard Medical School
Weston, MA
Torrey A. Creed, Ph.D.
Assistant Professor
Perelman School of Medicine at the University of Pennsylvania
Philadelphia, Pennsylvania
Shannon Wiltsey Stirman, Ph.D.
Associate Professor/Acting Deputy Director
Stanford University
Menlo Park, California
Assessing fidelity is an important part of assessing key implementation outcomes. Although observer ratings are the gold-standard for assessing fidelity, conducting these ratings is burdensome and often infeasible. Developing scalable measures of fidelity is a priority for implementation efforts. This study aimed to validate a more efficient measure of fidelity—clinician-rated checklists completed as part of routine treatment—to cognitive processing therapy (CPT) and to cognitive behavior therapy (CBT), which are frontline treatments for PTSD and depression/anxiety, respectively. It was hypothesized that for both CPT and CBT (1) clinician-rated fidelity would be associated with observer-rated fidelity and (2) clinician-rated fidelity would be associated with treatment outcome. Clinicians (N = 164) who treated patients (N = 403) with CPT for PTSD or CBT for depression or anxiety in routine care settings (e.g., VA, medical center, community mental health clinics) completed a clinician-rated checklist of fidelity to either CPT (N =687 checklists) or CBT (N = 312 checklists) at the end of each session. Checklists were scored for the percentage of required treatment elements that were applied (CPT: mean = 64%, SD = 29%; CBT: mean = 82%, SD = 17%). Trained observers also rated treatment sessions for fidelity (N = 552 [CPT]; 268 [CBT]). Treatment outcome was measured using the PTSD Checklist for DSM-5 (PCL-5), Beck Anxiety Inventory (BAI), and Patient Health Questionnaire-9 (PHQ-9). Multilevel modeling was used. In prelimiary analyses, clinician-rated fidelity was significantly associated with observer-rated fidelity for both CPT (Beta = 0.52, p < .000) and CBT (Beta = 0.57, p < .000). Analyses using time-lagged models revealed that clinician-rated fidelity predicted change on the PCL-5 following CPT (Beta = -0.04, p = 0.04). Analyses revealed that clinician-rated fidelity predicted change on the PHQ-9 following CBT (Beta=-0.12, p = 0.04) but did not predict change on the BAI. These results offer preliminary evidence that clinician-rated checklists can efficiently and accurately assess fidelity to CPT and CBT in routine care settings. Although clinician ratings were not associated with all outcome measures, they were highly concordant with observer ratings, which suggests they are an option for regularly monitoring fidelity. The findings also provide preliminary evidence of the validity of a new checklist that can be used across CBTs. Implications for different measurement options and treatment outcome will be discussed.