Technology
Are Behavioral interventions in The Field Delivered as Initially Intended? Data from 17k Sessions
Shiri Sadeh-Sharvit, Ph.D.
Chief Clinical Officer
Eleos Health
Waltham, Massachusetts
Simon A. Rego, ABPP, Psy.D.
Chief of Psychology
Montefiore Medical Center
Bronx, New York
Samuel Jefroykin, M.S.
Data Scientist
Eleos Health
Waltham, Massachusetts
Gal Peretz, M.S.
Data Scientist
Eleos Health
Waltham, Massachusetts
Tomer Kupershmidt, M.A.
Product Manager
Eleos Health
Waltham, Massachusetts
Although behavioral interventions have been found efficacious and effective in randomized clinical trials for most psychological disorders, the quality and effectiveness of their delivery in routine clinical practice “in the real world” remains inadequate for a variety of reasons, one of which is the scarcity of objective and standardized methods for evaluating treatments as they are provided. For example, “therapist drift” is a well established issue that is known to ultimately reduce the effectiveness of psychological treatments, however until recently there was limited opportunity to assess treatment adherence beyond controlled studies and at scale. The current study examined whether one key aspect contained in most empirically-supported practice guidelines was included in routine clinical practice in real-life behavioral healthcare settings: encouraging clients to review and summarize their treatment session. This is an essential component of most treatment protocols, as it is proposed to help the client conceptualize their understanding of the session and of their treatment plan. The dataset used for this study contained 17,607 behavioral treatment sessions delivered by 322 therapists to 3,519 patients in 37 mental healthcare settings across the U.S. Sessions were processed via an artificial intelligence (AI) therapy-specific platform. We first created a machine learning (ML) algorithm for speaker recognition to differentiate between the session participants and predict whether the speaker was the client or the therapist according to their utterances. Next, we applied the Valence Aware Dictionary and sEntiment Reasoner (VADER), a sentiment analysis model, to decode the emotional characteristics of the sessions. We found that despite clinical recommendations, only 0.30% (N = 54) of sessions included a review and summary. The few session summaries that we found most commonly addressed relationships (N = 27), work (N = 20), change (N = 6), and alcohol (N = 5). Further, in sessions that included summaries, the clients expressed more positive and negative emotions, but the therapists had not – compared to sessions with no summary. These findings suggest that fidelity with at least one of the main behavioral treatment components is not routinely performed in real-life behavioral settings – perhaps partly explaining the diminished quality and effectiveness of evidence-based treatments in real world clinical practice. As a potential remedy for this, ML and AI can be utilized to offer nuanced, timely feedback to providers, thereby improving the quality of behavioral healthcare services.