Smart Digital Therapeutics for Alcohol Use Disorder: Algorithms for Prediction and Adaptive Intervention

John J. Curtin, Ph.D.

University of Wisconsin-Madison

Mental Healthcare Needs are High and Unmet

• In 2019, 52 million Americans had an active mental illness
  • More than half did not receive any treatment

We have a mental health crisis in the U.S. and it is a crisis of unmet high need because our delivery of mental healthcare is deeply flawed.

In 2019, more than half of the 52 million Americans with an active mental illness did not receive any treatment. More than half!

Mental Healthcare Needs are High and Unmet

• In 2019, 52 million Americans had an active mental illness
  • More than half did not receive any treatment
• 20 million adults had an active substance use disorder
  • 9 out of 10 did not receive any treatment

And for the 20 million adults suffering with a substance use disorder, it was worse still.

9 out of 10 without any treatment

Mental Healthcare Needs are High and Unmet

• In 2019, 52 million Americans had an active mental illness
  • More than half did not receive any treatment
• 20 million adults had an active substance use disorder
  • 9 out of 10 did not receive any treatment
• Large treatment disparities exist by race, ethnicity, geography, and income

Our failure to treat is even more troubling for vulnerable groups. Black and LatinX adults receive mental healthcare services at only half the rate of whites.

And similar mental healthcare disparities exist for people living in rural communities and for those with lower incomes.

Mental Healthcare Needs are High and Unmet

• In 2019, 52 million Americans had an active mental illness

  • More than half did not receive any treatment

• 20 million adults had an active substance use disorder

  • 9 out of 10 did not receive any treatment

• Large treatment disparities exist by race, ethnicity, geography, and income

• Failure to treat is not surprising given many treatment barriers:

  • Access
  • Availability
  • Affordability
  • Acceptability

Our failure to treat is, unfortunately, not surprising. There are many well known barriers to receiving traditional mental healthcare.

These include problems with access that are particularly limiting for people living in rural communities

Problems with availability

Treatment costs are often prohibitive for those without health insurance

and stigma and related issues make traditional treatments for mental illness less acceptable to some patients.

Digital Therapeutics (DTx)

Digital therapeutics are smartphone “apps” that are designed to prevent, manage, or treat disease, including mental illness.

Can augment mental health services to address barriers

• Accessible everywhere
• Available 24/7
• Highly scalable (affordable?)

Fortunately, digital therapeutics are now being developed and used by patients to address many of these treatment barriers.

For those of you that are not yet familiar with this treatment modality, digital therapeutics are smartphone “apps” that are designed to prevent, manage, or treat disease, including substance use disorders and other mental illness.

These digital therapeutics can be combined with traditional treatments to reduce barriers because they are

-Accessible everywhere -Available everyday, 24/7 -and highlighly scalable, which may lower costs

Digital Therapeutics (DTx)

Digital therapeutics are smartphone “apps” that are designed to prevent, manage, or treat disease, including mental illness.

Can augment mental health services to address barriers

• Accessible everywhere
• Available 24/7
• Highly scalable (affordable?)
• Effective!

Of course, these benefits would be meaningless if digital therapeutics were not effective.

But they are.

For example, patients with substance use disorders who use a digital therapeutic have almost double the odds of being abstinent from alcohol or other drugs

These increases in abstinence from using digital therapeutics are observed not only when compared to patients on wait lists, who have yet to gain access to treatment but also when digital therapeutics are added on top of traditional treatments for substance use disorders.

And these benefits are durable - they have been documented up to 12 months after the start of treatment.

These apps are in use today with patients with SUD and at least two have recently received FDA approval. However, I think they are still best considered beta versions relative to their full potential.

Their power comes from easy, 24/7 access to their many supports - the treatments, tools, and services built into these smartphone apps. But this is also their Achilles heel. As the patient using these apps, you now have to tackle difficult questions like: - When should I use them? - For how long? - Which of their many supports are best for me? - And which are best for me right now, at this moment in time?

Smart Digital Therapeutics

“Could you predict not only who might be at greatest risk for relapse …
… but precisely when that relapse might occur …
… and how best to intervene to prevent it?”

My research team became interested in these issues when my colleague Dave Gustafson, the developer of a leading digital therapeutic for substance use disorders, approached us with a simple question. He asked…

“Could you predict not only who might be at greatest risk for relapse but precisely when that relapse might occur and how best to intervene to prevent it”

Dave had just completed a large study demonstrating the effectiveness of his app. However, he also noticed many of the people who relapsed hadn’t used the app in the days leading up to that relapse. And others who had relapsed hadn’t used the specific supports in the app that he would have thought would be most effective for them.

The next wave of digital therapeutics, lets call them smart digital therapeutics, must learn to know us well enough to recognize when we are at greatest risk for relapse and they must be smart enough to recommend the specific supports that would be most effective for us at that moment in time to prevent that relapse.

And these apps will do this through the use of built-in artificial intelligence or machine learning algorithms that are powered by personal sensing.

Lapse Prediction in Patients with AUD

• 151 patients with AUD
• Early in recovery (1-8 weeks)
• Committed to abstinence throughout study
• Followed for up to 3 months
• Collected active and passive personal sensing data streams

 
 

GOAL: Develop a temporally precise lapse monitoring (prediction) system for patients with AUD

So let me transition now to describing how we are taking the first baby steps toward developing smart digital therapeutics for SUD

We have recently completed a NIAAA funded project where we collected data from 151 participants who were in early recovery from a moderate to severe alcohol use disorder.

These participants were committed to abstinence at the start of the study and we followed them for up to 3 months, collecting a variety of active and passive personal sensing data streams.

Our first goal with this grant was to develop machine learning algorithms that can generate temporally precise predictions about when future lapses back to alcohol use will occur for patients with AUD.

Personal Sensing Data Streams

• 4X daily ecological momentary assessments (EMA)

• Monthly self-report

• Geolocation (GPS)

• Cellular communications (voice and text messages)

  • Meta data
  • Text message content

• Sleep sensor (Wake/sleep times; sleep efficiency; wakings; restlessness)

As I mentioned, in this project, we collected a variety of active and passive personal sensing data streams.

Participants completed brief (7-10 item) ecological momentary assessments or EMAs, 4 times per day

We also have - more temporally coarse, monthly self reports, - Moment by moment geolocation, - Meta data from their cellular communications and the actual content of their text messages, - and we had sleep sensors in their beds.

We are in the early stages of model building at this point and I will focus today on results from preliminary models using only EMA. However, we are actively working with GPS and cellular communications as well and I’ll end with some brief discussion of those preliminary models, as well as early plans on how to implement these models clinically to help patients.

4x Daily Ecological Momentary Assessments

 

• Current
  • Craving
  • Affect
  • Risky situations
  • Stressful events
  • Pleasant events
• Future
  • Risky situations
  • Stressful events
  • Confidence

So let me tell you a bit more about the 4x daily EMA we collected.

On each EMA, participants reported the date and time of any lapses back to alcohol use that they hadn’t previously reported.

All of the EMAs also asked them about their current craving, affective valence and arousal, recent risky situations, and recent stressful and pleasant events since their last EMA.

On the first EMA each day, they also reported any future risky situations and stressful events that they expected in the next week and their confidence that they would remain abstinent.

Feature Engineering

• Features based on recent past experiences (12, 24, 48, 72, 168 hours)

• Min, max, and median response (all items)

• History (count) of past lapses (item 1) and completed EMAs (compliance)

• Raw scores and change scores (from baseline/all past responses)

We used these raw EMAs to engineer about 300 features to use in machine learning models to predict future lapses

We formed features by aggregating EMA items over various past time periods ranging from 12 -168 hours in the past

We calculated mins, maxes and medians for the EMA items in these time periods

We also calculated counts of past lapses and counts of past EMAs completed to index compliance

And we included these scores both in raw form and as change from baselines for the participant based on all their previous responses since the start of the study.

Machine Learning Methods

• Predict hour-by-hour probability of future lapse

• Lapse window widths

  • 1 hour
  • 1 day
  • 1 week

For our purposes today I wont dive deep into the machine learning methods but let me highlight a few high level details

We used these features I just described to make predictions about the hour-by-hour probability of a future lapse. We are developing separate models for three future lapse windows – lapses in the next hour, lapses in the next day, and lapses in the next week.

For example, if I was in recovery from an AUD, I could use these models to generate the probability that I would lapse after this symposium starting at 4pm. One model would generate the probability of a lapse between 4pm and 5 pm today, the second would predict the probability of a lapse between 4 pm today and 4pm tomorrow and the third would provide the probability of a lapse between 4 pm today and 4pm next Tuesday.

And of course, all of the models would only use data collected prior to 4 pm today so that they are “predicting”, in the full sense of the word, into the future and not just demonstrating an association.

Machine Learning Methods

• Statistical Algorithms
  • ElasticNet GLM (e.g., LASSO, ridge regression)
  • Random Forest
  • XGBoost
  • KNN
• Model Tuning and Performance Evaluation
  • Area under ROC curve (AUC) as primary performance metric
  • Sensitivity, Specificity, Balanced accuracy, Positive predictive value
  • Using grouped (by participant) 10-fold CV

We are evaluating machine learning model configurations that differ by common statistical algorithms.

We are evaluating these models primarily using the area under the ROC curve but we also consider and report a variety of other common metrics.

And, of course, these performance metrics are calculated for new observations and new participants that the models have never seen and were not trained on by using grouped 10-fold cross-validation.

1 Week: Probabilities for No Lapse and Lapse

• Model predicts probability of lapse in next week for “new” observations in test set

• Can panel predictions by Ground Truth (i.e., true lapse vs. no lapse observations

• Want high probabilities to be high for true lapses and low for true no lapses

Ok, lets start with the model that provides the coarsest level of temporal specificity – 1 week, and let me take a moment to make the predictions that this machine learning model provides more concrete for you

On the right, you are looking at histograms of the lapse probability predictions that the model makes for all the weeks for all the patients in the held out folds.

I’ve paneled these histograms by whether a lapse did or did not happen in reality for each predicted week. The top panel is for weeks with lapses and the bottom panel is for weeks with no lapses.

Ideally, you want the predicted probabilities to be very high for weeks where there was a lapse and very low for weeks where there was no lapse.

And this is exactly what we see for the one week lapse window model

1 Week: Probabilities for No Lapse and Lapse

• Model predicts probability of lapse in next week for “new” observations in test set

• Can panel predictions for GROUND TRUTH lapse and no lapse observations

• Want high probabilities to be high for true lapses and low for true no lapses

• Need decision threshold for classification (.50 default)

Now to move from probabilities to actual categorical decisions – in other words, predicting a lapse or no lapse in some specific week, we need a decision threshold. A probability of .50 is often used for this threshold, but as we will discuss later, there are times when we might want to choose other thresholds.

But for now, I will use .5 such that the model will predict “lapse” for all weeks with probabilities > .5 and it will predict “no-lapse” for all weeks with probabilities < .5

Performance Metrics by Lapse Window Width

Using this decision threshold, we can now calculate the model’s sensitivity, specificity and balanced accuracy which is just the average of these two.

This one week lapse prediction model correctly predicts “lapse” for 79 percent of the weeks that contain a lapse and it correctly predicts “no-lapse” for 86 percent of the weeks that do not include a lapse.

1 Week: ROC Curve

Area under the ROC curve (AUC)

• Across all decision thresholds

• ~.5 (random) – 1.0 (perfect)

Coarse rules of thumb for AUC

.70 - .80 are considered fair
.80 - .90 are considered good
> .90 are considered excellent

But as I said, depending on the application, we may not want to always use a .5 decision threshold. And this is where the ROC curve and the area under this curve come into play as a performance metric. The ROC curve is a plot of the model’s sensitivity by its specificity across all possible decision thresholds.

The area under this curve can range from approximately .5 for a random classifier to 1.0 for a classifier that performs perfectly.

And the AUC for our 1 week model is .90, which is generally considered excellent performance.

1 Day: ROC Curve

Coarse rules of thumb for AUC

.70 - .80 are considered fair
.80 - .90 are considered good
> .90 are considered excellent

So we were very encouraged by the performance of this model because it exceeded the performance of the only other published week level lapse prediction model that we were aware of.

But we were also eager to see how well we could do if we required a higher level of temporal precision by developing a day level model.

1 Day: ROC Curve

Coarse rules of thumb for AUC

.70 - .80 are considered fair
.80 - .90 are considered good
> .90 are considered excellent

And here is the ROC curve and the AUC for the day level model in blue, superimposed on the week level model in purple.

The day level model performed as well as, if not better, than the week level model with an AUC of .91

[PAUSE]

Performance Metrics by Lapse Window Width

And consistent with this higher AUC, the day level model also had slightly better sensitivity and balanced accuracy, along with comparable specificity to the week level model

[PAUSE]

1 Hour: ROC Curve

Coarse rules of thumb for AUC

.70 - .80 are considered fair
.80 - .90 are considered good
> .90 are considered excellent

Given our success with this day level model, we next developed a model with the highest level of temporal precision we could build given how we measured lapses, which was the hour level model.

1 Hour: ROC Curve

Coarse rules of thumb for AUC

.70 - .80 are considered fair
.80 - .90 are considered good
> .90 are considered excellent

And it turns out that we can do somewhat better still with hour level predictions. Here the orange curve represents the ROC curve for the hour level model, which had the best AUC yet, .93

[PAUSE]

Performance Metrics by Lapse Window Width

And again, consistent with that AUC, the sensitivity, specificity, and balanced accuracy were higher still for this model.

[PAUSE]

Global Variable Importance by Model

• All EMA items impact lapse probability (both globally and locally)
• Local importance used for recommendation interventions?
• Lapse day and Lapse hour are useful for day and hour level models as expected
• Demographics not particularly important (but limited race/ethnicity diversity)

In the spirit of making this model more transparent and interpretable, lets briefly look under the hood at the feature importance

The plot on the right shows feature names and their associated importance. From this we see a few important characteristics of the model.

First, only a small number of the approximately 300 features contribute meaningfully to predictions. You can see that the importance is already skewing toward 0 even among the first 25 important features

Second, 8 of the 10 EMA items are represented among these features.

And counts, mins, maxes, and medians, as well as raw and chng scores across all time periods all make contributions.

Positive Predictive Value (PPV)

Lets return one more time to the performance of these models because I would be remiss if I didn’t complicate the story a bit before we close.

Clearly, the sensitivity, specificity, and balanced accuracy of all three of these models is encouragingly high

But what is often missed when we evaluate the performance of machine learning models is their positive predictive value or PPV.

PPV looks at the percentage of positive predictions from the model that are actually true lapses. And PPV is often lower when the positive event, in our case, lapses, is infrequent.

Positive Predictive Value (PPV)

Now we haven’t talked about the frequency of lapses across the three lapse windows, but it should be intuitively clear that the percent of weeks that contain a lapse will be higher than the percent of days that contain a lapse, and that the percent of hours that contain a lapse will be lower still.

And this is what we see in our data.

Positive Predictive Value (PPV)

Because of this, we see relatively low PPV for our models – and particularly the day and the hour level models.

These models catch most lapses (they have hi sensitivity), and they correctly label most no-lapse observations as well (hi specificity). But when you consider their positive predictions, a high percentage of these lapse predictions are false alarms.

This has two big implications.

First, we need to consider the impact of false alarms when using these predictions. It may not be problematic to encourage the patient to use their digital therapeutic in cases where the model makes a positive lapse prediction because the cost of a false alarm in that instance is low. More DTx use is likely always good.

But we may not want to encourage the use of more costly treatments if PPV is low.

And we may need to be careful about how we convey our predictions to patients. Perhaps it might be better to talk about level of risk rather than specifically predicting that a lapse will occur to avoid explicit false alarms that might decrease patient trust in the system.

Impact of Decision Thresholds: 1 day

 

Second, if we need higher PPV, we may be able to get it by using a decision threshold that is higher than .5

Here I am showing you the performance of the day level model when we use the default .5 threshold that you saw previously

Impact of Decision Thresholds: 1 day

 

But if we move the threshold up to .9 before we call an observation a lapse, we can increase the PPV from .32 to .83.

And this is what we will need to do if we want to recommend more costly treatments or other actions where false alarms may be problematic.

Precision - Recall Curves

 

But of course, as we increase the decision threshold for labeling a window as a lapse, we will trade off sensitivity. We can see this trade off directly in the precision-recall curves. If we decide we need PPV of at least .75, you can see that we still have reasonable sensitivity for the one week window but we start to miss many lapses in the 1day window and more still in the 1hour window.

I’ll return to this a bit more later when we discuss emerging plans for how best to implement these models within a digital therapeutic.

Key Take Home Messages

• Relatively high combined sensitivity and specificity

• Comparable performance (AUC) from 1 week down to 1 hour windows

• Will need to adjust decision thresholds to fit how we use the algorithm.

  • Lower PPV OK for low burden or low cost recommendations
  • Higher PPV needed to recommend “costly” interventions or actions

So to recap our main take-aways from this first project

• We were generally encouraged by these preliminary models.

• We’ve demonstrated that we can get relatively high combined sensitivity and specificity

• We can do this for not only coarse one week windows but also temporally precise windows down to one hour

• But we also recognize that depending on what we will use the predictions for, we may need to adjust decision thresholds to trade off sensitivity for greater positive predictive value.

(Selective) Next Steps

• Geolocation, cellular communications, and other passively sensed signals

One of the key next steps is to develop models that rely more on passive sensing rather than EMA to lower the patient burden of using these systems long term and to gain access to signals that may not be available by self-report.

Let’s take a look at two of the more revealing personal sensing methods that we are developing to provide you with some intuition about how we think this will work.

Here is a wide view of my moment-by-moment location detected by a GPS app over a month when we were first experimenting with this sensing method.  The app recorded the paths that I traveled, with movement by car in green and running in blue.

The red dots indicate places that I stopped to visit for at least a few minutes.

And although not displayed here, the app recorded the days and exact times that I was at each of these locations.

From these data, you can immediately see that I am runner, with long runs leaving from downtown Madison and frequent trail runs on the weekends in the county and state parks to the west and northwest.

Zooming in to the Madison isthmus, these data show that I drove my children halfway around the lake each morning to their elementary school.  And from these data we might be able to detect those stressful mornings when getting my young kids dressed and fed didn’t go as planned and we were late, sometimes very late, to school!

The app recorded my daily running commute through downtown Madison to and from my office.  From this, we can observe my longs days at the office and also those days that I skipped out.

Looking at the red dots indicating the places I visit, the app can detect the restaurants, bars, and coffee shops where I eat, drink and socialize. We can use public map data to identify these places and make inferences about what I do there.

…Imagine my smartphone communications…

 

In addition to geolocation, we also collected my smartphone communications logs and even the content of my text messages.

And no such luck, I don’t plan to show you my actual text messages!

But imagine what we could learn about me from the patterns of my communications - Who I was calling, when I made those calls, and even the content of what I sent and received by text message.

Context is Critical

We believe we can improve the predictive strength of these geolocation and communication signals even further by identifying the specific people and places that make us happy or sad or stressed, those that we perceive support our mental health and recovery and those who undermine it. 

Context is Critical

 

For example, consider the implications of this brief text message thread between a hypothetical patient and their drinking buddy for what you might predict for the probability that they might lapse back to drinking in the coming hours.

… And how would your prediction change if this wasn’t their drinking buddy but instead, their mom who was a big supporter of their recovery.

This interpersonal context matters!!!

Context is Critical

We can gather this contextual information quickly by asking a few key questions about the people and places we interact with frequently over the first couple of months that we record these signals. And we can identify these frequent contacts and locations directly from these signals.

In our current projects, we target people and places that we interact with at least twice a month or more for more detailed follow-up to gather context. And it turns out that this really isn’t that burdensome. Most of us are creatures of habit and if we set a threshold for 2x monthly interactions, we typically only have 10-30 people and places that meet this threshold. And it’s the same people and places each month so we can build this context up when the person first starts to use the system and after that it only needs to be updated occasionally when we go somewhere new or make a new friend.

Contextualized Geolocation

 

Contextualized Communications

 

Baseline Feature Engineering for GPS

• Focus on recent past experiences (6, 12, 24, 48, 72, 168 hours)

• Raw scores and change scores (from baseline)

• Time spent at important places (e.g, alcohol present, drank at location in past, risky, unpleasant)

 

To be clear, we have only just begun to train models using GPS. We have started by focusing on recent locations….. DESCRIBE MORE

There is clearly predictive signal in the geolocation signal but not enough to stand alone as the only features at this point. We are seeing AUCs in the low .7s when we use only geolocation to predict future lapses.

However, when we add GPS to the EMA models that I described today, we appear to need fewer unique EMA items to get the same level of model performance. So the addition of GPS may serve to lower patient burden while maintaining model performance.

[PAUSE]

The story is pretty much the same right now for cellular communications as well. At this point we have only worked with the meta data – the contact numbers, call times, durations, etc – combined with context. We haven’t yet done anything with the text message content. And models built from the meta data perform about as well at these preliminary GPS models.

But we are hoping to extract more signal from the both of these as we continue to work on feature engineering with it.

(Selective) Next Steps

• Geolocation, cellular communications, and other passively sensed signals

• Build models with lead times > 0 hours

NEED TEXT FOR THIS BUT ITS IMPORTANT. FOCUS ON lead = 0 for 1 hour and JIT – distraction, urge surfing but lead = 1 week for contact with therapist, sponsor, supportive friends and family, etc.

(Selective) Next Steps

• Geolocation, cellular communications, and other passively sensed signals

• Build models with lead times > 0 hours

• More diversity in training data

We are excited by the early performance of these machine learning models BUT I want to be clear that these are preliminary research studies. And they included mostly white participants from our local community in Madison, WI.

Models trained on these participants would be unlikely to work well with black and brown patients or patients from rural communities.

Machine learning models must be trained on diverse samples of patients or their use may exacerbate rather than reduce existing mental healthcare disparities.

Active Project: Lapse in patients with Opioid Use Disorder

• Recruiting 400 - 500 patients in recovery from Opioid Use Disorder (~ 300 so far)
• National sample (size; diversity: demographics, location)
• More variation in stage of recover (1 – 6 months at start)
• 12 months of monitoring
• Closer to real implementation methods

 

 

We are now collecting data for a NIDA funded project where we are specifically recruiting for racial, ethnic, and geographic diversity across the entire United States.

We are also recruiting for people at different stages in their recovery and following them for a longer period of time – up to 12 months

(Selective) Next Steps

• Geolocation, cellular communications, and other passively sensed signals

• Build models with lead times > 0 hours

• More diversity in training data

• Use models to improve DTx engagement and clinical outcomes

  • SMART DTx – algorithm guided use
  • How to craft patient feedback to encourage trust in the algorithm

Of course, accurately predicting future lapses is only useful if these predictions can be used to sustain engagement with treatments and improve clinical outcomes.

We have now started to consider how we can use these predictions to help patients optimize their use of a digital therapeutic.

These models may be able to suggest when more use of the DTx is needed because lapse risk is increasing or high but we are also hoping that we can use the models to recommend which tools and supports in the DTx are best for that patient at that moment in time given their lapse risk probability and important features contributing to the lapse prediction.

We are also sensitive to the fact that these recommendations from the machine learning models will need to be provided to patients in a transparent manner that also encourages them to trust the algorithm and follow its recommendations.

Relapse Prevention Model

 

Optimization/Evaluation of an Algorithm Guided Smart DTx

• Lapse probabilities updated daily based on EMA and Geolocation features
• Use lapse probability and locally important features to select optimal DTx modules – guided by Rela
• Provide recommendations designed to encourage engagement
  • Algorithm transparency (risk level, change, features)
  • Communication factors (empathy, feasibility)
• MRT to optimize recommendation message components
• RCT to evaluate Standard vs. Smart DTx on clinical outcomes

 

CRediTs

  ::: {.notes}

:::

Demographics

Alcohol Use History

Consort Diagram

ROC Posterior Probabilities

Model Comparison Posterior Probabilites