Left on Read: Strategies to Improve SafeSpace Participant Engagement and Retention

The holiday season is upon us and so is Year 4 of the SafeSpace rigorous evaluation! Our team is very proud of the work we have accomplished thus far and we are excited to share this update. Get cozy as you read about our progress and what we have learned about the SafeSpace app users. 

MyHealthEd, Inc. has completed the app usage data analysis for the first two cohorts of the SafeSpace study. Cohort 1 consists of 331 youth who completed the program from June - December 2023, and Cohort 2 includes 176 youth who completed the program from January - June 2024. We previously shared blog posts on how we developed an enrollment procedure and the valuable feedback we received from interviewing users. This blog post will describe the processes utilized to monitor user engagement and retain participants throughout the program and evaluation period. We will also share the results of these efforts and lessons learned.

SafeSpace is an innovative mobile app for youth that delivers 10 weeks’ worth of information on adolescent development, sexual and reproductive health, and overall well-being. Each lesson contains a youth-generated story, medically accurate facts, reputable resources, and a knowledge application activity. Delivering the SafeSpace material via a mobile app allows users the flexibility to learn at their own pace and convenience. This format also removes geographic barriers, extending our reach to a more diverse audience. Currently, SafeSpace participants come from more than 13 states throughout the U.S. and range from ages 14-18, with more than half identifying as LGBTQ+, Hispanic, and/or non-Hispanic Black. 

As with any intervention, high participant engagement and retention are key to success.  While we knew this would be challenging since we do not meet with participants in person, we saw this as an opportunity to be creative in how we keep participants interested in the program. Our team implemented two strategies to encourage engagement:

1. Providing incentives,

2. Strategizing outreach based on usage data analysis. 

Strategy #1: Incentives

The first method we use to increase retention is to provide monetary incentives. Participants have four opportunities to earn up to $100 in gift card rewards. All eligible participants earn the first gift card after successfully enrolling in the study. To earn the second gift card, they must complete at least two lessons within the first five weeks. This midpoint incentive rewards engagement during the first half of the program and encourages use for the remainder of the study. If a participant does not initially meet the requirements for the second gift card, they are able to earn the reward before the program ends by “catching up” and reviewing the lessons that they missed. The final two incentives are distributed when participants complete the short-term and long-term surveys at 10 weeks and nine months, respectively. 

We found these monetary rewards to be effective, with over 70% of participants receiving their midpoint incentive.

Strategy #2: Outreach

Once participants enroll in the study, their mobile device is registered in our monitoring system which tracks participant app usage. When the monitoring system detects that a user has not opened the app in five days, their status is changed to “inactive,” and a push notification is automatically delivered. This notification reminds the user of missed content and encourages them to return to the program. 

After monitoring the program for a few weeks, we noticed a sharp decrease in active users soon after enrollment. The sudden lack of participation urged our team to investigate other ways to encourage engagement. We decided to implement a personalized outreach strategy midway through Cohort 1, texting inactive users at two time points. Our objectives for this strategy included

1. Retaining engagement,

2. Increasing midpoint incentive distribution, and 

3. Streamlining communication efforts. 

Since the greatest decline in participation occurs after the first week, we began reaching out to inactive users three weeks post-enrollment. Texting them during Week 3 serves two purposes; it enables us to communicate with participants early in the program to address initial drop off, and it acts as a reminder for the midpoint incentive. We send a second text message out during Week 8 to encourage engagement in the latter half of the program and remind participants of the next reward. Choosing these two time points maintains an evenly distributed communication schedule that does not overwhelm participants with too many notifications. 

Our team created a spreadsheet monitoring participant status to track those who need outreach. The spreadsheet consists of each participant’s study ID number and their usage status at Week 3 and Week 8. Messages are sent via YakChat, and once participants are texted, our team checks their statuses every week to see if they have re-engaged. We also ensure to reply to messages from participants and address any questions or concerns they may have. These brief interactions are beneficial for re-engaging participants and help reveal any app issues that may have discouraged participation. 

To assess the effectiveness of our outreach strategy, we analyzed attendance, defined as engaging with at least one of the app’s three weekly lessons. First, we compared attendance between cohorts, since only a portion of Cohort 1 received these low-engagement texts as opposed to all of Cohort 2.

Cohort 2 had higher attendance rates for Week 3 and Week 8, suggesting that our reminders prompted re-engagement. 

We also calculated re-engagement numbers, looking at how many participants returned to the app after we texted them. Combining the two cohorts, we found that 25% of users returned after the Week 3 text and 23% after the Week 8 message.  

While these numbers may be small, we consider these results a promising indication of our outreach strategy’s effectiveness. Communicating with participants taught us that some lost access to SafeSpace due to accidentally deleting the app or acquiring new phones. Our team has been able to quickly address these issues and get users back into the app. It also brings us joy when participants thank us for the helpful reminder messages! We will continue exploring new ways to enhance engagement and encourage participants to use the SafeSpace app throughout the 10-week program. 

Lessons Learned

When our team identified a decline in participant engagement within the first few weeks of enrollment, we knew we had to act fast. We quickly implemented a new outreach strategy to keep users engaged throughout the second half of the program. After making these changes, we learned a few new things about engagement and retention:

Message Frequency

There are at least 10 text message reminders from our team (e.g.  incentives, inactivity, and surveys), in addition to push notifications that come through the app. This may cause communication overload and lead some participants to ignore the messages, disable push notifications, and/or block our texts. When this happens, participants do not receive important messages regarding the study, which impacts engagement. We are learning to strike a balance between reminding participants to stay engaged while not over-messaging them by spacing out our messaging schedule, as we did with the Week 3 and Week 8 texts.

Personalized Text Messages

Given our reliance on push notifications and text messages as the primary form of communication, youth may not feel as inclined to engage due to a lack of personal connection. We learned that participants sometimes think our messages come from bots. However, once they realize that we are real people with whom they can communicate, they feel more supported while using the app. Using youth-friendly language in text messages is crucial to successfully communicating with participants. We also aim to use more engaging language so they know our team is here and available to answer their questions or concerns. We are currently updating our messages to be more personalized and sound less automated.   

Program Length

Ten weeks is a long time for young people to keep up with a program on their own. We understand they are navigating school, work, and other life responsibilities. As such, a decline in engagement is expected. Not only may they become disinterested, but other priorities may also take precedence, inhibiting their app usage. The task is then to condense the content into a shorter time frame, which may be more beneficial to keeping youth engaged. 

We’re excited to contribute these lessons learned to the growing research on mobile health interventions and use them to inform future iterations of the SafeSpace app. 

Stay Tuned

SafeSpace is an innovative app that has allowed us to educate over 600 youth nationwide. We are working tirelessly to continue outreach to increase enrollments while maintaining participant engagement and retention. Stay tuned for more updates from our team as we navigate improving our program structure and addressing challenges that arise while implementing a completely digital sexual and reproductive health education intervention.

-Brittany, Bajha, & Marisa