Evaluating Young Women’s Engagement with an Online Reproductive Health App
Older teens and young adults have the highest rates of unplanned pregnancy in the United States, yet the majority of pregnancy prevention programs are designed for and implemented with younger teens.,, Technology-based interventions offer an innovative approach to providing pregnancy prevention programming for hard-to-reach populations, including older teens who are no longer in school (either graduated or dropped out), and for racial and ethnic minority populations, such as Latinx and black women, that experience birth rates twice as high as rates for white teens. As the use of mobile devices has become nearly universal, new pregnancy prevention programs have been designed with technology-based service delivery methods, such as mobile apps and text messaging, to better engage hard-to-reach populations.,,,,,
Child Trends is currently conducting a randomized controlled trial to evaluate the impact of one such technology-based pregnancy prevention program, Pulse. Forthcoming results of this evaluation will focus on the potential impact Pulse has on participants’ behaviors, attitudes, knowledge, and intentions related to reproductive health. In this research brief, we highlight the characteristics of women who participated in the Pulse program, their differing levels of engagement with the mobile app, and the characteristics of the participants who engaged with the app versus those who had little or no engagement. Results provide important lessons about the types of young women who can be effectively reached through a solely tech-based pregnancy prevention program.
- Pulse participants were self-identified females ages 18 to 20 who were not pregnant or seeking pregnancy. The majority were black (42 percent) or Latinx (44 percent). Compared to a nationally representative sample with similar demographics, at baseline, Pulse participants had:
- Higher levels of educational attainment
- Higher levels of current school enrollment
- Lower levels of reported pregnancy
- As a part of the evaluation design, participants were considered engaged users if they visited all six sections of the app and visited an activity within each section. Only 25 percent of Pulse participants were engaged users. Eight percent of participants were non-users and 67 percent were limited users.
- Engaged app users had higher response rates to follow-up evaluation surveys and were less likely to opt out of receiving program-related text messages.
- Participant baseline characteristics were associated with app engagement:
- Participants with higher levels of education, more knowledge about sexual and reproductive health, and positive attitudes about birth control were more likely to be engaged users.
- Participants working full-time and those with a child were less likely to be engaged users.
- Participants who reported receiving most of their sexual and reproductive health information online were more likely to be engaged users.
The study was conducted between November 2016 and April 2019 with continuous enrollment. Participants began their intervention window after being randomized. Each participant received the intervention for six weeks. During the six-week period, participants had unlimited access to Pulse and received text messages with app-related content. The program and evaluation were fully technology-based, meaning all recruitment, enrollment, intervention content, and pre- and post-survey participation took place online, primarily on smartphones. Program implementers and evaluators had no in-person contact with participants. To encourage participants to use the app, the study team developed targeted text message reminders that were sent to participants who never registered with the app, never logged in, or who did not meet the usage requirement for adequate dosage.
Not surprisingly, engaged users visited the app more frequently than limited users, an average of 4.4 times over the course of the six weeks, versus 1.7 times for limited users. Engaged users also had a significantly higher average number of section visits within the app (six sections for engaged users versus three for limited users). Additionally, engaged users had an average of 56 app clicks (number of clicks across sections and activities) versus limited users who had an average of 16 app clicks.
Engaged users were more involved in the study overall. They had higher response rates to follow-up evaluation surveys: 97 percent of engaged users completed the follow-up survey compared to 84 percent of limited users and 62 percent of non-users. As part of enrollment in the study, participants were automatically opted in to receive study-related text messages, which included program content as well as reminders to visit the app; however, they could opt out of receiving these text messages at any time. Engaged users were less likely to opt out of receiving these text messages than limited users (7 percent versus 13 percent). Engaged users were also less likely to experience text message bounce back (i.e., message delivery failure): 16 percent of engaged users versus 21 percent of limited users and 34 percent of non-users.
Participant characteristics linked to usage
Education, employment, and parenthood were associated with participants’ levels of engagement with the app. Figure 2 shows the percentage of participants with certain characteristics who were engaged users versus limited users and non-users. The findings (and statistical testing) focus on engaged users versus non-users and limited users combined. Patterns suggest similar findings for non-users and limited users (data not shown).
Participants with at least some college or technical education, participants who were not currently working, and participants without a child were more likely to be engaged users. For example, participants who did not have a child were more than twice as likely as those with a child to be engaged users (26 percent versus 11 percent; Figure 2).
Participants’ baseline knowledge of reproductive health and their attitudes about birth control were also associated with levels of engagement with the app. Figure 3 shows the level of participants’ knowledge across four true/false sexual and reproductive health questions (low knowledge = 0 correct answers, moderate knowledge = 1 or 2 correct answers, high knowledge = 3 or 4 correct answers) and participants’ attitudes about birth control (those with positive attitudes disagreed with the statement “Birth control is too much of a hassle to use”).
Participants with high levels of sexual and reproductive health knowledge were the most likely to be engaged users (30 percent compared with 26 percent for moderate knowledge and 17 percent for low knowledge). In addition, participants with low knowledge were more than twice as likely to be non-users than those with high knowledge (data not shown). Participants with more positive attitudes about birth control (who disagreed that birth control is a hassle to use) were more likely to be engaged users than those who had more negative views of birth control.
On the baseline survey, participants were asked, “Where do you get most of your information about health?” Those who reported receiving sexual and reproductive health information primarily from online sources (versus family, friends, and school; or a health care provider) were more likely to be engaged users (29 percent of those who received most information online were engaged users, compared to 19 to 21 percent of those who reported other sources of information; see Figure 4).
The Pulse program was designed to help users prevent unplanned pregnancies by choosing effective birth control and seeking reproductive health services. The program was designed for young black and Latinx women ages 18 to 20 who were not currently pregnant or seeking pregnancy. The study team successfully reached this target population, although we found that Pulse participants were not representative of black and Latinx women of similar age on a few key demographic characteristics. Compared to black and Latinx women in a nationally representative survey conducted at a similar point in time, Pulse participants were more highly educated, more likely to be in school, and less likely to have experienced a pregnancy. The fact that Pulse was most successful in connecting with more educated and less at-risk young adults could be due to the process of recruiting online through social media and the requirement that Pulse participants have daily access to a smartphone. Studies of social media users have found that a higher percentage of college-educated adults use Facebook and Instagram (the sites used for recruitment) than adults with a high school degree or less. Other studies have also found that recruitment through Facebook yields a disproportionately large number of youth with higher education., More specific to Pulse, the demographics of participants could also reflect the appeal of a sexual health app to young women who are more educated.
Among Pulse participants, only one-quarter were considered engaged users, visiting all six sections of the app at least once and completing one activity in each section during the six-week program period. This metric was chosen to ensure that participants received a variety of program content. Therefore, most participants did not receive a wide range of program content. These findings highlight both the strengths and limitations of online programming, which allows implementers to reach many racially, ethnically, and geographically diverse young adults—who do not generally receive programming—with less effort and expense than traditional school- or community-based teen pregnancy prevention approaches require., However, many youth may not receive the online program content as desired. While the populations that other approaches reach may be younger and less diverse, those youth receive a larger amount of program content.
Additionally, while overall response rates were high for an online study (86%), the follow-up survey rates were highest among engaged users, compared to rates for limited users and non-users. This has implications for conducting high-quality program evaluations, since high response rates across participants are required to detect statistically significant differences between treatment and control groups and to ensure that findings are not biased.
Overall, the young black and Latinx women who likely received the most program content through Pulse (through original enrollment and then engagement with the app) were women who may be less at risk of unintended pregnancy than other young black and Latinx women because of their higher levels of education, more likely enrollment in school, greater sexual and reproductive health knowledge, and more positive attitudes about birth control. Additionally, those who already sought reproductive health information online were more likely to engage in the app. Further, those who were working full-time or who had a child were less likely to have high levels of engagement in the Pulse app, highlighting their lower time availability for app-based content. Findings suggest that while online pregnancy prevention programs can connect with diverse young adult populations that would be hard to reach through school- or community-based programming, programs that are conducted entirely online through social media recruitment may not target at-risk populations as successfully. Future studies should examine ways to target populations to focus on racially and ethnically diverse young adults who have lower educational attainment and lower socioeconomic status.
This publication was made possible by Grant Number TP2AH000038 from the Office of
Population Affairs (OPA), U.S. Department of Health and Human Services (HHS). Its contents
are solely the responsibility of the authors and do not necessarily represent the official views of
the OPA or HHS.
 Finer, L. B., & Zolna, M. R. (2016). Declines in unintended pregnancy in the United States, 2008–2011. New England Journal of Medicine, 374(9), 843-852. doi:doi:10.1056/NEJMsa1506575
 Juras, R., Tanner-Smith, E., Kelsey, M., Lipsey, M., & Layzer, J. (2019). Adolescent Pregnancy Prevention: Meta-Analysis of Federally Funded Program Evaluations. Am J Public Health, 109(4), e1-e8. doi:10.2105/ajph.2018.304925
 Lugo-Gil, J., Lee, A., Vohra, D., Harding, J., Ochoa, L., & Goesling, B. (2018). Updated findings from the HHS Teen Pregnancy Prevention Evidence Review: August 2015 through October 2016. Retrieved from Washington, DC: https://tppevidencereview.aspe.hhs.gov/pdfs/Summary_of_findings_2016-2017.pdf
 Martin, J. A., Hamilton, B. E., Osterman, M. J. K., Driscoll, A. K., & Drake, P. (2018). Births: Final Data for 2017. Natl Vital Stat Rep, 67(8), 1-50.
 Antonishak, J., Kaye, K., & Swiader, L. (2015). Impact of an Online Birth Control Support Network on Unintended Pregnancy. Social Marketing Quarterly, 21(1), 23-36. doi:10.1177/1524500414566698
 Bull, S., Devine, S., Schmiege, S. J., Pickard, L., Campbell, J., & Shlay, J. C. (2016). Text Messaging, Teen Outreach Program, and Sexual Health Behavior: A Cluster Randomized Trial. American Journal of Public Health, 106(S1), S117-S124. doi:10.2105/AJPH.2016.303363
 Guse, K., Levine, D., Martins, S., Lira, A., Gaarde, J., Westmorland, W., & Gilliam, M. (2012). Interventions Using New Digital Media to Improve Adolescent Sexual Health: A Systematic Review. Journal of Adolescent Health, 51(6), 535-543. doi:10.1016/j.jadohealth.2012.03.014
 Kissinger, P., Schmidt, N., Green, J., Latimer, J., Madkour, A., Clum, G., & Johnson, C. (2015). Evaluation of BUtiful: An Internet pregnancy prevention for older teenage girls in New Orleans, Louisiana. Retrieved from New Orleans, LA: https://www.hhs.gov/ash/oah/sites/default/files/ash/oah/oah-initiatives/evaluation/grantee-led-evaluation/reports/tulane-final-report.pdf
 Steinberg, A., Griffin-Tomas, M., Abu-Odeh, D., & Whitten, A. (2018). Evaluation of a Mobile Phone App for Providing Adolescents With Sexual and Reproductive Health Information, New York City, 2013-2016. Public Health Rep, 133(3), 234-239. doi:10.1177/0033354918769289
 Widman, L., Nesi, J., Kamke, K., Choukas-Bradley, S., & Stewart, J. L. (2018). Technology-Based Interventions to Reduce Sexually Transmitted Infections and Unintended Pregnancy Among Youth. Journal of Adolescent Health, 62(6), 651-660. doi:10.1016/j.jadohealth.2018.02.007
 Pew Research Center. (2018). Social Media Fact Sheet. Retrieved from https://www.pewinternet.org/fact-sheet/social-media/
 Fenner, Y., Garland, S. M., Moore, E. E., Jayasinghe, Y., Fletcher, A., Tabrizi, S. N., . . . Wark, J. D. (2012). Web-Based Recruiting for Health Research Using a Social Networking Site: An Exploratory Study. J Med Internet Res, 14(1), e20. doi:10.2196/jmir.1978
 Loxton, D., Powers, J., Anderson, A. E., Townsend, N., Harris, M. L., Tuckerman, R., . . . Byles, J. (2015). Online and Offline Recruitment of Young Women for a Longitudinal Health Survey: Findings From the Australian Longitudinal Study on Women’s Health 1989-95 Cohort. J Med Internet Res, 17(5), e109. doi:10.2196/jmir.4261
 Thornton, L., Batterham, P. J., Fassnacht, D. B., Kay-Lambkin, F., Calear, A. L., & Hunt, S. (2016). Recruiting for health, medical or psychosocial research using Facebook: Systematic review. Internet Interventions, 4, 72-81. doi:https://doi.org/10.1016/j.invent.2016.02.001