Evaluating Pulse: Lessons from an online evaluation of an app-based approach to teen pregnancy prevention

Publication Date:

Jun 27, 2018

Pulse: An app-based teen pregnancy prevention intervention

From 2016 to 2018, Child Trends collaborated with Healthy Teen Network, Ewald & Wasserman, and Meta Media Training to evaluate Pulse, an innovative web-based mobile app designed to empower young women to take control of their health by providing on-demand access to comprehensive, medically accurate sexual health information. The Pulse evaluation is funded by the U.S. Department of Health and Human Services’ Office of Adolescent Health under a grant to rigorously evaluate “new or innovative approaches to prevent teen pregnancy.” Child Trends completed impact analyses and will publicly release the impact results and an extensive report on the evaluation’s implementation in the future.

This resource provides a brief introduction to the Pulse intervention and describes Child Trends’ unique approach to evaluating Pulse’s impact on teen pregnancy. For this intervention, Child Trends recruited a large, racially diverse sample of young women completely online. Most of these women were retained for the duration of the intervention and post-test survey administration using the app and pre-programmed text messages, and the study team did not ever have personal contact with them.

The need for technology-based interventions

Rates of unplanned pregnancy are highest among women in their older teens and early twenties,1 yet the majority of teen pregnancy prevention programs are designed for younger teens. Only 8 percent of the nearly 500,000 teens served by the Office of Adolescent Health’s national Teen Pregnancy Prevention Program from 2010 to 2014 were age 16 or older.2 And despite recent declines in teen birth rates among racial and ethnic minority populations in the United States, black and Latina teens continue to experience birth rates more than twice as high as their white peers,1 highlighting the need for program approaches that target racial and ethnic minority populations. To address these gaps, teen pregnancy prevention programs have increasingly incorporated technology-based components, but more conclusive research is needed on the efficacy of these approaches.

The Pulse program

Pulse is a web-based mobile health app designed for black and Latina women ages 18–20. Through culturally and age-appropriate content, Pulse provides information on birth control, healthy relationships, sexual health and physiology, pregnancy, and utilization of clinical services to encourage users to choose effective birth control, seek reproductive health services, and ultimately, prevent unplanned pregnancies.

App content

The Pulse intervention app has six interactive sections with information on sexual and reproductive health topics. It contains activities that engage young women with the content, such as videos featuring medical professionals, a clinic locator, and appointment reminders.

Intervention and Comparison App Main Pages and Intervention Content Page

Know Your Body promotes sexual health by explaining basic anatomy and physiology, and telling users how to prevent sexually transmitted infections.

Know Your Options supports sexually active women in assessing and selecting the best birth control method for them. It promotes birth control literacy, and includes information about all birth control methods with an emphasis on long-acting reversible contraception.

Get Personal supports women who are considering initiating sexual activity, helps them identify abusive behaviors in their relationships, and demonstrates negotiation of condom use with a partner.

Take Action empowers women to visit a health clinic by guiding them to a local clinic through a clinic finder, and orienting them on expectations in a clinic visit.

Make A Plan orients women who think they may be pregnant, or are pregnant, and who are currently navigating prenatal care services.

Get Savvy answers women’s additional questions in text and video format and directs them to additional web-based resources and hotlines.

Evaluation of Pulse

Child Trends used a randomized controlled design to evaluate the impact of Pulse on two measures of unprotected sex (sex without any birth control method, and sex without a hormonal/LARC method), as well as the program’s impacts on knowledge, attitudes, self-efficacy, and intentions. The intervention lasted six weeks and consisted of unlimited access to the app and receipt of text messages. The project team administered an online survey at the end of the six-week intervention, and 86 percent of the sample completed it.

Evaluation key features:

  • The evaluation was fully technology-based: All recruitment, enrollment, intervention content, and pre- and post-survey participation took place online. Most participants completed all sections on smartphones.
  • Participants were recruited through ads on Instagram, Facebook, Twitter, and Google AdWords.

Example of Pulse ads

  • Participants completed a screener; if they met eligibility criteria, they were directed to an online enrollment form, consent form, and baseline survey. Upon completion of the baseline survey, participants were randomized to either the intervention or comparison group.
  • Child Trends followed a protocol for identifying and removing scammer and duplicate accounts from the sample to ensure that only eligible individuals were in the study.
  • Pulse comparison and intervention apps looked and felt similar aesthetically.
  • The project team sent weekly text messages to participants. These messages contained sexual and reproductive health information (for intervention participants) or general health information (for comparison participants) with fun and engaging animations.

Example Text Messages

  • The study team sent reminder text messages to participants to encourage them to view the app and/or take the post-test survey (if they had not yet done so).
  • The team sent checks or gift card incentives to participants after they completed baseline and follow-up surveys.

Eligibility requirements

At enrollment, participants had to meet the following criteria:

  • Self-identify as female
  • Ages 18–20
  • Not currently pregnant or trying to become pregnant
  • Has daily access to a smartphone
  • Currently living in the United States or a U.S. territory
  • Speaks English


The screener was completed 5,722 times, 2,190 of which did not meet the study eligibility requirements. Of the 3,532 accounts that were eligible, 1,876 consented and enrolled in the study, and 1,588 were randomized and completed the baseline survey (797 to intervention and 791 to comparison). One hundred thirty-six (136) accounts assigned to the intervention group were identified as scammer or duplicate accounts and were removed from the sample, leaving 661 valid participants in the intervention group. One hundred forty-eight (148) accounts assigned to the comparison group were identified as scammer or duplicate accounts and were also removed, leaving 643 valid participants in the comparison group. Eight-five percent of the intervention group (n = 565) and 87 percent of the comparison group (n = 559) completed the follow-up survey.


Participants were recruited through ads on Facebook, Instagram, Twitter, and Google AdWords. Once participants clicked on an ad, they were taken to a screener. Participants completed the screener and, if they met eligibility criteria, were directed to an online enrollment form, consent form, and baseline survey. Upon completion of the baseline survey, participants were randomized to either the intervention or comparison group.

Enrollment and demographics

The evaluation team enrolled 1,304 participants online, most of whom were recruited through Instagram ads. The participants’ characteristics at the time of study enrolment were similar for the intervention and comparison groups.

Characteristics of the intervention sample:[1]

Participant app usage

Although participants were asked at enrollment to view the app, not all participants did. Since Pulse was an online evaluation, we relied on participants to view the app on their own time. We analyzed app usage data to see whether participants viewed the app, and how often they did so.

Design and methods

  • Participants were randomized to either the intervention or comparison group using a permuted block design.
  • The evaluation team used ordinary least squares regressions to assess the impact of Pulse. All analyses controlled for the baseline measure of each outcome, sexual experience at baseline, age, and race and ethnicity.
  • We adjusted standard errors to account for the permuted block randomization and multiple hypotheses testing within domains.

Evaluation successes

  • The evaluation team recruited a large, racially diverse sample completely online in less than one year. Of all 1,300 participants, 76 percent were black and/or Latina.
  • Most participants stayed in the study despite never having personal contact with study staff. Eighty-six (86) percent of the sample completed the follow-up survey.
  • The intervention and comparison groups were equivalent on important characteristics at the beginning of the study. Both groups were comparable on age, race and Hispanic ethnicity, employment, educational achievement, parental educational achievement, relationship status, number of children, recent exposure to reproductive health information, current participation in another teen pregnancy prevention program, and all behavioral, knowledge, attitudes, and intentions outcomes.
  • At the end of the study, more intervention group participants than comparison group participants remembered receiving information about reproductive health from any source. Six weeks after enrolling in the study, more intervention than comparison group participants said they had received information about abstinence from sex (57 vs. 51 percent), how to say no to sex (70 vs. 64 percent), how to talk to a partner about using birth control (66 vs. 56 percent), and methods of birth control (80 vs. 72 percent).

Lessons learned

Throughout the duration of the Pulse study, Child Trends identified several challenges and successes:

  • Streamlining the enrollment process improved recruitment numbers. The evaluation team changed the process so that users were automatically directed from social media ads to the screener, instead of to the study landing page. In addition, we changed the format of the screener questions so they were all located on a single page instead of a “question-per-screen” format. Since most participants took the screener on their phone, this resulted in a large increase in screener completions—from 9 percent to over 90 percent.
  • Targeted recruitment reduced recruitment costs. By using targeted recruitment—for example, by focusing our ad dollars on cities with high concentrations of blacks and Latinas—we reduced the cost of recruiting a single participant from over $100 at the beginning of the full study to $3 for the last three months of recruitment.
  • Targeted recruitment enabled specific sub-groups to be recruited. Enrolling such a large sample of racial and ethnic minority participants in such a narrow age group was a major study achievement. We carefully refined the content of the social media ads and the specified audience to target sub-groups, which resulted in a higher influx of participants within the preferred sample populations (18- and 19-year-old black and/or Latina women).
  • Identifying and removing ineligible participants is critical for online enrollment. We developed a streamlined procedure to identify and remove scammers (participants who did not meet our eligibility criteria in their first screener attempt, but who changed their responses until they passed the screener) and duplicate users (participants who enrolled more than once) from our study sample. In total, we identified and removed 284 scammer or duplicate users, which accounted for 18 percent of 1,588 screener attempts.

Contributors: Genevieve Martínez-García, Milagros Garrido, Nicholas Sufrinko, and Emily Miller.

This publication was made possible by Grant Number TP2AH000038 from the Office of Adolescent Health (OAH), U.S. Department of Health and Human Services (HHS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the OAH or HHS.

[1] “Hormonal/long-acting reversible contraceptive (LARC) methods” include the following methods: birth control pills, the shot (for example, Depo Provera), the patch (for example, Ortho Evra), the ring (for example, NuvaRing), IUD (for example, Mirena, Skyla, or Paragard), and implants (for example, Implanon or Nexplanon).


  1. Martin JA, Hamilton BE, Osterman MJ, Driscoll AK, Drake P. Births: final data for 2016. Hyattsville, MD: National Center for Health Statistics;2018.
  2. Results from the OAH Teen Pregnancy Prevention Program. Washington, DC: HHS Office of Adolescent Health;2015.