Federally funded teen pregnancy prevention programs aim to improve sexual and reproductive health outcomes for youth across the United States. To determine these programs’ effectiveness, evaluations often assess a common set of behavioral outcomes, including measures of sexual activity and unprotected sex. For a recent evaluation of the Pulse mobile health app focusing on Black and Latinx women ages 18 to 20, Child Trends incorporated measures of unprotected sex that were recommended by the program funder, the Office of Population Affairs (OPA). These measures included the following: 1) sex without using any method of contraception, 2) sex without using a condom, and 3) sex without using a hormonal or long-acting (LARC) method of contraception.[1] Child Trends researchers found inconsistent responses across these three measures of unprotected sex on the online baseline survey.

Before fielding the longer-term follow-up survey, Child Trends conducted cognitive interviews with evaluation participants to gain insight into how their understanding of question wording may have led to these inconsistent responses. We then revised the survey measures and field tested the new measures on a six-month follow-up survey to assess whether the revisions reduced inconsistencies in participants’ responses. This research brief presents findings from these cognitive interviews, along with recommendations for updating the survey questions that OPA grantees use to assess unprotected sex. Our findings provide valuable insight into the ways that survey measures can be adjusted to most accurately reflect teen and young adult behaviors.

Key Findings

The original unprotected sex questions may have led to inaccurate responses.

  • Participants misinterpreted negatively worded questions.
  • Participants had difficulty understanding long and detailed questions.
  • Participants felt compelled to offer what they perceived to be the socially acceptable answer even if it was not necessarily the accurate answer.

Implementing the following recommendations improved the accuracy of unprotected sex measures and was associated with more consistent responses in our six-month follow-up survey.

  • Phrase questions in the positive.
  • Keep questions short.
  • Provide a broader range of response options instead of binary yes or no answers.
  • Incorporate skip patterns to reduce cognitive burden for teens who are not using a hormonal or LARC method.

Inconsistency of Reporting Across Original Questions

Among Black and Latinx women ages 18 to 20 who participated in the Pulse evaluation, almost 10 percent of sexually active participants responded inconsistently to the three measures of unprotected sex on the baseline survey (listed below). The original questions were derived from the Office of Population Affairs’ survey items for teen pregnancy prevention program evaluations (all had three response categories: yes, no, or don’t know). In all cases, the inconsistent responders answered “yes” to having sexual intercourse without any method of contraception (question 1), but “no” to ever having vaginal intercourse without using a hormonal or LARC method (question 3) and/or “no” to ever having vaginal intercourse without a condom (question 2). In other words, these respondents answered that they had had sex at least once without using any method—but also that they always used a condom or a hormonal or LARC method of contraception when they’d had sex.

Original Survey Questions, Before Revision

1. In the past 3 months, have you had vaginal intercourse without you or your partner using any method of birth control? That means you had sex and didn’t use a condom or use birth control pills, the shot, the patch, the ring, IUD, or implant.

2. In the past 3 months, have you had vaginal intercourse without using a condom?

3.In the past 3 months, have you had vaginal intercourse without using any of the following methods of birth control?

Birth control pills
The shot (for example, Depo Provera)
The patch (for example, Ortho Evra)
The ring (for example, NuvaRing)
IUD (for example, Mirena, Skyla, or Paragard)
Implant (for example, Implanon or Nexplanon)

a. Yes
b. No
c. Don’t know

Methods

We conducted cognitive interviews to better understand the inconsistency in responses across measures of unprotected sex. Cognitive interviews were conducted with 13 of the 47 participants who had provided inconsistent responses. Interviews were conducted via phone and lasted 30 to 60 minutes. Respondents received a $25 Amazon gift card as a thank you for participating. We obtained IRB approval for all study activities.

Rooted in cognitive psychology, cognitive interviews are a method for identifying potential survey problems—especially in the response process—by tapping into the thought processes of respondents as they interpret and answer questions. During the semi-structured interviews, we asked participants to think aloud as they answered each question and probed for common challenges that people have when answering survey questions.

For the first seven interviews, we probed on potential issues with the original survey questions. Based on participant responses in this first round of cognitive interviews, we revised the survey questions and tested them in a second round of six cognitive interviews. We kept the original questions in our six-month follow-up survey and added the new questions to the end of survey. We ran cross-tabulations of 488 sexually active women to analyze the internal consistency of the new survey questions compared to the original questions.


Findings from Cognitive Interviews

Based on the cognitive interviews, we identified three primary challenges with the survey questions that could lead to inconsistent responses.

Negatively worded questions were easily misinterpreted.

The original survey questions asked participants if they have had vaginal intercourse without using any method of birth control, a condom, or a hormonal/LARC method. Participants found this wording confusing: It is counterintuitive to answer “yes” if one did not use a method each time they had vaginal intercourse, and “no” if they did use a method each time. For example, one participant [incorrectly] reworded Q3 (sex without hormonal/LARC method) as, “Am I taking any method of birth control?” and reported that someone would answer, “Yes … I take birth control.” Another participant noted “The ‘without’ is confusing … ‘without’ means you didn’t … but you have to read it and then re‐read it to understand.”

Lengthy or detailed questions imposed a cognitive burden on many respondents.

Participants said they had to go back and reread text-heavy questions like Q1 and Q3 multiple times. One participant said of Q1, “[the question] was medium difficulty to understand because you have to reread it … you have to keep going back to the question to understand it.” Another participant, after responding to Q1, noted the importance of keeping questions simple and to the point: “[This question] is easy if you just stop reading where the question mark is.”

Limited response options compelled participants to provide what they perceived to be the socially acceptable answer—even if it was not the most accurate.

Revised Survey Questions

1. In the past 3 months, have you used any of these methods of birth control? (yes/no responses)

a. Condom
b. Birth control pills
c. The shot (for example, Depo Provera)
d. The patch (for example, Ortho Evra)
e. The ring (for example, NuvaRing)
f. IUD (for example, Mirena, Skyla, or Paragard)
g. Implant (for example, Implanon or Nexplanon)
h. Other

If respondent answers “yes” to any of 1a-1g:

2. In the past 3 months, how often have you used any method of birth control, including condoms, during vaginal intercourse? (responses: all of the time, most of the time, some of the time, none of the time)

If respondent answers “yes” to 1a:

3. In the past 3 months, how often have you used condoms during vaginal intercourse? (responses: all of the time, most of the time, some of the time, none of the time)

If respondent answers “yes” to any of 1b-1g:

4. In the past 3 months, how often did you use at least one of the following methods when you had vaginal intercourse? (responses: all of the time, most of the time, some of the time, none of the time)

a. Birth control pills
b. The shot (for example, Depo Provera)
c. The patch (for example, Ortho Evra)
d. The ring (for example, NuvaRing)
e. IUD (for example, Mirena, Skyla, or Paragard)
f. Implant (for example, Implanon or Nexplanon)

A separate document with these questions, along with coding directions to create binary measures of unprotected sex, can be found here.


Findings from Updated Questions

Our findings from the second round of cognitive interviews suggest that the revisions improved participants’ comprehension of questions and their willingness to report instances of unprotected sex. Almost all participants understood the new questions and answered the way we would expect them to, based on their descriptions of their method use in the past three months. Participants also said that someone who used a condom or birth control method every time except once would likely answer “most of the time” or “some of the time” to the new measure of contraceptive use, instead of reporting (inaccurately) that they did not have sex without a method in the past three months on the previous measure.

Our analyses of the six-month follow-up survey data also found lower rates of inconsistency across the new survey questions. Ten percent of respondents had inconsistent responses across the original questions, while only 4 percent had inconsistent responses across the new questions. Additionally, we found higher reports of unprotected sex based on the new survey questions, as compared to the original survey questions, possibly because the option to report using a method “most of the time” or “some of the time” reduced the social desirability bias that may make young women reluctant to report having had unprotected sex.


Conclusion

Our work highlights that cognitive interviews are an effective way to assess participants’ understanding of survey questions and provide insight into reporting inconsistencies. In this study, cognitive interviews allowed program participants to directly inform the wording of questions, thus reducing their discomfort when taking the survey and increasing the accuracy of responses. Improved participation and response quality, in turn, can improve the reliability of evaluation findings.

Our research expands on existing research on the measurement of unprotected sex among teens. In line with findings from other studies, we found that young women had difficulty responding to a dichotomous measure of unprotected sex. Teens, especially those enrolled in a teen pregnancy prevention program, recognize that only one answer in a dichotomous yes/no scenario would be socially perceived as “desirable.” Teens who generally feel that their behavior aligns with what is considered socially desirable (i.e., using contraception)—including, perhaps, those teens who have only had sex one time without contraception but are otherwise consistent users of contraception—are likely to choose the socially desirable response because it fits with their self-perception. Providing more response options allows participants to identify a response that aligns with their behavior (e.g., using contraceptive most of the time or some of the time) and yields more accurate responses. The new questions allow evaluators to assess binary measures of unprotected sex, but also to examine nuances in consistency of method use. This aligns with calls from other researchers to move away from dichotomous measures of perfect contraceptive use. For example, a program can still be considered successful if participants increase the consistency of their contraceptive use, even if they do not achieve 100 percent contraceptive consistency.

The use of a broader range of response options in questions about contraceptive consistency may also help program implementers identify interim changes in behavior and help them tailor their message to the specific progress of program participants. For example, if an implementer notices at an interim survey that many participants have gone from using contraception none of the time to using it most of the time, they can reinforce the importance of contraceptive consistency. If they see that their participants are reporting using contraception most of the time or all of the time (on either a baseline or interim survey), they may choose to dispel common myths that lead teens to discontinue contraceptive use.


Acknowledgements

We would like to thank Mónica Caudillo from the University of Maryland and Elizabeth Cook from Child Trends for their thoughtful review and feedback on an initial draft of this research, and Kate Welti from Child Trends for her careful review of the research brief. We also thank Genevieve Martínez-García and Milagros Garrido of Healthy Teen Network for the development of Pulse.

Funding

This publication was made possible by Grant Number TP2AH000038 from the Office of Population Affairs (OPA), U.S. Department of Health and Human Services (HHS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the OPA or HHS.

Footnote

[1] Evaluators of teen pregnancy prevention programs often measure unprotected sex, rather than the percentage of teens using contraception, in order to include teens who are not sexually active in the analysis (these respondents can be coded as 0 rather than dropped from the analysis).