Recommendations for Developing Survey Items and Outcome Measures to Evaluate Teen Pregnancy Prevention Programs

Publication Date:

April 08, 2020

This brief gives evaluators of teen pregnancy prevention programs survey items and outcome measures to consider when assessing the impact of their programs.

The outcomes included are drawn from Child Trends’ evaluation of Re:MIX, a comprehensive sexual health education program that aims to reduce rates of unintended pregnancy and sexually transmitted infections (STIs) among primarily Latinx adolescents (grades 8-10) in Texas. Child Trends included measures from four mediating outcome domains (intentions, knowledge, self-efficacy, and attitudes about sexual activity and contraceptive use) that are linked to reproductive health behaviors.



The sections below outline the outcomes, related survey questions (full survey questionnaire available here), outcome response format, and survey question source within each outcome domain. In sections located below the evaluation outcomes, we give more detail about the development of the survey items and the development of the evaluation outcome measures.


Intentions

1

Outcome: Intend to have sex in the next year

Survey Question: Do you intend to have sexual intercourse in the next year, if you have the chance? (survey question 45)

Outcome Response Format: Binary: “yes, definitely” and “yes, probably” versus “no, probably not” and “no, definitely not”

Survey Question Source: OPA

Outcome: Intend to use hormonal or long-acting birth control if have sex

Survey Question: If you were to have sexual intercourse in the next year, do you intend to use (or have your partner use) any of these methods of birth control?

• Birth control pills
• The shot (for example, Depo Provera)
• The patch (for example, Ortho Evra)
• The ring (for example, NuvaRing)
• IUD (for example, Mirena, Skyla, or Paragard)
• Implant (for example, Implanon and Nexplanon)

(survey question 47)

Outcome Response Format: Binary: answered “yes, definitely” vs. “yes, probably,” “no probably not,” and “no, definitely not”

Survey Question Source: OPA

Knowledge

1

Outcome: Knowledge about condoms

Survey Question: For the following set of statements, please indicate whether you think the statement is true or false.

a. It is okay to use the same condom more than once.
b. Condoms have an expiration date.
c. When putting on a condom, it is important to leave a space at the tip.
d. When using a condom, it is important for the man to pull out right after ejaculation.
e. Wearing two latex condoms will provide extra protection.

(survey question 33a-e)

Outcome Response Format: Percent correct out of 5 questions

Survey Question Source: Fog Zone

Outcome: Knowledge about STI prevention

Survey Question: For the following set of statements, please indicate whether you think the statement is true or false.

a. If condoms are used correctly and consistently, they can decrease the risk of getting sexually transmitted infections (STIs).
b. If birth control pills are used correctly and consistently, they can decrease the risk of getting sexually transmitted infections (STIs).

(survey question 34a-b)

Outcome Response Format: Percent correct out of 2 questions

Survey Question Source: Adapted from PPA – Teen Prevention Education Program (PEP)

Outcome: Knowledge about birth control efficacy

Survey Question: For the following set of statements, please indicate whether you think the statement is true or false.

a. The condom is more effective at preventing pregnancy than the birth control pill.
b. The intrauterine device (IUD) is more effective at preventing pregnancy than the condom.

(survey question 34c-d)

Outcome Response Format: Percent correct out of 2 questions

Survey Question Sources: a. PPA – Gender Matters

b. PPA – Teen Prevention Education Program (PEP)

Attitudes


Attitudes about gender roles and romantic relationships

1

Outcome: Progressive attitudes about gender roles

Survey Question: 1. Please indicate if you agree or disagree with the following statements about girls.

a. A girl wearing revealing clothing deserves to have negative comments made about her.
b. Teenage girls who have a boyfriend feel better about themselves than girls who don’t have a boyfriend.
c. I think it is more important for a girl to look good than it is for a boy.

(survey question 22d-f)

2. These next questions are about what sex means to boys and girls your age and what should happen if a boy and girl have sex. Please indicate if you agree or disagree with the following statements.

a. A girl can suggest using condoms, just like a boy can.
b. If a girl and boy have sex, the girl should be more responsible for preventing pregnancy than the boy
c. A girl needs to have sex with a boy to keep him from finding someone else.
d. I think a boy should know more about sex than a girl.
e. I think cheating is expected more from boys than from girls.
f. It is okay for a boy to expect a girl to have sex with him if they have been dating for a long time

(survey question 24a-f)

3. Please indicate if you agree or disagree with the following statements about boys.

a. In a healthy dating relationship, the boy gets his way most of the time.
b. I think it is okay for boys to say and do whatever it takes to have sex.
c. I think it is important for a boy to act like he has had sex, even if it is not true.

(survey question 32d-f)

Outcome Response Format: Mean of 12 items (range 1-4)

Alpha = .81a

Survey Question Sources: 1a. Adapted from Gender-Equitable Men (GEM) Scale; Adolescent Masculinity Ideology in Relationships Scale (AMIRS)

1b. Adapted from PPA – Gender Matters

1c. Adapted from Scale for the Assessment of Sexual Standards Among Youth (SASSY)

2a. Adapted from Gender-Equitable Men (GEM) Scale

2b. Developed by the study team

2c. PPA – Gender Matters

2d. Adapted from Scale for the Assessment of Sexual Standards Among Youth (SASSY)

2e. Adapted from Scale for the Assessment of Sexual Standards Among Youth (SASSY)

2f. Adapted from PPA – Gender Matters

3a. Adapted from Gender-Equitable Men (GEM) Scale

3b. Adapted from Gender-Equitable Men (GEM) Scale

3c. Adapted from Scale for the Assessment of Sexual Standards Among Youth (SASSY)

Outcome: Progressive attitudes about gender and sexual identities

Survey Questions:

1. Please indicate if you agree or disagree with the following statements about girls
a. I would be friends with a girl who is gay.
b. It bothers me when a girl acts like a boy.

(survey question 22a-b)

2. Please indicate if you agree or disagree with the following statements about boys.
a. I would be friends with a boy who is gay.
b. It bothers me when a boy acts like a girl.

(survey question 32a-b)

Outcome Response Format: Mean of 4 items (range 1-4)

Alpha = .80 a

b. It bothers me when a boy acts like a girl.

(survey questions 22a-b and 32a-b)

Survey Question Sources: 1a and 2a. Adapted from Adolescent Masculinity Ideology in Relationships Scale (AMIRS)

1b.Adapted from Gender-Equitable Men (GEM) Scale

2b. Gender-Equitable Men (GEM) Scale

Outcome: Equitable attitudes about sexual responsibility

Survey Questions: 1. In a teen dating relationship, who should be responsible for the following:

a. Deciding whether to have sex?
b. Using birth control, like the pill or condom?
c. Preventing pregnancy?

(survey question 25a-c)

2. In a teen dating relationship, who should be responsible for the following:

a. Deciding on a birth control method

(survey question 28b)

Outcome Response Format: Binary: “you and your partner” for all questions versus “all you,” “mostly you,” “mostly your partner,” “all your partner”

Survey Question Source: 

Adapted from The Healthy Marriage and Relationship Education Models and Measures project, National Survey of Adolescent Males (NSAM) 1995, and Giordano 2006

Attitudes about early sex and contraception

1

Outcome: Negative attitudes about early sex

Survey Question: Please indicate if you agree or disagree with the following.

a. At my age, having sex would create problems.
b. At my age, not having sex is important to be safe and healthy.
c. Sex is a good thing for me to do at my age.
d. At my age, it is okay to have sex if you use birth control, like a condom.
e. At my age, it is okay to have sex if you are dating the same person for a long time.

(survey question 23a-e)

Outcome Response Format: Mean of 5 items (range 1-4)

Alpha = .84 a

Survey Question Source: Adapted from PPA – Gender Matters

Outcome: High perceived need for consistent condom use

Survey Question: Please indicate whether you agree or disagree with the following.

  1. It is okay if you forget to use condoms sometimes.

(survey question 35b)

Outcome Response Format: Binary: answered “strongly disagree” vs “disagree,” “agree” and “strongly agree”

Survey Question Source: Developed by the study team

Attitudes about teen pregnancy and parenting

1

Outcome: Low perceived risk of experiencing a teen pregnancy

Survey Question: How likely is it that you will get pregnant (or get someone pregnant) between now and age 20?

(survey question 42)

Outcome Response Format: Binary: answered “not at all likely” versus “a little likely,” “somewhat likely,” “very likely”

Survey Question Source: PPA – Teen Prevention Education Program (PEP)

Outcome: Agree that teen parenting is challenging

Survey Question: Please indicate if you agree or disagree with the following statements.

a. When you have children as a teen, you have to give up things you enjoy.
b. Teens don’t have enough money to take care of a baby.

(survey question 40)

Outcome Response Format: Binary: answered “strongly agree” to both questions

Survey Question Source: a. Adapted from National Survey of adolescent Males (NSAM) 1995

b. Adapted from Fog Zone

Self-efficacy

1

Outcome: Know where to get birth control

Survey Question: Do you feel that you know where to go to get birth control methods like the pill or the shot for you or your partner?

(survey question 37)

Outcome Response Format: Binary: “yes, I definitely know where to go” versus “no, I definitely do not know where to go,” “no, I probably do not know where to go,” “yes, I probably know where to go”

Survey Question Source: Adapted from PPA – Gender Matters

Outcome: Confident in negotiating condom use

Survey Question: Whether or not you’ve ever had sex, how confident are you that you could.

a. Talk to a partner about using condoms?
b. Say no to sex if your partner won’t use a condom?

(survey question 36a-b)

Outcome Response Format: Binary: answered “Very confident” to both questions

Survey Question Source: Adapted from Bradford & Beck 1991

Outcome: High perceived ability to avoid unwanted sex and sexual contact

Survey Question: Imagine you are alone with someone you like very much. How likely is it that you could.

a. Avoid having sex if you didn’t want to?
b. Stop them if they wanted to touch your private parts and you did not want them to do that? Private parts are the parts of the body covered by underwear or a bra.

(survey question 38a-b)

Outcome Response Format: Binary: answered “Very likely” to both questions

Survey Question Source: Adapted from PPA – Gender Matters

 

Outcome: High perceived ability to ask for and give consent

Survey Question: Sexual consent means that each person agrees to a sexual activity. Please indicate if you agree or disagree with the following statements.

a. I feel confident that I could verbally state my consent to someone I want to have sex with.
b. I feel confident that I could ask for consent from someone I want to have sex with.

(survey question 39a-b)

Outcome Response Format: Binary: answered “Strongly agree” to both questions

Survey Question Source: Adapted from Humphreys & Brousseau 2010

Outcome: Perceived ability to communicate with partner

Survey Question: Whether or not you’ve dated, how confident are you that you could.

a. Say no to a date?
b. Break-up with someone you no longer like?
c. Tell your boyfriend/ girlfriend how you want to be treated?
d. Talk out a disagreement with your boyfriend/ girlfriend?

(survey question 27a-d)

Outcome Response Format: Mean of 4 items (range 1-5)

Alpha = .77 a

Survey Question Source: Adapted from Toledo Adolescent Relationships Study (TARS)

Outcome: Positive future orientation

Survey Question: Please indicate how much these statements describe you.

a. If I set goals, I take action to reach them.
b. It is important to me that I reach my goals.
c. I know how to make my plans happen.

(survey question 21a-c)

Question Response Format: Mean of 3 items (range 1-5)

Alpha = .80 a

Survey Question Source: Lippman 2014

Reproductive Health Behaviors

1

Outcome: Ever had vaginal sex

Survey Question: Have you ever had vaginal sex? (survey question 51)

Outcome Response Format: Binary: Yes/No

Survey Question Source: OPA

Outcome: Unprotected sex in the past three months

Survey Question: In the past 3 months, have you had vaginal intercourse without you or your partner using any of these methods of birth control?

• Condoms
• Birth control pills
• The shot (for example, Depo Provera)
• The patch (for example, Ortho Evra)
• The ring (for example, NuvaRing)
• IUD (for example, Mirena, Skyla, or Paragard)
• Implant (for example, Implanon and Nexplanon)

(survey question 55)

Outcome Response Format: Binary: Yes / No. Students who did not have sex in the past 3 months or never had sex were coded as No.

Survey Question Source: OPA

a – Alphas were determined based on Re:MIX baseline sample.


Student survey item development

The survey, administered to students participating in the Re:MIX program and to those in the control condition, assessed students’ demographic characteristics, sexual activity, and contraceptive use. It also included questions about intentions, knowledge, attitudes, and self-efficacy related to sexual activity and contraceptive use—factors that are associated with early and unprotected sex and are targeted by the Re:MIX curriculum.

The study team included survey items that reflected the goals of Re:MIX and were appropriate for the age and background of the adolescents in the study sample. Questions were based on established measures from various sources, including those mandated by the Office of Population Affairs (OPA) (formerly known as the Office of Adolescent Health, or OAH), surveys used to evaluate programs that were part of the federal Adolescent Pregnancy Prevention Approaches (PPA) Study (specifically, Teen PEP and Gender Matters), and existing national surveys that have tested measures with adolescent populations, such as the National Survey of Adolescent Males and the Survey of Young Adults 2009 (also known as The Fog Zone).

Before the study team distributed the survey to Re:MIX evaluation participants, they conducted cognitive interviews over the phone with White, Black, and Latinx students in grades 8 through 10, located in three Texas counties. The goal of these interviews was to ensure that participants easily understood and accurately interpreted the survey items, and to confirm that respondents had the information needed to answer the questions. Survey questions were adapted based on the findings.

Evaluation outcome measures development

To develop the evaluation outcome measures, the study team recoded survey items and often combined several survey items into measures that would capture the underlying constructs of interest. For outcomes that utilized three or more question items, we formed unique constructs using principal component analysis with varimax rotation. We kept items with a factor loading of 0.5 or higher. Internal consistency for created scales (three or more items) was assessed with Cronbach’s alpha. All scales had an alpha of at least 0.7, considered the threshold for internal consistency. For several constructs, we created binary measures based on theory and/or the distribution of our sample. The response distributions and item loadings that informed the study team’s decisions around the final measures were dependent on the Re:MIX evaluation sample—specifically, the full sample with baseline survey data. Future evaluators will want to test these measures against their own samples. All outcomes, apart from intentions to have sex and the behavioral outcomes, were coded such that a higher value is negatively associated with early and unprotected sex.