Equity in evaluation research means that researchers evaluate programs in a way that is intended to benefit those who receive programmatic services. Approaches to integrating equity and cultural responsiveness into evaluations often include the following characteristics: They attend to issues of power imbalances between researchers and research participants, engage community partners, reflect on assumptions and biases, consider different worldviews, ensure that evaluation methods are multiculturally valid and oriented toward participants, strive for intentionality in data collection and analysis, and aim for accessible and actionable findings.
Although there is considerable published research on the value and importance of equity-focused evaluation approaches, there is less clarity about how these approaches are incorporated in practice, especially among federally funded evaluations of sexual and reproductive health programs. Child Trends received evaluation funding from the Family and Youth Services Bureau’s (FYSB) Personal Responsibility Education Innovative Strategies (PREIS) program to evaluate a gender-inclusive sexual and reproductive health program, Many Ways of Being (MWB) (see “MWB program” textbox). This funding provides an opportunity to embed equity-focused practices and principles into evaluation design and implementation.
MWB is an innovative gender-transformative sexual health program for youth of all genders and sexual orientations. The MWB program was developed to increase positive sexual and reproductive health (SRH) behaviors and avoid promoting shame and stigma around these topics. Program session topics include Welcome; Exploring Gender and Identity; Understanding and Expressing My Emotions; Exploring Power and Relationships; Sex and Sexuality in Media; Keeping Yourself and Others Healthy and Safe; Contraceptives and Seeking Services; and The Journey Ahead.
Child Trends collaborated with Equimundo and the Latin American Youth Center (LAYC) to lay the groundwork for a successful and equitable evaluation of MWB. Each partner played a unique and critical role in advancing equity in evaluation planning and preparation. In this brief, we highlight key equity-focused strategies used within our first year of funding in three central project preparation activities led by a different program partner: program design and adaptation, program implementation, and evaluation. We hope the approaches described throughout this brief can generate ideas for other federally funded programs as they begin their evaluations.
Below, we summarize the key equity-focused approaches described in more detail throughout this brief. For each activity, we highlight the program partner that led that effort.
Equimundo redesigned the MWB program to prioritize youth’s desires and needs. Specifically, they partnered with Black and Latinx youth to adapt the program to better reflect the lived experiences of youth, honor intersectional identities, acknowledge historical harm by the medical community, and incorporate content on the influence of media on youth today.
Latin American Youth Center prioritized reciprocity and the needs of the youth and schools. Specifically, they:
Child Trends explored internal biases and took a strengths-based approach to evaluation. Specifically, they:
In designing the MWB program, Equimundo drew from two existing programs, Manhood 2.0 and Sisterhood 2.0. These programs aimed to help students identify gender norms that operate in their community, observe the consequences of inequitable gender norms, and explore the benefits of reimagined (more equitable) gender norms. However, offering single-gendered spaces was not inclusive of all identities and did not meet the needs of youth participants. To ensure that the program would be relevant to youth’s needs, Equimundo partnered with Healthy Teen Network to conduct formative research.
Using a human-centered design approach, Healthy Teen Network conducted a hybrid of ethnographic interviews and digital focus groups to learn from 1) non-cisgender and cisgender Black and Latinx youth ages 15 to 19, and 2) former facilitators from similar sexual and reproductive health programs. The goal of getting feedback was to better understand the priorities and needs of the populations that will be served through the program and to redesign the program with those needs in mind.
In response to formative research findings, Equimundo strengthened the core content and methodologies by promoting inclusivity, addressing intersectionality, discussing media and technological contributions to messaging, and adapting to youth’s learning preferences.
Equimundo ensured that the MWB program was representative of non-heterosexual relationships and sexual orientations and incorporated discussion of gender identities beyond the male/female binary. For example, in many roleplays and scenario examples throughout program sessions, characters have a variety of pronouns and sexual orientations. This not only allows students to become familiar with diverse gender identities and sexual orientations, but also to think about the ways in which these multiple identities may influence the situations presented through roleplays or scenario activities. Other examples from the program include:
The MWB program considers various identities—such as age, wealth, disability, geography, immigration—that may impact young peoples’ lived experiences. The program incorporates elements of intersectionality into program activities and content by:
Focus group participants emphasized that any program that addresses sexual and reproductive health must also address modern media, including pornography and sexting. Therefore, the MWB developers included an activity that encourages youth to reflect on the subtle and overt messages about relationships and sexuality that they absorb in social media and advertising. Youth are also provided an opportunity to critically think about and openly discuss the purpose, benefits, and risks associated with pornography and sexting.
Focus group participants indicated that they did not want a program that simply provided information they could find online; instead, they wanted to learn skills and have opportunities to apply those skills. Therefore, the MWB developers adapted existing activities to include more opportunities to apply the knowledge gained during program sessions through roleplaying scenarios and case studies. The formative research for this project reinforced that youth connect with one another predominantly in digital spaces (texting, social media, etc.). Therefore, the MWB team designed a complementary digital component, including a set of 34 ready-to-go Instagram posts and stories that reinforce the program’s messaging and create a digital space for participants to further engage on program topics.
In preparing to implement the first year of the evaluation, LAYC focused on hiring and building an implementation team of facilitators, as well as recruiting and developing relationships with partners. LAYC relied on existing partnerships and connections to create awareness about the program and connect with potential partners.
Throughout implementation, LAYC embedded equity into the project by prioritizing reciprocal relationships with implementation sites, knowing and advocating on behalf of youth participants’ needs, and hiring staff with community- and population-relevant experience.
LAYC invested in strong and lasting relationships with several schools in the Washington, DC metropolitan area by identifying and strategizing ways to meet students’ and schools’ needs. Prior to meeting with schools, LAYC staff reviewed websites and, when possible, scheduled informational interviews with LAYC staff in different departments who may have interacted with the schools previously to learn about the schools and their students. LAYC sought to partner with schools that were not currently offering sexual health programming at the time of partnership development; the organization intends to continue working with schools to implement MWB even after the project ends to help meet these schools’ need for sexual reproductive health programming. From the beginning, LAYC consistently asked school staff about their needs and requirements and prioritized reciprocity by offering to provide a results document to school partners to share outputs and outcomes during implementation.
LAYC has been embedded in the Washington, DC community for 55 years and has extensive experience engaging youth with a variety of racial and cultural backgrounds and diverse identities. As such, LAYC staff provided input throughout the program development process, particularly to ensure that the language used was inclusive of nonbinary and trans youth. Further, LAYC raised the idea of—and advocated for—conducting programming in Spanish because the Washington, DC area has a large population of Latinx youth. As a result, Equimundo translated the program and all evaluation materials into Spanish to better serve the youth in the community.
As the implementation partner and organization working directly with youth for the MWB project, LAYC hired staff and program facilitators who understood the needs of the populations being served. LAYC focused on local candidates who understood the challenges and issues faced by Black and Latinx youth in the Washington, DC metropolitan area. For example, LAYC hired two curriculum facilitators who had previously participated in other LAYC programming. Further, when interviewing staff, LAYC asked questions to ensure that facilitators had considered and understood their role in promoting social justice and equity through facilitation:
Child Trends staff prepared to evaluate MWB by assessing their own evaluation team’s internal biases and ensuring that the evaluation design was strengths-based, culturally appropriate, and responsive to the needs of Black and Latinx teens in Washington, DC.
To prepare to evaluate MWB, Child Trends evaluators assessed the research team for cultural responsiveness and biases, matched data collection protocols to participants’ needs, and provided a high-quality control program.
To encourage an open dialogue around equity, the Child Trends team completed culturally responsive evaluation self-assessments and discussed their responses, with the goal of furthering equity practices in the evaluation. This activity allowed the team to critically think about practicing cultural responsiveness and to examine potentially interfering biases at an individual and group level. These conversations also inspired discussions with LAYC and Equimundo around what success in this study would mean to the team and to the target populations. As a result, the research questions have shifted away from a deficits-based, prevention focus and toward a sexual health promotion and empowerment focus. This ensures that the outcomes of interest align with agreed-upon benchmarks of success and are appropriate, responsive, and equitable to the populations we aim to serve (more below).
An important component of equity in evaluation is developing questions to assess program impacts that are meaningful to the stakeholders. The MWB program takes an asset-based approach and has shifted the framing of program content away from deterring youth engagement in “unhealthy” or “negative” sexual behaviors and toward encouraging youth to engage in positive, healthy sexual behaviors. The evaluation team built the participant survey and focus group protocols to align with this ideology. For example:
To ensure that the evaluation design is centered on participants and the needs of the priority community, the project team set aside resources to implement high-quality and beneficial comparison programs. One example includes the International Youth Foundation’s Passport to Success® (PTS) program. By offering a high-quality comparison program, youth who are not randomly selected to receive MWB still receive benefits from participating in the evaluation. PTS equips young people with a range of skills to help them stay in school and acquire the education, professional skills, employment readiness, and confidence they need to succeed in life and in the workplace.
Child Trends, in collaboration with its partners at Equimundo and LAYC, used the evaluation planning year to align MWB’s program, implementation, and evaluation design with key equity principles and to lay the groundwork for a successful and equitable evaluation. Each partner played an important role in this process, highlighting the advantage of a collaborative and multipronged effort to advance equity.
Following the project planning period, the team piloted the MWB program in Fall 2022 and began full implementation in 2023. Since then, the team has continued to prioritize equity in design, implementation, and evaluation by updating the implementation and evaluation approach based on evaluation findings and input from program participants and staff. For example, Equimundo updated the MWB program to better respond to the needs of youth participants and school staff by 1) translating it into Spanish to accommodate Spanish-speaking participants and 2) creating a version of the program with 16 one-hour sessions (instead of eight two-hour sessions) to allow for in-school implementation. To create a peer-peer approach for recruitment, LAYC asked interested program alumni to serve as youth ambassadors. In this role, youth ambassadors support program recruitment by discussing their experiences in the program with their peers. To guide future project improvements, the team created and continually updates a “lessons learned” document to report on what is working well and what can be improved about the MWB program, as well as implementation and evaluation practices.
Although the team incorporated equity principles into the MWB project during the project planning period and continues to make changes to further promote equity, we have identified several areas of opportunity for improvement in future work. For example, a key component of the program and evaluation design processes focused on ensuring that the program is inclusive of students of all genders and sexual orientations. However, through the pilot, the team noted that the programming and implementation could be more supportive of other youth populations, including youth with learning differences or challenges. We have addressed this in part by researching, discussing, and implementing best practices on supporting youth with intellectual and developmental disabilities—for example, by allowing youth to take additional breaks, providing an extra adult in the room to support the youth, and having direct discussions with the youth on how facilitators can best support them. We recognize, though, that we can do even more in future work—for example, by providing staff trainings on serving youth with learning differences or disabilities and partnering with school counselors to gain insight on how to better to support youth.
Another important component of equity-focused evaluations of sexual and reproductive health programs for youth is to intentionally incorporate youth voices into all processes. The MWB evaluation has incorporated youth participants into formative research to inform program design, recruited youth ambassadors to help with recruitment and retention, and conducted youth focus groups to obtain information about student responsiveness to the program and facilitators. However, future work would ideally incorporate youth voices in all stages, including selection of research questions and evaluation design. In later stages of the evaluation, we plan to engage youth and school partners in interpreting evaluation findings through brief presentations and data walks. During data walks, we will ask small groups of students or school partners to review study findings and then participate in a discussion to interpret, discuss, and reflect on these findings.
While we have presented strategies for federally funded programs to advance equity through evaluation, we recognize that funders are critical for any program to advance equity. The PREIS grant incorporated an up to 18-month planning period that allowed the team to adapt the MWB program, partner with implementation sites, and design and pilot the evaluation. We recommend that funders continue incorporating a similar planning period within evaluations to allow project teams time to design an equitable evaluation. It’s important to have funders who champion diversity, equity, and inclusion (DEI) in evaluation work and who are willing to fund a role focused on DEI to better support equitable implementation and evaluation. We also understand that including youth voices in all stages—including to provide feedback on research questions—would require additional funds and flexibility from funders. Lastly, while federal program evaluations are designed to expand the pool of evidence-based programs available to communities, researchers must go beyond incorporating equity into evaluation to incorporate equity and inclusion in program development. As a result, we recommend funding innovation development that incorporates youth and community voices to ensure the development of programs that are responsive to the needs of communities.
This brief was made possible by Grant Number 90AP2693 from the Department of Health and Human Services, Administration for Children and Families. Its contents are solely the responsibility of Child Trends, Equimundo, and LAYC and do not necessarily represent the official views of the Department of Health and Human Services, Administration for Children and Families.
As authors, we would like to thank our colleagues, without whom this work would not have been possible. First, we would like to thank our wonderful program facilitators Altagracia Kubinyi, Dereck Myers, Alma Figueroa, France Robinson, and Sokka Asif. We also would like to thank our former project officer, Kathleen Derrick, for thought partnership throughout the project planning period; our current project officer, Katherine Godesky, for her review of this brief; Lauren Kissela for her fact check; Brent Franklin for his edits; and Catherine Nichols for her design work. Finally, we are deeply grateful to the staff at our implementation sites and to our youth participants, who offered their time and perspectives—all of which were critical to making the MWB evaluation project stronger.
Parekh, J., Ciaravino, S., Welti. K., Ragonese, C., Lapointe, L., Manlove, J., Gilbertsen, J., & Katz , R. (2023). Equity-focused strategies in a federally funded evaluation of a sexual health program. Child Trends. https://doi.org/10.56417/4593q1135m
[i] Expressed sexual agency items used in the surveys were developed by the Child Trends team based on conversations with Equimundo and researcher Spring Cooper. Perceived sexual agency items were adapted from the Female Sexual Subjectivity Inventory and edited to better fit the age and gender diversity of the MWB evaluation sample.
[ii] Self-efficacy items were adapted from the Re:MIX program evaluation survey and the Access to Sexual and Reproductive Health Information measure in Upadhyay et al.’s Sexual and Reproductive Empowerment Scale for Adolescents and Young Adults.
[iii] Sexual empowerment measures were adapted from Upadhyay et al.’s Sexual and Reproductive Empowerment Scale for Adolescents and Young Adults.
© Copyright 2023 ChildTrendsPrivacy StatementNewsletter SignupLinkedInThreadsYouTube