Implications of COVID-19 on the Research and Evaluation Provisions of the Family First Act

Publication Date:

September 09, 2020

Topic:

COVID-19

This fact sheet was updated in December 2020 to include new federal guidance and new research from Child Trends.

The federal Family First Prevention Services Act of 2018 (Family First Act) was designed to help children at risk of entering foster care remain safely with their families, and to help children already in foster care live with families rather than in group homes or congregate care settings. One major way in which the Family First Act achieves this goal is through expanding the largest federal funding source for child welfare activities—Title IV-E of the Social Security Act—to include evidence-based mental health, substance abuse, and in-home parenting programs that prevent entry to foster care for children at imminent risk of being removed from their homes. As the COVID-19 pandemic continues to impact families and the child welfare system, this brief describes anticipated implementation challenges within the evidence and evaluation requirements of the Family First Act, including implications for programs with an existing evidence base and programs currently undergoing or planning for evaluation.

As child welfare leaders and policymakers rebuild from the impacts of COVID-19, they will also need to account for the historic and current impacts of structural racism. It will be important that these decision makers create more equitable systems and supports for children and families who have experienced racism. The prevention provisions of the Family First Act can be a powerful tool in that work. Throughout this brief—and as we discuss the implications of COVID-19 on program purveyors, implementers, evaluators, and system stakeholders—we also elevate specific racial equity considerations.

What is the Family First Act?

Passed in 2018, this landmark federal legislation advances several key priorities:

  • Funding services for families to prevent foster care entry
  • Supporting kinship caregivers
  • Establishing criteria for appropriate use of residential treatment
  • Strengthening services for older youth

For details on the Family First Act, see FamilyFirstAct.org.

For information on the evidence-based requirements of the Family First Act, see Applying the Research and Evaluation Provisions of the Family First Prevention Services Act.

For information on the changes to the evidence-based requirements from the Family First Transition Act, see The Family First Transition Act Provides New Implementation Supports for States and Tribes.


Background

The prevention provisions of the Family First Act are optional: States that choose to opt in must file (and have approved) Title IV-E prevention program five-year plans that outline their proposed work.[1] To qualify for Title IV-E funds, prevention programs must meet certain evidentiary criteria and be evaluated and rated by the Title IV-E Prevention Services Clearinghouse (Clearinghouse) as promising, supported, or well-supported.[2] The Family First Act requires that a certain proportion of state expenditures be used for programs that receive the highest rating from the Clearinghouse, while the subsequent Family First Transition Act phases in these requirements. As of December 2020, eight states and the District of Columbia have approved plans and six states and two tribal IV-E agencies have submitted plans to the Children’s Bureau for review and approval.[3] Kinship navigator programs, which connect kin who are caring for children to services and supports, can also qualify for Title IV-E funds if the Clearinghouse rates the program as promising, supported, or well-supported. The Clearinghouse is required to review “culturally specific, or location- or population-based adaptations” to ensure that programs designed to address the needs of communities that meet the Clearinghouse standards qualify for Title IV-E funds.

In March 2020, most areas of the United States began to practice social distancing measures to reduce the spread of COVID-19. As in other public sectors, child welfare services drastically changed to protect children and families, as well as agency staff. Every facet of the child welfare system was impacted; as case workers shifted to remote service delivery, parent-child visitation schedules were disrupted, reunification services experienced disruptions, and court timelines were delayed. Although it is unclear as to whether, or how, rates of maltreatment may have changed, the closures of schools and daycares also led to a deep reduction in child welfare hotline reporting.

Implementation of the Family First Act, including aspects of its evidence-based provisions, has similarly been impacted by the pandemic and resulting recession. In addition to the significant scope of changes to services, programs, and budgets required during the pandemic, state and local budgets are in crisis, resulting in limited capacity and reduced resources for agency leaders and staff to focus on a new initiative. From a practical perspective, implementing a major initiative requires a great deal of time, energy, and funding from multiple stakeholders. Implementing new prevention programs will likely cause a huge strain on child welfare systems already in crisis due to the pandemic.

While the pandemic has unfolded, child welfare leaders and stakeholders have reexamined existing inequities for families and begun to strategize new ways to support all families and children, with a particular focus on those who have been marginalized by the child welfare system. There are persistent racial and ethnic disparities and disproportionality in the child welfare system. Nationally, Black and American Indian/Alaska Native children are more likely to be removed from their families than white, Asian, and Hispanic children; once in foster care, children of color experience poorer outcomes than their white peers. These same families are also facing disproportionate health and economic obstacles due to the impact of COVID-19 and the resulting recession. As a component of policy and programmatic responses to these challenges, evidence-based prevention and kinship navigator programs can serve as key resources for creating a more equitable child welfare system.

Although we do not yet know the full extent of the impact of the pandemic, or know how child welfare systems may adapt to be more racially equitable, we describe implications for programs along the evidence-building continuum below.

The following key evaluation terms will aid readers’ understanding of this brief’s implications for program evaluations, as well as our recommendations to evaluators, program leaders, and policymakers.

Key evaluation terms

Fidelity: “The extent to which an intervention is implemented as intended by the designers of the intervention. Thus, fidelity refers not only to whether or not all the intervention components and activities were actually implemented, but whether they were implemented in the proper manner.” (California Evidence Based Clearinghouse, 2020)

Formal program adaptation: For the purposes of the Clearinghouse’s reviews, formal program adaptation entails “alternative manualized versions of the original program designed to address particular issues or populations” (Wilson et al., 2019, p. 4). Formal adaptations include changes to a program’s content or approach. The Clearinghouse has not released guidance on whether switching from in-person to remote service delivery is considered a formal adaptation for evidence-building purposes; however, the Children’s Bureau allows states and tribes with approved prevention plans to adapt programs for remote service delivery when necessitated by the pandemic (Children’s Bureau, 2020).

Interrupted time series design: An evaluation in which “data are collected at multiple and evenly spaced time points (e.g., weekly, monthly, or yearly) before and after an intervention … The main objective is to examine whether the data pattern observed post-intervention is different to that observed pre-intervention” (Hudson, Fielding, & Ramsay, 2019).

Outcome evaluation: “Evaluations that investigate whether the program or activity is associated with changes for program participants. This type of evaluation looks at whether, to what extent, and in what direction outcomes change for those in the program” (Bronte-Tinkew et al., 2007).

Process evaluation: “Examines the extent to which a program is operating as intended by assessing ongoing program operations and determining whether the target population is being served” (Bowie & Bronte-Tinkew, 2008).

Program/service: A prevention program (i.e., mental health prevention and treatment service, substance abuse prevention and treatment service, or in-home parent skill-based program) or kinship navigator program eligible for Title IV-E funding under the Family First Act.

Program purveyor: The individual or entity responsible for disseminating a program or service, which may be the original developer of the program model or a separate entity. Program purveyors may support implementation through training, technical assistance, and fidelity monitoring.

Program implementer: The public or private agency delivering a program or service to children and families.

Remote service delivery: The use of video technology and/or phones to deliver services to families.


Implications for Evidence-based Programs that Have Been Rated by the Clearinghouse

As of December 2020, 48 programs have been reviewed by the Clearinghouse. Of those, 20 did not currently meet criteria for having evidence of effectiveness, one was not eligible for review, and the remaining 27 were reviewed and received ratings of promising (13 programs), supported (5 programs), or well-supported (9 programs). Two additional programs have been approved for transitional payments through independent systematic reviews until the Clearinghouse completes its review. As the Clearinghouse continues its reviews, it is likely that more programs will be rated as promising or higher, given existing research evidence. However, as states continue to respond to COVID-19, implementers of programs currently meeting evidence requirements (i.e., rated as promising, supported, or well-supported) may be making:

  • Changes in service delivery and content. Program purveyors and implementers are changing the ways in which they serve children and families to keep children, families, and staff safe during the pandemic. The Children’s Bureau issued guidance in November 2020 to allow states and tribes with approved prevention plans to claim federal funding for prevention services adapted for remote service delivery during the pandemic, when these services would otherwise not be available to families due to pandemic-related obstacles such as stay-at-home orders or office closures. Such adjustments to delivery formats are also promoted in joint guidance from the Administration for Children and Families and the Health Resources and Services Administration; this guidance describes the ways programs are using remote service delivery, such as video conferencing, to continue serving families. Program purveyors have responded quickly to support families and staff. For example, Family Centered Treatment, a home-based family therapy program, released a variety of resources to their providers on engaging families and completing certain program activities (e.g., structured family assessment) via telehealth. While these changes are necessary and important, they do change the way services are delivered and have the potential to change their effectiveness. The changes may also raise unique equity challenges when children or families lack ready access to the technology or high-speed internet needed to fully participate in services. Although some programs may rely on evidence of the effectiveness of telehealth in medicine and behavioral health, many prevention programs are being delivered remotely for the first time, and some changes have not yet been tested. Beyond service delivery, program purveyors and implementers may find that certain program components cannot be delivered effectively remotely, leading to adaptations to program content.
  • Changes that may interfere with fidelity. Changes to service delivery and/or content may have fidelity implications, and the Family First Act requires documented processes for implementing programs with fidelity. Several program purveyors (for example, in the home visiting field) have released guidance that allows services to be delivered remotely during an emergency while still maintaining program fidelity. Federal programs such as the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program have accepted these changes to modality. Additionally, changes to the core components of a program may impact fidelity—and, in turn, impact anticipated child or family outcomes.

Implications for Programs Planning or Currently Undergoing Evaluation

The Family First Act promotes evaluation by tying Title IV-E funds to evidence-based programs reviewed by the Clearinghouse and by requiring ongoing evaluation of programs that are being implemented under approved state prevention plans.[4] As a result, prevention programs and kinship navigator programs across the country are in various stages of evaluation (including planning stages), as required by their state’s prevention plans or with the goal of receiving or improving a Clearinghouse rating. The pandemic and resulting recession are likely to affect existing and future program evaluations in myriad ways, including the following:

State examples

State leaders across the country are working to ensure that families have access to the services and supports they need to prevent entry into foster care. Child Trends is grateful for the input of our partners at APHSA and NCSL for tracking and sharing this work.

State legislators are introducing legislation to enhance families’ economic stability—for example, through paid leave and the provision of access to food banks and housing security. For a full listing of state legislative action supporting child welfare systems during COVID-19, see  https://www.ncsl.org/research/human-services/child-welfare-system-covid-19-resources.aspx.

State agency leaders are using existing flexibilities to shift toward remote service delivery and virtual training formats. This flexibility is keeping families safe while allowing services and trainings to continue. For descriptions of current policy levers that state agency leaders should consider in their responses to COVID-19, see this resource from APHSA: https://files.constantcontact.com/391325ca001/09c14e28-eba2-4042-8d4f-c039289689bd.pdf.


Recommendations

As they plan their responses to COVID-19, state and federal leaders, program purveyors, program implementers, and evaluators must be aware of the aforementioned implications related to the Family First Act’s evaluation provisions. These implications also play a role in how states develop prevention programs that support the families disproportionately impacted by COVID-19 and leverage their prevention work to address racial disparities and inequities within the child welfare system. Although there are challenges, the prevention provisions of the Family First Act provide a new funding opportunity for states to provide critical services to children and families by keeping them together and reducing the trauma associated with foster care entry. We urge evaluators, program leaders, and policymakers to do the following:

  • Monitor for program and federal guidance. Given their responsibility for ensuring fidelity to their program models, program purveyors are key partners in Family First Act implementation. Whenever possible, consult program purveyors about their plans and seek their guidance on the impact of program adaptations on fidelity, as recommended by the Children’s Bureau. Similarly, monitor the Clearinghouse and federal agencies for updated guidance and potential changes to Family First Act implementation timelines.
  • Invest in documenting what is happening now. As program purveyors and implementers innovate and adapt to changing circumstances, it is important that they understand the timing and substance of programmatic changes. Process evaluations can help the child welfare field understand how programs are being adapted and implemented during the pandemic. Subsequent outcome evaluations can explore whether (and how) the adapted program helps children and families (i.e., improves outcomes). Program implementers may choose to continue delivering services remotely in the future (e.g., to reach families with transportation barriers and respond to families’ needs and preferences). By evaluating these programs now and sharing findings with practitioners and policymakers, the field can better understand the effectiveness of this approach.
  • Critically examine and build the evidence for programs that are effective for children and families of color. As evaluators document how programs are affected by the pandemic, it will be important to assess whether these programs are effective for families disproportionately affected by the pandemic and by existing inequities. Ask critical questions to understand whether existing evidence speaks to the effectiveness of the program for children and families of color. Consider, for example, the racial and ethnic background of children and families in prior evaluations, whether the evaluations used culturally appropriate measures, whether the study provided contextual/historical reasons that may explain differences, and whether the study tested differential effectiveness for families based on their races and ethnicities. Child welfare leaders and policymakers should also prioritize the evaluation of culturally relevant prevention programs for the children and families who live in their jurisdictions. For example, it is important to continue building the evidence for culturally based prevention services for American Indian/Alaska Native families to fully understand whether services are effective in supporting these unique populations.
  • Collaborate across agencies. Agencies are collaborating in new ways as they strive to keep children, families, and communities safe during the pandemic. There may be opportunities to leverage and sustain these strengthened relationships to implement prevention services. At the same time, collaborative partnerships may face increased stress due to shrinking budgets and increased competition for flexible resources.
  • Partner closely with evaluators to understand the specific implications for each unique circumstance. This brief has outlined potential implications of COVID-19 on the evaluation provisions of the Family First Act. However, each program and evaluation is unique. State and community leaders, as well as program implementers, should work closely with their evaluation partners to explore ways in which the pandemic and recession may affect their ongoing work.

Conclusion

Prevention services, including those included in the Family First Act, can strengthen families’ protective factors to address those stressors which have been introduced or exacerbated by the pandemic. In addition to responding to families’ immediate needs, prevention services may result in future fiscal benefits by reducing the long-term costs associated with children’s entry into foster care. Implementing the prevention provisions of the Family First Act also presents an opportunity to promote race equity and address the disproportionate representation and disparate outcomes of families of color in the child welfare system.

Acknowledgments

We are thankful for our colleagues at Child Trends and in the child welfare field who provided insights and ideas during the development of this brief. We are particularly grateful for the thoughtful review and feedback from Jan DeCoursey, Esther Gross, Alyssa Liehr, Karin Malm, and Lauren Supplee at Child Trends; Meghann Dygert at APHSA; Nina Williams-Mbengue and Donna Wilson at NCSL; and Marla McDaniel at The Urban Institute.

Additional resources


Footnotes

[1] Other important provisions of the Family First Act are not discussed in the brief. For example, the Title IV-E prevention funds are linked to Title IV-E funding for certain congregate care settings for children in foster care. This provision both incentivizes and supports state investment in prevention while encouraging placement of children in family settings (foster and kinship), rather than in congregate care settings. All states are required to comply with the new funding limitations on congregate care by October 1, 2021, including those states that do not adopt a prevention plan. For a detailed summary of the law and updates on federal guidance and state implementation, see FamilyFirstAct.org.

[2] For more information on the evaluation requirements and evidence ratings, see Applying the Research and Evaluation Provisions of the Family First Prevention Services Act.

[3] Most recent data from the Children’s Bureau: https://www.acf.hhs.gov/cb/resource/title-iv-e-five-year-plan. Approved states: Arkansas; Washington, DC; Kansas; Kentucky; Maryland; North Dakota; Utah; Washington; West Virginia. Submitted states and tribes: Alaska; Aleut Community of St. Paul Island, Alaska; Colorado; Eastern Band of Cherokee Indians, Cherokee, North Carolina; Illinois; Iowa; Nebraska; Virginia.

[4] States may request a waiver of this requirement for well-supported programs.