Lessons Learned When Building the Evidence for a Child Welfare Practice Model

Publication Date:

July 17, 2019

The passage of the Family First Prevention Services Act (FFPSA)[1] offers the potential for funding to implement evidence-based programs for children at risk of maltreatment, including children who are reunified with their parents. Many extant programs have not been studied with sufficient rigor to qualify as evidence-based. This new legislation offers an opportunity for agency leaders and practitioners to begin to build evidence of success.

The purpose of this brief is to highlight for the child welfare field the evidence-building lessons learned from the evaluation of Success Coach, a post-reunification program developed by Catawba County, North Carolina. This evidence building began in 2013 with an evaluation of the Success Coach program’s process and preliminary outcomes. Child Trends collaborated on a series of briefs highlighting lessons learned from this initial work,[2] and then designed a more rigorous outcome and implementation evaluation of the program.

In this brief we describe the background and development of Success Coach and discuss learnings from the earlier process and outcome evaluation as they relate to building the evidence for Success Coach. In more detail, we describe the design of the outcome evaluation and present preliminary findings from this second study. We also discuss implications for other jurisdictions wanting to implement and evaluate post-reunification services similar to Success Coach.

The need for post-reunifications programs and evidence for them

In the United States, the number of children removed from their parents’ custody due to maltreatment has risen in recent years, from 400,394 in fiscal year (FY) 2013 to 442,995 in FY 2017.[3] Nationally, approximately 217,000 (49%) of children who exited care in FY 2017 were reunified (custody is returned) with their parents, and based on findings from other studies,[4] an estimated 6,500 (3%) to 60,700 (28%) will subsequently re-enter care at some point due to repeat maltreatment. Typically, child welfare agencies end contact with families after custody is returned. Often, however, families still face challenges in caring for their children—challenges that continued supports and services could help to ameliorate.

There are very few post-reunification programs available to families; moreover, the evidence base for these programs is limited. Only one, the Homebuilders program, is rated by the California Evidence Based Clearinghouse (CEBC) as supported by research evidence.[5] CEBC rates another post-reunification program, Family Centered Treatment, as having promising research evidence. Although other post-reunification programs show promise in preventing re-allegations of maltreatment and re-entry into care, they lack an evidence base (Carnochan et al., 2013).[6] To bridge this gap, Catawba County in North Carolina initiated their Child Wellbeing Project in 2006, in partnership with The Duke Endowment.

The Child Wellbeing Project focused on post-permanency stabilization and child well-being and resulted in the development and implementation of two principal intervention strategies: the Success Coach program and the Educational Advocate service. Three smaller-scale psycho-educational and clinical interventions were also implemented: Parent Child Interaction Therapy (PCIT), Strengthening Families Program (SFP), and Adoption Support Groups (ASG).[7] The Success Coach program addresses post-care, both following reunification of children with their birth parents, and following adoption finalization.

In this brief, we discuss implications for the field based on these findings and offer suggestions and approaches to addressing and managing these barriers in order to effectively evaluate post-reunification programs.


The Success Coach Model

Catawba County, North Carolina staff developed the Success Coach program based on needs identified by conducting a comprehensive literature review, as well as focus groups with reunified parents, adoptive parents, guardians, and child welfare staff. Catawba County also consulted with the National Implementation Research Network on how to develop an array of services for post-reunification and post-placement families.

What is the Success Coach model?

Success Coach is a voluntary service, offered to families at the point of reunification or adoption/guardianship. When a family agrees to work with a Success Coach, they negotiate together to develop a Success Plan that lasts for up to two years. The Success Coach meets with the family according to the agreement, usually in their home, at least twice monthly. Success Coaches work with families to assess their needs and develop and implement the Success Plan. They also assist families by coordinating with other needed services, and helping families manage crises that may arise. They work with families to enhance a range of skills essential to achieving stability and well-being.

Below, we provide a description of the model, including staffing, the target population, and the array of activities conducted and services provided.

Staffing. Success Coaches are employed by Catawba County Social Services and have a bachelor’s degree in social work or a related field. Most of them have prior experience in child welfare before becoming a Success Coach. All Success Coaches are trained in the model, receive regular supervision, and are observed in the field at least annually.

Target population. Success Coaches serve families of children under age 18 who exited foster care, and who exited to a permanent placement (reunification or adoption).[8]

Program activities and services. Success Coach has a practice manual and clear guidelines for the work.[9] The program includes activities and services for building families’ resilience and increasing their well-being in the following domains:

  • Mental/emotional health
  • Family functioning
  • Caregiver self-sufficiency
  • Child’s education
  • Environment
  • Social/community capital or support

From the outset, the Catawba County developers embraced the theory that when post-permanency placements are stable, and caregivers can support children and promote their well-being, children are less likely to suffer repeat maltreatment. With stable, supportive placements, children are also less likely to re-enter the child welfare system and more likely to make successful transitions to adulthood. The theory has a basis in the literature,[10] but there is not yet any research evidence that supports it. For Catawba County, it was important to establish reliable evidence for the efficacy of their practice. The county took its first step toward gathering evidence for the Success Coach program in 2013, with an evaluation of process and preliminary outcomes. To expand on the 2013 evaluation and address its limitations, Catawba County has now undertaken a rigorous impact evaluation of Success Coach.

Intervention approaches:

  • Review Success Plan
  • Work on Success Plan
  • Engagement
  • Assessment
  • Skill Building
  • Service Coordination
  • Advocacy
  • Educational Services
  • Crisis Prevention/ Intervention

Types of skills Success Coaches focus on with families:

  • Financial Planning
  • Social Skills
  • Parenting Skills
  • Employability Skills
  • Resiliency Building

Path to a Rigorous Evaluation

In December 2013, Catawba County undertook an evaluation of the Child Wellbeing Project, including the Success Coach program. Child Trends and the National Implementation Research Network (NIRN) consulted on project design, service development, and program implementation. Independent Living Resources, Inc. conducted a process and preliminary outcomes evaluation. Evaluators found that the presence of a Success Coach in the lives of post-adoptive and post-reunified families appeared to decrease the risk of re-allegations of abuse and neglect and re-entries into out-of-home care. However, due to the study design as well as many co-occurring factors, including low enrollment and families’ receiving services in addition to Success Coach, evaluators were unable to establish a direct association between Success Coach programs and positive outcomes for the families served.

Expanding services

Catawba County was encouraged by findings from the 2013 evaluation that indicated the Success Coach program had potential to increase child and family well-being and reduce repeat maltreatment. The next step in building an evidence base for their program was a more rigorous study, which they initiated in 2015. At that time, to serve more families and increase the sample size for the more rigorous study, Success Coach program staff decided to expand their reach to neighboring counties. Five counties agreed to collaborate with Catawba County by referring families, at the time the child(ren) were returned home (i.e., trial home placement), to the Catawba County Success Coach program. These Catawba County Success Coaches provided services to the families referred by the expansion counties, in addition to families referred by Catawba County. Figure 2 depicts the initial phase, in which only Catawba County families received the service, and the expansion of the service to five neighboring counties. The rigorous design of the 2015 study included an implementation study that would examine the successes and challenges of expanding services to additional sites.

Determining the evaluation design

The Success Coach developers wanted to know whether the program worked for families, and to identify the factors that were instrumental in supporting the implementation and expansion of the Success Coach program. Therefore, Child Trends designed an evaluation that includes both an outcome and an implementation study.

Outcome study. This component of the evaluation examines impacts of the Success Coach program on child safety (as measured by rates of re-allegations of maltreatment and re-entry into care) and child well-being (as measured by parents’ attitudes and access to supportive people and services that promote child and family well-being). To detect any statistically significant impacts on outcomes that might exist, we anticipated needing to enroll at least 200 families into the study. Families would be randomly assigned to either receive Success Coach services (treatment) or to receive “services as usual” (control).

Randomization. To be able to attribute differences in any outcomes we observed between the two groups to the intervention, we carried out a randomized controlled trial. With a sufficiently large sample size, randomly assigning families to either the treatment or control group makes the two groups statistically equivalent in terms of other factors that might influence outcomes of interest. Families’ motivation is an example of such a factor; that is, without random assignment, families who are particularly motivated might be more likely volunteer for the intervention, and this motivation—rather than the intervention itself—might explain more favorable outcomes observed in the treatment group.

The various DSS county offices referred families to the Success Coach program at the point when the child returned home for a trial placement prior to reunification. When families were referred, we randomly assigned them to either the treatment or control group. Success Coaches then approached the referred families, to explain the Success Coach program and the possibility of receiving the service if the family decided to join the study. For families who consented to participate in the study, the Success Coach informed them of their study assignment, and then offered services to those families assigned to the treatment group.

We retained all families who consented to participate in the study in our analytical sample, even if they were assigned to the treatment group and did not take up services. This “intent to treat” design is critical for maintaining the statistical equivalence of the treatment and control groups.

Measures. The outcome study relies on a variety of measures to assess differences between control and treatment families in family well-being, repeat maltreatment, and placement stability. These measures include the following:

A parent survey administered at baseline and every six months for two years. The survey included:

  • Mini Child and Adolescent Needs and Strengths (Mini-CANS)
  • Protective Factors Survey
  • ACES Survey
  • Miami Dade Parent Survey

Data from the Success Coach program database, for example:

  • Demographic data about the parents
  • Number and types of contacts with the Success Coaches

Administrative data from the state child welfare data system, for example:

  • Reports of maltreatment by parents
  • Outcome of any investigations done in response to these reports
  • Re-entry of children into foster care, if applicable, and length of stay in care

Implementation study. This component of the evaluation complements the impact study by providing context and an understanding of factors driving the implementation, both in the Success Coach program and the expansion counties. Factors studied include what supports were present or absent, and how this affected the implementation. The research team also wanted to know what factors influence whether families consent to participate in the study, and, for treatment group families, what influences whether they take up the service. Lastly, the research team needed to understand all other services and supports available in the community that families who have recently been involved with child protective services could access.

Measures. The research team developed focus group and interview protocols to gather information from key Success Coach program staff and county workers. Research team members conducted two rounds of interviews and focus groups, one in September 2017 and one in September 2018. They met with Success Coach program staff, including Success Coaches, program leadership, the data manager, and the education advocate. Additionally, they conducted interviews and focus groups with all participating county staff, including supervisors, caseworkers, mental health therapists, and guardians ad litem (GALs).

Identifying the study population

In participating counties, children removed from their home are typically returned home for a “trial home placement” once the safety concerns that existed are alleviated. After successful completion of the trial home placement, the children are legally reunited with their parents. Success Coaches provide services at this point—at the time custody of the child(ren) is returned to the parent(s). Overall, in North Carolina, 4,756 children were in out-of-home care during FY 2017. Among those children, rates of reunification vary. On average, 39 percent of children were reunified after being in care for less than one year, 35 percent were in care for one to two years, and 25 percent were in care for more than two years. As shown in Table 1, the number of children reunified with their parents across the six counties being served increased from 2010 to 2014. While there were some increases in individual counties, across the six counties reunification rates remained stable from 2014 to 2015 and decreased from 2015 to 2017 (our study period).

Parents were predominantly female (64%) and white (73%). The largest group of parents were single (43%), but a third were married (34%). Over half (57%) had a high school diploma, GED, or less, and almost two-thirds were employed (60%). At baseline, families in both treatment and control groups reported similar needs and strengths (mini-CANS), similar protective factors, ACES scores, and community supports. There were a few significant differences: control families reported higher family functioning/resiliency (p = 0.017) than treatment families. They were more likely to believe their child misbehaved just to upset them (p = 0.036) compared to treatment families.We started enrolling families in the study in October 2015 with counties referring families with a child on trial home placement to the Success Coach program. As of January 2018, 168 families were referred by the six counties: 30 families, with 49 children, were assigned to the control group and 23 families, with 42 children, to the treatment group. Parents completed paper-and-pencil, self-administered surveys (Protective Factors, ACES, and Miami Dade Parent) during the initial visit (baseline data). These measures were also collected at 6, 12, 18, and 24 months after the initial visit. Caseworkers completed the Mini-CANS at the point of reunification, and identified areas where families needed support as well as strength-building in order to help the Success Coaches begin working with the parents on a treatment plan.

Parents in both groups reported they experienced more than three adverse childhood events. They rated their communities as safe places for their children, where people usually watch out for each other’s children, and they usually have people on whom they can rely for support.

At baseline, there were no significant differences between treatment and control families on the Mini-CANS. Caseworkers rated most families as having a mixture of needs and strengths. They rated families strongest in the following areas: being a support for the child, dealing with the child’s reaction to trauma, helping children feel comfortable with their cultural identity, being able to manage household activities, having strong family relationships with good communication among family members, helping the child function well in their living arrangement; and in the family in general. The areas in which caseworkers rated most families needing at least some help included building social assets, knowing about their child’s strengths and problems, planning for and providing mental health or other services needed by family members, monitoring and disciplining their child, developing a sense of belonging in a cultural group, and ensuring the child has a positive school experience.


Early Implementation Findings

As mentioned earlier, the implementation study complements our rigorous outcome study. While impact findings from random assignment designs require analyses after the accumulation of the sample is complete, preliminary findings from implementation studies can be disseminated far earlier. Below we discuss findings related to our implementation evaluation questions.

What key factors support expansion of the Success Coach program?

We examined factors that support expansion of the Success Coach program, including whether public agencies are best suited to delivery of the program during its expansion, and whether private funding is needed to expand post-reunification services. In addition, we describe challenges and facilitators to the referral system during expansion of services.

Success Coach, both within their own jurisdiction and when expanding to neighboring jurisdictions. Success Coach staff believe the service is best provided by a child welfare agency and should not be outsourced to private agencies or small counties. They were concerned that small counties would be unable to support the practice effectively, and that private service providers in this region would not have the in-depth knowledge of the child welfare system needed to work with parents whose children have been in state custody due to substantiated allegations of maltreatment.

There is a need for private funding. Private funding was needed to support and expand post-reunification services. Success Coach staff reported that in many communities, a post-reunification program cannot be supported without grant funding because it is a non-mandated program with no state funding available. Also, it is a resource-intensive program, both in terms of staffing costs and costs associated with training staff and maintaining model fidelity. Success Coach staff were able to offer services to the expansion counties because of their external funding.

Effective communication aids the referral process. The referral process was strengthened by having open, regular communication between the Success Coaches and expansion counties. Staff in the expansion counties were not accustomed to referring families to another child welfare agency for services. When they learned about the Success Coach program, many caseworkers and supervisors in the expansion counties wanted their families to have the opportunity to work with a Success Coach. The Success Coach manager worked with the expansion counties to develop procedures to ensure all eligible families were referred.

With the study occurring simultaneously with program expansion, referring caseworkers wanted to know whether the families they referred were selected to receive the service (i.e., assigned to the treatment group) and, if selected, whether the families accept the service. Some caseworkers initially referred families that were assigned to the control group and not offered services, so they were reluctant to refer more families. Once the Success Coach staff became aware of these concerns, they sought to allay them: Staff began sharing more information about the status of the referred cases, including the families’ group assignments. This response helped strengthen communication between workers in the expansion counties and the Success Coach staff, which in turn strengthened the referral process.

What are key challenges related to the implementation of the Success Coach program and evaluation?

The research team identified several challenges affecting both implementation of the Success Coach program and the evaluation. These include the effect of staff turnover on referral rates, lower than expected rates of reunification, and the impact of the rigorous evaluation design on recruitment.

There were fewer referrals than anticipated. Fewer children/families were referred to the Success Coach program than originally projected. One reason for the low numbers of referrals was caseworker turnover, which was high in all participating counties. As described earlier, program referrals were dependent on caseworkers in the five-county area referring families starting a trial home placement. On average, 56 percent of staff in the participating counties were replaced between the start of the study in 2015 and the end of 2018. As a result of turnover, Success Coaches had to regularly reintroduce themselves and the program to new casework and supervisory staff, and spend time training them in study procedures. Casework staff positions often went unfilled for months, which left the remaining caseworkers with high caseloads and less time per family. Because of these extra demands, they were less likely to take time to invite the Success Coaches to join them on their final visits with families prior to reunification (as a way to introduce the Success Coach to the family and promote the program). They were also less likely to remember to refer families to the program and encourage families to meet with the Success Coaches.

Reunification rates were lower than expected. Local courts were slow to reunify some families, due in part to lack of court personnel, which caused continuations and delays. In addition, we heard from respondents that the complex needs of families necessitated longer trial home placements before judges deemed them stable enough to return child custody to the parents. Success Coaches and caseworkers believed the extended trial home placements were related to an increase in parental drug use disorders, involving opioids and methamphetamines.

Study procedure affected family engagement in services. Some Success Coaches initially reported that the research process (e.g., administering consent procedures) interfered with their ability to engage families. They expressed feeling awkward about introducing the research study, including the possibility of the family having access to the service, which the family would not actually receive if they were assigned to the control group. After discussion among the Success Coaches, management, and evaluators, the Success Coaches became more comfortable with the study process and discussing the study with the referred families.

Take-up of the Success Coach program was slow. Child welfare agencies typically struggle to engage families, who have an inherent distrust of an agency with authority to remove their children. Given that families referred to the Success coach program recently had custody of their child returned to them, it is understandable that some families were not eager to continue their involvement with the agency that removed their child. Success Coaches used many engagement techniques to connect with families. They brought food to visits and toys for the children and took their cues from the family about the style of communication that worked best. They maintained an awareness of cultural, linguistic, or racial barriers that may exist and responded sensitively to the level of comfort families had with accepting a new service. Success Coaches who had more job experience seemed better able to connect with families than those new to the position.

How similar are the Success Coach programs to the “services as usual” received by all children?

In evaluations involving comparison groups, if people in both groups obtain similar services, any effects of the service being studied (i.e., treatment) are likely diluted. In addition to understanding exactly what services and supports Success Coaches provide to families, we need to understand what services and supports are available to families in the control group. Below, we describe what Success Coaches provide and other services available in the participating counties.