Path to a Rigorous Evaluation

In December 2013, Catawba County undertook an evaluation of the Child Wellbeing Project, including the Success Coach program. Child Trends and the National Implementation Research Network (NIRN) consulted on project design, service development, and program implementation. Independent Living Resources, Inc. conducted a process and preliminary outcomes evaluation. Evaluators found that the presence of a Success Coach in the lives of post-adoptive and post-reunified families appeared to decrease the risk of re-allegations of abuse and neglect and re-entries into out-of-home care. However, due to the study design as well as many co-occurring factors, including low enrollment and families’ receiving services in addition to Success Coach, evaluators were unable to establish a direct association between Success Coach programs and positive outcomes for the families served.

Expanding services

Catawba County was encouraged by findings from the 2013 evaluation that indicated the Success Coach program had potential to increase child and family well-being and reduce repeat maltreatment. The next step in building an evidence base for their program was a more rigorous study, which they initiated in 2015. At that time, to serve more families and increase the sample size for the more rigorous study, Success Coach program staff decided to expand their reach to neighboring counties. Five counties agreed to collaborate with Catawba County by referring families, at the time the child(ren) were returned home (i.e., trial home placement), to the Catawba County Success Coach program. These Catawba County Success Coaches provided services to the families referred by the expansion counties, in addition to families referred by Catawba County. Figure 2 depicts the initial phase, in which only Catawba County families received the service, and the expansion of the service to five neighboring counties. The rigorous design of the 2015 study included an implementation study that would examine the successes and challenges of expanding services to additional sites.

Determining the evaluation design

The Success Coach developers wanted to know whether the program worked for families, and to identify the factors that were instrumental in supporting the implementation and expansion of the Success Coach program. Therefore, Child Trends designed an evaluation that includes both an outcome and an implementation study.

Outcome study. This component of the evaluation examines impacts of the Success Coach program on child safety (as measured by rates of re-allegations of maltreatment and re-entry into care) and child well-being (as measured by parents’ attitudes and access to supportive people and services that promote child and family well-being). To detect any statistically significant impacts on outcomes that might exist, we anticipated needing to enroll at least 200 families into the study. Families would be randomly assigned to either receive Success Coach services (treatment) or to receive “services as usual” (control).

Randomization. To be able to attribute differences in any outcomes we observed between the two groups to the intervention, we carried out a randomized controlled trial. With a sufficiently large sample size, randomly assigning families to either the treatment or control group makes the two groups statistically equivalent in terms of other factors that might influence outcomes of interest. Families’ motivation is an example of such a factor; that is, without random assignment, families who are particularly motivated might be more likely volunteer for the intervention, and this motivation—rather than the intervention itself—might explain more favorable outcomes observed in the treatment group.

The various DSS county offices referred families to the Success Coach program at the point when the child returned home for a trial placement prior to reunification. When families were referred, we randomly assigned them to either the treatment or control group. Success Coaches then approached the referred families, to explain the Success Coach program and the possibility of receiving the service if the family decided to join the study. For families who consented to participate in the study, the Success Coach informed them of their study assignment, and then offered services to those families assigned to the treatment group.

We retained all families who consented to participate in the study in our analytical sample, even if they were assigned to the treatment group and did not take up services. This “intent to treat” design is critical for maintaining the statistical equivalence of the treatment and control groups.

Measures. The outcome study relies on a variety of measures to assess differences between control and treatment families in family well-being, repeat maltreatment, and placement stability. These measures include the following:

A parent survey administered at baseline and every six months for two years. The survey included:

• Mini Child and Adolescent Needs and Strengths (Mini-CANS)
• Protective Factors Survey
• ACES Survey
• Miami Dade Parent Survey

Data from the Success Coach program database, for example:

• Demographic data about the parents
• Number and types of contacts with the Success Coaches

Administrative data from the state child welfare data system, for example:

• Reports of maltreatment by parents
• Outcome of any investigations done in response to these reports
• Re-entry of children into foster care, if applicable, and length of stay in care

Implementation study. This component of the evaluation complements the impact study by providing context and an understanding of factors driving the implementation, both in the Success Coach program and the expansion counties. Factors studied include what supports were present or absent, and how this affected the implementation. The research team also wanted to know what factors influence whether families consent to participate in the study, and, for treatment group families, what influences whether they take up the service. Lastly, the research team needed to understand all other services and supports available in the community that families who have recently been involved with child protective services could access.

Measures. The research team developed focus group and interview protocols to gather information from key Success Coach program staff and county workers. Research team members conducted two rounds of interviews and focus groups, one in September 2017 and one in September 2018. They met with Success Coach program staff, including Success Coaches, program leadership, the data manager, and the education advocate. Additionally, they conducted interviews and focus groups with all participating county staff, including supervisors, caseworkers, mental health therapists, and guardians ad litem (GALs).

Identifying the study population

In participating counties, children removed from their home are typically returned home for a “trial home placement” once the safety concerns that existed are alleviated. After successful completion of the trial home placement, the children are legally reunited with their parents. Success Coaches provide services at this point—at the time custody of the child(ren) is returned to the parent(s). Overall, in North Carolina, 4,756 children were in out-of-home care during FY 2017. Among those children, rates of reunification vary. On average, 39 percent of children were reunified after being in care for less than one year, 35 percent were in care for one to two years, and 25 percent were in care for more than two years. As shown in Table 1, the number of children reunified with their parents across the six counties being served increased from 2010 to 2014. While there were some increases in individual counties, across the six counties reunification rates remained stable from 2014 to 2015 and decreased from 2015 to 2017 (our study period).

Parents were predominantly female (64%) and white (73%). The largest group of parents were single (43%), but a third were married (34%). Over half (57%) had a high school diploma, GED, or less, and almost two-thirds were employed (60%). At baseline, families in both treatment and control groups reported similar needs and strengths (mini-CANS), similar protective factors, ACES scores, and community supports. There were a few significant differences: control families reported higher family functioning/resiliency (p = 0.017) than treatment families. They were more likely to believe their child misbehaved just to upset them (p = 0.036) compared to treatment families.We started enrolling families in the study in October 2015 with counties referring families with a child on trial home placement to the Success Coach program. As of January 2018, 168 families were referred by the six counties: 30 families, with 49 children, were assigned to the control group and 23 families, with 42 children, to the treatment group. Parents completed paper-and-pencil, self-administered surveys (Protective Factors, ACES, and Miami Dade Parent) during the initial visit (baseline data). These measures were also collected at 6, 12, 18, and 24 months after the initial visit. Caseworkers completed the Mini-CANS at the point of reunification, and identified areas where families needed support as well as strength-building in order to help the Success Coaches begin working with the parents on a treatment plan.

Parents in both groups reported they experienced more than three adverse childhood events. They rated their communities as safe places for their children, where people usually watch out for each other’s children, and they usually have people on whom they can rely for support.

At baseline, there were no significant differences between treatment and control families on the Mini-CANS. Caseworkers rated most families as having a mixture of needs and strengths. They rated families strongest in the following areas: being a support for the child, dealing with the child’s reaction to trauma, helping children feel comfortable with their cultural identity, being able to manage household activities, having strong family relationships with good communication among family members, helping the child function well in their living arrangement; and in the family in general. The areas in which caseworkers rated most families needing at least some help included building social assets, knowing about their child’s strengths and problems, planning for and providing mental health or other services needed by family members, monitoring and disciplining their child, developing a sense of belonging in a cultural group, and ensuring the child has a positive school experience.

---

Early Implementation Findings

As mentioned earlier, the implementation study complements our rigorous outcome study. While impact findings from random assignment designs require analyses after the accumulation of the sample is complete, preliminary findings from implementation studies can be disseminated far earlier. Below we discuss findings related to our implementation evaluation questions.

What key factors support expansion of the Success Coach program?

We examined factors that support expansion of the Success Coach program, including whether public agencies are best suited to delivery of the program during its expansion, and whether private funding is needed to expand post-reunification services. In addition, we describe challenges and facilitators to the referral system during expansion of services.

Success Coach, both within their own jurisdiction and when expanding to neighboring jurisdictions. Success Coach staff believe the service is best provided by a child welfare agency and should not be outsourced to private agencies or small counties. They were concerned that small counties would be unable to support the practice effectively, and that private service providers in this region would not have the in-depth knowledge of the child welfare system needed to work with parents whose children have been in state custody due to substantiated allegations of maltreatment.

There is a need for private funding. Private funding was needed to support and expand post-reunification services. Success Coach staff reported that in many communities, a post-reunification program cannot be supported without grant funding because it is a non-mandated program with no state funding available. Also, it is a resource-intensive program, both in terms of staffing costs and costs associated with training staff and maintaining model fidelity. Success Coach staff were able to offer services to the expansion counties because of their external funding.

Effective communication aids the referral process. The referral process was strengthened by having open, regular communication between the Success Coaches and expansion counties. Staff in the expansion counties were not accustomed to referring families to another child welfare agency for services. When they learned about the Success Coach program, many caseworkers and supervisors in the expansion counties wanted their families to have the opportunity to work with a Success Coach. The Success Coach manager worked with the expansion counties to develop procedures to ensure all eligible families were referred.

With the study occurring simultaneously with program expansion, referring caseworkers wanted to know whether the families they referred were selected to receive the service (i.e., assigned to the treatment group) and, if selected, whether the families accept the service. Some caseworkers initially referred families that were assigned to the control group and not offered services, so they were reluctant to refer more families. Once the Success Coach staff became aware of these concerns, they sought to allay them: Staff began sharing more information about the status of the referred cases, including the families’ group assignments. This response helped strengthen communication between workers in the expansion counties and the Success Coach staff, which in turn strengthened the referral process.

What are key challenges related to the implementation of the Success Coach program and evaluation?

The research team identified several challenges affecting both implementation of the Success Coach program and the evaluation. These include the effect of staff turnover on referral rates, lower than expected rates of reunification, and the impact of the rigorous evaluation design on recruitment.

There were fewer referrals than anticipated. Fewer children/families were referred to the Success Coach program than originally projected. One reason for the low numbers of referrals was caseworker turnover, which was high in all participating counties. As described earlier, program referrals were dependent on caseworkers in the five-county area referring families starting a trial home placement. On average, 56 percent of staff in the participating counties were replaced between the start of the study in 2015 and the end of 2018. As a result of turnover, Success Coaches had to regularly reintroduce themselves and the program to new casework and supervisory staff, and spend time training them in study procedures. Casework staff positions often went unfilled for months, which left the remaining caseworkers with high caseloads and less time per family. Because of these extra demands, they were less likely to take time to invite the Success Coaches to join them on their final visits with families prior to reunification (as a way to introduce the Success Coach to the family and promote the program). They were also less likely to remember to refer families to the program and encourage families to meet with the Success Coaches.

Reunification rates were lower than expected. Local courts were slow to reunify some families, due in part to lack of court personnel, which caused continuations and delays. In addition, we heard from respondents that the complex needs of families necessitated longer trial home placements before judges deemed them stable enough to return child custody to the parents. Success Coaches and caseworkers believed the extended trial home placements were related to an increase in parental drug use disorders, involving opioids and methamphetamines.

Study procedure affected family engagement in services. Some Success Coaches initially reported that the research process (e.g., administering consent procedures) interfered with their ability to engage families. They expressed feeling awkward about introducing the research study, including the possibility of the family having access to the service, which the family would not actually receive if they were assigned to the control group. After discussion among the Success Coaches, management, and evaluators, the Success Coaches became more comfortable with the study process and discussing the study with the referred families.

Take-up of the Success Coach program was slow. Child welfare agencies typically struggle to engage families, who have an inherent distrust of an agency with authority to remove their children. Given that families referred to the Success coach program recently had custody of their child returned to them, it is understandable that some families were not eager to continue their involvement with the agency that removed their child. Success Coaches used many engagement techniques to connect with families. They brought food to visits and toys for the children and took their cues from the family about the style of communication that worked best. They maintained an awareness of cultural, linguistic, or racial barriers that may exist and responded sensitively to the level of comfort families had with accepting a new service. Success Coaches who had more job experience seemed better able to connect with families than those new to the position.

How similar are the Success Coach programs to the “services as usual” received by all children?

In evaluations involving comparison groups, if people in both groups obtain similar services, any effects of the service being studied (i.e., treatment) are likely diluted. In addition to understanding exactly what services and supports Success Coaches provide to families, we need to understand what services and supports are available to families in the control group. Below, we describe what Success Coaches provide and other services available in the participating counties.

Other available services. There are few other services and supports available to families in the participating counties, and none as comprehensive as those available through the Success Coach program. Caseworkers typically have very little or no involvement with families once they are reunified with their children, beyond giving them information about what services and supports exist in the community. Occasionally, after reunification has occurred, a family may call a caseworker to ask for assistance, at which time the caseworker may make a referral if a service is available, but most caseworkers reported there are few options available.

During interviews and focus groups, respondents mentioned the following resources were available to families involved in the child welfare system: Iredell Stop Child Abuse Now,[12] the Children’s Home Society,[13] drug treatment programs, mental health services, and several churches that have food pantries and offer counseling to families. Catawba County offers the Family NET program,[14] a unique mental health service that bridges the gap between trial home placement and post-reunification. The program is co-located with Catawba County Social Services, which is very convenient for families. Therapists are also on site to provide support and guidance to the Success Coaches.

---

Implications for Implementation and Evaluation

Establishing an evidence base for a program model can be challenging. We learned several important lessons that may be useful to the child welfare field as program developers and evaluators continue to develop and implement new program models and evaluate their efficacy.

Revisit the service model and manuals when expanding to new sites

Program managers may choose to expand their service area for a number of reasons. One reason is to reach more families with a promising service; another is to ensure a large enough sample to evaluate program outcomes. Although the Success Coach program had a comprehensive practice manual and training and mentoring model, the information included did not cover outreach to expansion counties. Program managers expected Success Coaches to consistently reach out to staff in the expansion counties, and they regularly discussed progress on this task during performance practice reviews. The county learned that program expansion can alter the service delivery dynamic, such as by increasing the need for additional outreach to expansion counties. Program managers may need to revise practices and procedures to accommodate these changes.

Be aware that context affects the ability to evaluate program outcomes

Child welfare agencies often struggle with worker recruitment and retention, and the counties participating in this study were no exception. In addition, during the study period, the total number of children reunified in the participating counties decreased, reducing referrals to the program. This, in turn, reduced enrollment of families into the study. Success Coaches had to work hard to overcome these obstacles, and it took time to develop procedures to consistently reach out to new staff in each county, ensure the new workers understood and supported the study, and remembered to refer eligible families. Even though Success Coach managers were aware of these challenges, they may have underestimated how the expansion into additional counties would magnify the challenges. As the expansion progresses, reviewing and revising what is understood about the context of the programs may be helpful for making course corrections when needed.

Family engagement requires staff with specialized skills

Despite their continued need for services and supports post reunification, families are reluctant to engage in services for many reasons. Success Coaches had to use all their skills and expertise to gain the trust of families. The more experienced Success Coaches (those who had worked in the field for a longer period of time) better understood the challenges that families faced. They were also more skilled at ascertaining what was important to each family and tailoring their message accordingly. Social workers with more practice experience may be better able to promote a service to families, so staffing decisions are especially important.

Agencies need to secure funding and support from key partners

Child welfare agencies are required to provide mandated services, frequently on limited budgets, with no funding available for voluntary services provided after the child returns home. Catawba County leveraged its use of private funding to fill this gap, which allowed them to offer an innovative, voluntary program. Even with funding, child welfare agencies that want to expand their services into new jurisdictions need the support and buy-in of potential partners. Partner engagement and buy-in is also essential for evaluation purposes. The requirements of a rigorous study, one involving randomization of families, necessitate successful outreach and education regarding the importance of random assignment in establishing an evidence base for a program. Rigorous designs rely on adequate sample sizes, further heightening the importance of a steady source of referrals. Consistent, regular contacts with referral sources to establish and maintain positive relationships is important for identifying eligible families and encouraging them to consent to the service and the study.

---

Conclusion

What we learned from the process of building evidence for the Success Coach program can provide broader lessons for the child welfare field. The expansion of an existing program while evaluating program outcomes is a complex process and can be affected by many factors. Understanding the service model, the context in which the service is offered, and the characteristics of the staff delivering the service (such as level of experience and training received) can provide insight into whether implementation is successful. Planning and forethought related to potential challenges and how to address them, along with perseverance and tenacity (such as regular and persistent outreach to referral sources), can increase the chances of successful implementation, expansion, and evaluation of a program.

---

End Notes

[1] http://www.ncsl.org/research/human-services/family-first-prevention-services-act-ffpsa.aspx

[2] Redmond, P., Ball, H., Wilson, W., Brandes, B., Metz, A., & Malm, K, (2012) Brief #1: Implementing a Post-Care Service System in Child Welfare: The Catawba County Child Well-Being Project. Child Trends, Bethesda, Maryland.

Metz, A., Blasé, K., Wilson, D., Redmond, P., & Malm, K. (2012). Brief #2: Using implementation science to support and align practice and system change: a case study of the Catawba County Child Wellbeing Project.  Child Trends, Bethesda, Maryland.

Wilson, D., Brandes, B., Ball, H., & Malm, K.  (2012). Brief #3: Building the Child Wellbeing Project: Practitioners’ perspectives on the role of implementation science in strengthening post-care child welfare services. Child Trends, Bethesda, Maryland.

[3] Adoption and Foster Care Analysis and Reporting System (AFCARS) FY2017data https://www.acf.hhs.gov/sites/default/files/cb/afcarsreport25.pdf

[4] Carnochan, S., Rizik-Baer, D., & J. Austin, M. (2013). Preventing Re-Entry to Foster Care. Journal of Evidence Based Social Work, 10: 196-209.

[5] Homebuilders serves families with children (birth to 18) at imminent risk of placement into, or needing intensive services to return from, foster care, group or residential treatment, psychiatric hospitals, or juvenile justice facilities. http://www.cebc4cw.org/program/homebuilders/

[6] Carnochan, S., et al. (2013).

[7] Kirk, R. S. & Conner, N. W. (2013). Catawba County Child Wellbeing Project Final Report. Durham, NC, Independent Living Resources.

[8] When Success Coach began, Success Coaches served families with guardianship and legal custody arrangements. They no longer serve these families.

[9] For more information see the Child Wellbeing project website http://www.postadoptionsuccesscoach.org/

[10] Carnochan, S., et al. (2013).

[11] Duncan, D. F., Stewart, C. J., Vaughn, J. S, Guest, S., Rose, R. A, Malley, K., and Gwaltney, A. Y. (2018). Management Assistance for Child Welfare, Work First, and Food & Nutrition Services in North Carolina (v3.21). Retrieved [month day, year], from University of North Carolina at Chapel Hill Jordan Institute for Families website. URL: http://ssw.unc.edu/ma/

[12] SCAN provides parenting education and counseling programs, both in the home and in the office https://www.iredellscan.org/programs.html

[13] The Children’s Home Society offers parent education programs. https://www.chsnc.org/

[14] http://www.catawbacountync.gov/county-services/social-services/family-net-clinical-services/about-family-net/