Schools across the country are beginning new school-based initiatives to address and improve the social, emotional, and academic needs of their students. Whether with the help of the recently funded “Now is The Time” grant programs (Project PREVENT, Project AWARE, and the School Climate Transformation Grants) or developed internally, schools are implementing new programs, collecting new data, and working towards better outcomes. In the process, it is easy to overlook perhaps the most central of questions – how do we know whether the programs that are being implemented are actually improving outcomes? Even the most evidence-based program may be ineffective for certain schools  or groups of students(Bumbarger, Perkins, & Greenberg, 2010).

It is critical, then, to continually evaluate both program implementation and impact. Not all evaluations are useful, however. Here are five things to consider for evaluating new initiatives:

1)      Move beyond required outcomes data

Schools that receive federal funding are required to submit certain metrics to demonstrate their progress under the grant. The Government Performance and Results Act (GPRA) requires federal agencies to collect performance data on its activities and submit reports to the White House Office of Management and Business. Specific GPRA data requirements are included for nearly every federal grant, and are often limited to three or four outcome measures. For example, for the Department of Education’s School Climate Transformation Grant, grantees are required to report on (1) attendance rates, (2) disciplinary referral rates, (3) suspension and expulsion rates, and (4) the number of schools implementing multi-tiered behavioral frameworks with fidelity.

An effective evaluation, however, cannot be limited to just these outcomes. School staff should consider the goals they hope to accomplish and how they think the program will help them achieve those outcomes. Often progress on outcomes such as behavior change may take a long time to accomplish, but shorter term or “proximal” outcomes can give indication of progress.  These shorter term outcomes often include changes in attitudes or knowledge. For example, to reach a school’s ultimate goal of reducing bullying, students’ attitudes about bullying and their willingness to stand up for others must first improve. These processes and goals will look different from school to school, based on the school’s needs and the specific implemented program.

2)      Use multiple, reliable measures

Too often, evaluations limit indicators of program effectiveness to incident data, or counts of reports of certain student behaviors. Although incident data is certainly valuable, it misses the broader context. Incident data only reflects incidents that are reported to the school and cannot capture incidents which are not reported or otherwise not recorded. There is evidence, for instance, that only 1 in 3 youth who have been bullied has reported the bullying (Roebers et al., 2013) and teachers significantly underestimate the amount of bullying in their schools when compared to student reports (Bradshaw, Sawyer & O’Brennan, 2007). Utilizing multiple sources of data, then, helps ensure the validity of findings. Other sources of data such as student and staff surveys, focus groups, interviews, and observations can all be used to supplement incident data.

3)      Consider implementation

Although the most central evaluation question for most schools is whether a program achieved the desired outcomes, along the way it is important to ask whether the program is the right fit for a given school and whether it is able to be implemented successfully. These questions are inherently related – the best outcomes are often only achieved when there is program fit and the program is implemented with fidelity (Bumbarger, Perkins, & Greenberg, 2010). Identifying the factors that both facilitate and hinder successful program implementation is key to improving current outcomes and informing future initiatives. Schools that monitor program implementation on an ongoing basis are better able to quickly identify and address quality shortfalls.

4)      Seek external help

Conducting a valid evaluation requires the ability to be objective about the findings. Partnering with an external evaluator can help schools increase the rigor, objectivity and efficiency of their evaluations, and external evaluations are often required for federal grantees. Schools should select their external evaluators carefully, considering past evaluation experience, subject matter expertise, and the quality of past work. External evaluators should also be able to help translate findings for the easy comprehension of school staff.

5)      Utilize data to make improvements

An evaluation is only useful when it is used to inform both current and future work. Interim evaluation results should be used to make adjustments in program implementation. For instance, if no significant progress is being made on a particular outcome, schools can consider adding or changing content around that outcome. Following the conclusion of the evaluation, the results can help inform whether or not to sustain the program. Program evaluation results can also be used to bolster support for on-going initiatives and demonstrate the need for continued investment.

For more information about Child Trends’ school evaluation work, please see https://childtrends.org/our-research/research-services/or email Deborah Temkin at dtemkin@childtrends.org.