Evaluating the Expansion of School-Based Behavioral Health Supports in DC Schools

Evaluating the Expansion of School-Based Behavioral Health Supports in DC Schools is a 5-year project to evaluate the District of Columbia Department of Behavioral Health’s (DBH) efforts to expand access to behavioral health supports in schools via promotion of partnerships between schools and community-based providers. The project has two main components: 1) a mixed methods evaluation of the expansion efforts that included development of a logic model, and 2) technical assistance provided to the DBH team to increase their capacity to engage in continuous quality improvement.

Project background

In 2018, the District of Columbia Department of Behavioral Health and several partners launched a phased approach to increase access to behavioral health supports among students attending four school years of public and public charter schools across the city. Partners included community-based organizations, the Office of the State Superintendent of Education, the District of Columbia Public Schools, and the District of Columbia Public Charter School Board, among others. Child Trends was contracted in 2020 to evaluate the expansion and build capacity to increase DBH’s expertise in collecting and using data to make program improvements.

Approach

Development of an initiative logic model

To guide the evaluation and monitoring of the expansion, Child Trends worked with DBH to develop a logic model that visually depicts the resources needed to expand behavioral health, any activities being implemented to advance this goal, and expected outputs and outcomes—at both the school and citywide levels. Child Trends held listening sessions with partners across the expansion to get feedback on the logic model and ensure that it reflects a shared understanding among the many stakeholders in the district.

Large-scale data collection

To understand the experiences of students, families, school staff, and school behavioral health coordinators, Child Trends developed a set of surveys that were administered annually across four school years. Child Trends researchers worked with school staff to support survey administration and achieved participation from approximately 200 schools each year, from which an average of about 4,500 student surveys, 2,100 staff surveys, 1,300 family surveys, and over 100 school behavioral health coordinator surveys were completed. We also collected additional data during the first year via focus groups with students, families, and school staff, as well as members of the Coordinating Council for School Behavioral Health.

Youth-engaged evaluation

In the evaluation’s third year, a high school intern led focus groups with LGBTQ+ youth to better understand significant disparities observed in the survey data; a conference poster summarizing the focus groups was accepted to the Annual Conference on Advancing School Mental Health and co-presented by the high school intern and the project’s principal investigator.

Continuous quality improvement

In addition to the evaluation, Child Trends provided capacity-building support to the DBH team, helping them strengthen their data infrastructure, establish processes for monitoring data to make program improvements, and make meaning of data and share insights with expansion partners.

Informing policy and practice

Throughout the project, Child Trends has submitted annual reports to DBH that describe implementation and outcomes, including system-level and school-level recommendations. In addition, each school that participated in data collection and achieved a minimum of 10 responses received an individual report with their data as well as the citywide average for schools serving the same grade level for comparison; more than 400 reports were developed over the project period. In the third year of the project, Child Trends also provided DBH with a report outlining several potential program improvements, based on a review of best practices in the field of school-based behavioral health and a series of discussions with key stakeholders. The report was presented to the Council of the District of Columbia to inform legislative and budget priorities.

Intended impacts

Child Trends’ evaluation and capacity-building supports are intended to improve the use of data at both the system and school levels. In the first four years of the project, the DBH team was able to establish a practice of reviewing the activities of school-based behavioral health clinicians on a monthly basis to determine whether expected benchmarks were being achieved, and then regularly reporting those data to the Coordinating Council for School-Based Behavioral Health to guide program improvement discussions. At the school level, teams reported using their data reports to plan and monitor efforts to promote behavioral health in their schools. The final evaluation will include longitudinal analyses to provide DBH and its partners with information about the relationship between program implementation and student outcomes to inform their efforts to sustain and improve the delivery of behavioral health supports in schools.

Timeline

The project period is March 2020 through March 2025. Notably, this project started during the same month in which DC schools transitioned to virtual learning as a result of the COVID-19 pandemic. As a result, much of the first year of the project was spent developing an evaluation plan that reflected the impacts of the pandemic and the pandemic response and recovery efforts.

• Year 1 (March 2020 – March 2021): Start study, including adaptions made to reflect the reality of the COVID-19 pandemic.
• Year 2 (March 2021 – March 2022): Conduct first two rounds of survey data collection in Spring 2021 and Winter 2022 and produce summary report of School Year 2020-2021.
• Year 3 (March 2022 – March 2023): Conduct third round of survey data collection in Spring 2023 and produce summary report of School Year 2021-2022.
• Year 4 (March 2023 – March 2024): Conduct fourth round of survey data collection in Spring 2024 and produce summary report of School Year 2022-2023.
• Year 5 (March 2024 – March 2025): Produce a final report, including longitudinal analyses.

Project team