Recently, nearly 400 people gathered in Washington D.C. for a national meeting – sponsored by the Build Initiative with funding from the Alliance for Early Success and other partners – to discuss and plan for the next generation of Quality Rating and Improvement Systems (QRIS). The participants included state, territory and federal administrators, QRIS leaders, staff from organizations working directly with early care and education programs, and QRIS researchers who gathered to take stock of the past, present and future of QRIS. Many states and territories are still in the early years of designing or piloting a QRIS, while others are making revisions to QRIS that have been operating for more than five years. Attendees discussed challenges and opportunities in QRIS implementation and offered lessons and recommendations from QRIS practice and research.

As QRIS researchers, we have always appreciated the openness of the QRIS community to new evaluation findings, new strategies, and new methods for increasing the effectiveness of the work. Discussions in the recent meeting focused on using evidence and implementation science to improve the techniques used to measure and rate quality, to support early care and education programs in improving their quality, and to disseminate information to parents and other consumers. QRIS administrators and staff are seeking ways to make adjustments or significant changes to their QRIS, and they are looking for guidance from research and practice to make these decisions. They also recognize the resource and practical constraints that characterize work in early care and education.

In this spirit of learning and improving, we reviewed a recent Science article on QRIS to ask how the findings contribute to our understanding of QRIS design and implementation. Briefly, the Science article addressed the question, “Can pre-kindergarten program ratings predict children’s learning?” The authors addressed this question using existing data from a multi-state study of pre-kindergarten in 2001-2004. Although the data were not collected in a QRIS context, the dataset is one of the few large ones available that include both quality indicators and measures of children’s development. The authors combined data on staff qualifications (pre-k teachers and administrators), classroom environment, family partnerships in the pre-k classroom, as well as pre-k ratio and group size to mimic the QRIS ratings from nine states. They also developed a generic QRIS that included an additional quality indicator for teacher-child interactions. They then examined the relationship of the overall ratings and the individual indicators with measures of children’s academic and social skills. The authors concluded that ”on most measures of children’s learning, programs rated high by QRIS produce outcomes that are not significantly better than those of low-rated programs” (p. 845), and that the measure of teacher-child interactions was most strongly associated with children’s learning. The use of secondary data analysis is an innovative strategy, and the results of the study have already spurred much fruitful discussion about the implications and future research needs for better understanding QRIS.

What should we consider when reviewing research about QRIS and what can – and can’t – we conclude about QRIS based on this new study? The first two points below focus specifically on the Science article; the others apply more generally to QRIS research.

Limitations of the Science study sample and research strategy make it difficult to draw broad conclusions about the effectiveness of state QRIS.

The Science article addresses a central aspect of QRIS – the individual measures of quality components that are combined to produce a quality rating for a program – and the extent to which they correlate with measures of children’s learning and development. When interpreting the results, it is important to consider the study context (a research study of pre-kindergarten programs, not QRIS-rated programs, conducted nearly a decade ago) and to recognize that these data were not collected for the purpose of producing quality ratings. It is also important to remember that pre-kindergarten programs represent only a subset of programs typically participating in QRIS. We look forward to future studies that include QRIS-rated programs and use designs and methods that can address questions of QRIS effectiveness and outcomes. For example, the Race to the Top—Early Learning Challenge grant funds will likely yield over the next few years several datasets to address QRIS-relevant research questions.

Observational measures of quality are associated with measures of children’s development and learning.

The Science article highlights the potential importance of including in QRIS observational measures that can capture elements of interactions and activities to support play and learning. Consistent with previous research, Table S3 in the Science Supplementary Materials suggests that the Early Childhood Environment Rating Scale-Revised (ECERS-R), a global measure of observed classroom quality, and the Classroom Assessment and Scoring System (CLASS), a measure of interaction quality, were similarly related to children’s academic/language skills. The CLASS was also related to children’s social-emotional skills.

Defining and measuring quality accurately and efficiently is challenging.

State and territory QRIS stakeholders make choices about what components of quality to measure and rate in a QRIS. The ECERS-R has been used for years by states in their QRIS to promote improvements in global quality; use of CLASS is also increasing. Although these observational data are expensive to collect, they are often included because of their demonstrated relationship with children’s learning. The current array of quality measures needs further testing in a QRIS context to evaluate the benefits, challenges, and best practices for including observational measures to rate all or a subset of QRIS programs. States are also looking for guidance on effective protocols to verify the use of appropriate curriculum and assessment strategies that can support individualized instruction for children. These dimensions of quality are not captured fully in the commonly-used, aggregate measures that look across the children in a classroom or group setting. While there is much to build on, further work on QRIS measures is warranted.

As a framework to support multiple activities aimed at quality improvement, QRIS may promote children’s learning through direct and indirect paths. Other desired outcomes are also valuable to articulate so that expectations for QRIS activities can be aligned appropriately.

We agree with Linda Smith, Deputy Assistant Secretary of Early Childhood Development, Administration for Children and Families, Department of Health and Human Services, and others that QRIS is a framework to support quality improvement in early learning and development programs. In a system strategy like QRIS, multiple outcomes for children, families, and the early care and education workforce may be possible if sufficient resources and supports for implementation are available. QRIS-related activities such as individual coaching that directly promote improvements in teachers’ and caregivers’ interactions with children may link to measures of children’s learning. QRIS activities that facilitate health and safety provisions in programs (e.g., through enhanced ratio requirements) or provide supports for improving staff qualifications may be necessary in states with less stringent licensing provisions that are using a QRIS to raise the threshold for basic quality provisions in programs caring for young children. Lower rates of children’s accidents and injuries and increases in staff knowledge of best practices are important intermediate outcomes to track when immediate, direct links to children’s learning would not necessarily be predicted. At the system level, QRIS outcomes such as improved coordination and increased efficiency among agencies that support early care and education programs are also valuable to states as they may lead to increased resource availability and the opportunity to serve more children in high quality programs.

QRIS models are diverse. Some QRIS approaches may work better than others.

There is not one QRIS model. Though some QRIS infrastructure may look alike across states, currently QRIS have as many differences as they do similarities (see the QRIS Compendium for details). New research examining and comparing QRIS models will be important to build evidence about what works well (and less well) for particular programs and populations. This comparative research must take into account the unique contexts and resources available across diverse state QRIS.

Taking a long-term perspective, QRIS is a relatively new policy lever—and research about QRIS measures and effectiveness is just beginning to emerge.

The Race to the Top – Early Learning Challenge will help generate more research by requiring and funding state QRIS validation studies that examine how well QRIS levels differentiate program quality and are related to measures of children’s development and learning. These studies should be conducted carefully with attention paid to the characteristics of participating programs and families that may limit the findings and with clear guidance provided to stakeholders about the conclusions that can be drawn. We encourage states and other funders to invest not only in validation studies but also evaluations of the implementation of QRIS and the effectiveness of quality improvement strategies as well as research to strengthen the various measures included in QRIS. Through our work with the Quality Initiatives Research and Evaluation Consortium (INQUIRE) funded through a contract with the Office of Planning, Research and Evaluation (OPRE), we will continue to collaborate with other researchers, policymakers and practitioners to build the QRIS evidence base. It is impossible for any one study to provide all the answers—but a collection of research will enable us to learn more about QRIS and support states’ efforts to continually improve the quality of programs for young children. We’re glad to be part of the QRIS community that seeks to improve, learns from research, and tests innovative approaches to support young children and their families.