Abriendo Puertas/Opening Doors Parenting Program: Summary Report of Program Implementation and Impacts
Culture Counts: Engaging Black and Latino Parents of Young Children in Family Support Programs
DataBank: Reading to Young Children
Early literacy skills are the foundation for school success. This is particularly important for groups of children at heightened risk of poor educational outcomes, such as English language learners and children from low-income families. Informed by a growing body of research and evaluation studies that point to the importance of home literacy habits, some early literacy programs have increased their focus on parenting practices that support children’s academic success. Although the evidence base for these programs is promising, large-scale experimental studies to comprehensively assess whether they work across multiple settings and with different groups of children are essential next steps.
This report examines the case of Raising A Reader (RAR), which has been steadily building its evidence base over many years and is now positioned to undertake such a comprehensive evaluation. RAR is a national nonprofit literacy organization which, through work with direct service agencies, helps develop sustainable home literacy routines essential to language and literacy development. Currently active in 34 states, the program reached more than 130,000 children and their families in 2013. RAR’s programming is grounded in a large body of research that substantiates the critical role that parents play in nurturing their children’s educational success through home-based reading practices. Raising A Reader has for many years informed its program planning and expansion with independent evaluation studies. These evaluations across a wide range of implementation settings found that families participating in RAR increased their home literacy behaviors and children improved their reading skills. These useful but small-scale studies led the organization to prepare for taking the next steps aimed at looking comprehensively at the RAR program through a random assignment evaluation, the most rigorous method of evaluation research, often called a randomized controlled trial or impact evaluation.
Raising A Reader is working with Child Trends, an independent evaluator, to use research to continue to inform and improve its program and to prepare for a large-scale impact evaluation in typical RAR settings. This report offers a case study that may be informative to other programs in the parent engagement/early literacy field that seek to use a variety of informal and formal data effectively for continuous quality improvement and to build their evidence base. The report first summarizes the research base for family literacy programs and the emerging evidence base for RAR. It then describes the RAR program and how it uses a variety of data to regularly improve its program, inform programming integrity, and prepare for a randomized controlled trial. In closing, the report addresses the lessons for the broader field.
Why Early Literacy Matters
In the United States, education is the surest route to economic and life success. Research finds that educational success depends in part on the early development of literacy skills.1 For example, kindergartners who enter school with basic knowledge of reading and math are more likely to fare well in elementary school,2 and students’ grades in elementary and middle school are good predictors of school completion.3
By the end of third grade, reading proficiently is critical since children rely on those reading skills in later grades to learn and to integrate information across content areas.4 This is the stage when children move from “learning to read” to “reading to learn.” Unfortunately, two thirds of all children, 80 percent of low-income children, more than 80 percent of black and Latino children, and 93 percent of English language learners are not reading at grade level by the end of third grade.5
Parents Play a Central Role in their Children's Early Literacy Development
One of the ways to significantly increase the likelihood that children will be proficient readers by the end of third grade is to equip parents with the skills and materials they need to help their children develop the language and literacy essential for reading acquisition.6 Research finds positive effects when children are read to at least three times per week,7 and a large body of existing research finds that home-based literacy practices, such as shared reading routines and direct teaching of reading and writing skills, contribute to children’s developing literacy and other positive academic and social-emotional outcomes. Further, a growing body of research evaluations point to the success of programs that teach parenting skills, including the establishment of home literacy habits, to promote their children’s literacy development.
Family Engagement—The Evidence: A recent review of research conducted over the past 10 years confirms that family engagement in home-based literacy activities, such as interactive book sharing, positively affects children’s literacy skills.8 The studies show links between parental engagement in children’s literacy, and improved vocabulary, listening comprehension, and decoding skills,9 as well as more advanced reading ability, spelling, and comprehension.10 Home-based literacy practices are also positively associated with social-emotional skills, such as increased motivation to learn, and attention to, and persistence with, difficult tasks.11 The specific practices that support these skills include reading aloud to the child, engaging the child in a conversation about the story, asking the child to connect stories to pictures and to predict events, and calling attention to the words on the page.12 The link between parent engagement and child outcomes related to a developing skill reflects a basic tenet of child development: the parent-child relationship and the family context are critically important for children’s developmental outcomes.13
Family Engagement Programs—The Evidence: Family engagement intervention programs have responded to the strong research supporting the importance of early literacy and the key role that parent-child book sharing routines play in developing reading skills. Results from four systematic reviews of multiple research studies found that parent engagement programs designed to increase the quantity and quality of home-based shared reading practices can improve children’s literacy skills.14 The interventions reviewed were typically intensive, for example, offering parents nine training sessions15 or as many as 25 intervention sessions over the course of one year.16 Although the literature on less intensive programs is only now emerging, researchers are finding that lower dose, and more affordable programs may also impact literacy outcomes.17
What is Raising a Reader?
Raising A Reader is a family engagement program that is grounded in basic research on child development and draws on the growing evidence base that establishes the need for and effectiveness of early literacy programs. However, RAR differs from many of these programs in important ways. First, it focuses on the role families play in language and literacy development rather than offering direct training or instruction to children. Second, it is intentionally designed to extend and enhance existing early childhood programs. The program is adaptable and applicable across a wide variety of settings and promotes and facilitates the involvement of parents.
“Even if [I] don’t want to, [my child] grabs the bag and says ‘Mommy! Mommy! Come read to me.'”
“I [didn’t] read to my child before, but now my son is the one that gets the book and tells me to read with him, so we sit down and read together. He is now used to it, so now he wants to read a book every day”
-Parents comments on the child’s role in RAR, made during Child Trends focus groups
RAR’s program is designed to help families who may otherwise lack the knowledge, skills, or resources to develop home-based book sharing routines. RAR works with schools and other affiliates throughout the country to deliver the program. Parents participate in a series of training sessions aimed at building awareness and understanding of the importance of their role and the value of ‘parent-friendly’ research-based home literacy practices. Each week, teachers send their students home with a bright red RAR bag filled with high-quality, multicultural, developmentally appropriate books. The program relies on children’s excitement about books and their natural interest in them as a motivating factor leading to requests of caregivers to share books with them. In order to promote the continuity of these routines beyond the life span of the program, RAR connects families to school or public libraries to reinforce and help them maintain family literacy behaviors.
Parent training workshops are essential to the program. Local affiliated RAR partners offer between two and five workshops which focus on the importance of book sharing for brain and language development. Further, the workshops help adult caregivers learn and practice specific evidence based techniques for book-sharing routines, such as asking interactive questions and making connections between stories in the books and the child’s life. Additionally, each bag includes a DVD, available in fifteen different languages, that reinforces ideas communicated to parents during training, and promotes the value of these literacy practices in supporting their child’s success. Paired with the training, the books that children take home each week provide families with the resources to practice the strategies learned in the workshops and develop the home literacy routines essential for language development and literacy acquisition.
Evaluation Findings Build an Evidence Base and Inform and Improve the Program
Since the program’s inception in 1999, RAR has developed and shared with its local partners a standard set of assessment tools to evaluate whether the RAR program is being implemented with fidelity, and whether the RAR program is influencing parents’ shared reading behaviors. Between 2000 and 2014, RAR and its funders commissioned numerous small-scale evaluations of RAR programs in 16 locations across the country. These evaluations were conducted independently of one another, measured different outcomes, and used different methodologies.
Child Trends reviewed 21 evaluations of the RAR program, one of which was a random controlled trial. Taken as a whole, the evaluation findings constitute a strong emerging evidence base for RAR’s effectiveness in improving parent-child reading activities and behaviors and for improving children’s reading skills. These evaluations have also provided useful information to Raising A Reader as it continues to refine aspects of its intervention to increase program effectiveness.
Of the twelve pre- and post-test designs, eleven found significant positive changes in parents’ home-based reading behaviors. Together, these 11 evaluations found that parents who completed the RAR program were more likely to share books with their children more frequently, more likely to have established a reading routine, and had an increased awareness of the importance of reading with their children. Two of these evaluations also found that parents reported positive changes in their children’s reading behavior. For example, following participation in a RAR program, parents reported that their children were more likely to ask questions, and more likely to turn the pages in a book while reading with the parent.
Eight evaluations also compared the literacy skills of Raising A Reader participants with non-participants by using existing educational data from standardized assessments of Pre-K students, and six reported significant effects on children’s skills such as print and book knowledge and story comprehension. All of these evaluations found a significant effect on parent reading behaviors, including increased frequency of sharing books with their children.
In 2014, the W. K. Kellogg Foundation funded a randomized control trial of RAR that compared the literacy skills of children whose families had participated in a traditional RAR program with the skills of children whose families participated in an enhanced version of the program. The study also included a control group that had no exposure to RAR. This kind of randomized control experimental study is considered the “gold standard” for evaluating an intervention’s impact. The study found that a version of RAR’s program, augmented with five parent training workshops, had significant impacts on measures of oral language skills compared with RAR’s then-standard program with only two parent training workshops.18 Subsequent to this evaluation finding, RAR amended the program to offer the five workshop model.
Similarly, Raising A Reader considers program suggestions from the field, and then examines the evidence to support these recommendations. Such was the case that led RAR to expand the program’s focus, from children birth to age five to children birth to age eight. RAR conducts an annual affiliate survey to inform such decisions, and the 2010 survey found that many affiliates supported expanding the RAR program from kindergarten into early elementary school. This was consistent with research on family engagement and shared reading interventions which suggested that RAR could continue to be beneficial for children in the early elementary grades.19 In fact, approximately 40 percent of U.S. children between three and six years did not attend preschool in 2012, and between 50 and 60 percent of children in families living in poverty, or with mothers with low educational levels, had not gone to preschool.20 Such families are much less likely to read to their children three or more times a week. Offering RAR in the early elementary grades, therefore, reaches a population that otherwise may not have access to any preschool or early elementary family literacy programs.
“Before, I didn’t know how to read a picture book, so now we do that, and create a story based on the pictures…”
“I learned to ask questions when we read. I learned to ask questions about things that rhyme… It makes reading more exciting, more fun.”
“I don’t know English very well. Now I make the effort to read in English. My son also helps me to pronounce words correctly. I didn’t do that before because I was ashamed. I feel more comfortable now.”
-Parents comments on the child’s role in RAR, made during Child Trends focus groups
In summary, although the earlier evaluations of the RAR program varied in rigor, the findings provide preliminary evidence of RAR’s effectiveness and important information about specific aspects of the program. The pre-post design studies and the quasi-experimental studies suggest that the program has potential for improving parents’ booksharing practices. Several studies also suggest RAR’s effectiveness in improving children’s reading development. The 2014 randomized controlled trial provided the strongest evidence for the program’s theoretical model. As the trial was conducted in high-quality early childhood development centers that are not typical RAR settings, care must be taken not to generalize those findings to all RAR programs around the country. The types of settings more typical for RAR are educational environments with varying levels of quality and resources, and serving a range of populations. Overall, however, prior studies indicate RAR’s readiness for a more comprehensive, substantive assessment of its programs in order to better understand their potential impact across diverse populations and the more typical settings in which RAR operates.
Preparing for a Large-Scale Impact Evaluation
In partnership with Child Trends, Raising A Reader is preparing for a large-scale comprehensive examination of its program—a multi-site random assignment evaluations study. Previous small-scale evaluation studies helped inform aspects of the RAR program and shed light on the need for a more rigorous study to examine the effectiveness of RAR across its multiple settings. This study will provide key information to guide further program improvement. For example, the findings of such a rigorous study could help guide RAR’s program’s plan for scaling up for maximum impact, while keeping the program affordable—a conundrum facing many programs designed to enhance children’s development and well-being.
The preparation for a large-scale, multi-site evaluation entails assessing the quality of Raising A Reader’s operational model in ways that would improve the ability of the study to determine the program’s impact. Towards this end, Child Trends undertook two evaluations of the RAR program in 2012. One of these studies, funded by The Annie E. Casey Foundation and The Masons of California and conducted in the San Diego Unified School District, focused on RAR programs in elementary schools. In addition to RAR’s new school-age expansion, the study also investigated the challenges of family literacy programs serving a dual-language (English and Spanish) student population. The project included a small pilot study in three schools to assess random assignment procedures and examine family literacy practices in kindergarten classrooms.
The second Child Trends evaluation of RAR in 2012 was funded by the Department of Education as part of RAR’s Innovative Approaches to Literacy (IAL) award. The project was primarily an implementation study of a large scale up of RAR in three locations: Phoenix, the greater Boston area, and Baltimore. As part of the IAL study, Child Trends worked with the RAR national office to refine its performance measures, improve its data management system, and establish measurable goals to assess affiliate program fidelity. Research has found that multiple evaluations of the same programs often find different program effects, and that these variations may be attributed to differences in the populations served and in the quality of the program on the ground. Therefore, given the diversity of program sites, it is critical that RAR’s performance measures are designed to account for such variables as the program scales up.
The partnership between Child Trends and Raising A Reader includes key elements for the advancement of both RAR and the field of early literacy programs. Guided by data and observations of RAR programs in multiple sites, Child Trends is helping RAR prepare for a large-scale, random assignment study and to take its program to scale with integrity. Working with RAR offers Child Trends ongoing access to the dynamic and challenging realities of program implementation, which has deepened Child Trends’ inquiry around rigorous yet practical evaluation approaches. This level of engagement is a critical contribution to strengthening the ‘bridge’ between research and practice.
By working together, the Child Trends studies have prepared RAR for a rigorous impact evaluation through:
- Piloting a Randomized Controlled Trial in San Diego: The San Diego pilot study took place in three elementary schools in the 2013–2014 school year and was designed to test procedures for conducting a random assignment study, such as the feasibility of randomly assigning schools to RAR or to the control condition, and incentives for teachers to get parents to return surveys both at the beginning and at the end of the school year. The study was also intended to ensure that program implementation was sufficiently strong to conduct a larger study in an urban school district. Importantly, unlike the randomized controlled study (Anthony 2014) that compared children’s literacy skills, the San Diego study compared family reading practices, since RAR’s theoretical model assumes that practices in the home should change prior to seeing changes in children’s literacy skills. The findings were promising, both with respect to parents’ participation in activities to learn book sharing skills, and in their reported use of those skills.
- Exploring Strategies to Increase Parent Attendance: Parent engagement programs face the common challenge of ensuring parent attendance in activities. RAR is no exception. In focus groups with parents who attended RAR workshops, parents reported how the program helped them learn specific behaviors related to reading. But an examination of RAR’s administrative data found that in the three RAR sites funded by the Innovative Approaches to Literacy award— Baltimore, Massachusetts, and Phoenix—attendance at parent workshops on shared reading practices was generally below 40 percent during the 2013–2014 school year, numbers that are consistent with enrollment and attendance data for parent engagement programs in general.21 In response to these findings and to reports from the field, RAR continues to search for additional ways to reach parents. While face-to-face workshops may be desirable, and attendance can be increased through pairing workshops with existing events and providing child care and food, the organization is also exploring a wide range of innovative approaches to teach and reinforce shared reading practices and home literacy routines.
- Measuring Program Performance: When the evaluation for sites funded under the Department of Education’s IAL award began, RAR’s monitoring process permitted the organization and its partners to examine progress in parents’ self-reported reading practices. While helpful, this approach did not necessarily indicate whether the progress is meaningful. In addition, parents reported to RAR’s local partners that they felt the original survey was too long. To determine meaningful progress, Child Trends first worked with RAR to shorten the survey and focus it on specific family literacy practices linked with success in reading. Then, when data from the IAL sites became available, Child Trends and RAR began to define success by identifying the type and degree of changes in home literacy practices that RAR would like to see parents achieve after participation in the program. Relying on research about family literacy practices and RAR’s experiences in the field, meaningful success was defined as parents using multiple reading practices—such as asking children questions about the story while reading books—in combination with reading regularly to children for 20 minutes, a minimum of 3 times a week.
Having defined success for families, RAR and its partner agencies can use the parent survey data to assess the proportion of families who achieve success. With that information, they can identify specific RAR program locations that may need additional supports. The next phase of RAR’s work can now focus on building the capacity of affiliates to set realistic goals that respond to both the characteristics and specific needs of local populations being served, and align agency ‘success’ metrics with national RAR goals.
In addition to assessing outcomes to define success, Child Trends has worked with RAR to understand the elements of program implementation that predict program success. This work is critical for understanding variations in implementation, which in turn indicates the program locations that may need additional help in implementing the program. It is also critical for understanding the particular aspects of a program that may need to be improved in order to increase the probability of program success.
Next Steps and Conclusion
Raising A Reader is grounded in a strong foundation of research that highlights the importance of family involvement in promoting children’s literacy.
It has also established a strong emerging evidence base through numerous small-scale evaluations in a wide variety of sites across the country. And it is using this emerging evidence base as the springboard for an impact evaluation in order to provide an even higher standard of evidence for its effectiveness, as RAR continues its trajectory toward national expansion.
The experiences of RAR as it practices evidenced-based planning also exemplify how programs develop the ability to use both data and evidence to increase their effectiveness and impact. Building on a significant number and array of small studies, RAR is now well-positioned to undertake a more comprehensive evaluation study that will strengthen the program’s understanding of how to increase its overall effectiveness. To prepare for such a study, RAR will continue to integrate lessons learned from its knowledge base to solidify strong program operations and consistency of implementation. By continuing to choose and refine performance metrics and measurement tools, RAR will increase the reliability and validity of the data it collects and, in turn, enhance its ability to link program fidelity to desired outcomes.
While this is the story of one program, there are lessons for the broader field. For practitioners facing the ongoing challenge of integrating evidenced-based planning into their programs, understanding how other organizations successfully use research to improve their effectiveness can provide a road map for future success. And, for researchers, developing partnerships with organizations such as Raising a Reader yields valuable new perspectives on the dynamic and challenging realities of program implementation–leading to new lines of inquiry and strengthened connections between research and practice.
Carol Emig, President, Child Trends
Angela Rojas, Senior Research Analyst , Child Trends
Vanessa Harbin Sacks, Senior Research Analyst, Child Trends
Gabrielle Miller Ed.D., National Executive Director, Raising A Reader
Georganne Morin, Senior Director, West Coast Programs, Raising A Reader
Rebecca Armstrong, Senior Director, East Coast Programs, Raising A Reader
This research was funded by the Annie E. Casey Foundation. We thank them for their support but acknowledge that the findings and conclusions presented in this report are those of the authors alone, and do not necessarily reflect the opinions of the Foundation.
© Child Trends 2014. May be reprinted with citation.
Child Trends is a nonprofit, nonpartisan research center that studies children at all stages of development. Our mission is to improve outcomes for children by providing research, data, and analysis to the people and institutions whose decisions and actions affect children. For additional information, including publications available to download, visit our website at childtrends.org.
1 The Annie E. Casey Foundation. (2010). Early warning: Why reading by the end of the third grade matters. A Kids Count Special Report. Baltimore, MD: Author. Retrieved from http://www.aecf.org/m/resourcedoc/AECF-Early_Warning_Full_Report-2010. pdf#page=43
2 Bernstein, S., West, J., Newsham, R., & Reid, M. (2014). Kindergartners’ skills at school entry: An analysis of the ECLS-K. Princeton, NJ: Mathematica Policy Research.
3 Rumberger, R., & Lim, S. (2008). Why students drop out of school: A review of 25 years of research. California Dropout Research Project Report #15. Santa Barbara, CA: California Dropout Research Project.
4 Hernandez, D. J. (2012). Double-jeopardy: How third-grade reading skills and poverty influence high school graduation. Baltimore, MD: The Annie E. Casey Foundation. The Annie E. Casey Foundation. (2013). Early warning confirmed: A research update on third-grade reading. Baltimore, MD: Author. Retrieved from http://www.aecf.org/m/resourcedoc/AECFEarlyWarningConfirmed-2013.pdf
5 National Center for Education Statistics. (2013). The nation’s report card: First look: 2013 mathematics and reading (NCES 2014–451). Washington, D.C.: Institute of Education Sciences, U.S. Department of Education. Retrieved from http://www. nationsreportcard.gov/reading_math_2013/#/student-groups
6 The Annie E. Casey Foundation. (2010).
7 Kalb, G., & Van Ours, J. C. (2014). Reading to young children: A head-start in life? Economics of Education Review, 40, 1–24.
8 Van Voorhis, L. F., Maier, M. F., Epstein, J. L., & Lloyd, C. M. (2013). The impact of family involvement on the education of children ages 3 to 8: A focus on literacy and math achievement outcomes and social-emotional skills. New York, NY: MRDC. Retrieved from http://www.mdrc.org/sites/default/files/The_Impact_of_Family_Involvement_FR.pdf
9 Hindman, A. H., & Morrison, F. J. (2011). Family involvement and educator outreach in Head Start: Nature, extent, and contributions to early literacy skills. Elementary School Journal, 111, 359–386. Rodriguez, E. T., Tamis-LeMonda, C. S., Spellmann, M. E., Pan, B. A., Raikes, H., Lugo-Gil, J., & Luze, G. (2009). The formative role of home literacy experiences across the first three years of life in children from low-income families. Journal of Applied Developmental Psychology, 30, 677–694. Sénéchal, M., and LeFevre, J. (2002). Parental involvement in the development of children’s reading skills: A five-year longitudinal study. Child Development, 73, 445–460.
10 Hood, M., Conlon, E., & Andrews, G. (2008). Preschool home literacy practices and children’s literacy development: A longitudinal analysis. Journal of Educational Psychology, 100, 252–271. Reese, E., Leyva, D., Sparks, A., & Grolnick, W. (2010). Maternal elaborative reminiscing increases low-income children’s narrative skills relative to dialogic reading. Early Education and Development, 21, 318–342.
11 Fantuzzo, J., McWayne, C., Perry, M. A., & Childs, S. (2004). Multiple dimensions of family involvement and their relations to behavioral and learning competencies for urban, low-income children. School Psychology Review, 33, 467–480.
12 Justice, L. M., Kaderavek, J. N., Fan, X., Sofka, A., & Hunt, A. (2009). Accelerating preschoolers’ early literacy development through classroom-based teacher–child storybook reading and explicit print referencing. Language, Speech, and Hearing Services in Schools, 40(1), 67–85. Lovelace, S., & Stewart, S. R. (2007). Increasing print awareness in preschoolers with language impairment using non-evocative print referencing. Language, Speech, and Hearing Services in Schools, 38(1), 16–30.
13 National Scientific Council on the Developing Child (2004). Young children develop in an environment of relationships: Working Paper No. 1. Retrieved from www.developingchild.harvard.edu
14 Jeynes, W. (2012). A meta-analysis of the efficacy of different types of parental involvement programs for urban students. Urban Education, 47, 706–742. Mol, S. E., Bus, A. G., de Jong, M. T., & Smeets, D. J. H. (2008). Added value of dialogic parent-child book readings: A meta-analysis. Early Education and Development, 19, 7–26. Sénéchal, M., & Young, L. (2008). The effect of family literacy interventions on children’s acquisition of reading from kindergarten to grade 3: A meta-analytic review. Review of Educational Research, 78, 880–907. van Steensel, R., McElvany, N., Kurvers, J., & Herppich, S. (2011). How effective are family literacy programs? Results of a meta-analysis. Review of Educational Research, 81(1), 69–96.
15 St. Laurent, L. & Giasson, J. (2005). Effects of a family literacy program adapting parental intervention to first graders’ evolution of reading and writing abilities. Journal of Early Childhood Literacy, 5, 253–278.
16 St. Clair, L., Jackson, B., & Zweiback, R. (2012). Six years later: Effect of family involvement training on the language skills of children from migrant families. School Community Journal, 22, 9–19.
17 High, P. C., LaGasse, L., Becker, S., Ahlgren, I., & Gardner, A. (2000). Literacy promotion in primary care pediatrics: can we make a difference?. Pediatrics, 105(Supplement 3), 927–934. Moore, K., Caal, S., Lawner, E., Rojas, A. &, Walker, K. (2014). Abriendo Puertas/ Opening Doors parenting program: Summary report of program implementation and impacts. Child Trends Publication 2014–24. Bethesda, MD: Child Trends.
18 Anthony, J. L., Williams, J. M., Zhang, Z., Landry, S. H., & Dunkelberger, M. J. (2014). Experimental evaluation of the value added by Raising a Reader and supplemental parent training in shared reading. Early Education and Development, 25(4), 493–514.
19 Sénéchal (2002).
20 Child Trends. (2014) Preschool and Prekindergarten. Bethesda, MD: Author. Retrieved from http://www.childtrends. org/?indicators=preschool-and-prekindergarten
21 Baker, C. N., Arnold, D. H., & Meagher, S. (2011). Enrollment and attendance in a parent training prevention program for conduct problems. Prevention Science, 12, 126–138.