The results of this secondary data analysis simulating a QRIS validation using six large early care and education datasets demonstrate several issues that should be considered when constructing, validating, and making changes to existing quality ratings.