A recent report on Tennessee’s state-funded pre-k program has led to some controversy over the program’s effects.
The report’s authors, Strategic Research Group, concluded that “the results provide evidence that the objective of Tennessee’s Pre-K program – school readiness is being met…[T]he results should clearly inform stakeholders that not only should the Pre-K program continue to serve at-risk students in Tennessee, it should be complemented with additional support and intervention for students over time to help sustain the early academic growth observed among Pre-K students in Kindergarten.” (pp. 40-41)
However, state representative Bill Dunn argues that the report shows that Tennessee’s pre-K program lacks long-run effectiveness. Rep. Dunn said in a press release that “This report should serve as a revelation for individuals who still believe pre-K is some sort of answer for long-term achievement in education. The fact is, it just isn’t. It may be the largest hoax ever perpetrated on the people of Tennessee.”
The basis of this dispute is a report that shows some short-run evidence of pre-k’s effects as of the end of kindergarten and first-grade, particularly for students eligible for free and reduced price lunch. However, these effects appear to disappear by 3rd to 5th grade.
What can be said about this dispute, and the effectiveness of pre-K in Tennessee?
First, the basis for this dispute is a report that does NOT have a randomly chosen control group. Rather, the report matches pre-K participants with non-pre-K participants of the same gender, race, free-or- reduced- price lunch (FRL) status, and school district (or in some cases school). This is important because in general, reports that do not have a randomly chosen control group, or a comparison group that is a good comparison group, may be seriously biased.
Second, there is some reason based on the study’s design to think that such bias may have occurred in this particular study. For free-or-reduced price lunch (FRL) status, the study matched pre-K participants to non-pre-K participants based on whether the student had EVER been designated as FRL, from pre-K on up through 5th grade.
For Tennessee’s pre-K program, however, first priority for admission was based on whether student’s families would meet the income guidelines for free or reduced price lunch status as of the pre-K year. After such children are enrolled, programs may enroll students with disabilities, or English language learners, or who are at risk of abuse or neglect.
What this means is that Tennessee pre-K participants must either have low income in the pre-K year as well, or have other risk factors. The matched non-pre-K participants do not need to have low income in the pre-K year or have these other risk factors. This means that the income status of the pre-K participants who are eligible for FRL is more likely to be persistently poor. For pre-K participants who are non-FRL status, they are more likely to have other risk factors.
It should be remembered that a zero-one indicator for whether someone is eligible for free-or-reduced price lunch is only a rough control for family income. Within the FRL category, families with lower income will tend on average to have students who will have more problems in school. (Obviously this is just an average tendency, not inevitability, and it can be affected by public policy.) More persistent FRL status is more likely to be associated with lower family income.
If the pre-K participants are on average somewhat lower income or have unobserved higher risk factors or higher income, this will tend to bias downwards the estimated effects of Tennessee’s pre-K program. This bias will occur even in the end of kindergarten test score results. However, this bias will tend to grow over time, because unobserved family income effects or risk factors will have a cumulative effect on student test score performance that will grow over time.
Third, the report’s findings for pre-K’s effects in Tennessee are far weaker than the effects found in a recent Vanderbilt study that has a better comparison group. This Vanderbilt study uses both a randomly chosen control group, and also does a regression discontinuity analysis. The Vanderbilt study’s statistics suggests some problems with the random assignment portion of the study due to missing data. However, the regression discontinuity portion of the study uses a very strong methodology that has been shown in other studies to usually lead to comparison groups that are truly comparable.
Measured in terms of “effect sizes”, the Vanderbilt study shows effects at the beginning of kindergarten that are 15 to 60 times as large as the Strategic Research Group results for effects at the end of kindergarten. It seems highly implausible that the “true effects” could differ so much between the beginning and end of kindergarten.
One plausible interpretation of these discrepancies is that the Strategic Research Group’s results are biased downwards due to unobserved lower income and higher risk factors in the pre-k group relative to the SRG comparison group. This bias could be already apparent at the end of kindergarten, and then grow over time in later grades.
Fourth, our interpretation of Tennessee’s results should be informed by previous research studies. We have good evidence, from the Perry Preschool program and the Chicago Child-Parent Center program, that high-quality preschool can yield long-term benefits. These studies use good comparison groups. While these preschool programs are not identical to Tennessee’s program, the preschool programs do have some similarities.
On the whole, given some of the issues with the SRG comparison groups, I don’t think the SRG results for long-term effects are sufficiently reliable to overcome the evidence from prior research studies that high-quality preschool has long-term benefits. In an ideal world, it would be good to have evidence with a more rigorously chosen comparison group for the specific long-term effects of Tennessee’s pre-K program. Such evidence may be forthcoming if the Vanderbilt random assignment study is able to be pursued over the long-term, and if the Vanderbilt random assignment study is able to correct for some of its problems with missing data.