A just-released paper, by Chloe Gibbs, Jens Ludwig, and Douglas Miller, provides a somewhat different analysis of recent research on Head Start, and its implications for policy. (Unfortunately, for many potentially interested readers, the paper is probably not available for free. However, I think most university libraries would have free access to NBER working papers.)

Among the important points of the paper (henceforth called GLM), with my comments, are the following:

1. GLM strongly defend the “quasi-experimental” research literature that has shown long-run effects of Head Start. (As GLM note, Ludwig and Miller have contributed to this literature.)  Steve Barnett has argued that such research may give a misleading impression if only papers with significant positive effects end up getting published. GLM argue that this quasi-experimental research has in fact thoroughly looked at the only national data sets that allow such long-run quasi-experimental analysis, and that there are not unexamined data sets that yield different long-term effects of Head Start.

My comment: I think GLM have a strong point here. The methodology of these long-term studies is sound, and they use the available data bases. I think the results should be deemed valid unless someone re-analyzes these data bases and shows the results are sensitive to estimating assumptions.

2. GLM point out that what is really unusual about the results from the randomized national Head Start Impact Study is not the initial impacts of Head Start, which are similar to past studies, but rather that the results so quickly fade by Grade 1. They argue that this increased fade-out of academic test score effects of Head Start could be due to the K-12 system becoming more effective in helping lower achievement students, who will be disproportionately in the control group, to catch up to higher achievement students.  Therefore, these fade-out effects of Head Start understate the true effects of Head Start, as part of Head Start’s benefits are that by helping some students avoid the need for remedial catch-up efforts, they free up resources which can be used to help other students.

My comments: This is a plausible hypothesis. However, it would be nice to confirm this by seeing how Head Start expansions affect who receives extra attention in catching up. The problem is that to really do such a study, we would have to have detailed information on how teachers allocate their time with different students before and after a major Head Start expansion.

3. GLM express some skepticism about the “regression discontinuity” studies of state pre-K programs, which show larger immediate test score effects of state pre-K programs than for Head Start.  They argue that such studies could be biased if more able students who just miss the age cut-off are enrolled by their parents in private preschool and private kindergarten, and are thereby excluded from the sample of students in these analyses. Essentially, the argument is that regression discontinuity studies may understate the effects of student age on test scores near the age cutoff, which will exaggerate the “jump” in test scores due to participation in state pre-K programs.

My comments: I am hardly a disinterested observer, because my recent paper with Gormley and Adelstein applies this regression discontinuity model to Tulsa’s pre-K program. I think that GLM’s objection is theoretically possible. However, in practice, I think it is unlikely that it is a major source of bias in studies of the kindergarten readiness effects of state pre-K programs. In most cases, and in particular in our study in Tulsa, the analysis examines whether there is any sign that observable characteristics of students and parents show a “jump” at the age cut-off.  If there was severe selection bias based on student ability for students who just missed the age cut-off, we would expect to see some sign of this in a jump in student or family characteristics at the age cut-off.  I know we do not see this in Tulsa. In addition, in the case of Oklahoma, private schools do not have a large market share, so this selective attrition to private schools seems unlikely. My bottom line: we should believe the finding that many state pre-K programs have larger immediate test-score impacts than Head Start.

4. GLM argue that based on other studies, such as those by Deming and Chetty et al, early intervention programs may have large effects on adult outcomes even if their test score impacts fade. These long-run impacts may be due to effects on soft skills, which are harder to measure. Therefore, in making reforms to Head Start, we need to be careful not to inadvertently weaken the program’s impacts on soft skills.

My comments: As I have argued in numerous blog posts, how early childhood programs develop soft skills is a major issue. I think GLM’s concern is a valid one. Our processes for evaluating Head Start programs, or adopting new approaches to providing federal aid for early childhood programs, should be balanced in that they should stress both “hard skills” and “soft skills”.  This may require that we use more subjective approaches to assessing whether a particular Head Start program, or a particular reform approach, is successful in increasing “soft skills”.

The bottom-line: The Gibbs, Ludwig, and Miller paper makes an important contribution in arguing forcefully that the recent Head Start Impact Study findings should not be interpreted as meaning that Head Start is ineffective. That interpretation conflicts with other well-done research, and may lead to counter-productive reforms to Head Start.