Two recent research papers, by Kline and Walters, and by Feller et al., suggest that Head Start has much larger impacts when it is compared to the alternative of “no preschool”. This finding tends to increase the likelihood that Head Start has benefits greater than costs.

The Kline/Walters and Feller et al. papers are in part responding to an important research puzzle: how to reconcile the results of the randomized Head Start experiment with prior Head Start research. The Head Start randomized experiment found relatively small immediate effects of Head Start on test scores, and these effects quickly faded. In contrast, prior Head Start research has tended to find larger short-term effects, and more persistent effects on other outcomes, including adult outcomes.

For example, a meta-analysis (by Shager et al.) of numerous studies of Head Start found short-term test score effects that were around twice the effects estimated in the Head Start experiment (0.27 standard deviations versus effects generally between 0.1 and 0.2 standard deviations). In addition, while  the Head Start experiment’s results faded by over 70% by grade 3, and were no longer statistically significant, , other research (for example, by Deming) has found  predicted effects on adult earnings that exceed the earnings effects that would be predicted based on early test score effects (Figure 4.1 in my recent book, From Preschool to Prosperity ).

How can these results be reconciled? One possibility, raised by numerous researchers, is that the alternative preschool options to Head Start have expanded and improved over time. Most Head Start studies are estimating the effects of Head Start relative to whatever choice would otherwise be made by Head Start participants, whether that choice is home care or preschool. Over time, non-Head Start preschool options have expanded, and may have improved in quality. This would tend to reduce Head Start’s net impact in newer studies compared to prior studies. This is particularly important for studies of Head Start’s long-term impact, as these studies necessarily are analyzing Head Start as it existed some time ago.

In the Head Start experiment, it appears that in many cases, assignment to the “treatment group” shifted children from other preschool programs to Head Start.  For example, Kline and Walters estimate that among the 4-year-old participants in the Head Start experiment, 41% of the families induced by the treatment assignment to enroll in Head Start would have otherwise enrolled their 4-year-old in some other preschool. The net impact of the Head Start experiment on child outcomes will be a weighted average of Head Start’s effects relative to preschool, for families who otherwise would have enrolled their child in preschool, and relative to home care for families who would have chosen that option. If the other preschool is similar in its effects to Head Start, this reduces the estimated net effects of Head Start.

The new research by Kline and Walters, and by Feller et al. al. , explicitly analyze the choice by families among options of Head Start versus other preschool versus home care, and how this is affected by treatment group assignment in the Head Start experiment. These studies explicitly estimate how this choice might be influenced by a wide variety of variables. For example, if a state has invested more in state preschool programs, we would expect the counterfactual alternative to Head Start to be state-funded preschool for more families (and this is found in both the Kline/Walters and Feller et al. studies). In addition, if the child has younger siblings, we expect (and we find in Feller et al.) that a greater proportion of the Head Start treatment group would have enrolled their 3-year old or 4-year old in some preschool anyway.

These two new studies, after explicitly trying to control for what alternatives parents would have chosen to Head Start, find much larger effects of Head Start versus home care than for Head Start versus other preschools. Both Kline and Walters, and Feller et al., find very small or non-existent effects of Head Start on test score outcomes when those Head Start effects are compared to the test score effects of other preschools that would have been chosen. In contrast, the effects of Head Start versus home care on test score outcomes are much larger.

As a result, Feller et al. estimate short-run test score effects of Head Start versus no preschool that are about 60% greater than the average impact of simply being assigned to the Head Start treatment group (0. 23 standard deviations versus 0.14 standard deviations when using the same test measures).  Kline and Walters also find short-run test effects of Head Start versus no preschool that are about 60% greater than the average effect of simply being assigned to the Head Start treatment group (0.37 standard deviations versus 0.23 standard deviations using the same test score measures).

Both Feller et al and Kline/Walters also conclude that for longer-term Head Start analyses, when proper controls are done for the alternatives facing families, estimates of test score impacts become much more imprecise, and cannot rule out more sizable outcomes. Therefore, it is not true that the Head Start impact study definitely showed that Head Start impacts quickly faded.

How is all of this important for public policy? In two ways. First, for public policy, the most important analysis is of the net benefits and costs of preschool compared to its alternatives. This will be a weighted average of: Head Start’s benefits versus no preschool compared to Head Start’s costs; Head Start’s net benefits, if any, over other preschool programs, compared to Head Start’s extra costs, if any, over other preschool programs.  Kline and Walters present some analyses that when benefit-cost analyses properly adjust for the reality that Head Start substitutes for other government-financed preschool programs, which have some costs and benefits, then Head Start probably has benefits that are 15% greater than costs.

Second, this suggests that policy analysis of the benefits and costs of any preschool program, Head Start or otherwise, needs to consider who will be drawn into the program, and the program’s relative benefits and costs compared to the alternatives that families would otherwise have chosen for their child. What the preschool program substitutes for may vary with how the program is designed and promoted, or where program sites are located. For example, Kline and Walters present evidence that some of those who are least likely to sign up for Head Start may be least likely to sign up for it, so expansions of the programs may have greater benefits. Similarly, another recent analysis of the benefits and costs of Head Start, by Bitler et al., finds that children who otherwise would have had very low test scores benefit the most from Head Start.

Finally, a more technical note. The recent research by Feller et al. and Kline/Walters reaches different conclusions compared to some other research on Head Start’s impacts versus no preschool, but I find the Feller et al. and Kline/Walters methodologies to be more convincing on this topic. First, the recent research’s conclusions differ from those of a dissertation by Peter Bernardy, which compares Head Start treatment group members with control group members not in any preschool by matching on observables, and finds no evidence of lasting Head Start impact from this comparison. But Kline and Walters show that simply controlling for observable variables does not make as much of a difference as actually looking at variations in the alternatives facing families due to different state preschool programs or different family circumstances. Second, in the recent Bitler et al. paper, they do not find that Head Start’s net impacts vary with the variation in alternatives to Head Start chosen by different demographic groups. (This is a side point in the Bitler et al. paper, which mainly focuses on the distribution of test score impacts.) But the Feller et al. and Kline/Walters paper show that controlling for the alternatives that would be selected by families probably requires controlling for a larger set of potential selection variables than demographic characteristics, such as the local availability of preschool and family circumstances.

The bottom line is that the conclusion that the recent Head Start experiment shows that preschool does not work is a misinterpretation of the results. We do need to think about how we can structure expansions of Head Start or other preschool programs so that those who may benefit the most are reached by these expansions. This may require that these program expansions be structured to help attract families who otherwise would not be enrolled in any preschool.