FactCheck’s recent column criticizing President Obama’s claims about his preschool program gave a misleading description of the overall research evidence on preschool.

FactCheck describes itself as “a project of the Annenberg Public Policy Center of the University of Pennsylvania”, and says that it is  “a nonpartisan, nonprofit “consumer advocate” for voters that aims to reduce the level of deception and confusion in U.S. politics.”

According to FactCheck, “In his State of the Union address, Obama exaggerated the effects of universal preschool by comparing results from small, expensive programs targeted to disadvantaged youth to a universal program for which such results are unproven.”

There are two major problems with FactCheck’s coverage.

First, FactCheck’s implication that there is no evidence of high benefit-cost ratios for programs of similar costs to large-scale state pre-K programs is incorrect.

FactCheck correctly points out that the Perry Preschool program, which shows high benefit-cost ratios, is “far different than any universal preschool program currently run by any state”, and in particular, that Perry’s $19,000 per student cost is far greater than most state’s preschool program.

However, FactCheck ignores that other preschool programs have very high benefit cost ratios yet are of similar cost to large-scale state preschool programs. For example, the Chicago Child-Parent Center program is of similar cost to these state programs, and has very high benefit-cost ratios, based on data collected on former participants up to their late 20s.

In Chicago, about half the students participated for one year at age 4 for a half-day program, and about half for two years at both ages 3 and 4. The benefit cost ratio was higher for the one-year program. The estimated benefit-cost ratio was 13.58 for the one-year version of the CPC program (see Table 5 of a 2011 paper by Arthur Reynolds and his colleagues).

One year of this program had a cost of $5,597, according to this same paper.  This is comparable to the $6,100 that FactCheck cites for average costs of one year of preschool in different states, based on a chapter by Duncan, Ludwig, and Magnuson. However, some of these figures may be for a mix of half-day and full-day programs.  The Chicago figure is greater than the $4403 per student spent in Tulsa, Oklahoma for a half-day program (Bartik, Gormley, and Adelstein).  But our Tulsa figure was in 2005-2006 prices, whereas the Chicago CPC figures are in 2007 prices. Also, Chicago prices are higher than Tulsa prices by about 14%.  If we adjust our Tulsa figures to “year 2007 Chicago prices”, Tulsa costs for a half day program would be $5,229 per student, which is not much below CPC’s costs.

In addition, the Chicago program was run at a large scale, similar to state programs, so it is not true that we do not have good research evidence for large-scale programs.

Second, FactCheck is incorrect to imply that there is no good evidence of high benefit-cost ratios for middle-class students.

They refer only to studies that “suggest the benefits [from preschool] that accrue to middle income students are far less dramatic”. They quoted Rich Neimand, of the Neimand Collaborative, as saying that “To the best of our knowledge, there has been no evidence-based scientific research on the value of universal preschool. However, that doesn’t necessarily mean that universal has no value.”

This ignores the research done on Tulsa by me and Bill Gormley and Shirley Adelstein. To the best of my knowledge, this is the only research on universal pre-K programs to compare preschool’s effect across low-income children and middle-class children using a well-regarded methodology (regression discontinuity).  Regression discontinuity is not quite as rigorous as random assignment, but it does provide good evidence.

The intuition is that we look for jumps in test scores comparing students who were just old enough to attend pre-K the previous year, and who are now entering kindergarten, with children who just missed the cut-off for 4-year-old pre-K the previous year, and who therefore are just entering pre-K. The argument is that these children are of almost the same age, and therefore should be similar in observed and unobserved characteristics. In fact, the data show the children are similar in characteristics that we can observe. The main difference is that one group of children has had a year of high-quality preschool, and the other group has not.

The limitation of our methodology is that we can only directly estimate the effects of universal preschool on kindergarten entrance scores for low-income and middle-class children. However, we use the available data on how kindergarten test scores predict adult earnings to do a partial benefit-cost analysis of universal pre-K. The benefit-cost analysis is partial in that we only look at predicted effects on future earnings. We ignore effects of preschool in reducing crime, which in many preschool studies are equal to or greater in dollar value than benefits in increasing earnings.

In our study, the “partial”, earnings-based benefit to cost ratio for half-day preschool for the lowest income children, who are eligible for a free lunch, is 4.08  — that is, each dollar invested in preschool increases the present value of these children’s future earnings by over four dollars. For more middle-class children, who are ineligible for any lunch subsidy, the earnings-based benefit to cost ratio is 3.44. This is below the benefit-cost ratio for low-income children, but not much below.

One could argue that the total social benefits to low-income children from Tulsa’s preschool program may be greater, and in fact we make that argument in our article. We point out that percentage effects on expected future earnings are greater for low-income children, which may mean that in some sense these dollar earnings gains are more socially valuable. In addition, it is possible that the anti-crime benefits of preschool may be greater for low-income children, although there is no direct evidence for this.

However, contrary to FactCheck, I would regard a benefit-cost ratio of 3.44 for earnings benefits for middle-class children as “dramatic benefits”. The estimated real “rate of return” to investing in preschool for middle-class children, based on earnings benefits alone, is 6.7%. This compares favorably with many other social investments and private investments.

It is fair to say that we have more long-term evidence from more rigorous studies for the benefits of preschool for low-income children.  But we do have some good evidence for a high rate of return to preschool programs for middle-class children.  And it’s fair to say that some of our evidence for high returns to preschool comes from small programs such as the Perry Preschool program. But we also have evidence from large-scale programs.

It is perhaps unreasonable to expect every social and educational program to have evidence from large-scale randomized experiments involving middle-class children. This is not the kind of research that government agencies or foundations tend to fund – such a large-scale program would be expensive, and these agencies tend to focus on needier children with their research dollars, for understandable reasons.

If the standard for proof for benefits for middle-class children is large-scale randomized experiments, we are rarely if ever going to have such evidence.  I sometimes wonder what would have happened to the “common school” movement of the 19^th century if we had demanded that Horace Mann produce random assignment evidence from large-scale programs that the common school was needed for children of all income classes.