investinginkids

Ezra Klein had an interesting column recently that argued for stronger evaluation as the key to the government making smart public investments while avoiding excessive deficits.  The argument is that rigorous evaluation of government programs is simple common sense, equivalent to a dieter using a scale to measure progress in losing weight.

As an economist, I’ve often put on my policy wonk hat over the years and advocated for more and better evaluation of government programs.  Perhaps this could be argued to be self-interest. We economists are considered to have some expertise in evaluation, and so arguing for evaluation could be seen as a way of drumming up business.

But there really is a common sense argument for evaluation. If we really want more and effective government policies, surely we should want to know how current policies are working, and what program variations work the best.

However, I sometimes wonder whether the U.S. has the right political system to appropriately use the results of evaluation. Evaluation results logically should be used to improve public programs. If a program evaluation is negative, this does not necessarily mean that the program should be eliminated. Rather, if the goal of the program is good, we should use the evaluation results to explore what might work better to achieve that policy goal.

In the U.S., that is often not the way evaluation results are used. In the U.S. context, any negative evaluation results are often used to attempt to kill the program rather than reform the program.

In my opinion, this occurs because for many government programs, there is no overall consensus that the goals being pursued are legitimate for government to pursue. Therefore, philosophical opponents of the programs are never persuaded by positive evidence. Any slightly negative evaluation approaches are mainly used by opponents as ammunition to attempt to kill the program.   In reaction, supporters of the program resist evaluation, or seek to overinterpret positive results as showing more than they actually show.

Let’s consider an example of this problem with respect to early childhood programs. As I have mentioned in earlier blog posts, pre-k programs appear to have lasting effects on adult outcomes for former child participants even though there is considerable fading of effects on academic test scores in middle school and high school. These long-run effects on adult outcomes may be due to effects of pre-k programs on “soft skills” that tend to appreciate over time rather than depreciate, and which are not well measured by academic test scores in middle school and high school.

Yet even though this fading of test score effects of pre-k is well-known, every time it is rediscovered in some other state, it is used as a reason not to provide state funding of pre-k.   However, this fading of effects of pre-k on “hard skills” could equally well be used to rationalize more reforms of K-12 schools, to make sure that the hard skill gains from pre-k are not lost. Or, these findings could be used as a reason to develop better measures of “soft skills” and how they change from pre-k through the K-12 system.

For evaluation to be constructively used, we need a political culture that has some consensus on appropriate government goals, and some willingness to use evaluation results to improve government programs. In a political culture without any consensus on government goals, it is very hard for evaluations to become anything more than talking points in bitter political fights.

What ultimately matters to preschool quality is what goes on in the classroom, between teacher and child.  This makes intuitive sense, and is also backed by research. Several studies indicate that preschool achievement gains are higher if more classroom time is devoted to teachers interacting with children individually, particularly with high quality questions and feedback that develops thinking skills.  In the preschool research field, such improvements are referred to as improvements in “process quality” in that they improve the process of what goes on in the classroom.

In chapter 5 of Investing in Kids , I use these research results to estimate the benefits of improvements in preschool “process quality”. I consider what would happen if a preschool classroom, on these process quality measures, changed from the median classroom to a classroom that would be ranked in the 84^th percentile. That is, the classroom would change from being better than half of all preschool classrooms, to being better than 84% of all preschool classrooms. (This particular improvement was considered because it is equal to what statisticians call a “one standard deviation” improvement.) The resulting test score improvements were used to project future economic development benefits for the local economy.

These calculations suggest that such process quality improvements would increase the present value of state residents’ earnings – which is called “economic development benefits” in my book – by 29% to 47% of the typical cost of preschool programs. Therefore, such quality improvements would have economic development benefits exceeding costs as long as making such process quality improvements did not increase per student costs of preschool programs by more than 29% to 47%.

It seems likely that good preschool management could attain such process quality improvements at a cost increase of less than 29%.  Presumably such process quality improvements require more attention to teacher training and teacher hiring, and more expert interventions by preschool management. Although this would cost something, it seems unlikely that it would raise preschool costs as much as 29%.

In short, preschool process quality pays off for the local economy, and by quite a bit compared to the costs of quality.

One feature of pre-k programs that can “easily” be changed by public policy is class size.  When I say class size can be changed “easily”, I mean that it can be changed with a simple to implement change in law or program rules. Reducing class size is NOT easy in terms of money. Reducing class size is expensive. Are the large costs of reducing pre-k class size justified by the resulting economic benefits for state or local economic development?

As reviewed in chapter 5 of Investing in Kids, lower class size in a pre-K program probably significantly improves program effects on child development. The best direct evidence comes from the National Day Care Study. Some good indirect evidence comes from the Tennessee Class Size Study. Although the Tennessee Study looked at class size in kindergarten through 3^rd grade, and not pre-K, it seems reasonable that this study’s effects for kindergarten might be extrapolated to pre-K. The Tennessee Class Size Study’s effects are of interest because they were estimated based on random assignment of students to different class sizes.  Random assignment makes these estimates less likely to be biased by unobservable variables.

These studies estimate effects on early test scores. In the book, I use these estimated effects on early test scores, and research relating adult earnings to early test scores, to estimate the effects of lower class size in pre-K on local economic development.  The pre-k program I am analyzing is a half-day, school-year program for four-year-olds.

I find that lowering pre-K class size from 20 students per class to 15 students per class increases costs of pre-K by 28%. However, lower class size provides local economic development benefits that are three times these increased costs.  As defined in the book, these local economic development benefits are the present value of the future increase in per capita earnings in the local economy.  This increase in local per capita earnings is what local economic developers and many local policymakers should be trying to increase when they pursue the goal of economic development.

Therefore, lowering pre-k class size from 20 students to 15 students per class clearly passes a benefit-cost test, even when we only include local economic development benefits. Benefits of this lower class size would be even greater if we include economic development benefits that accrue outside the local economy, or if we included non-economic development benefits such as lower crime.

Many aspects of pre-k programs affect their quality. “Quality” in turn affects the magnitude of economic development benefits for a state or local economy. I’ll be exploring this important issue of pre-k quality in several blog posts.

One of the most controversial quality issues is the issue of teacher quality.  Teacher quality is a controversial issue because a lot of money is at stake, and it affects people’s jobs. Potentially, measures to increase teacher quality in pre-k education could significantly increase both pre-k teacher pay and pre-k program costs.  Requiring particular teacher credentials could threaten some current pre-k teachers’ jobs. Some teacher quality policies might affect how pre-k teachers are evaluated.  (For example, test scores at entry to pre-k and entry to kindergarten could be part of the evaluation.)

Susan Ochshorn of ECE PolicyWorks explores this issue of teacher quality in a recent blog entry at Huffington Post.   She points to recent research by economist Raj Chetty of Harvard (along with colleagues  John  Friedman, Nathaniel Hilger, Emmanuel Saez, Diane Whitmore Schanzenbach, and Danny Yagan) that uses data from the Tennessee Class-Size Study to show that higher teacher quality in kindergarten raises earnings of former kindergartners at age 27.

Chetty et al show that higher test scores in kindergarten are significantly correlated with adult earnings at age 27. Interestingly, these adult earnings effects occur even though test score gains in kindergarten tend to fade later in K-12.  By first grade, over 75% of the kindergarten test score gains have vanished, and by fourth grade, the kindergarten test score gains have completely disappeared. However, their estimates suggest less fade-out in effects on various behavioral measures in grades 4 and 8. These better “non-cognitive skills” or “soft skills” (see my earlier blog post on soft skills) may explain the long-term effects of better kindergarten experiences on adult earnings.

Chetty et al use their results on kindergarten test scores and adult earnings, together with previous research on how teacher quality affects kindergarten test scores, to calculate the adult earnings effect of a specified improvement in kindergarten teacher quality.  For example, consider a “1 standard deviation” improvement in kindergarten teacher quality. This corresponds to changing from the “median” kindergarten teacher to a kindergarten teacher at the “84^th percentile” of teacher quality, that is from a teacher who is better than half of all kindergarten teachers, to a teacher better than 84% of all kindergarten teachers.   For a kindergarten class size of 20, this increase in teacher quality is estimated to increase the present value of future earnings of these former kindergartners by $214,000.

As Susan Ochshorn mentions, these high effects of kindergarten teacher quality on adult earnings also suggest the importance of teacher quality in pre-k programs and child-care programs.  Surely if teacher quality is so important at age 5, it should be of considerable importance at age 4 and younger ages.

In chapter 5 of Investing in Kids, I take a somewhat different approach to estimating the importance of teacher quality. I ask the following question: suppose we increase lead teacher salaries in pre-k from what are typical salaries for an individual with a high school degree plus a Child Development Associate credential, to typical salaries for a public school kindergarten teacher. How much would learning in a one-year pre-k program at age 4 have to go up for the local economic development benefits from this change to exceed the increase in program costs?

I take the pre-k salary figures for teachers with different credentials, and pre-k program costs, from the excellent publication by Barbara Gault and her colleagues at the Institute for Women’s Policy Research, Meaningful Investments in Pre-K: Estimating the Per-Child Costs of Quality Programs (2008). Gault et al calculate annual straight salaries, in 2007 dollars, before payroll taxes and benefits, at about $18,000 for a lead teacher with only a CDA credential, versus $45,000 for a lead pre-k teacher who is paid similarly to a typical public school kindergarten teacher.  Based on that research, I calculate that these higher salaries end up increasing pre-k’s cost per student by a little over one-fourth.

I use estimates that are similar to Chetty et al for how early test scores are related to adult earnings. Based on these estimates, I conclude that these higher teacher salaries “pay off” in higher economic development benefits even with only very slight effects on student learning in pre-k programs.  Student learning gains must only go up by 5% compared to typical student learning in pre-k for this salary increase from $18,000 to $45,000 to pay off in higher present values of earnings per capita in a state’s economy.  This calculation adjusts the effects downward to account for former pre-k students who move out of state as adults.

Now, there is certainly room for debate on how such improvements in pre-k teacher salaries should best be used to improve pre-k teacher quality. How much of that increase in teacher salaries should be used to allow for higher credential requirements in terms of education and training? How much of that increase in teacher salaries should be used to allow for stronger evaluation of pre-k teachers by programs based on student outcomes or other mechanisms?  I suspect that as we explore how to improve teacher quality in pre-k, we will find that both increases in credentials and strengthening of evaluation will be helpful in improving pre-k teacher quality.  And higher salaries may be a part of a personnel policy that will allow for more selective recruitment and retention of pre-k teachers, and will reduce the problems caused by teacher turnover.

However, the underlying point should not be forgotten. Early learning is so important to adult success that even very slight improvements in early learning can justify large costs incurred to increase early learning.  Even modest improvements in early learning, whether due to higher teacher credential requirements, or stronger evaluation procedures, can justify the considerable costs that may be needed to implement these reforms.