Jeremy Wallace

Social scientists tend to propose and test hypotheses of the following type:

Hypothesis: As [our explanatory variable] increases, [our outcome variable] also increases, all else equal.

The way that hypotheses of this sort are tested is by compiling a large data set and examining if in a multivariate framework there exists a positive coefficient on the explanatory variable. If so, then the hypothesis is confirmed (well, the null hypothesis is rejected) and all is right with the world.

Those conducting survey or field experiments operate similarly, although there is less concern in those situations about the interaction of other potentially confounding factors as the treatment has been administered randomly to a sample that resembles the non-treated sample (i.e. it is balanced).

This is fine. Obviously there are numerous questions where we are trying to establish the direction and magnitude of causal effects of one factor on an outcome of interest, and this methodology tends to produce compelling evidence in support of these kinds of hypotheses.

What I would like to suggest, however, is that we do not limit the work of social science to the investigation of average treatment effects. Those interested in pushing randomized controlled trials as the only acceptable methodology appear less concerned on this point. A positive mean effect can result from a consistent positive treatment effect across all individuals or a bimodal treatment effect with a larger share of individuals being effected in one direction than in the other (for a ludicrously outsized example, see the Pax in Serenity).

Politics is a complicated domain (as Einstein said) with creative and independent thinking actors that can change over time and respond to similar stimuli in wildly different directions.

The singular focus on means can cause us to ignore the variety of responses inside of the distribution from which the mean emerges. For example, Dean Baker has a great post on his Beat the Press blog about the reality of a very low–but positive–inflation rate.

Expressing concern over deflation (i.e. the inflation rate turning negative) is the way in which people tell you that they have no clue about economics and just repeat what they heard others say. The inflation indexes we use are an aggregation of millions of different price changes. There is a substantial amount of dispersion around the overall inflation number. This means that when the inflation rate is near zero, there are many goods and services whose prices are already falling.

The almost total focus on the mean number ignores that in order to produce a number so close to zero implies a broad set of goods where deflation is already in progress. If deflation is terrifying, then a very low overall inflation rate implies not that deflation is not occurring but that the share of the economy in deflation is relatively large.

Focusing solely on the meaning of the mean effect can lead us astray when the real story is elsewhere.