(This is a "permanent" page, featuring a shorter, edited version of this blogpost)
Why do some economists producing sophisticated, intelligent work end up supporting claims about the real world that can seem – at times – bizarre and clearly unsupported by evidence? (And for those who disagree that this is the case - what would be an example where this clearly happens?)
A colleague and I have tried to understand why this happens in a recently published paper. An essay presenting the same ideas in a shorter, simpler, and more readable form is here, and for those who prefer to get “the gist of it” through a video, you can do so here. An even shorter version follows:
Consider the research process as an “evolutionary” process: Initial "variation" comes from hunches and ideas turned into models and arguments and papers. "Selection" happens when these are “attacked” by colleagues who read drafts, attend seminars, perform anonymous peer-reviews or respond to published articles. Claims surviving this process are seen as “solid” and “backed by research.” If this selection process is "stricter" for some types of claims than for others, we might find that some types of “accepted” claims tended to be well supported (e.g., formal, theoretical models and certain types of statistical fitting) while other types of “accepted claims” would be more poorly supported on average (e.g., claims about how the real world actually works or what policies people would actually be better off under). These claims would be more poorly supported on average for two reasons: A higher share of poor-quality claims would make it through, and as researchers learn that claims of this type are less strictly evaluated they may reduce the effort they put into evaluating and supporting these claims in their own work.
Our starting point is that different types of claims require different types of evidence: The Pythagorean theorem, for instance, must be supported using quite different methods and evidence relative to a claim about the permeability of shale rock. In our paper, we pursue this line of thought by identifying four types of claims that are commonly made and that require different types of evidence. We then apply this to the literature on rational addiction in economics. Some types of claims made in this literature seem well supported: It has extended theory (by deriving unstable consumption from stable preferences) and it has shown that, to some extent, it is “as if” the market data was generated by these models. On the other hand, there are other types of claims made that are poorly supported: There is (as good as) no evidence that these models capture the actual mechanism underlying an addiction or that they are credible, valid tools for predicting consumer welfare under addictions. All the same – these claims have been made too – and we argue that such claims are allowed to piggy-back on the former claims provided these have been validly supported.
Finally, we discuss a survey mailed to all published rational addiction researchers which provides indicative support – or at least is consistent with – the claim that the “culture” of economics acknowledges the relevant criteria for evaluating claims of pure theory and statistical fit better than the relevant criteria for evaluating claims of causal or welfare “insight”. To see this, just compare the Bradford-Hill criteria for establishing causality in medicine/epidemiology with the evidence presented in modern macro or rational addiction theory or a game-theoretic model of the climate treaty negotiation process.
If this explanation holds up after further challenges and research and refinement, it would also provide a way of changing things – simply by demanding that researchers state claims more explicitly and with greater precision, and that we start discussing different claims separately and using the evidence relevant to each specific one. Unsupported claims about the real world should not be something you`re allowed to tag on at the end of a work as a treat for competently having done something quite unrelated.
Anyway, this is also an experiment in spreading research – and in addition to this blogpost you can pick from three different levels of interest: The full paper, the essay or the video.
Comments welcome :-)
A blind spot in economics? Unjustified claims about reality