Friday, February 25, 2011

Fighting publication bias #1

Short version: By having a hierarchy of journals that accept work partly based on a prediction of how important/novel the work seems to be to a few referees and an editor, researchers will

  • try too hard to find results that will seem to be novel/important
  • try too hard to reproduce new results and show that they too have found this new novel/important thing
  • shelve their work (because it seems flawed or because will at best be publishable only in less interesting lower-tier journals) if they fail to reproduce the new novel/important things

The current academic publishing system with peer-reviewed journals is an attempt to achieve a lot of different goals at the same time:

  • Facilitate scientific progress, by
    • ensuring quality of published research by weeding out work that is riddled with errors, poor methodology etc. through anonymous peer-review by relevant experts
    • assessing/predicting importance of research and thus how “high up” in the journal hierarchy it should be published,
    • making research results broadly accessible so that disciplines can build their way brick-by-brick to greater truths
    • promoting a convergence towards consensus by ensuring reproducibility of research and promoting academic dialogue and debate
  • Simplify the evaluation of individual researchers (given the above, the number of articles weighted by journal type is a proxy for the importance and quality of your research)
  • Generate huge profits for publishing houses (To quote an article from Journal of Economic Perspectives, “
  • The six most-cited economics journals listed in the Social Science Citation Index are all nonprofit journals, and their library subscription prices average about $180 per year. Only five of the 20 most-cited journals are owned by commercial publishers, and the average price of these five
    journals is about $1660 per year.

Now, clearly, not all of these goals are compatible – most obviously, it is hard to square rocketing subscription costs with the goal of making research results more accessible. However, the ranking of academics based on where in a hierarchy of journals they have published seems likely to lead to issues as well.

If you want to get ahead as a researcher, you need to be published, preferably in good journals. If you want to be published in a good journal you need to do something surprising and interesting. You need to either show that something  people think is smart is stupid, or that something people think is stupid is smart. As a result, you get a kind of publication bias that can be illustrated by a simple thought experiment:

Imagine that the world is exactly as we think it is. If you drew a number of random samples, the estimates for various parameters of interest would tend to be distributed rather nicely around the true values. Only the researchers “lucky” enough to draw the outlier samples whose estimated parameters were surprising would be able to write rigorously done research that supported new (and false) models of the world that were in line with these (non-representative) results. This is actually not a very subtle point: One out of twenty samples will by definition have results that reject a true null hypothesis at 5% significance level.

OK, so let us say ideological bias, fashions and trends in modeling approaches etc. are irrelevant, so the result is published. Right away, this becomes a hot new topic, and anyone else able to reproduce it (read: anyone else drawing random but non-representative samples) get published. And then, gradually, the pendulum shifts – and the interesting and novel thing is to disprove the new result.

Now, clearly the above thought model is too simple. For one thing, we don’t know the truth. But the recent New Yorker essay on “The decline effect” sounds like this might be part of what’s going on:

all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants

The essay discusses a number of explanations (some of them sort of mystical and new-agish), but also notes the explanation above. When biologist Leigh Simmons failed to replicate a new interesting result, he failed to replicate it:

“But the worst part was that when I submitted these null results I had difficulty getting them published. The journals only wanted confirming data. It was too exciting an idea to disprove, at least back then.” For Simmons, the steep rise and slow fall of fluctuating asymmetry is a clear example of a scientific paradigm, one of those intellectual fads that both guide and constrain research: after a new paradigm is proposed, the peer-review process is tilted toward positive results. But then, after a few years, the academic incentives shift—the paradigm has become entrenched—so that the most notable results are now those that disprove the theory.

It seems to me that this is an almost unavoidable result of the current journal system, but not an unavoidable result of peer-reviewed journals as such. The problem seems to me to stem from the hierarchy of journals, and from the two tasks we give to referees (assess quality and assess importance/interest). The new open-access mega-journals (PLOS One, Sage Open, etc) that aim to publish all competently done research independently of how “important” it seems should at least mitigate the problem. Not necessarily by making it less important to have a “breakthrough” paper with a seemingly important result, but by making it easier to publish null-results.

Monday, February 21, 2011

Rewards and incentives can be OK

There’s a result from behavioral economics that increasing rewards and incentives for a behavior (e.g. to get kids to read more books) “crowds out” intrinsic motivation and leaves them less interested in books than before once the rewards dry out. Barking up the wrong tree notes a study that fails to find this when it comes to getting kids to eat vegetables. Good to know for those of us with kids.

Liking and intake of the vegetable were assessed in a free-choice consumption task at preintervention, postintervention, 1 month after intervention, and 3 months after intervention. Liking increased more in the three intervention conditions than in the control condition, and there were no significant differences between the intervention conditions. These effects were maintained at follow-up. Children in both reward conditions increased consumption, and these effects were maintained for 3 months; however, the effects of exposure with no reward became nonsignificant by 3 months. These results indicate that external rewards do not necessarily produce negative effects and may be useful in promoting healthful eating.

Sunday, February 20, 2011

Why support free trade?

Economists are usually in favor of free trade. I myself am both an economist and usually in favor of free trade. But I thought this post on “Kids prefer Cheese” which Mark Thoma recently re-blogged had a good and valid point that economists do well to remember: Even if trade benefits the trading partners, that does not mean that a large number of people in a country may not be hurt by allowing free trade. And even if the monetary gains of the winners are bigger in sum than the monetary losses of the losers, that isn’t always a big help for the losers since there is no redistribution automatically triggered making everyone at least as well off as before.

Economists usually defend their stance on such issues by talking about Pareto efficiency, saying that making someone better off is always good provided someone else isn’t made worse off by it. Then they switch from talking about Pareto improvements (probably rare in actual policy) to talking about potential Pareto improvements, where the winners could compensate the losers and achieve a true Pareto improvement. Of course, they won’t do so in actuality, which makes the policy also have a redistributive element. A common reply is that redistribution should not be solved through trade measures, but through redistributive policies. But, guess what, most of those aren’t that popular amongst economists either: They distort incentives and reduce efficiency and involve moral hazard problems and, besides, inequality isn’t that horrible anyway. I may be completely wrong, but my guess is many of the economists most adamant about the glories of completely free trade are also amongst those staunchest in opposition to redistributive taxation and public welfare schemes. Though, being a guess, that may be just based on stereotypes and shouldn’t be given too much weight.

Anyway, here’s an excerpt:

People, the United States is not a person! Only in DSGE models do we assume that all individuals are identical! There is no "our" to which general statements can be attached.
Yes, going from autarky to free trade will raise the GDPs of both nations, but that is a very far cry from saying that a large number of individuals will not be made worse off in the process. I figure that NGM is familiar with the Stolper-Samuelson theorem, so I guess he is assuming the political process always provides adequate compensation for the losers??

ROFLMAO, anyone?
Here's a case for free trade:

Individuals should be allowed to contract with whoever they wish, without government interference based solely on geography.

Now, that is not much of an economic argument, but, to tell the ugly truth, THERE ISN'T MUCH OF AN ECONOMIC ARGUMENT.

Once you factor in agent heterogeneity, imperfect competition, increasing returns, and an arbitrarily large number of traded goods, the welfare economics of free trade is murky at best.

More good stuff making the same point here

Friday, February 18, 2011

Manipulating maths for whose amusement?

Amplify’d from www.technologyreview.com

Q&A: The Experimenter

Gary Loveman, the CEO of Caesars Entertainment, says there are three ways to get fired from the hotel and casino company: theft, sexual harassment, and running an experiment without a control group.

Loveman, who has a PhD in economics from MIT and was a professor at Harvard Business School, has impressed the importance of data analysis on his employees, who are expected to quickly scale small tests into company-wide initiatives. For example, they might test which is likelier to get customers to spend more: a free meal or a free night in a hotel.

When you got your economics PhD from MIT in 1989, subdisciplines like behavioral economics and experimental economics had a mixed reputation. Now—a couple of Nobel Prizes in the field later—they seem to be cornerstones of how many businesses and industries try to innovate.

My impression is that when I got my PhD, we were really manipulating mathematics for our own amusement, and we weren't producing all that much to help real people make real decisions. That was dissatisfying to me and, frankly, frustrating. The notion that we could do experiments based on the central tenets of economics and have that make a real-world difference was exciting. Of course, with Freakonomics and Predictably Irrational these themes have become more popularized and accessible. It's a very heartening development, and it's increased my enthusiasm for my own discipline enormously. 

What do you like to tell your academic colleagues about the challenges of real-world experimentation and innovation?

Honestly, my only surprise is that it is easier than I would have thought. I remember back in school how difficult it was to find rich data sets to work on. In our world, where we measure virtually everything we do, what has struck me is how easy it is to do this. I'm a little surprised more people don't do this.

Read more at www.technologyreview.com
 

Experimental evidence on infinitely repeated games??? Infinity is a loooong time!

Surely an ongoing study by definition, reporting on some results from a work in progress. And from the most prestigious economics journal - the American Economic Review - no less.

My own criticism of the predictions from the theory of infinitely repeated games would be more directed towards their lack of applicability in my (AFAIK) finite life.

However, if I could gain immortality only by agreeing to spend it sitting in a laboratory playing prisoner's dilemma for ever - then I think I would pass. My guess is they have a sample selection problem.

And yes, I know I'm being dumb.

And no, I'm not being serious.

Amplify’d from www.ingentaconnect.com
The Evolution of Cooperation in Infinitely Repeated Games: Experimental Evidence

Authors: Bó, Pedro Dal; Fréchette, Guillaume R.

Source: The American Economic Review,
Volume 101, Number 1, February 2011, pp. 411-429(19)

Abstract: A usual criticism of the theory of infinitely repeated games is that it does not provide sharp predictions since there may be a multiplicity of equilibria. To address this issue, we present experimental evidence on the evolution of cooperation in infinitely repeated prisoner's dilemma games as subjects gain experience. We show that cooperation may prevail in infinitely repeated games, but the conditions under which this occurs are more stringent than the subgame perfect conditions usually considered or even a condition based on risk dominance.

Read more at www.ingentaconnect.com

Monday, February 14, 2011

Economists should not be unduly concerned with reality?

It’s “Quotes out of context” day today. Here’s a couple of interesting quotes by prominent economists that I came across in a blog-post I stumbled onto. None of them really say anything factually wrong, but they seem (out of context, at least) indicative of an attitude valuing logically correct, sophisticated and elegant mathematical systems over pragmatically useful and informative, well-supported theories about the world. One danger of this is that if we use the word “economic theories” about both logical systems and theories-of-the-world, and if we also say that logical systems are correct or true when they are logically consistent and valued by economists, then it is only a small slip of the mind before we allow our views of the world to be colored and influenced by the logical systems that have yet to be related to reality.

There’s one by Samuelson:

Nobel Prizewinner Paul Samuelson's conclusion in his famous 1939 article on "The Gains from International Trade":

"In pointing out the consequences of a set of abstract assumptions, one need not be committed unduly as to the relation between reality and these assumptions."[3]

This attitude did not deter him from drawing policy conclusions affecting the material world in which real people live.

And one from

the textbook Microeconomics by William Vickery, winner of the 1997 Nobel Economics Prize:

"Economic theory proper, indeed, is nothing more than a system of logical relations between certain sets of assumptions and the conclusions derived from them... The validity of a theory proper does not depend on the correspondence or lack of it between the assumptions of the theory or its conclusions and observations in the real world. A theory as an internally consistent system is valid if the conclusions follow logically from its premises, and the fact that neither the premises nor the conclusions correspond to reality may show that the theory is not very useful, but does not invalidate it. In any pure theory, all propositions are essentially tautological, in the sense that the results are implicit in the assumptions made."[4]

Thursday, February 10, 2011

Should we see it coming?

Michael Lewis has a wonderfully engaging, well-written (long) article about the Irish economic catastrophe in Vanity Fair. Worth reading for all sorts of reasons.

Here, I just want to point out the simple arguments and observations used by an economics professor during the boom to argue that there was a housing bubble. It’s puzzling how something that seems obvious in retrospect, based on simple, big-picture statistics that were easily googled at the time, could be so ignored or downplayed or rejected by economists and others alike at the time. The sense that “this time is different,” “past cases don’t apply,” and that all sorts of more or less good “small” arguments are enough to (psychologically?) weaken the impact of the big-picture items. Sometimes, the difficult thing is to just keep pounding on the big, strong, clear argument instead of allowing yourself to get derailed into lots of smaller-scale discussions of all sorts of details that don’t really count for much in the big picture. (It seems to me, for instance on the basis of this graph, that Norwegian house prices are grossly inflated today (the red curve is Norway, the blue US, both in real terms and normalized to 1890 levels)– but when I present this graph to others I constantly get derailed into side-tracks like “building standards are more stringent now than in the past, which might have increased costs”)

Morgan Kelly is a professor of economics at University College Dublin, […] Kelly saw house prices rising madly and heard young men in Irish finance to whom he had recently taught economics try to explain why the boom didn’t trouble them. And they troubled him. “Around the middle of 2006 all these former students of ours working for the banks started to appear on TV!” he says. “They were now all bank economists, and they were nice guys and all that. And they were all saying the same thing: ‘We’re going to have a soft landing.’ ”

The statement struck him as absurd: real-estate bubbles never end with soft landings. A bubble is inflated by nothing firmer than expectations. The moment people cease to believe that house prices will rise forever, they will notice what a terrible long-term investment real estate has become and flee the market, and the market will crash. It was in the nature of real-estate booms to end with crashes—just as it was perhaps in Morgan Kelly’s nature to assume that, if his former students were cast on Irish TV as financial experts, something was amiss. “I just started Googling things,” he says.

Googling things, Kelly learned that more than a fifth of the Irish workforce was employed building houses. The Irish construction industry had swollen to become nearly a quarter of the country’s G.D.P.—compared with less than 10 percent in a normal economy—and Ireland was building half as many new houses a year as the United Kingdom, which had almost 15 times as many people to house. He learned that since 1994 the average price for a Dublin home had risen more than 500 percent. In parts of the city, rents had fallen to less than 1 percent of the purchase price—that is, you could rent a million-dollar home for less than $833 a month. The investment returns on Irish land were ridiculously low: it made no sense for capital to flow into Ireland to develop more of it. Irish home prices implied an economic growth rate that would leave Ireland, in 25 years, three times as rich as the United States. (“A price/earning ratio above Google’s,” as Kelly put it.) Where would this growth come from? Since 2000, Irish exports had stalled, and the economy had been consumed with building houses and offices and hotels. “Competitiveness didn’t matter,” says Kelly. “From now on we were going to get rich building houses for each other.”

The endless flow of cheap foreign money had teased a new trait out of a nation. “We are sort of a hard, pessimistic people,” says Kelly. “We don’t look on the bright side.” Yet, since the year 2000, a lot of people had behaved as if each day would be sunnier than the last. The Irish had discovered optimism.

Their real-estate boom had the flavor of a family lie: it was sustainable so long as it went unquestioned, and it went unquestioned so long as it appeared sustainable. After all, once the value of Irish real estate came untethered from rents there was no value for it that couldn’t be justified. The 35 million euros Irish entrepreneur Denis O’Brien paid for an impressive manor house on Dublin’s Shrewsbury Road sounded like a lot until a trust controlled by the real-estate developer Sean Dunne’s wife reportedly paid 58 million euros for a 4,000-square-foot fixer-upper just down the street. But the minute you compared the rise in prices to real-estate booms elsewhere and at other times, you re-anchored the conversation; you biffed the narrative. The comparisons that sprung to Morgan Kelly’s mind were with the housing bubbles in the Netherlands in the 1970s and Finland in the 1980s, but it almost didn’t matter which examples he picked: the mere idea that Ireland was not sui generis was the panic-making thought. “There is an iron law of house prices,” he wrote. “The more house prices rise relative to income and rents, the more they subsequently fall.”

Tuesday, February 8, 2011

Why scientists are liberals – some speculative comments

The Freakonomics blog discusses political bias in sciences, and quotes a social psychologist who estimated that 80% of attendees at a conference were liberals (he asked for a show of hands). Dubner seems worried that political views will shape research conclusions, and writes:

How can it be that an academic field is so politically homogeneous? What kind of biases does such homogeneity produce? What sort of ideas get crowded out? And how homogeneous are other disciplines?

I have to say that I was surprised at the overt political (leftward) bias exhibited by several prominent economists at the recent American Economics Association meetings, although my sample set was quite small.

It is interesting — and sobering — that two fields, psychology and economics, that we rely upon to describe and amend bias in the world are themselves so susceptible to bias within the ranks of their practitioners.

Krugman disagrees, implying that research conclusions probably push attitudes towards the liberal side, saying

Biologists, physicists, and chemists are all predominantly liberal; does this reflect discrimination, or the tendency of people who actually know science to reject a political tendency that denies climate change and is broadly hostile to the theory of evolution?

Now, I don’t mean to say that political bias in the academy is absent, although it’s not consistent: I can well imagine that it’s hard to be a conservative in some social sciences, but in economics, the obvious bias in things like acceptance of papers at major journals is towards, not against, a doctrinaire free-market view. But the point is that doing head counts is a terrible way to assess that bias.

It might be that the most recent amusing statistical post on the blog for dating service OK Cupid has the answer. The post analyzes its database to identify the most unthreatening, innocent questions that best predict characteristics that you may not want to ask about directly (whether they’re religious, would have sex on a first date, their political ideology etc.). Based on their national US data they write that the question identifying politics is

  • Do you prefer the people in your life to be simple or complex?
Because...

We were very surprised to find that this one question very strongly predicts a person's ideas on these divisive issues:

Should burning your country's flag be illegal?

Should the death penalty be abolished?

Should gay marriage be legal?

Should Evolution and Creationism be taught side-by-side in schools?

In each case, complexity-preferrers are 65-70% likely to give the Liberal answer. And those who prefer simplicity in others are 65-70% likely to give the Conservative one.

Seems to me that this is pretty consistent with the “bias” in academia. Academia is often very much concerned with complexity – finding nuances in interpretations and methods, considering alternative explanations for patterns in data, etc. If you prefer simplicity as a general trait in people and thoughts you would probably be pretty frustrated as an academic. And a 2:1 ratio is roughly 66%, which isn’t that far away from the estimate of 80% that we started with.

Sidenote: Interesting that Dubner is so worried by the left-wing attitudes of economists he encountered, given that his freakonomics podcast on how the world would look if it was driven by the gloriously rational economists basically said they would implement Milton Friedman’s pretty libertarian proposals. His (presumably unbiased and representative?) economist picked to answer on the behalf of the profession was Russ Roberts at George Mason University, who answered that his policy program would

start with some obvious things. I would get rid of the Department of Commerce. The Department of Commerce doesn’t do anything except subsidize exports, which is just a way of saying it makes certain companies rich at the expense of the rest of us. So I don’t think the Department of Commerce does anything particularly useful, I would get rid of that. I’d get rid of the Department of Education. I don’t think that the Federal Government has any productive role to play in the school system. I’d get rid of all tariffs. I’d let people be free to buy whatever they wanted from all around the world. What else? I would get rid of the minimum wage law, which I think makes it hard for low-skilled people to find work; it makes them artificially expensive. I’d change the Federal Reserve. We spend a lot of time trying to find the right interest rate. That’s a fool’s game that has contributed to the current crisis. So I would change the Federal Reserve. I would certainly at a minimum require it to only care about price stability. Right now it cares about price stability, unemployment, the health of the stock market, Wall Street salaries, evidently. So I would get all of those things out. It’s going to be hard to do legislatively, so I would probably replace the the Fed with a Friedmanite fixed growth and money supply or just abolish it entirely and let private money emerge. I’m getting out of control here.

You don’t say…


Update: More links and discussion here from McArdle in the Atlantic. Her take seems to be that there is a bias, that it is amusing to see conservatives (usually dismissive of bias accusations) believe it and liberals (usually sympathetic to bias accusations) dismissive, that it is unsolvable, and that we should all just try harder to get along and see each other's point of view.

Thursday, February 3, 2011

Why intuitive stories are important and dangerous

Below are some excerpts from a blogpost on the importance of "simple" stories/models that Paul Krugman praised on his NYT blog. I fail to find a simple moral to the story as it seems (to me) to involve a lot of different views on this issue, such as (in my formulations):



"Simple case-stories/thought-experiments are a necessary adjunct to sophisticated models/theories - because we cannot reason using models but need simplified versions that our brains can grasp"

"Simple case-stories/thought-experiments are rhetorically convincing in discussions/debates"

"Simple case-stories/thought-experiments trigger the psychological feeling of understanding/insight which is a better signal of truth than other kinds of evidence"

"Formal/standard models in economic theory are accepted because other economists accept them (emperor's new clothes) but people who accept them don't really understand the mechanisms they involve"

"Economists are confused concerning what it takes to evaluate claims about the real world"



Here's part of Krugman's comment on the same post (http://krugman.blogs.nytimes.com/2011/02/02/models-plain-and-fancy/ ):



"I have nothing against mathematical models and econometrics. But my experience is that many misunderstandings in economics come about because people don’t have in their minds any intuitive notion of what it is they’re supposed to be modeling. The whole notion of an economy-wide shortfall in demand is just hard to grasp — by famous economists as well as the lay public; quite a lot of our hopeless public debate reflects the fact that many people, some of them imagining themselves to be sophisticated about the issue, just can’t visualize what Keynesian ideas are about. But the baby-sitting coop offers a human-scale example, and makes the whole thing clear."

Amplify’d from modeledbehavior.com
more than any other analysis the baby-sitting coop story made me a confident Keynesian. Before then I could parrot the New Keynesian models and understood that this was more or less what a smart economist was supposed to say.

However, I didn’t know how to counter the logic of Laizze Faire except to say, “well there are sticky prices and an Euler equation and so the household will adjust consumption . . . “  This is compelling to virtually no one – not even, on a deep level, to myself.

When it really came down to it, I would have been left with “Great Depression! Want it to happen again? No? Then we need to spend more money or cut taxes! Why? Because I am very smart and I have a whiteboard. Do you have a whiteboard?”

However, a simple story about baby-sitting and it all fell into place
Read more at modeledbehavior.com