Gelman on the Replication Crisis and Social Media

This Andrew Gelman post on the replication crisis and the role that blogs have played in generating that crisis starts off slow but just builds and builds until by the end it’s like holy rolling thunder. Here is just one bit:

Fiske is annoyed with social media, and I can understand that. She’s sitting at the top of traditional media. She can publish an article in the APS Observer and get all this discussion without having to go through peer review; she has the power to approve articles for the prestigious Proceedings of the National Academy of Sciences; work by herself and har colleagues is featured in national newspapers, TV, radio, and even Ted talks, or so I’ve heard. Top-down media are Susan Fiske’s friend. Social media, though, she has no control over. That’s must be frustrating, and as a successful practioner of traditional media myself (yes, I too have published in scholarly journals), I too can get annoyed when newcomers circumvent the traditional channels of publication. People such as Fiske and myself spend our professional lives building up a small fortune of coin in the form of publications and citations, and it’s painful to see that devalued, or to think that there’s another sort of scrip in circulation that can buy things that our old-school money cannot.

But let’s forget about careers for a moment and instead talk science.

When it comes to pointing out errors in published work, social media have been necessary. There just has been no reasonable alternative. Yes, it’s sometimes possible to publish peer-reviewed letters in journals criticizing published work, but it can be a huge amount of effort. Journals and authors often apply massive resistance to bury criticisms.

If you are interested in the replication crisis or the practice of science read the whole thing.

Aside from the content, I also love Gelman’s post for brilliantly mirroring its metaphor in its structure. Very meta.

Comments

Andrew Gelman crushing it.

Do read the whole thing. Self-recommending.

In this Straussian age, it's good to see a straightforward mauling from time to time.

Never heard of Gelman. I've heard of Geller, a Hungarian GM. "Ioannidis" in Gelman's timeline a certain Scott Sumner claims (if I recall correctly) of having never heard of him, despite being a pioneer in the replication bias controversy. And the bias in economics is just as real. The famous "Euler Equation" used in DSGE econ models has been shown to be fake (not observed by the data), see http://noahpinionblog.blogspot.gr/2014/01/the-equation-at-core-of-modern-macro.html What does that say about e-CON-o-mics, my former roommate, BD?

This is like a late night bull session from the good old days!

DSGE, and the whole microfoundations obsession, always struck me as very ambitious. We don't learn about human behavior by studying quarks, do we?

Which isn't to say that the Keynesian habit of blowing off micro-effects is any better.

Macro has always been something of a mess. Dog bites man.

Anyway, I'm sure Noah is thrilled to have you in his corner.

But DSGE is based on equilibrium assumptions which treat entire sectors of economies in line with Newtonian physics.

Hasn't it always been this way? Some upstart dares contradict or question the master. Hasn't it been said that science advances one death at a time.

What I find fascinating is how the smartest people are as stupid as people have always been and in spite of (or maybe because of) their exquisite education they are profoundly self unaware. The history of science is the history of correcting flaws and mistakes and misunderstandings.

I don't think social psychologists are a particularly smart bunch, even compared to other academic psychologists.

Then again, as Gelman (who is reasonably smart) writes, not even the smartest people realized the true extent of the problem until recently, with the exception of a few like Meehl. (I made the same point here).

They deal with more complex problems in a context of a general absence of guinea pigs.

The problem is the word 'science'. Science is a way of proceeding. It's very effective at improving our understanding of the physical world and, to a large extent, the biological world. Much less effective in the human world.

What we see here is an entire discipline that is basically over its head, having staked out all kinds of overbroad conclusions under the mantle of 'science' for a generation, but it was never any such thing.

Part of the essence of science is a desire to disprove your theory in any way imaginable. What we have to day is a collection of 'social marekting' disciplines, in which children wield statistics they don't understand.

Even someone as scrupulous and possessed of deep understanding as Daniel Kahneman looks to have swallowed some dubious stuff.

I'm not saying it's hopeless, but that practitioners in the study of human beings need to be at least as humble about their theories as real scientists. The economics profession is far and away the best of a very bad lot in this sphere.

Almost all published "science" is this way - i,e, largely useless. It's not just the social sciences. The problem is that true insights worthy of papers are very very rare and you have to be smart and lucky to find one. But the incentive system in Universities is to publish as many papers as you can on as unique effects. So people do what is expected of them and everyone agrees to play the game. It is a really sad waste of talent, but I guess every society has it elements of this - probably it is better than studying the bible in monasteries. Of course none of this would be happening if it wasn't for state taxation.

Lots of CRISPR replication going on. All wasted?

Related hilarious Twitter account:

https://twitter.com/RealPeerReview

I'll take the monasteries.

Henry VIII said the same thing.

They are not useless. Most papers represent serious efforts to explore a few variables in an N-variable problem, which runs into the inability to describe an N-dimensional space in N-x dimensions (think conic sections in 2-D and 3-D).

However, when solving real-world problems, those papers provide a lot of information that allows you to eliminate possible solutions and focus on the solution space that may actually work. This is especially true for practical decisions where a 51% probability of being right (well below 95% confidence levels) is much better than no information, but you just have to keep in mind that you really don't know for sure.

"Part of the essence of science is a desire to disprove your theory in any way imaginable."

This is a very important problem, the fact that it often does not apply.

But, in many things, one instance of falsification does not prove that a theory or empirical demonstration might not still be relevant or true for many other cases. Because, very fortunately from the perspective of freedom and treating people as ends unto themselves and not mere object to be studied or used, it continues to be true that social scientists (or potentially those with nefarious intentions) cannot control all variables while studying people stuff.

Yes, Kuhn in his extremely readable "Structure of Scientific Revolutions" points out that often paradigms supporting existing normal science are often difficult to change and its only when their practioners die away that the new paradigm can get entrenched.

That's pretty much what Planck said. Yet I remember reading about a psychologist (alas!) who studied the matter and found that the New Physics was accepted in much the same proportions, and at much the same speed, by old and young physicists. I suspect that the issue is that only old physicists had the power to try to impede the acceptance of the New Physics.

There is little evidence for Planck's hypothesis:
http://lesswrong.com/lw/14/does_blind_review_slow_down_science/

What a fine citation. Tip o' the hat.

I think it's easy to think smart people are stupid when you don't really understand their work, and know it mostly through the words of critics and stone-throwers.

The problem isn't that people in the system aren't smart, the problem is that they are. So they are very good and innovative at responding to the incentives to publish papers, even when the effect being described isn't really there or is not really significant. To really see whether a particular science is valid, see if it is being used commercially (or at least by other people not in the same discipline). Stealing from Steve Sailer a bit here, if priming for instance was such a powerful effect, you can bet that advertisers would be all over it. And good social scientists would be hired for millions. But that's not really a thing.

Two things. First it is important to remember that governments are not complete idiots. They actually allocate the least research money to the least documented results. If I recall correctly most government research is medical, which while having a known statistics problem, is seriously about "the affect being described."

Second, I think advertisers actually had "priming" first. Do images on cereal boxes lead you in any way to think cereal might be fun? What does a jumping bunny actually have to do with cereal? Or what does it have to do with "activation of a node in this network, which may serve as a filter, an interpretive frame, or a premise for further information processing or judgment formation?"

Advertisers use priming all the time.

First, there's lots of sexy people, or in situations which are appealing ... all sorts of associations built up, generally in social contexts where the product is most likely to be consumed or bought. And then they just present the brand name at the end.

How effective is priming in political propaganda? Imagine for a minute.

There is a man. Any conclusions?

There is a man in the desert. Any conclusions?

There is a man in the desert with a turban or something wrapped around his head and covering his face. Any conclusions?

Now, imagine that I flash this imagine in a political ad for 0.2 or 0.5 seconds.

Both advertisers and political propagandists are all over this stuff. It's one of the main reasons that I refuse to get my information from videos and TV, although avoiding involuntary associations is increasingly difficult due to new technologies which you cannot even turn off, if targeted by them.

"Priming" is much like "marketing."

One difference, however, is that social scientists are expected to search out Enduring Truths, while marketing researchers don't want you to believe the research they did for you will last forever. They want you to pay them more money each year to see if things are changing.

http://takimag.com/article/the_replication_crisis_and_the_repetition_crisis_steve_sailer/print#axzz4KhA9XJyh

The econoblogosphere in a nutshell.

Stupid is as stupid does. Someone who considers the number of citations or publications as a measure of anything except the number of citations or publications is missing something fundamental in the endeavor they are involved in.

The whole scientific method and tradition is to work around the stupidity of people. When it doesn't work around them, what can you call it?

Believe me I understand the challenges. It is extraordinarily difficult to overcome human tendencies in any human endeavor. But if it takes outsiders in their pyjamas to challenge the foundations of your edifice, and they threaten it to that extent, what do you call the supporters and defenders of the edifice?

Status markers are a damper on progress.

"Work by herself and har colleagues is featured in national newspapers, TV, radio, and even Ted talks, or so I’ve heard." -Andrew Gelmon.

I am certain that her is spelled h-e-r, and not h-a-r.

I've got to wonder how much of behavioral economics is based on flawed studies.

Replication is high for the big stuff. (Time preferences, risk biases.) Probably some overreach at the edges.

Glass half full

http://www.sciencemag.org/news/2016/03/about-40-economics-experiments-fail-replication-survey

Replicability varies a lot by field. Behavioral economics replicates relatively well. Try some of the classic experiments from Kahneman/Tversky and their disciples with a group of people and it will replicate. Perception biases also replicate easily. The problems are with many social psychology experiments where the response is very noisy and the original papers got through with small number of subjects per cell and very specific experimental conditions.

Shorter Fiske: Get off our lawn

The amazing thing is taht people on social media would waste time on a faux-science like hers.

Good post.

Personally I'm of the opinion that some sort of wiki-like process will eventually supplant the traditional peer-reviewed closed-journal model in science. The conventional peer review process both takes way too long and is corrupted by internal politics and social hierarchies within the scientific community. The wikipedia type process has the benefit of allowing outsiders who have no particular political or social stake to enter the discussion and critique scientific papers in detail. As long as there's a commitment among the participants to being ruthlessly honest and rational it works fairly well. The model of having discussion pages for every article where debates are hashed out allows for much more thorough vetting of content then the conventional peer review process, in which two or three people (generally which the author selects) are roped into commenting on work they might not be too familiar with, or which they mgiht be predisposed to approve or disapprove of.

More importantly, science that the public can actually see is going to have a greater impact, which will ultimately drive scientists to want to do it that way. Right now there's far too much bullshit masquerading as science (see GMOs) infecting what sort of information the public has easy access to. That's entirely because conspiracy websites can immediately publish anything they want but the real science is behind firewalls.

Also, people who like to engage in debate may be drawn to the wiki process because it provides an online forum where people can actually argue about this science.

The trick is going to be controlling access so that discussions get whittled down to a group of people who are capable of contributing positively and rationally to debates, without excluding newcomers or non-dominant-paradigm points of view.

I'd say the problem of "internal politics and social hierarchies" is minimized by the existence of several journals dealing with similar topics. Even the most specialized scientist can choose among 5+ different journals. If the scientist is an idiot and picks fights with all editors, that signals poor social skills....and being ignored in publications is 1 more problem in a long list.

Ps. everyday is a good day to spread the word of pirate science ;) www.sci-hub.cc

Well, having everyone publishing in a friendly journal is part of the problem. Science needs debate between different camps, not everyone spiraling off into closed little groups. Also, the problem of with the political and social hierarchies in science goes way beyond publishing in journals and way up into how research funding gets allocated by groups in the NIH and NSF , etc. There's a feedback loop there, where labs that are connected to the top people at the NSF have a better chance of getting funding. And it's hard to go to conferences and meet people and get connected unless you have the funding. It becomes very insular at the top.

Friendly journal? Not at all. Possible choices are among: the Nature group, Elsevier, Wiley, Springer, all of them have at least 2 journals per topic. Then, there are the regional scientists associations for every discipline (US, Europe, Latin America, Asia) that also have journals. With the big 4 publishers, review bias is minimized. A few colleagues have stories with the regional scientist association journals (such as PNAS): long review times, etc.

If research is good, it will succeed. Problems arise when a lot of scientists compete for the same position in a hip city where you need 20+ publications. This is the incentive to cheat in the publication rat race. If your care about science, the US have universities and positions in very nice places. However, cool people does not like research positions in Texas, New Mexico or Arkansas.

Budget for conferences? Domestic US travel is cheap, most scientist's associations have young researcher grants, this is a good time to be a scientist. The system can always be improved, but I seriously doubt if 30-40 years ago it was better.

Hahaha. Young researcher grants offer a couple of grad students each year a discount on the cost of registration. So it's $500 instead of $2000 to get in the door, not counting travel costs. That's diddly squat. Plus there are restrictions on eligibility, like you have to be an undergrad, or under 25, so a post-doc wouldn't qualify. Plus lots of conferences are international. You can't expect them to be in the US every year, so you might have to wait 3-4 years for the next "friendly" conference to locate somewhere in the US near you, by which time your research is going to be out of date. And even if you get in the door at the conference, your chance of getting invited to a private lunch with the head of the NSF so you can beg for grant money are remote.

It's just a fact of reality in academia. The big money is allocated to big, well known labs with big name researchers at them. If you aren't part of a well-connected set of insiders with name recognition it's difficult to both get the funding and get published.
Yeah, sure if you're a super brilliant outsider you can publish something innovative on a shoestring budget and break in. But if you're a merely smart well-connected person you can get into Harvard and have your hand held until you land in a comfortable mid-tier academic slot where you can publish a string of mediocre papers until you get tenure.

" However, cool people does not like research positions in Texas, New Mexico or Arkansas"

Austin is cool and has sufficient cool people. Thank You for your attention.

"""Nature owner merges with publishing giant (Springer)"""

http://www.nature.com/news/nature-owner-merges-with-publishing-giant-1.16731?WT.ec_id=NEWS-20150120

I can think of a couple ways to decide if things are better or worse for young scientists now vs in the past:

a. What fraction of people who do a PhD end up working in the field in a job that actually pays the bills and lets them (say) raise a family? If a large fraction of people with PhDs in your field end up bagging groceries or making squat teaching a couple classes at the local community college while living with their parents, that looks like things are pretty rough.

b. How many years past the PhD do successful people in the field have to spend, before they get the kind of job that lets them have a reasonable middle-class existence? Is it right after the PhD? Or is it PhD plus two postdocs, then you get a real full-time job with benefits and a salary you can live on?

True, it is said that Brian Josephson as a PhD student had his paper ( which later got him a Nobel) published in Physics Letters , because chances of it getting done in Phys Rev or Phys Rev Letters was low. If the twice Nobel prize winning physicist Bardeen was the referee , it could have been rejected (per Wikipedia)

Even with a high quality open-source means of accumulating and arranging knowledge from scientific processes, you still have the matter that rigorous and effectively implemented studies are often needed to provide these inputs for review.

So, the online argument is one thing, but at the end of the day, unless theorizing about things for which we have not yet devised ethical experimental methods, it is all contingent on results and/or replicability of previous studies.

My point is that crowd-sources methods of review would point out where studies are not rigorous and not effectively implemented rather quickly. Really shoddy work would not make it through the peer review process of a million nerds living in their basements who live to destroy your self-esteem.

Gelman:

"People such as Fiske and myself spend our professional lives building up a small fortune of coin in the form of publications and citations, and it’s painful to see that devalued, or to think that there’s another sort of scrip in circulation that can buy things that our old-school money cannot."

That simile points to an underlying complaint that there may be (probably is) a "Gresham's Law" operative in the Information Age.

That simile points to an underlying complaint that there may be (probably is) a “Gresham’s Law” operative in the Information Age.

Clever analogy, but I'm not sure that people get rid of shoddy research by circulating it quickly, or that they hide the good research behind firewalls to preserve it's value.

If a career or an individual is "destroyed" by mean online comments.......that's the definition of weak. I remember being trashed in public after a presentation, it hurts, you feel inadequate......a few beers and work harder for the next.

It's the weakness of the people around him that matters. If your boss fires you because people freaked out over some innocuous twitter comment you made, it's your boss that is weak. Maybe people should toughen up and stop regarding mob sentiment as something that needs to be respected.

Responsiveness to mob sentiment is a problem, as it further motivates the mob to do more of the same.

I think we should respect that people with no formally recognized power are willing to take to the streets and otherwise become activist to have their positions recognized. In the meantime, it is important that they be encouraged to ensure that their actions are based on reason, or at least reasonable emotional responses, and not just ... following the mob, whether to belong or for lack of better things to do.

No disagreement there. As I wrote above the trick is going to be managing popular discussions on social media in a way the whittles down the number of unreasonable participants.
The Wikipedia community has some pretty high standards for participation, which is why I think it's a better paradigm than (say) twitter.

> I will not be giving any sort of point-by-point refutation of Fiske’s piece

You mean, you're not going to fisk it?

Here's s relevant article. Scientists don't even have to try to game the system, and can in fact work with complete integrity as individuals, and the research system as a whole still degrades:

http://www.theatlantic.com/science/archive/2016/09/the-inevitable-evolution-of-bad-science/500609/?utm_source=atltw

Fortunately, science's replication crisis admits a market-based solution: http://ssrn.com/abstract=2835131

Comments for this post are closed