How social science got the “psychoticism” factor wrong

Remember the paper that said “conservatives” were on average more likely to exhibit “psychoticism,” but then it turned out there was a statistical mistake and this should have been attributed to “liberals,” at least within the confines of the paper’s model?  How did it all happen, and why did it take so long to correct?  Jesse Singal has the scoop, here is one excerpt:

Hatemi is convinced that Ludeke is out to get him. In our phone conversation, he repeatedly impressed on me just how minor the error is, how few times the papers in question had been cited, and how much of an overreaction it was for anyone to care all that much. “This error is freaking tangential and minor and there’s nothing novel in the error, whether [the sign on the correlation] was plus or minus,” he told me. “There’s no story. And I wish there was — if there’s any story, it’s, Should people be allowed to honestly correct their errors, or should you lampoon them and badmouth them for everything they didn’t do because they had a real error they admit to?”

Yes it’s that kind of story.  There is much more at the link, including tales of academics acting “like dicks.”  Here is the conclusion of the piece:

…the social-science landscape isn’t yet as embracing as it could be — and should be — of the replicators, challengers, and other would-be nudges like Ludeke who tend to make science better and more rigorous, who make it harder for people to coast by on big names and sloppy research.

For the pointer I thank Daniel Klein.

Comments

Is my name big enough that I can coast by on it?

That would be quite atrocious.

"Hatemi is convinced that Ludeke is out to get him. "

That's pretty funny. In other words, Hatemi whose probably a liberal, happens to be a bit psychotic himself.

The depressing part of the story is that if Ludeke had it to do over again, he says that he definitely wouldn't:

“Frankly, my advice to all the other fellow peons out there is until that happens [and data openness is more common], my experience is not supportive of engaging in this,” said Ludeke. “The amount of time that this project took for me, the amount of time and the amount of stress, makes this clearly a bad choice to have pursued, unequivocally. If you have a chance to say something to that effect in the article — ‘I do not find this to be a recommendable experience’ — I would like that.”

There was also the somewhat overblown

http://www.vox.com/2016/7/14/12120746/science-challenges-fixes

I think the glass is half full here as well. While there is certainly room for improvement, science advances, three or for steps forward for one back. And yes, some of those are baby steps, safe projects with inflated press releases.

And as an aside, when twin studies etc do show some genetic connection to politics, what is normal?

An unfortunate use of the word "hardwired" here for results that show anything but:

http://www.pewresearch.org/fact-tank/2013/12/09/study-on-twins-suggests-our-political-beliefs-may-be-hard-wired/

I would say that when contributions are ~50% that says 1)we should be patient with people who might just be different but 2) expect some adaptability as well.

We are neither hardwired nor blank slates, something that strikes me as both reasonable and comforting.

Ah, a brand new TED from Pinker on this:

How Do Nature And Nurture Combine To Make Us Who We Are?

Sorry, not new, just newly linked from NPR.

The standard social science model, in practice, continues to insist on something close to the blank slate as a practical matter. Virtually all of sociology and anthropology, for example.

Anyone who has been the parent of more than one child knows, on the other hand, that, to a surprising and humbling extent, kids come 'a certain way'. Each one is of course unique, but children are so far from blobs of indistinguishable clay as to make the blank slate idea absurd.

Pinker wrote a whole book on the subject, called The Blank Slate. By 2016, Pinker has become a dangerous enemy to large swathes of the orthodox social science world.

"The standard social science model, in practice, continues to insist on something close to the blank slate as a practical matter. "
I hear people say that people say this more than I hear people say this.

Really? Look at how many people routinely use 'social construct' as a Theory of Everything.

How about 'disparate impact' as prima facie evidence of unfair discrimination, case closed?

I get it. It's ok to talk about how my kids are different from one another, but this 'they come a certain way' observation gets really touchy really quick. Tact and decorum squash scientific inquiry. Tricky.

Isn't one of the primary justifications for a welfare state that people are not blank slates, that some are born to unlucky circumstances and morally deserve support from the more fortunate? If one is hard wired for low intelligence and drive, no amount of hard work will be enough....

If someone like a bushman is in a state of nature, then just about everything that separates him and me is a social construction, yes.

Americans burned witches ~300 years ago. What separates them from me are new social constructs.

Conservatism is the belief that social constructs are just right. Progressives believe culture should be hacked.

The current PBS "The Nine Months that Made You" merges lots of threads into a "radical liberal" narrative of fetal development. Science advances by looking for deviations from what is believed to be the status quo. The series picks the deviations that most challenge the status quo conservatives try to keep constant.

I'm sure it's pure chance, but the example of girls who think they are boys who go on to grow a penis at puberty is particularly timely. Genetic AIS is much higher in one community so transgender children are common enough to be accepted as a normal. Why would a baby raised as a girl know she was a boy before she grew a penis? Would conversion therapy make the penis not grow?

@anon,

I mostly agree with your last line.

Progressive: Well, why do we do it that way?
Conservative: We've always done it that way.
Progressive: That's not a reason. Look, I've reasoned this thing out from first principles...
Conservative: There's your mistake.

Conservatives view their culture as an organic thing. Things are the way they are because they work, having been honed over time. Excising bits of the culture that serve no obvious purpose on a Progressive clean sheet of paper sounds reckless at best.

But of course, if everyone were conservative, we'd still be in caves, right?

It's almost like it's a good thing there are different kinds of people.

I hear people say that people say this more than I hear people say this.

And you hang around here, and not around anthropologists and sociologists, right?

This is getting more common in journals. Perhaps the status of open data journals should increase.

"authors are expected to curate the above data for at least 5 years after publication and provide a transparent process to make the data available to anyone upon request........._____ reserves the right to refuse publication when authors are unwilling to make the underlying data available or otherwise refuse to comply with this Data Policy."

But that's really not good enough (as the IRS amazing epidemic of 'lost' and 'damaged' hard drives demonstrated) -- in cases of controversy like this, authors will find that they have -- oops, so sorry, our bad -- lost the data. So data needs to be submitted with the paper as part of the review process or, at least, provided to the journal once the paper has been accepted for publication. Subsequently, the journal is the one that should be archiving and providing the data and it should not notify the authors when the data has been requested or who requested it.

Someday public science will reach the github level of granularity, with daily incremental results pushed to a public server.

Dozens of "We need a GitHub of Science" articles point the way.

Right, today's system is slow. The "judge" is the publisher, so it's necessary to: 1) contact the publisher, 2) the publisher contacts the author, 3) the author answers with Yes but not when..........while it would be much more efficient to have a link in the article to download the data.

Today's system presumes that data hoarding is fair and advantageous. If you get a government grant to perform some regime on mice, and examine them weekly, the presumption is that you own that data, and not society as a whole. You can publish it, delay it, or not, with your best interpretation and no one else's.

A better system would publish the regime first, and then incremental results, and then the interpretation if any.

Of course that would be asking researchers to give up the my-data, my-interpretation, monopoly.

It also would eliminate the career incentives for researchers to gather or generate the data in the first place.

You don't think agencies would fund for data collection, or you don't think anyone would accept funds for data collection?

As I understand it, there are some domains where, because of their structure, data are already produced and shared this way. Earthquakes and astrophysics?

(If you are saying that you recognize the perverse incentives in the vox piece, I am saying I recognize them too, but prefer not to continue them.)

Conservatism is the belief that social constructs are just right. Progressives believe culture should be hacked.

Sayeth a man who knows nothing of the conservative critique of contemporary social relations.

Dod and NASA contractors are required to maintain all the design data and validation programs. Thus there is an largely volunteer hobbyist archivist community who maintain old equipment hacked to enable data transfer to new media where computer simulators of old computers, in some cases running computer simulators of old computers, provide access to knowledge bases for things like nuclear weapons and reactors, the Shuttle, Saturn V, earth and planet sensing satellites.

As an engineer responsible for maintaining such records because I worked on COTS products sold to the US government under contract, the genius MBAs outsourced data retention to outside parties with cost cutting mandates that ensured old data would not be retained. The bean counters would pretty much respond, "we aren't paid to keep old records and my responsibility is shareholder value", and tell the outside contractors to trash data and the systems needed to access it. Thus the old engineers had disk drives, tapes, listings stashed under their desks in boxes they moved from office to office as they moved from project to project, with occasional flurries of activity looking for some old data or some old hardware so old data could be retrieved.

When it came to email, we were told to purge it because it cost too much to store it. I worked with disk drives which we were making exponentially bigger, so that was bogus. But never try logic with MBAs focused on shareholder value - ie, figuring out who to fire.

"a big-name researcher with an aggressive, hard-charging reputation": I take it that in British colloquial English that would be "arsehole"?

Penn State? Maybe he and Michael Mann are buddies. They both are arseholes.

I'm not clear on what the problem is here. Some people continue to believe, despite overwhelming evidence to the contrary, that social science is Science. Like flat-Earthers, I doubt it's worthwhile to spend much time in their world. The fact that some nasty piece of work like Hatemi can hide his incompetence and dishonesty by using the system for years clearly indicates the system isn't much interested in either competence or honesty. The fact that PLOS accepted work by such hacks is clear evidence of the venality of its editors, as well as an indication of the crisis in scientific publication in general. The fact that Hatemi continued to be highly regarded, despite his clear unwillingness to admit to error, is a indictment of the discipline, not simply one academic. IMHO, one of the hardest and one of the most important, if not the most important, traits a Scientist must have is intellectual honesty and a willingness to admit one's own errors. Without that, it is just a beauty contest - but then, that's just exactly what the "social sciences" are, innit.
As an aside, Ludeke did make his own errors. His advisor appears to have handled the first exchange well, with the exception of injecting a time frame for Hatemi's substantive response. He then should have handed the problem back to Ludeke to handle. Ludeke should have (and apparently did) request the data for his analysis but again with a (reasonable) time limit. At the end of that time, he should have reminded Hatemi that he hadn't received the data (as far as I saw, Hatemi never agreed to provide it) and if the journal of record required published papers to be backed with open data, to notify that journal of a violation of terms, also he should have written a letter to the editor expressing his surprise and concern - essentially calling out Hatemi. And then left it alone. It was the responsibility of the Community of Scholars, not Ludeke, to fix the problem. They failed miserably, but Ludeke was partly to blame for not publicly documenting his concern. John Stuart Mills:"Bad men need nothing more to compass their ends, than that good men should look on and do nothing."

There are two ways to view data generated in public science. One extreme would be to say it is all public, a work for hire. The other extreme would be to say that money is nothing, and intellectual property is created in the mind of the investigator. We lean too far to the second extreme.

A scientist should be rewarded for good work, but if they ask for a grant, they are also working for hire. A combination of the two.

Oh, I remember that evil leftist big government view back in the early 80s especially.

The virtuous conservative view was that the private contractors paid to build advanced defense system stuff for the taxpayers should be given exclusive rights to it so they can make big profits in the private sector, and then they will bid a lot less for future contract because they will make so much private sector profit from what they needed to invent to meet the government contracts.

The problem to be solved was all the public domain inventions that anyone could turn into products, but only if they figured out how and did so at pure labor cost and no profit. After all, if you turn a public patent into a desirable product, competitors will set up factories churning out products they sell for less.

Thus all the changes in patent policy mostly in the 80s, changes that Alex complains about.

Somehow, the focus shifting from engineers deciding what should be built based on what they thought was cool no matter the cost to shareholder value overseen by MBA bean counters has failed to increase the amount of invention and innovation over the bad old WWII to 1980 liberal tax and spend big government era.

Note, the government was prohibited from advocating the Internet because the private sector would invent superior products like SNA, RJE, Aol, MSN, DECnet, converging on a superior industry standard like ISO/OSI. Al Gore made maximum use of an obscure paragraph in a reauthorization bill in 1992 for technology training to derail Microsoft in 1994, IBM, Aol, ...

Everything Internet originated pre-80 in institution organization so it could not be controlled by a for-profit private contractors in terms of data and methodology.

In contrast, the ISO/OSI was staffed by "volunteers" who pay their way and support the standards institution, meaning everyone is an employee of a big corporation paying millions to be involved in developing industry standards. And the public data access requires paying the costs of archiving and publishing it. Thus access to one chapter of the ISO standards typically costs $25,000 a year.

The Internet standards were known to be flawed in serious ways in the 70s so the institution Congress looks to, the National Bureau of Standards was tasked with recommending better standards working with industry. In the 80s, one way to cut costs was buying COTS products instead of government funding custom products by the hundreds which did pretty much the same thing. A name change and more responsibility did not increase funding, so progress on standards for COTS products progressed at the rate the corporations getting the cost plus contracts for non-COTS work spent just enough on industry standards to ensure no competitor could benefit.

MBAs focused on shareholder value do not like paying for innovation because it disrupts highly profitable monopolies.

Engineers love innovation because they love the excitement of disruption, especially when they can pay lots of friends more money gaining lots more friends.

"It's ok to fuck up because the paper hasn't been cited all that much" is an incredibly lame excuse.

How did the paper gets so much coverage in the mainstream media, which is different from being cited in journals? Who pushed it out there half-baked?

The box paper suggests that research incentives encourage safe projects, small increases in knowledge, that can also be spun for high publicity.

This seems to fit that.

Comments for this post are closed