Researchers speaking on the scientific process

When looking at success indicators, we found that indicators related to openness, transparency, quality, and innovation were perceived as highly important in advancing science, but as relatively overlooked in career advancement. Conversely, indicators which denoted of prestige and competition were generally rated as important to career advancement, but irrelevant or even detrimental in advancing science. Open comments from respondents further revealed that, although indicators which indicate openness, transparency, and quality (e.g., publishing open access, publishing negative findings, sharing data, etc.) should ultimately be valued more in research assessments, the resources and support currently in place were insufficient to allow researchers to endorse such practices. In other words, current research assessments are inadequate and ignore practices which are essential in contributing to the advancement of science. Yet, before we change the way in which researchers are being assessed, supporting infrastructures must be put in place to ensure that researchers are able to commit to the activities that may benefit the advancement of science.

That is from a recent paper by Noémie Aubert Bonn and Wim Pinxten, via Michelle Dawson.

Comments

Incentive structures affect decisions on both an individual and institutional level.

News at 11

Circa 1980, the theory advanced by economists was monopoly profits was the best incentive to increase innovation, ensuring that the private for profit sector would be more innovative in delivering cheap nuclear power, cheap human sace flight, cheaper more plentiful advanced health care, more advanced transportation, better infrastructure for every person everywhere, especially rural areas globally, but especially the US, and so much more, than government policies that directly paid more and more workers to innovate, or made the cost in taxes for the for profit sector paying workers to build new infrastructure driving innovation in how infrastructure is built, and innovation in what the infrastructure delivers to customers.

For example, the regulation of telcos, with AT&T being the biggest investor in infrastructure, had to pay more and more workers to build exponentially more capital because capital value was capped at labor costs, and that was depreciated constantly to determine invested capital that capped business profits. If a telco built out a system then stopped building new capital on a per customer basis, rates would be forced down every year.

Thus, the telcos became skilled, as did electric utilities, in getting customers to use the services more and more. State regulators, when rural America was more than half the population, rural customers got telco services from telcos at the same price as urban customers. Initially, most telephone lines were party lines and humans switched calles,, but in figuring out how to get customers to make more calls, the telcos had to innovate in making connections, innovate in how to connect to customers. I remember when party lines were common, but party lines were virtually eliminated by 1970 everywhere in the US. AT&T invented more services to sell to customers requiring building completely new capital to replace the modern crossbar switches installed only a decade before in the AT&T strongholds like Chicago.

By 1980, AT&T was trying to figure out how to charge every custom more to provide data services, like the French government telco was doing with Telex. A big growth area for AT &T was long distance, and fiber optics was the key to exploding the call volume which was increasingly data.

In the 80s I started working with a former Bell Labs engineer on connecting every computer with fiber optics at the same cost/price as coax, but we had to compete with twisted pair. There was lots of effort, but limited innovation that benefited the majority of the American people because telcos were encouraged to increase profits, which meant not paying workers to build new infrastructure driving rapid innovation.

Profits is the money not paid to workers.

Note, Milton Friedman argued telcos and power utilities should not build more capital, but instead raise prices to increase profits from those who wanted more that the status quo. He thought picking up the phone and always getting a dial tone, flipping the light switch and getting light was to costly to most consumers. If anyone wanted reliable service, they should pay extra.

Which is where we are today with Internet access. If you want good Internet you pay extra by buying in the right part of urban areas, or pay millions to have private fiber lines run to your rural compound. Businesses should focus on picking the most high profit customers, not serving the public.

There isn’t even the barest thread of logic to that wall of text.

It’s like someone pasted a Boomer culture war GPT-3 output

How else does one soapbox effectively if they can't be in the top 10 early comments on some topic close to their heart that has finally come up in MR?

Paper: patents advance employers profits (and are favored) while non-patented publications advance whatever metrics the paper authors are tracking. Never mind if we had better patents we'd have flying cars by now. And a cure for cancer.

As for Mulp's observation that phone monopolies are bad, it's not true, notice how China, Japan have national telecoms that do just fine for innovation and consumer prices, and in the USA, after the breakup of Ma Bell, a dozen telecoms have gradually reformed into 2 or 3 carriers, proving Ma Ball was efficient after all (economies of scale). Arguably with Ma Bell we'd have 5G by now.

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

That first paragraph might be the longest sentence I’ve read in my life.

Are you?

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

I would argue that 'Sensational' science such as Astronomy, where career advancement and advancing science is proportional to the Glory of the Finding over comparing openness, etc., with competition, etc. Witness the advancements communicated through a new fabulous publishing: "A Catalog of Celestial Exotica". This intends to push careers and science through its unique listings of all 'superlatives', therefore 'frontier' science. You want career and scientific progress - mainstream it.

It's not exactly clear what you are arguing here. Astronomy is surely subject to the fadishness that affects other areas of science (currently exoplanets), and there is a certain amount of gee-whiz discoveries that make the popular press. But in its practices it is a model of openness and transparency. Most of the data is public, codes are overwhelmingly open source, publications are regularly posted on preprint servers or otherwise freely available, and the scientists in the community are the most collaborative and collegial of any other field I know.

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

Big business is the worst offender in preventing advancements in science. How? By preventing innovations from being exploited, through restrictive covenants of those who create the innovations and abuse of the patent system to name two ways. That's what this blog post is describing. Nothing can ruin a promising career like revealing the secrets of big business.

-1, this blog post doesn't support this comment in any way. Indeed, it appears to be a study into the practices of Flemish researchers working for the government.

Respond

Add Comment

wayward
the scientists said they have attend too many meetings and it interferes with their research productivity

Respond

Add Comment

Respond

Add Comment

What about open letters saying that racism is a bigger problem than a pandemic? Where do those fit in?

Career preservation

Respond

Add Comment

Respond

Add Comment

"In other words, current research assessments are inadequate and ignore practices which are essential in contributing to the advancement of science. "

No, that's the wrong conclusion. The correct conclusion is that people signal that they support openness, transparency, quality, and innovation, but that their actions indicate other items have higher value.

+1

Pithy, accurate

Respond

Add Comment

"....support openness, transparency, quality, and innovation,..."
I have actually seen those word on posters and emails at my employer, alongside those signaling diversity, ethics, etc.
I call these "world peace" arguments because no one can/will argue against them. The only thing that matters is how individuals are compensated.

Respond

Add Comment

Being fundamentally built on a foundation of t openness, transparency, quality, and innovation.

OK, the quality part is open to some debate, but free software as a whole is certainly not lower in quality than commercial software when it comes to the framework of the Internet and its infrastructure.

Respond

Add Comment

Respond

Add Comment

"Academics shocked to learn that their jobs are a lot like jobs".

Respond

Add Comment

Are the researchers referenced working in government or industrial settings? Anyone who works in industrial R&D knows that openness and prestige are always in conflict.

"Actors included research institutions, research funders, scientific editors or publishers, researchers, research students, and several other actors who play an important role in academic research. "

It looks as if it's Flemish researchers in academia.

Respond

Add Comment

Respond

Add Comment

I remember science. It was a jolly good thing. D'ye think they'll bring it back at all?

not until we are done
decolonizing light & reeducating comrade houseplant biden
also newtons laws will now be called newtons rape manual

Respond

Add Comment

Respond

Add Comment

The problem is that open access publishing, negative findings, and sharing data have no value as signals of research ability. Anyone can do it, in fact, closed-source publishing has more value as a signal, because you at least have to have the prestige or peer-reviews to be granted a space in some of the journals that are selective, while arxiv is open to all. But, we have mostly succeeded to getting to the point where, all else equal, a lack of openness will count against you.

Respond

Add Comment

This sounds similar to a lot of work. Doing the right thing isn't always consistent with getting on aka dicks often prosper

Question: should we be trying to design better incentive mechanisms as the researchers seem to suggest? Or should we be trying to create conditions that encourage intrinsic motivation to do the right thing?

"should we be trying to design better incentive mechanisms as the researchers seem to suggest? Or should we be trying to create conditions that encourage intrinsic motivation to do the right thing?"
The two are inextricably linked, you effect one, you effect the other: Incentivizing exogenously will effect behavior endogenously and vice versa... Also, seeking clarification your version of "the right thing" especially in the face of "multiple truths". If you're Kantian, you're highly likely to consider that morality alone does not oblige you to "the right thing", no matter how transparent and innovative the researcher is.

"getting on aka dicks often prosper"
And also...huh?

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

I think my comment has bern deleted. It seems a very racist thing to do.

+1

Respond

Add Comment

Respond

Add Comment

Every signal can be corrupted. Every signal where it pays to be corrupted will be corrupted. Every corrupted signal will be corrupted to the degree where further corruption is more expensive than honest improvement.

It is particularly bad in more "meritocratic" fields as everything is done in rank order where being slightly "superior" to the next best candidate results getting most all of the spoils.

A far better option would be to just make more use of weighted sortition so goosing the numbers has a far lower expected payout. This of course would require admitting that a large portion of the meritocracy is a sham but would likely be the single best thing we could do to stop every metric from being endlessly corrupted.

"It is particularly bad in more "meritocratic" fields as everything is done in rank order where being slightly "superior" to the next best candidate results getting most all of the spoils."

+10, this really is an excellent point that I hadn't considered previously.

Though to be fair, I think the real problem is pseudo-meritocratic fields. Fields where meritocracy is claimed but that direct output is hard to measure. Or even in the cases where for social reasons we refuse to measure hard output or ignore the results.

You will be hard pressed to find a more wasteful "meritocracy" than medical training. We have a couple of tests, explicitly calibrated to be more of a pass/fail nature with the best predictive power at the bottom end of the curve that have grown in importance to corrupt all of medical education.

Prior to step exams schools drop didactic and clinical instruction for months to allow students to cram for the tests. Sellers of study aids ask for, and get, thousands of dollars (paid for by the loans that will eventually come back as higher physician fees) per student. Test takers have, of course, been caught cheating but also memorizing the test questions which then are handed off to future test takers.

And, of course, the majority of medical schools have retooled the entire early (and increasingly late) curricula around goosing their scores.

Depending on how you count the numbers (e.g. 2 months of lost physician time being worth something like $40,000) we are talking about billions being wasted on creating ranked order lists.

These Step exams were recently estimated to be around 70% of the determinant for who gets which residency (and the second biggest slot is med school grades which are increasing just mini-Step exams).

And what do we get at the end of the day? Rank orderings not dissimilar from those we had before Step exams existed. Oh and around 6% of the trained and qualified docs not matching and being unable to practice any given year.

For everyone inside the system it makes sense. You do not want to hand high value residencies to less qualified applicants (lawsuit bait if nothing else). You do not want residents competing on price (a lot of lower tier places would have trouble filling and likely the AHA would lose a major cash cow). And the residents know that getting just a bit ahead of the next guy can be a million dollar advantage. The fact that it ends up creating billions in losses, well that's just tragedy of the commons.

And none of this meritocracy has ever been shown to correlate with anything we actually care about: not patient mortality, not cost effective medical practice, not even rates of malpractice claims. I have seen limited data suggesting that the whole works is actually more likely to kill patients (e.g. memorization is of such high yield during training that docs who are good at it overly rely on it in practice and inevitably make an error that a quick reference check would prevent).

And all of that is discounting the massive wastes where physicians now routinely spend years doing far less useful work just to get into medical school. Which ultimately means that our average physician is in practice for fewer years, increasing medical costs and decreasing experience gained (which tends to correlate with lower patient mortality).

Maybe there is some other field our there where they burn through lives and money on silly ranking games, but I would be surprised.

"memorization is of such high yield during training that docs who are good at it overly rely on it in practice and inevitably make an error that a quick reference check would prevent"

Decades ago I swapped to a new GP (General Practitioner). He impressed me by having not just some drug catalogues in his office: he had textbooks and reference books. In this he seemed to be in a minority.

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

Is this a Straussian take on Global Warming? On Michael Mann's last set back in Court? After all, what did the East Anglia guy say - that he was not going to share his data or his code or something because people he did not like would just pick it apart?

Compare with the late recently departed Derek Parfit. Who published virtually nothing himself but who would comment, at length, on anything other people sent to him including undergraduates or even random strangers. Legend has it he provided comment on a book manuscript that was longer than the book. Great for their careers, not so great for his. But his reputation? Maybe if he published more he would not have such a great reputation?

Respond

Add Comment

Respond

Add Comment