Category: Science
The early history of peer review
By the 1950s, the Royal Society was asking reviewers to respond to standardized questions, including whether a study contained “contributions to knowledge of sufficient scientific interest” and simply whether the society should publish it.
These questions could prompt brief responses even to significant pieces of work. Chemist Dorothy Hodgkin wrote barely 50 words when asked to review the full manuscript of the structure of DNA by Francis Crick and James Watson in 1953, which was published in Proceedings of the Royal Society in April 19541. (A shorter paper announcing the discovery had already appeared in Nature2.)
In her sole comment, beyond a series of yes and no answers, Hodgkin suggests the duo should “touch up” photographs to eliminate distracting reflections of “chairs in the perspex rod” — a technical fix that modern cameras perform routinely. Crick and Watson seemed to follow the advice.
The archive is also littered with long reports, many in handwritten scrawl. In 1877, reviewer Robert Clifton finished a 24-page report on two related papers on optics, with an apology: “How you will hate me for bothering you with this tremendously long letter, but I hope before we meet time will have softened your anger.”
Ferlier says that the introduction of the standardized referee questions significantly reduced the amount of time and effort put in by reviewers. “There’s really this understanding in the nineteenth century and very early twentieth century that the peer review is a real discussion,” she says. “After that, it becomes a way of managing the influx of papers for the journal.”
The article, by David Adam in Nature, is interesting throughout. Via Mike Rosenwald.
Letters of recommendation
We analyze 6,400 letters of recommendation for more than 2,200 economics and finance Ph.D. graduates from 2018 to 2021. Letter text varies significantly by field of interest, with significantly less positive and shorter letters for Macroeconomics and Finance candidates. Letters for female and Black or Hispanic job candidates are weaker in some dimensions, while letters for Asian candidates are notably less positive overall. We introduce a new measure of letter quality capturing candidates that are recommended to “top” departments. Female, Asian, and Black or Hispanic candidates are all less likely to be recommended to top academic departments, even after controlling for other letter characteristics. Finally, we examine early career outcomes and find that letter characteristics, especially a “top” recommendation have meaningful effects on initial job placements and journal publications.
That is from a new paper by Beverly Hirtle and Anna Kovner. Via the excellent Kevin Lewis.
U.S.A. facts of the day
The U.S. Senate Committee on Commerce, Science, and Transportation minority staff (Committee), which oversees federal science agencies including NSF, analyzed 32,198 Prime Award grants NSF awarded to 2,443 different entities with project start dates between January 2021 and April 2024.
Committee analysis found 3,483 grants, more than ten percent of all NSF grants and totaling over $2.05 billion in federal dollars, went to questionable projects that promoted diversity, equity, and inclusion (DEI) tenets or pushed onto science neo-Marxist perspectives about enduring class struggle. The Committee grouped these grants into five categories: Status, Social Justice, Gender, Race, and Environmental Justice. For the purposes for this report, “DEI funding,” a “DEI grant,” or “DEI research” refers to taxpayer dollars NSF provided to a research or engagement program that fell into one of these five groups.
Here is the full report. Note that by early 2024, that figure had risen to 27 percent.
Model this
Doctors were given cases to diagnose, with half getting GPT-4 access to help. The control group got 73% right & the GPT-4 group 77%. No big difference.
But GPT-4 alone got 92%. The doctors didn’t want to listen to the AI.
Here is more from Ethan Mollick. And now the tweet is reposted with (minor) clarifications:
A preview of the coming problem of working with AI when it starts to match or exceed human capability: Doctors were given cases to diagnose, with half getting GPT-4 access to help. The control group got 73% score in diagnostic accuracy (a measure of diagnostic reasoning) & the GPT-4 group 77%. No big difference. But GPT-4 alone got 88%. The doctors didn’t change their opinions when working with AI.
Podcast on science policy
It is titled ARPAS, FROs, and Fast Grants, Oh My! The host is the excellent Tammy Winter, and the other guests are Patrick Hsu and Adam Marblestone, plus yours truly.
Here is the link, with transcript. Excerpt:
Tyler Cowen: In virtually all institutions, we should be taking more chances on quite young people, giving them more authority, in general. My background is quite different from the rest of you at this meeting. I spent a big chunk of my career studying the financing of the creative arts, economics of the arts. That’s always my mental touchstone. When I hear about Focused Research Organizations that expire when the project is over, I think of Hollywood movies. We’ve been doing that for a long time.
You can almost always find parallels in the arts, which makes you much more optimistic about what you can do. Rapid patronage was a big thing during the Renaissance, and it worked really well. I knew when we started Fast Grants, “Oh, we can do this” because of historical examples.
And when you think of young people running things — well, who ran the Beatles? There was George Martin and Brian Epstein, but the Beatles ran the Beatles. Paul McCartney had to figure out the recording studio. We don’t call that science, but that was an extremely difficult scientific project that had never been done before. And this guy, who hadn’t gone to college, at age 23 starts figuring it out and becomes a master. When you see those things happen in the arts — frequently, they happen — you become way more optimistic. “How many people can do this? How can we scale it? Can super young people contribute? Can this all work?”
You are not saying it’s easy — most projects in the arts fail, too — but you think, “Yes, yes, yes, we can do this.” And you do it, or you try to do it.
Recommended, interesting throughout. We had great fun taping this at Stripe headquarters.
How tenure should be granted, circa 2024
Not just on the basis of what you publish, but on what you contribute to the major AI models. So if you go to a major archive and, in some manner, turn it into AI-readable form, that should count for a good deal. It is no worse than publishing a significant article, though of course depending on the quality of the archive. As it stands today, you basically would get no credit for that. You would instead be expected to turn the archive into articles or a book, even if that meant unearthing far less data for the AIs. Turning data into books takes a long time — is that always what humans should be doing?
Articles still count under this standard, as jstor seems to be in the literary “diet” of the major AI models. Wikipedia contributions should count for tenure, and any “hard for the AI to access data set” should count for all the more. Soon it won’t much matter whether humans read your data contribution, as long as the AIs do.
So we’re all going to do this, right? After all, “how much you really contribute to science” is obviously the standard we use, right? Right?
What should I ask Neal Stephenson?
Yes I will be doing another Conversation with him, in honor of his forthcoming book Polostan, which initiates a new series. It is set in the 1930s, has some spies in it, and parts are set in the town of Magnetogorsk in the Ural mountains, as well as Montana and WDC in the U.S. So far I like the first thirty pages very much.
Here is my 2019 Conversation with Neal Stephenson. So what should I ask him?
Avian Flu is Bad for Cows
FarmProgress: With a closed herd and all his heifers artificially inseminated — no outside bulls needed — Nathan Brearley was confident his 500-cow dairy farm in Portland, Mich., would be spared from the avian flu strain that’s affecting dairies.
He was wrong. Nearly six months later after an infection on his farm, milk production still hasn’t recovered.
“I was quite surprised. I never saw any other disease this widespread affect the cattle like it did,” Brearley said during a recent webinar on dairy avian flu, put on by the Pennsylvania Center for Dairy Excellence.
…Brearley said the first signs of problems were in April when the SmaxTec boluses in his cows, which keep track of temperature and other health parameters, started sending high-temperature alarms to his phone and computer. Half the herd looked like it was getting sick.
“Looking at data, the average temperature rise was 5.1 degrees above normal,” he said. “Outlying cows were even higher with temperature.”
The cows were lethargic and didn’t move. Water consumption dropped from 40 gallons to 5 gallons a day. He gave his cows aspirin twice a day, increased the amount of water they were getting and gave injections of vitamins for three days.
Five percent of the herd had to be culled.
“They didn’t want to get up, they didn’t want to drink, and they got very dehydrated,” Brearley said, adding that his crew worked around the clock to treat nearly 300 cows twice a day. “There is no time to think about testing when it hits. You have to treat it. You have sick cows, and that’s our job is to take care of them.”
Testing eventually revealed that his cows did indeed contract H5N1. But how they contracted it, he said, is still a mystery.
Brearley said an egg-laying facility a mile and a half away tested positive for H5N1 and had to depopulate millions of birds. The birds were composted in windrows outside the facility, “and I could smell that process.”
The farm averaged 95-100 pounds of milk per head with 4.0% butterfat and strong solids before the outbreak. During the first three weeks of infection, milk production fell to 75 pounds a head and has been slow to recover.
“Honestly, we haven’t recovered since, though my forages have been stable,” Brearley said. “I cannot get back to our baseline again.”
Reproduction was also challenged. Right off the bat, his cows aborted their calves.
And how about this kicker:
He didn’t test his cows until two weeks after the first high temperatures entered his herd, fearing that his milk processor wouldn’t accept his farm’s milk.
Why do I get the feeling that we are sleepwalking?
How has human DNA evolved?
The full title of this paper is “Pervasive findings of directional selection realize the promise of ancient DNA to elucidate human adaptation.” It truly has an all-star cast of authors, including David Reich and Eric S. Lander, and also numerous others at top schools. I did read through this paper, but understood it only in part. In any case, here is the abstract:
We present a method for detecting evidence of natural selection in ancient DNA time-series data that leverages an opportunity not utilized in previous scans: testing for a consistent trend in allele frequency change over time. By applying this to 8433 West Eurasians who lived over the past 14000 years and 6510 contemporary people, we find an order of magnitude more genome-wide significant signals than previous studies: 347 independent loci with >99% probability of selection. Previous work showed that classic hard sweeps driving advantageous mutations to fixation have been rare over the broad span of human evolution, but in the last ten millennia, many hundreds of alleles have been affected by strong directional selection. Discoveries include an increase from ∼0% to ∼20% in 4000 years for the major risk factor for celiac disease at HLA-DQB1; a rise from ∼0% to ∼8% in 6000 years of blood type B; and fluctuating selection at the TYK2 tuberculosis risk allele rising from ∼2% to ∼9% from ∼5500 to ∼3000 years ago before dropping to ∼3%. We identify instances of coordinated selection on alleles affecting the same trait, with the polygenic score today predictive of body fat percentage decreasing by around a standard deviation over ten millennia, consistent with the “Thrifty Gene” hypothesis that a genetic predisposition to store energy during food scarcity became disadvantageous after farming. We also identify selection for combinations of alleles that are today associated with lighter skin color, lower risk for schizophrenia and bipolar disease, slower health decline, and increased measures related to cognitive performance (scores on intelligence tests, household income, and years of schooling). These traits are measured in modern industrialized societies, so what phenotypes were adaptive in the past is unclear. We estimate selection coefficients at 9.9 million variants, enabling study of how Darwinian forces couple to allelic effects and shape the genetic architecture of complex traits.
I can report that nothing in their exposition seemed unreasonable or unsupported to me. But also the paper didn’t much change my worldview? There is the usual Twitter speculation about how this might apply to different groups, but note the data aggregation methods of the paper in fact require that various human groups (Europe only in the dataset) evolved in tandem and in similar ways over time. Without that assumption, the entire piece of work collapses.
Exciting economics is often misguided economics
In my latest Bloomberg column, I weigh in on the issues surrounding the latest David Deming piece in The Atlantic. Here is one excerpt:
…economics is a relatively mature science, and even surprising results are typically consistent with the laws of supply and demand. Innovations tend to be subtle — they could also be described, less generously, as underwhelming — concerning the relative size of effects. So it is hard for radical new ideas to come out of nowhere, and that does lead to some geographic concentration, centered in the highest-reputation schools…
Can economics come up with truly novel remedies or ideas? Probably not. If there is a recession, or say hyperinflation, there is a standard kit of tools involving monetary policy, fiscal policy, deregulation and some other policy changes. Economists can and do argue about the right mix of those policies in a particular case. But there is no “new drug” waiting to be discovered.
And:
As for microeconomics, if there is too much traffic on a highway, congestion pricing usually works. If there isn’t enough housing, deregulating construction or eliminating rent control are worth a try. No brilliant outsider will come along and say, “The way to get more housing is for everyone to drink two shots of vodka,” or some other novel or wild idea.
The point is not that economists have all the answers. It’s that we have a pretty exhaustive list of possible remedies.
And in sum:
The good news is that economists have already achieved a lot. The bad news is that a lot of the remaining work is doomed to be pretty boring and marginal. So one lesson is simply to appreciate the dullness of economics, because exciting economics is often misguided economics.
There is further content at the link.
What Fusion Energy Can Learn From Biotechnology
Fusion energy is currently facing many of the same opportunities and challenges as the biotechnology industry of the 1970s: exciting scientific and engineering breakthroughs that could change the course of human history, with sufficient public and private funding, more effective business models, and appropriate regulatory oversight. A number of lessons can be learned from the last 50 years of biotechnology industry history, which lead to five proposed initiatives for accelerating progress in fusion: the creation of a university intellectual-property consortium; the standardization of fusion energy milestones along with fusion rating agencies to certify their achievement; the development of new financing and business models to fund the various stages of fusion progress; a coordinated plan for two-sided outreach, education, and engagement at all levels from K–12 to policymakers and the general public; and managing fusion initiatives as part of a broader ecosystem. Applying these historical lessons today can accelerate the development of fusion towards the same level of commercial success and human impact that biotech has achieved.
That is from a new paper by Andrew W. Lo and Dennis Whyte.
AI and Biology
I think AI is going to have some if its biggest effects on biology. Biological pathways are among the most complex in all of science. People are good at handling two or maybe three variable problems but just keeping three variables and their interactions in one’s head is difficult. AIs with access to vast databases of genes, proteins, networks and so forth will enable new simulations and learning as has already happened with protein folding.
LLMs are Creative Reasoners
It’s bizarre to me that there are still people claiming that LLMs are not reasoning or are not creative when by any objective measure they are obviously creative reasoners! By objective measure I mean a test that evaluates creativity and reasoning by evaluating outputs not by idle philosophical speculation that rules AIs out by definition. Here’s a good paper, Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers, which illustrates one such test. The authors asked top researchers in the field of natural language processing to propose research ideas which were then presented in a standardized format to a ratings panel of other NLP experts. The AI created ideas were judged more creative than the human ideas.
Now one might argue that the humans weren’t giving their best ideas–some data in the paper suggests they were giving ideas at the median of those for top researchers–and humans might also be looking for ideas that were perhaps easier to get funding precisely because they were less creative but more doable. Either way, however, the AIs are coming up with good ideas that could usefully supplement human generated ideas.
My excellent Conversation with Philip Ball
Here is the audio, video, and transcript. Here is part of the episode summary:
Tyler and Philip discuss how well scientists have stood up to power historically, the problematic pressures scientists feel within academia today, artificial wombs and the fertility crisis, the price of invisibility, the terrifying nature of outer space and Gothic cathedrals, the role Christianity played in the Scientific Revolution, what current myths may stick around forever, whether cells can be thought of as doing computation, the limitations of The Selfish Gene, whether the free energy principle can be usefully applied, the problem of microplastics gathering in testicles and other places, progress in science, his favorite science fiction, how to follow in his footsteps, and more.
Here is one excerpt, namely the opening bit:
TYLER COWEN: Hello, everyone, and welcome back to Conversations with Tyler. Today I’ll be chatting with Philip Ball. I think of Philip this way. We’ve had over 200 guests on Conversations with Tyler, and I think three of them, so far, have shown they are able to answer any question I might plausibly throw their way. Philip, I believe, is number four. He’s a scientist with degrees in chemistry and physics. He’s written about 30 books on different sciences. Both he and I have lost count.
He was an editor at Nature for about 20 years. His books cover such diverse topics as chemistry, physics, the history of experiments, social science, color, the elements, water, water in China, Chartres Cathedral, music, and more. But most notably, he has a new book out this year, a major work called How Life Works: A User’s Guide to the New Biology. Philip, welcome.
PHILIP BALL: Thank you, Tyler. Lovely to be here.
COWEN: What is the situation in history where scientists have most effectively stood up to power, not counting Jewish scientists, say, leaving Nazi Germany or the Soviet Union?
BALL: Gosh, now there’s a question to start with. Where they have most effectively stood up to power — this is a question that I looked at in a book (it must be about 10 years old now) which looked at the response of German physicists during the Nazi era to that regime. I’m afraid my conclusion was, the response was really not very impressive at all.
On the whole, the scientists acquiesced to what the regime wanted them to do. Very few of them were actively sympathetic to the Nazi party, but they mounted no real effective opposition whatsoever. I’m afraid that looking at that as a case study, really, made me realize that it’s actually very hard to find any time in history where scientists have actively mounted an effective opposition to that kind of imposition of some kind of ideology, or political power, or whatever. History doesn’t give us a very encouraging view of that.
That said, I think it’s fair to say, science is doing better these days. I think there’s a recognition that at an institutional level, science needs to be able to mobilize its resources when it’s threatened in this way. I think we’re starting to see that, certainly, with climate change. Scientists have come under fire a huge amount in that arena. I think there’s more institutional understanding of what to do about that. Scientists aren’t being so much left to their own devices to cope as best they can individually.
But I think that there’s this attitude that is still somewhat prevalent within science, that’s a bit like, “We’re above that.” This is exactly what some of the German physicists, particularly Werner Heisenberg, said during the Nazi regime, that science is somehow operating in a purer sphere, and that it’s removed from all the nastiness and the dirtiness that goes on in the political arena.
I think that that attitude hasn’t gone completely, but I think it needs to go. I think scientists need to get real, really, about the fact that they are working within a social and political context that they have to be able to work with, and to be able to — when the occasion demands it — take some control of, and not simply be pushed around by.
That, I think, is something that can only happen when there are institutional structures to allow it to happen, so that scientists are not left to their own individual devices and their own individual sense of morality to do something about it. I’m hoping that science will do better in the future than it’s done in the past.
COWEN: Which do you think are the power structures today that current scientists, say in the Anglo world, are most in thrall to?
Recommended, there are numerous topics of interest. I also asked GPT how much money it could earn if it had the powers of Wells’s Invisible Man.
From Reed and Logchies
Introduction. Your analysis produces a statistically insignificant estimate. Is it because the effect is negligibly different from zero? Or because your research design does not have sufficient power to achieve statistical significance? Alternatively, you read that “The median statistical power [in empirical economics] is 18%, or less” (Ioannidis et al., 2017) and you wonder if the article you are reading also has low statistical power. By the end of this blog, you will be able to easily answer both questions. Without doing any programming.
An Online App. In this post, we show how to calculate statistical power post-estimation for those who are not familiar with R. To do that, we have created a Shiny App that does all the necessarily calculating for the researcher (CLICK HERE).
Here is the link and the full story.