Category: Science
Overreaching causal language in the social sciences
Across the social sciences, many studies use cross-sectional designs that reveal associations but are generally unable to support direct causal claims, yet authors of such articles may make or imply causal claims anyway. Here, to examine the prevalence of such ‘overreaching’ causal language, we analysed 194,631 cross-sectional articles using large language models. Over the period 1980–2024, an average of 46% of articles contained causal language in their titles or abstracts, where the annual rate has risen almost threefold since 2000 from 20% to 60%. To examine the effects of such language, we conducted a human-subjects experiment (N = 1, 105), finding that readers frequently indicate abstracts with this phrasing provide causal evidence but that methodological labels (β = −0.4, 95% confidence interval −0.56 to −0.19) and associational wording (β = −0.3, 95% confidence interval −0.43 to −0.07) reduce this tendency. Experiments with five LLMs revealed that model summaries of these articles (N = 100 each) can amplify causal overstatement, removing hedges and introducing causal claims where articles used strictly associational phrasing; however, prompting caution diminishes this pattern.
That is from a recent paper by Calvin Isch, Timothy Dörr, Neil Fasching, Grace Jennings & Duncan J. Watts. Note that Isch is on the job market this year, working with Tetlock and Watts.
Indian documentary covers EV winners
A new short documentary (22 mins) film called The 22nd Century Indian by Shaurya Sinha offers an optimistic take on India, and also covers five (!) Emergent Ventures winners. Congratulations to them, and to Shruti too.
EV India winners featured: Naman Pushp https://x.com/therealnamzoo?s=11
Khushi Mittal: https://khushimittal.com
Shreeporna Rao: https://x.com/shreepoorna365?s=11
Samay Sanghvi: https://www.thealmanac.ai/article/samaysanghvii
Angad Daryani: https://www.linkedin.com/in/angaddaryani?utm_source=share_via&utm_content=profile&utm_medium=member_ios
How economics is changing
As a proportion of the literature, research about econometric theory, monetary policy and corporate governance has slumped. Meanwhile, papers on development, crime and gender are on the up. This could perhaps be because of changing intellectual interests, or maybe demand-side pressures, such as policymaker priorities and external funding…
Research from Prashant Garg, a postdoctoral researcher at Bocconi University, and Thiemo Fetzer, economics professor at Warwick University, finds causal claims in economics have jumped. In 1990, 7.7 per cent of claims made in the literature were causal. In 2023, that hit 32.6 per cent. Additionally, papers with more causal claims are more likely to receive citations and wind up in top five journals, the research suggests.
Here is more from Harvey Nriapia at the FT.
How well does AI peer review work?
Claude and I planted 100 known errors into 10 open-access psychology papers and then ran them through frontier models and two commercial AI review tools. In brief:
- The best single system caught 71 of 100 errors, while the worst caught 30.
- Pooling every system’s output caught 93 of 100. Models are only partly correlated in the errors they find, making ensembling a big lever for finding issues in papers. Check your papers against multiple models!
- Seven errors could not be caught by any system. All were omissions — information deleted from a paper rather than mistakes inserted into it.
- Refine.ink contributes more unique catches than any other single system, though it’s expensive.
- I didn’t measure false positives and I don’t know how this error distribution compares to the distribution of errors in real papers.
- I’ve made the papers, errors, model outputs, and the full experiment log public. I hope people can build on this work to create a comprehensive eval benchmark across disciplines.
That is from Paul Litvak, here is more. Note that is not even using the very latest generation of models.
The Queen song ’39
I only recently learned what it is really about, namely very rapid travel into space and time dilation mattering for the return voyage.
I had never before listened carefully to the lyrics. I heard the “’39” reference at the beginning, the mention of volunteeers sailing away, and the general nostalgic British music hall mood to the piece, and assumed it concerned the Second World War. But no, the volunteers sailing away are going to the stars, and their eventual return to the Earth will be for them a sad and tragic event, as everything they had known will be gone. You can listen to the song here.
Here is one of the most direct excerpts:
In the year of ’39 came a ship in from the blueThe volunteers came home that dayAnd they bring good news of a world so newly bornThough their hearts so heavily weighFor the Earth is old and grey, little darling, we’ll awayBut my love, this cannot beOh, so many years have gone though I’m older but a yearYour mother’s eyes, from your eyes, cry to me
Here are the full lyrics. Brian May, who wrote the song, is not only a wonderful guitarist but he has a PhD in astrophysics.
p.s. no synthesizers!
Solve for the (Refine) equilibrium
We’re proud to announce that Refine has signed partnerships with two leading publishers in economics. Both the American Economic Association and the Econometric Society now use Refine’s AI-assisted technical verification as part of their publication processes.
Here is the thread. And more comments here.
More math breakthroughs from GPT models
The cost of generating the proofs for all 10 of these breakthroughs combined was under $2,000 at Sol API prices. We’re excited to see what scientists and researchers are able to create with our upcoming Astra models!
That is from Noam Brown, at the link you will find his list of the ten breakthroughs. Here are some views on the import of these.
Reforms to the PhD?
The Trump administration is pairing top universities with corporate giants in an effort to steer traditional PhD programs toward national scientific priorities.
Today, the White House and National Science Foundation are announcing a $47 million pilot program to fund around 250 four-year doctorates in technical fields, starting this fall. Students in the pilot will be placed at an industry partner, where they will spend at least one year doing “practical research” that can go towards their dissertation. NSF’s investment will fund three years of these PhDs, with 31 universities and 20-plus companies funding the rest.
“About the same amount of basic research is done today in academia as in the private sector,” Michael Kratsios, the director of the White House Office of Science and Technology Policy, told Pirate Wires. “Over time, while the ecosystem has changed and different players have emerged where important scientific work is being done, the institutions which train our next generation of scientists have not really been updated.”
…The pilot represents a small fraction of the more than 45,000 science and engineering doctorates awarded annually — but the administration says it’s a “scalable model” that can expand to more universities and companies.
Here is the full story, I will pass along more details as I learn them.
Data on Chinese innovation
China’s technological progress in recent decades has been viewed with admiration, alarm, and (in some cases) doubt. To better understand the Chinese innovation ecosystem, we compile a dataset of almost 14 million domestic Chinese patent publications. We focus on the subset of critical technologies identified by the U.S. Department of Defense. Several surprising patterns emerge from the data: Chinese patenting is strongly associated with other measures of innovative progress; patents are not concentrated in corporate giants such as Huawei; universities have played a key role in innovation, much greater than state-owned enterprises or government-owned facilities; and fewer than one in ten Chinese critical technology patents involves an inventor with U.S. experience or training. Finally, using four text-based measures of patent quality, we show that the rise of Chinese patenting in critical technologies has not been associated with a decline in quality relative to the U.S. awards.
That is from a recent paper by Josh Lerner, Namrate Narain, Dimitris Papanikolaou, Amit Seru, and Zunda Winston Xu. Via the excellent Kevin Lewis.
The AEA presidential election
AEA is the American Economics Association, and a new president is needed, here is the candidate and likely winner:
PINELOPI (PENNY) KOUJIANOU GOLDBERG, William Nordhaus Professor of Economics and Global Affairs, Yale University.
Statement of Purpose: I am honored to stand for President of the American Economic Association. Economics has long contributed powerful tools to understand human behavior, markets, institutions, and public policy, but the environment in which we conduct research, teach, and engage with society is evolving rapidly. New technologies, especially artificial intelligence, are reshaping research methods, classroom instruction, and the evaluation and dissemination of scholarship. These changes create exciting opportunities, but they also raise difficult questions about the future of peer review in our journals, transparency, incentives, and research and teaching standards more generally. The profession also faces broader challenges: maintaining rigorous, credible, and independent research, supporting young scholars in an increasingly uncertain academic environment, and ensuring that economics remains open to diverse perspectives, methods, and global talent. If elected, I hope to use my research, teaching, policy, and editorial experience to help address these challenges, while promoting the core values of our discipline.
Here is the link. Hilarious as always that there is only one candidate allowed (which model predicts this as optimal?), but for the first time I have seen a meaningful statement of purpose that I also like. That said, I would like to see her endorse my core proposal, namely turning over all submissions, referee reports, and revisions (anonymized is fine) to the major AI companies for use as training data. After all, such an act would further the mission of the AEA, right?
The Decline in the Transmission of Scientific Ideas
We document that the diffusion of new scientific ideas beyond their field of origin has declined substantially over the past four decades. This contraction is closely linked to increasing specialization in scientific language: research that employs more technical terminology tends to be adopted less broadly. We develop a theory of scientific discovery in which the diffusion of new ideas depends on the degree to which potential adopters can understand and process them. When introducing their discoveries, scientists face a tradeoff between technical communication targeted at their immediate peers and more accessible language meant to reach broader audiences. As knowledge accumulates and research at the frontier builds on deeper layers of prior work, this tradeoff increasingly favors specialized language, limiting diffusion. Policy interventions that align scientists’ incentives can broaden adoption and increase the social value of scientific research.
That is from a new NBER conference paper by Enrico Berkes and Ruben Gaetani.
A natural experiment in economics
To study whether and how academics respond to political pressure, we exploit a natural experiment: the publication in early 2025 of a “blacklist” of words flagged by the U.S. government. We find that the release of this list led to a sharp reduction in the use of these flagged words among economists at universities that rely heavily on federal funding, relative to scholars from institutions that are less dependent on federal funding or based in the UK. The drop is driven by content related to gender, race, and environment. We show that changes are not simply semantic but reflect actual paper content and that neither the individual funding status nor time-invariant author characteristics are driving the effects. We also document interesting heterogeneous effects by department quality and author gender and ethnicity. Our findings are consistent with the idea that scholars respond strongly to political pressure.
That is from a new paper by Dominic Rohner, Oliver Vanden Eynde, and Philine Widmer. I should note that the authors frame their results in terms of “Science under threat,” which indeed is in the title of their paper. I do see some of that in operation, but I also see a lot of “removing incentives for pandering.” Your own weights here may vary.
Alec Stapp on the new Science report from Michael Kratsios
Major new report from the White House Office of Science and Technology Policy. Five things in the report I really liked:
1. Proposes metascience units as a way to advance experimentation in science agencies. IFP recommended OSTP take this forward in our response to a 2025 RFI.
2. Advocates for a portfolio approach to federal science funding. Right now, basic research funding largely goes to incremental, project-based grants. While important, they can’t be the only mechanism we use to fund science.
3. Recognizes the need to launch new institutions. It specifically highlights X-Labs as an experiment with independent labs that can take on ambitious challenges. This is a bipartisan idea whose time has come.
4. Focuses on a mix of innovation funding mechanisms, including fast grants, prizes & challenges, and advance market commitments. We described and contextualized these ideas in the Atlas of Innovation, which can help policymakers design and implement these approaches.
5. Seeks to reduce burden for American scientists, who face mountains of paperwork. Scientists should spend more time doing science and less time writing/reporting on grant proposals and working to meet regulatory requirements.
Here is the link, here is the report itself, by Michael Kratsios, Science a New Golden Age. Overall, less money will be given to universities and more will be spent on AI-assisted science. Here are further observations from Seth Bannon.
Persistent Inequality in Publishing in Economics
This paper documents new facts about concentration in publishing in economics. First, the profession grows downward . The number of economists grew almost sixfold since 1990, but new entrants publish in lower-tier journals while incumbents hold the top. Second, there is high and persistent concentration at the top. Along with the downward growth, the top-1% authors accounted for 38.4% of top-5 publication credit in 1990 and for 78.3% in 2025. Third, the persistence is widespread within cohorts, within subfields, and within gender. Fourth, new journals only slightly dilute concentration. Fifth, elite authors diversify on topics faster than the rest of the profession. We interpret the findings with a screening model of attention under information overload. The evidence is consistent with the model: as the field grows, citations concentrate on established work and the conditional citation premium of top-author papers narrows.
By Ricardo Dahis, via the excellent Samir Varma.
The Trump Administration’s Threat to Scientific Research
In The Nationalization of American Science I warned that the Trump administration’s rewriting of the seemingly mundane Regulation for Federal Financial Assistance was a tremendous threat to America’s historically successful decentralized system of science funding. Many others are now sounding the alarm.
It’s not surprising that organizations like the AAAS oppose the rule, albeit with unusually strongly worded dissents:
This latest move is a brazen power grab by the Director of the Office of Management and Budget to buck the will of Congress and the American people and will make future discoveries less likely. If this rule becomes final, Americans’ hopes for future cures, national security and economic strength will rely on the scientific sensibilities of the nation’s chief bureaucrat. Alzheimer’s disease will not be cured by a budget analyst from either political party.
But we are now seeing strong pushback from independent thinkers such as:
Grayson Logue writing at The Dispatch:
A sweeping new rule proposed by the Trump administration could remake how that money is awarded and give the president and his political appointees discretion to cancel funding or target recipients for virtually any reason—with little opportunity for recourse.
White House officials argue the new rule is necessary to assert more accountability over federal grantmaking, but observers fear the shift will expand opportunities for politicization, abuse, and even corruption for an administration that has already demonstrated a penchant for using the levers of the federal government to punish partisan enemies and reward ideological allies.
if I was trying to ruin American leadership in scientific research this is pretty much the kind of rule I would write…One of the genuine difficulties with observing the second Trump term is that the assault on state capacity and impartiality has been so multipronged that it is difficult to keep track of everything going on. But these proposed rule changes are monumental and catastrophic.
and Noah Smith:
MAGA’s attack on science is even worse than it looks…despite science’s overwhelming popularity and public trust, Trump and his administration are launching an unprecedented and devastating attack on American science — cutting funding, and forcing science projects to undergo ideological review by government commissars.
It may be that the Trump administration has pushed too far, but my real worry is that we are losing an equilibrium. Science was never completely independent of politics, of course, but even at the worst of times, funding was decentralized and the culture-war material that dominated the headlines was never more than a tiny fraction of the whole. Like an independent judiciary, independent science has been an American virtue. COVID policy, gender policy, and now the Trump administration’s weaponization of these mistakes may have destroyed that equilibrium.
As I wrote in my original post, we are adopting the loser policies of authoritarian nations but those policies are the norm elsewhere for a reason. Centralized control of science is the default because it serves the people in power of whatever party. Decentralization is the fragile exception—a historically unusual achievement that is easier to destroy than rebuild.
Addendum: And here is Andrew Gelman.