Category: Science
Language Models and Cognitive Automation for Economic Research
From a new and very good NBER paper by Anton Korinek:
Large language models (LLMs) such as ChatGPT have the potential to revolutionize research in economics and other disciplines. I describe 25 use cases along six domains in which LLMs are starting to become useful as both research assistants and tutors: ideation, writing, background research, data analysis, coding, and mathematical derivations. I provide general instructions and demonstrate specific examples for how to take advantage of each of these, classifying the LLM capabilities from experimental to highly useful. I hypothesize that ongoing advances will improve the performance of LLMs across all of these domains, and that economic researchers who take advantage of LLMs to automate micro tasks will become significantly more productive. Finally, I speculate on the longer-term implications of cognitive automation via LLMs for economic research.
Recommended.
Tap the Yellowstone Super-volcano!
I applaud this kind of big think!
The USA is confronted with three epic-size problems: (1) the need for production of energy on a scale that meets the current and future needs of the nation, (2) the need to confront the climate crisis head-on by only producing renewable, green energy, that is 100% emission-free, and (3) the need to forever forestall the eruption of the Yellowstone Supervolcano. This paper offers both a provable practical, novel solution, and a thought experiment, to simultaneously solve all of the above stated problems. Through a new copper-based engineering approach on an unprecedented scale, this paper proposes a safe means to draw up the mighty energy reserve of the Yellowstone Supervolcano from within the Earth, to superheat steam for spinning turbines at sufficient speed and on a sufficient scale, in order to power the entire USA. The proposed, single, multi-redundant facility utilizes the star topology in a grid array pattern to accomplish this. Over time, bleed-off of sufficient energy could potentially forestall this Supervolcano from ever erupting again. With obvious importance to our planet and the research community alike, COMSOL simulation demonstrates and proves the solution proposed herein, to bring vast amounts of green, emission-free energy to the planet’s surface for utilization. Well over 11 Quadrillion Watt hours of electrical energy generated over the course of one full year, to meet the current and future needs of the USA is shown to be practical. Going beyond other current and past research efforts, this methodology offers tremendous benefits, potentially on a planetary scale.
From Yellowstone Caldera Volcanic Power Generation Facility: A new engineering approach for harvesting emission-free green volcanic energy on a national scale by Arciuolo and Faezipour.
Hat tip: nvo.
My science remarks at the AEI metascience forum
Jim Pethokoukis has transcribed some of them, here is one bit:
There’s a commonly noted productivity slowdown in the Western world starting in 1973 that I and some other people here have written about. I think one neglected factor behind this slowdown is just the destruction of the German-language speaking and central European scientific world, which starts in the 1930s and culminates in World War II. On top of that, you have the Holocaust. The fruits of science take a long time. So if you’re entranced with AI, that is ultimately the result of someone earlier having come up with electricity — Tesla, Edison, others. So before the 1930s, central Europe and Germany is by far the world’s most productive scientific area. And which factors organizationally were behind those successes? To me, that feels dramatically understudied. But you have a 10-, 15-year period where essentially all of that goes poof. Some of it comes over here to the US, a bit to Great Britain, but that’s really a central event in the 20th century. You can’t understand 20th century science without thinking very hard and long about that event.
But in terms of more approximately and more recently, here’s how I would put the importance of metascience or how science is done. It’s how science is done better that will get us out of the Great Stagnation. I don’t think that doing it worse is what got us into the Great Stagnation. So when it comes to what got us into the Great Stagnation, I think the number one culprit is simply we exhausted a lot of low-hanging fruit from combining fossil fuels with powerful machines. At some point, enough people have cars and you get the side airbag. Your CD player has better sound, now you hook in your smartphone or whatever. Those are nice advances, but they’re not really fundamental to the act of driving. So there’s just an exhaustion that happens.
There are other factors, there’s an increase in the level of regulation in a number of ways, some good, some bad, but it’s still going to slow down some amount of progress. And then the energy price shock in 1973 combined with our unwillingness to really go large with nuclear power, right? So all that happening more or less at the same time. And then a few decades earlier, the world took this huge, you know, wrenching gut blow. Now this all means actually, you should be quite optimistic about the future. Here’s the trick to the whole hypothesis, the counterintuitive part. So let’s say the Great Stagnation is not caused by science being done worse. As we’re probably now coming out of the Great Stagnation, we have mRNA vaccines, right? We have ChatGPT, a lot of advances in green energy. Maybe they’re not all sure things yet. Other areas like quantum computing, not a sure thing yet, but you can see that a lot might be coming all based in this new fundamental technology. You could call it computing.
So the hurdle that science has to clear to get us out of those accumulated institutional and low-hanging fruit barriers, that’s a higher hurdle than it used to be. So if the current scientific advances are clearing that higher hurdle, you should actually be quite optimistic about them because they’ve passed through these filters. So you have these other developments. Oh, ‘90s internet becomes more of a thing. Well, that was nice for a few years, you know, Walmart managed its inventory better. 1995 to 1998 productivity goes up, wonderful. That dwindles away. It’s then all worse again. If we really are now clearing all the hurdles, you should be especially optimistic. But it also means science policy, how science is done, how it’s organized, how it’s funded is way more important than during all those years. Those years when we were stuck, you were reshuffling the deck chairs. Not on the Titanic, but what’s like a mediocre company that just keeps on going. I don’t want any name names, but there’s a bunch of them. You were reshuffling the deck chairs there, and now we’re reshuffling the hall enterprise. It’s a very exciting time, but science matters more than ever.
There is more at the link, good throughout.
A new and possibly important paper
The title is the somewhat awkward: “A Macroscope of English Print Culture, 1530-1700, Applied to the Coevolution of Ideas on Religion, Science, and Institutions.” The abstract is informative:
We combine unsupervised machine-learning and econometric methods to examine cultural change in 16th- and 17th-century England. A machine-learning digest synthesizes the content of 57,863 texts comprising 83 million words into 110 topics. The topics include the expected, such as Natural Philosophy, and the unexpected, such as Baconian Theology. Using the data generated via machine-learning we then study facets of England’s cultural history. Timelines suggest that religious and political discourse gradually became more scholarly over time and economic topics more prominent. The epistemology associated with Bacon was present in theological debates already in the 16th century. Estimating a VAR, we explore the coevolution of ideas on religion, science, and institutions. Innovations in religious ideas induced strong responses in the other two domains. Revolutions did not spur debates on institutions nor did the founding of the Royal Society markedly elevate attention to science.
By Peter Grazjl and Peter Murrell, here is the paper itself. Via the excellent and ever-aware Kevin Lewis.
Alex Epstein’s *Fossil Future*
Bryan Caplan asked me to read this book, and Alex Epstein was kind enough to provide me with a copy of it. The subtitle is Why Global Human Flourishing Requires More Oil, Coal, and Natural Gas — Not Less.
My overall view is this: it is a good rebuttal to “the unrealistic ones,” who don’t see the benefits of fossil fuels. But it does not rebut a properly steelmanned case for a transition away from fossil fuels.
I view the steelmanned case as this: we cannot simply keep on producing increasing amounts of carbon emissions for centuries on end. We thus need some trajectory where — at a pace we can debate — carbon emissions end up declining. I’ve stressed on MR many times that climate change is not in fact an existential risk, but it could be a civilization-destroying risk if we just keep on boosting carbon emissions without end. I don’t know a serious scientist who takes issue with that claim.
In a number of places, such as pp.251-252, and most significantly chapter nine, Epstein denies the likelihood of climate apocalypse, but I just don’t see that he has much of a counter to the standard, more quantitative accounts. He should try to publish his more optimistic take using actual models, and see if it can survive peer review. Why should I be convinced in the meantime? I found chapter nine the weakest part of the book. Maybe he feels he wouldn’t get a fair knock by trying to publish his alternative take through “the standard process,” but as it stands his casual take doesn’t come close to overturning what I consider to be the most rational, consensus-based Bayesian estimate of the consequences of making no transition to green energy.
I am also impressed by how many different kinds of scientists accept these conclusions, and see these conclusions mirrored in their own research. If you ask say the oceanographers, they will give you a broadly consistent account as the climate scientists proper.
Nor is there, for my taste, enough discussion of how much climate risk we should be willing to take on. It is not just about “beliefs most likely to be true.” Note that the less you believe in climate models, the more you should be worried about tail risk. In these matters, do not assume that uncertainty is the friend of inaction.
So I really do think we need to deviate from the world’s recent course with respect to fossil fuels. Now, we can believe that claim and simultaneously believe it would be better if Burkina Faso were much richer, even though that likely would be accompanied by more fossil fuel use, at least for a considerable period of time.
Epstein focuses on the Burkina Faso sort of issue, and buries the long-term risk of no real adjustment. But we do have to adjust. Why could he not have had the subtitle: “Why Global Human Flourishing Requires More Oil, Coal, and Natural Gas for a while, and Then Less”? Then I would be happier. In economic language, you could say he is not considering enough of the margins.
I think he is also too pessimistic about the long-run and even medium-run futures of alternative energy sources. More generally, I don’t think a few book chapters — by anyone with any point of view — can really settle that. I find the market data on green investments more convincing than his more abstract arguments (yes, I know a lot of those investments are driven by subsidies and regulation, but there is genuine change afoot).
I worry about his list of experts presented on pp.29-30. Mostly they are very weak, and this returns to my point about steelmanning.
In his inscription to the book Epstein calls me a contrarian — but he is the contrarian here! And I believe his position is likely to retain that designation. There is a lot in the book which is good, and true, nonetheless I fear the final message of the work will lower rather than raise social welfare.
For another point of view, here are various Bryan Caplan posts defending Epstein’s arguments. In any case, I thank Alex for the book.
This is actually quite a common attitude toward science
Scarlett decided to call for backup.
A few days later, the fourth-grader sent the evidence she’d collected to local police with a letter explaining her investigative methodology and what she was trying to find out: “Dear Cumberland Police department, I took a sample of a cookie and carrots that I left for Santa and the raindeer on christmas eve and was wondering if you could take a sample of DNA and see if Santa is real?”
Here is the full story by Jonathan Edwards, via the excellent Kevin Lewis.
Indoor Air Quality and Learning
More on the surprisingly large effects of air pollution on cognition from Palacios, Eichholtz, Kok and Duran:
Governments devote a large share of public budgets to construct, repair, and modernize school facilities. However, evidence on whether investments in the physical state of schools translate into better student outcomes is scant. In this study, we report the results of a large field study on the implications of poor air quality inside classrooms − a key performance measure of school mechanical ventilation systems. We continuously monitor the air quality (i.e., CO2), together with a rich set of indoor environmental parameters in 216 classrooms in the Netherlands. We link indoor air quality conditions to the outcomes on semi-annual nationally standardized tests of 5,500 children, during a period of five school terms (from 2018 to 2020). Using a fixed-effects strategy, relying on within-pupil changes in air quality conditions and test results, we document that exposure to poor indoor air quality during the school term preceding a test is associated with significantly lower test results: a one standard deviation increase in the school-term average daily peak of CO2 leads to a 0.11 standard deviation decrease in subsequent test scores. The estimates based on plausibly exogenous variation driven by mechanical ventilation system breakdown events confirm the robustness of the results. Our results add to the ongoing debate on the determinants of student human capital accumulation, highlighting the role of school infrastructure in shaping learning outcomes.
Note that the authors have data on the same students in high and low pollution episodes, allowing them to control for a wide variety of other factors.
Here are previous MR posts on air pollution including Why the New Pollution Literature is Credible and our MRU video on the Hidden Costs of Air Pollution. Note that you can take lower bounds of these effects and still think we are not paying enough attention to the costs of air pollution.
What should I ask Jess Wade?
I will be doing a Conversation with her, and here is her work:
Wade, a research fellow at Imperial College London, centers her work on Raman spectroscopy, a technique often employed in chemistry to identify molecules, among other uses. She has received several awards for her scientific contributions, and her Wikipedia page is robust with her many achievements.
Here is a good interview with her. And from The Washington Post:
Since 2017, Wade has written more than 1,750 Wikipedia pages for female and minority scientists and engineers whose accomplishments were not documented on the site.
Here is her own Wikipedia page. So what should I ask her?
*When the Heavens Went on Sale: The Misfits and Geniuses Racing to Put Space Within Reach*
The wisdom of Alex Tabarrok
The new AEA policy which requires all data to be legally acquired has costs. As I read it, it would prohibit analysis of say the Panama papers or ProPublica tax data. Satellite data could also be considered questionable in some countries. https://t.co/kQJPL4nwbe
— Alex Tabarrok (@ATabarrok) January 18, 2023
This will in relative terms help the larger, better established researchers, right? And how will it handle GDPR, the right to be forgotten, data storage under EU law, and so on? What is the chance this has all been thought through properly?
Is it Possible to Prepare for a Pandemic?
In a new paper, Robert Tucker Omberg and I ask whether being “prepared for a pandemic” ameliorated or shortened the pandemic. The short answer is No.
How effective were investments in pandemic preparation? We use a comprehensive and detailed measure of pandemic preparedness, the Global Health Security (GHS) Index produced by the Johns Hopkins Center for Health Security (JHU), to measure which investments in pandemic preparedness reduced infections, deaths, excess deaths, or otherwise ameliorated or shortened the pandemic. We also look at whether values or attitudinal factors such as individualism, willingness to sacrifice, or trust in government—which might be considered a form of cultural pandemic preparedness—influenced the course of the pandemic. Our primary finding is that almost no form of pandemic preparedness helped to ameliorate or shorten the pandemic. Compared to other countries, the United States did not perform poorly because of cultural values such as individualism, collectivism, selfishness, or lack of trust. General state capacity, as opposed to specific pandemic investments, is one of the few factors which appears to improve pandemic performance. Understanding the most effective forms of pandemic preparedness can help guide future investments. Our results may also suggest that either we aren’t measuring what is important or that pandemic preparedness is a global public good.
Our results can be simply illustrated by looking at daily Covid deaths per million in the country the GHS Index ranked as the most prepared for a pandemic, the United States, versus the country the GHS Index ranked as least prepared, Equatorial Guinea.
Now, of course, this is just raw data–maybe the US had different demographics, maybe Equatorial Guinea underestimated Covid deaths, maybe the GHS index is too broad or maybe sub-indexes measured preparation better. The bulk of our paper shows that the lesson of Figure 1 continue to apply even after controlling for a variety of demographic factors, when looking at other measures of deaths such as excess deaths, when looking at the time pattern of deaths etc. Note also that we are testing whether “preparedness” mattered and finding that it wasn’t an important factor in the course of the pandemic. We are not testing and not arguing that pandemic policy didn’t matter.
The lessons are not entirely negative, however. The GHS index measures pandemic preparedness by country but what mattered most to the world was the production of vaccines which depended less on any given country and more on global preparedness. Investing in global public goods such as by creating a library of vaccine candidates in advance that we could draw upon in the event of a pandemic is likely to have very high value. Indeed, it’s possible to begin to test and advance to phase I and phase II trials vaccines for every virus that is likely to jump from animal to human populations (Krammer, 2020). I am also a big proponent of wastewater surveillance. Every major sewage plant in the world and many minor plants at places like universities ought to be doing wastewater surveillance for viruses and bacteria. The CDC has a good program along these lines. These types of investments are global public goods and so don’t show up much in pandemic preparedness indexes, but they are key to a) making vaccines available more quickly and b) identifying and stopping a pandemic quickly.
A final lesson may be that a pandemic is simply one example of a low-probability but very bad event. Other examples which may have even greater expected cost are super-volcanoes, asteroid strikes, nuclear wars, and solar storms (Ord, 2020; Leigh, 2021). Preparing for X, Y, or Z may be less valuable than building resilience for a wide variety of potential events. The Boy Scout motto is simply ‘Be prepared’.
Read the whole thing.
The Extreme Shortage of High IQ Workers
At first glance it seems peculiar that semiconductors, a key item of national strategic interest, should be produced in only a few places in the world, most notably Taiwan, using devices produced only in Eindhoven in the Netherlands by one firm, ASML. Isn’t the United States big enough to be able to support all of these technologies domestically? Yes and no.
Semiconductor manufacturing is the most difficult and complicated manufacturing process ever attempted by human beings. A literal spec of dust can ruin an entire production run. How many people can run such a factory? Let’s look at the United States. The labor force is approximately 164 million people which sounds like a lot but half of the people in the labor force have IQs below 100. More specifically, although not everyone in semiconductor manufacturing requires a PhD, pretty much everyone has to be of above average intelligence and many will need to be in the top echelons of IQ.
In the entire US workforce there are approximately 3.7 million workers (2.3%) with an IQ greater than two standard deviations above the mean. (Mean 100, sd, 15, Normal dist.) Two standard deviations above the mean is pretty good but we are talking professor, physician, attorney level. At the very top of semiconductor manufacturing you are going to need workers with IQs at or higher than 1 in a 1000 people and there are only 164 thousand of these workers in the United States.
164 thousand very high-IQ workers are enough to run the entire semiconductor industry but you also want some of these workers doing fundamental research in mathematics, physics and computer science, running businesses, guiding the military and so forth. Moreover, we aren’t running a command economy. Many high-IQ workers won’t be interested in any of these fields but will want to study philosophy, music or English literature. Some of them will also be lazy! I’ve also assumed that we can identify all 164 thousand of these high-IQ workers but discrimination, poverty, poor health, bad luck and other factors will mean that many of these workers end up in jobs far below their potential–the US might be able to place only say 100,000 high-IQ workers in high-IQ professions, if we are lucky.
It’s very difficult to run a high-IQ civilization of 330 million on just 100,000 high-IQ workers–the pyramid of ability extends only so far. To some extent, we can economize on high-IQ workers by giving lower-IQ workers smarter tools and drawing on non-human intelligence. But we also need to draw on high-IQ workers throughout the world–which explains why some of the linchpins of our civilization end up in places like Eindhoven or Taiwan–or we need many more Americans.
Are scientific breakthroughs less fundamental?
From Max Kozlov, do note the data do not cover the very latest events:
The number of science and technology research papers published has skyrocketed over the past few decades — but the ‘disruptiveness’ of those papers has dropped, according to an analysis of how radically papers depart from the previous literature.
Data from millions of manuscripts show that, compared with the mid-twentieth century, research done in the 2000s was much more likely to incrementally push science forward than to veer off in a new direction and render previous work obsolete. Analysis of patents from 1976 to 2010 showed the same trend.
“The data suggest something is changing,” says Russell Funk, a sociologist at the University of Minnesota in Minneapolis and a co-author of the analysis, which was published on 4 January in Nature. “You don’t have quite the same intensity of breakthrough discoveries you once had.”
The authors reasoned that if a study was highly disruptive, subsequent research would be less likely to cite the study’s references, and instead cite the study itself. Using the citation data from 45 million manuscripts and 3.9 million patents, the researchers calculated a measure of disruptiveness, called the ‘CD index’, in which values ranged from –1 for the least disruptive work to 1 for the most disruptive.
The average CD index declined by more than 90% between 1945 and 2010 for research manuscripts (see ‘Disruptive science dwindles’), and by more than 78% from 1980 to 2010 for patents. Disruptiveness declined in all of the analysed research fields and patent types, even when factoring in potential differences in factors such as citation practices…
The authors also analysed the most common verbs used in manuscripts and found that whereas research in the 1950s was more likely to use words evoking creation or discovery such as, ‘produce’ or ‘determine’, that done in the 2010s was more likely to refer to incremental progress, using terms such as ‘improve’ or ‘enhance’.
Here is the piece, and here is the original research by Michael Park Erin Leahey, and Russell J. funk.
Is publication bias worse in economics?
Publication selection bias undermines the systematic accumulation of evidence. To assess the extent of this problem, we survey over 26,000 meta-analyses containing more than 800,000 effect size estimates from medicine, economics, and psychology. Our results indicate that meta-analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta-analyses in psychology, whereas meta-analyses in medicine are contaminated the least. The median probability of the presence of an effect in economics decreased from 99.9% to 29.7% after adjusting for publication selection bias. This reduction was slightly lower in psychology (98.9% −→55.7%) and considerably lower in medicine (38.0% −→ 27.5%). The high prevalence of publication selection bias underscores the importance of adopting better research practices such as preregistration and registered reports.
Here is the full article by František Bartoš, et.al, via Paul Blossom.