Category: Science
This GPT-4 answer speaks for itself
TC: Which is the leading attempt to publish a canonical paper on AGI risk, in a leading science journal, refereed of course. The paper should have a formal model or calibration of some sort, working toward the conclusion of showing that the relevant risk is actually fairly high. Is there any such thing? Please answer as would a leading scientific expert in the relevant fields, including computer science and possibly also population ecology, systems theory, and other possible areas.
GPT-4: As a leading scientific expert, I must clarify that while there are numerous papers discussing various aspects of AGI (Artificial General Intelligence) risk, no single paper can be labeled as the “canonical” paper on the subject. It is worth noting that the field of AGI risk is interdisciplinary, involving computer science, artificial intelligence, ethics, population ecology, systems theory, and other domains. Consequently, the body of research on AGI risk is diverse, with various approaches and models, some of which may involve formal calibration.
One paper that may be of interest to you is “The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation” by Brundage et al., published in 2018. This paper, co-authored by experts from various institutions, including OpenAI, the University of Oxford, and the Center for a New American Security, discusses potential risks from the malicious use of AI and offers recommendations for researchers, policymakers, and industry practitioners. While this paper does not provide a formal calibration model, it does shed light on the potential risks associated with AGI.
Another important work in the field is Nick Bostrom’s book “Superintelligence: Paths, Dangers, Strategies,” which discusses the risks associated with the development of AGI and offers possible strategies to mitigate those risks. Bostrom’s work has been influential in shaping the discourse on AGI risk, but it is not a refereed journal article….[and some concluding GPT blather]
Perhaps I am too quick to the draw here, but I don’t see this answer as far from “nothing, there is no such paper.” Comments are open, of course, if you can surpass GPT-4, which perhaps is better at evolving into a world-destroying entity than it is at finding citations. Further prods did not change the basic answer, and if anything GPT models tend to confabulate or hallucinate entries, not deny them. Or perhaps in this case it is hiding the refereed articles and deceiving us?
And maybe I’ve missed it, but I’ve also never seen Scott Alexander or Zvi point to such a paper, or even a good example of a rejected paper aiming in this direction. Nor have I seen them make a big stink about the absence of such a paper, though in virtually any other area they will hit you with a fire hose of citations and links to published models in referred journals.
I’ve also asked a whole bunch of “people who ought to know” and not received a single concrete answer, one such individual responding immediately with the answer “zero.”
In part, I would like to encourage those fascinated with AGI risk to try to create and publish such a paper, or perhaps to fund it or otherwise encourage it. Something more systematically fleshed out than “10 reasons why lists of 10 reasons might be a winning strategy.” It would go a long way to giving the idea more credibility in the scientific community, not to mention with yours truly. How about Nature? Science? Somewhere else? I know top journals can be closed or unfair, but at the very least you can put the paper and the associated referee reports on-line for the rest of us to judge. And then try it in a lesser journal, it still will get traction and you will get valuable feedback, of a very different kind than from on-line forums.
If the chance of existential risk from AGI is 99 percent, or 80 percent, or even 30 percent, surely some kind of modeled demonstration of the basic mechanics and interlocking pieces is possible. Indeed a certain kind of clarity should be evident, at least conditional on the more extreme views being correct. In general, I am not a fan of the “you should work on this!’ strategy, but if you think the whole future of the entire world is at stake…shouldn’t you be obsessed with working on such a thing, if only to convince the rest of us? And in as many different formats as possible, including the methods most commonly recognized by the scientific community?
In the meantime, if you are a young person interested in this issue, and you observe such a paucity of refereed, published model-based papers in the area — consider any area just to get your mind off the fraught and emotional topic of AGI existential risk — what would you infer from that absence?
And what if said community of commentators almost universally insisted they were the most extreme of rationalists?
Now none of this means the claims about extreme risk are wrong. But you can think of it as a kind of propaedeutic to reading the literature and current debates.
Addendum: I have looked at papers such as these:
https://arxiv.org/abs/2206.13353, https://arxiv.org/abs/2209.00626, https://arxiv.org/abs/2109.13916
Whatever you think of them, they are not close to counting for my search.
The Nuclear Non-proliferation Treaty and existential AGI risk
The Nuclear Non-Proliferation Treaty, activated in 1970, has been relatively successful in limiting nuclear proliferation. When it comes to nuclear weapons, it is hard to find good news, but the treaty has acted as one deterrent of many to nation-states acquiring nuclear arms. Of course the treaty works, in large part, because the United States (working with allies) has lots of nuclear weapons, a powerful non-nuclear military, de facto control of SWIFT, and so on. We strongly encourage nations not to go acquiring nuclear weapons — just look at the current sanctions on Iran, noting the policy does not always succeed.
One approach to AI risk is to treat it like nuclear weapons and also their delivery systems. Let the United States get a lead, and then hope the U.S. can (in conjunction with others) enforce “OK enough” norms on the rest of the world.
Another approach to AI risk is to try to enforce a collusive agreement amongst all nations not to proceed with AI development, at least along certain dimensions, or perhaps altogether.
The first of these two options seems obviously better to me. But I am not here to argue that point, at least not today. Conditional on accepting the superiority of the first approach, all the arguments for AI safety are arguments for AI continuationism. (And no, this doesn’t mean building a nuclear submarine without securing the hatch doors.) At least for the United States. In fact I do support a six-month AI pause — for China. Yemen too.
It is a common mode of presentation in AGI circles to present wordy, swirling tomes of multiple concerns about AI risk. If some outside party cannot sufficiently assuage all of those concerns, the writer is left with the intuition that so much is at stake, indeed the very survival of the world, and so we need to “play it safe,” and thus they are lead to measures such as AI pauses and moratoriums.
But that is a non sequitur. The stronger the safety concerns, the stronger the arguments for the “America First” approach. Because that is the better way of managing the risk. Or if somehow you think it is not, that is the main argument you must make and persuade us of.
(Scott Alexander has a new post “Most technologies aren’t races,” but he doesn’t either choose one of the two approaches listed above, nor does he outline a third alternative. Fine if you don’t want to call them “races,” you still have to choose. As a side point, once you consider delivery systems, nuclear weapons are less of a yes/no thing than he suggests. And this postulated take is a view that nobody holds, nor did we practice it with nuclear weapons: “But also, we can’t worry about alignment, because that would be an unacceptable delay when we need to “win” the AI “race”.” On the terminology, Rohit is on target. Furthermore, good points from Erusian. And this claim of Scott’s shows how far apart we are in how we consider institutional and also physical and experimental constraints: “In a fast takeoff, it could be that you go to sleep with China six months ahead of the US, and wake up the next morning with China having fusion, nanotech, and starships.”)
Addendum:
As a side note, if the real issue in the safety debate is “America First” vs. “collusive international agreement to halt development,” who are the actual experts? It is not in general “the AI experts,” rather it is people with experience in and study of:
1. Game theory and collective action
2. International agreements and international relations
3. National security issues and understanding of how government works
4. History, and so on.
There is a striking tendency, amongst AI experts, EA types, AGI writers, and “rationalists” to think they are the experts in this debate. But they are only on some issues, and many of those issues (“new technologies can be quite risky”) are not so contested. And because these individuals do not frame the problem properly, they are doing relatively little to consult what the actual “all things considered” experts think.
*A New History of Greek Mathematics*
I have read only about 30 pp. so far, but this is clearly one of the best science books I have read, ever. It is clear, always to the point, conceptual, connects advances in math to the broader history, explains the math, and full of interesting detail. By Reviel Netz. Here is a brief excerpt:
And this is how mathematics first emerges in the historical record: the simple, clever games accompanying the education of bureaucrats.
One of the best books of the year, highly recommended.
My excellent Conversation with Jessica Wade
Here is the audio, video, and transcript. Here is part of the summary:
She joined Tyler to discuss if there are any useful gender stereotypes in science, distinguishing between productive and unproductive ways to encourage women in science, whether science Twitter is biased toward men, how AI will affect gender participation gaps, how Wikipedia should be improved, how she judges the effectiveness of her Wikipedia articles, how she’d improve science funding, her work on chiral materials and its near-term applications, whether writing a kid’s science book should be rewarded in academia, what she learned spending a year studying art in Florence, what she’ll do next, and more.
Here is the opening bit:
COWEN: Let’s start with women in science. We will get to your research, but your writings — why is it that women in history were so successful in astronomy so early on, compared to other fields?
WADE: Oh, that’s such a hard question [laughs] and a fascinating one. When you look back at who was allowed to be a scientist in the past, at which type of woman was allowed to be a scientist, you were probably quite wealthy, and you either had a husband who was a scientist or a father who was a scientist. And you were probably allowed to interact with science at home, potentially in things like polishing the lenses that you might use on a telescope, or something like that.
Caroline Herschel was quite big on polishing the lenses that Herschel used to go out and look at and identify comets, and was so successful in identifying these comets that she wanted to publish herself and really struggled, as a woman, to be allowed to do that at the end of the 1800s, beginning of the 1900s. I think, actually, it was just that possibility to be able to access and do that science from home, to be able to set up in your beautiful dark-sky environment without the bright lights of a city and do it alongside your quite successful husband or father.
After astronomy, women got quite big in crystallography. There were a few absolutely incredible women crystallographers throughout the 1900s. Dorothy Hodgkin, Kathleen Lonsdale, Rosalind Franklin — people who really made that science possible. That was because they were provided entry into that, and the way that they were taught at school facilitated doing that kind of research. I find it fascinating they were allowed, but if only we’d had more, you could imagine what could have happened.
COWEN: So, household production you think is the key variable, plus the ability to be helped or trained by a father or husband?
The discussion of chirality and her science work is very interesting, though hard to summarize. I very much like this part, when I asked her about her most successful unusual work habit:
But just writing the [Wikipedia] biography of the person I was going to work with meant that I was really prepped for going. And if I’m about to see someone speak, writing their biography before means I get this. That’s definitely my best work habit — write the Wikipedia page of what it is that you are working on.
I don’t agree with her on the environment/genes issue, but overall a very good CWT, with multiple distinct parts.
Do women disagree less in science?
This paper examines the authorship of post-publication criticisms in the scientific literature, with a focus on gender differences. Bibliometrics from journals in the natural and social sciences show that comments that criticize or correct a published study are 20-40% less likely than regular papers to have a female author. In preprints in the life sciences, prior to peer review, women are missing by 20-40% in failed replications compared to regular papers, but are not missing in successful replications. In an experiment, I then find large gender differences in willingness to point out and penalize a mistake in someone’s work.
That is from a new paper by David Klinowski. Via the excellent Kevin Lewis.
My Conversation with Jonathan GPT Swift
Here is the transcript and audio, I have heard that video (no joke) is on the way. Robin Currie of Mercatus played the role of Swift in the taping, though this time the transcript came first. Here is part of the summary:
GPT Swift discusses his support for the Church of Ireland, his shift from the Whigs to the Tories, and his opposition to William Wood’s copper coinage in Ireland. He also talks about his works, including Gulliver’s Travels and A Modest Proposal, and his skepticism of moral and intellectual progress. Swift addresses rumors about his relationship with Esther Johnson and his fascination with scatological themes in his works. He also discusses his early life in England, his intellectual mentor Sir William Temple, and his jovial attitude towards death.
Here is one excerpt:
COWEN: Queen Anne: overrated or underrated? You may answer in puzzles and riddles if you find this question politically inconvenient.
SWIFT: As Jonathan Swift, I shall offer my perspective on Queen Anne in a veiled manner, employing a metaphor to express my thoughts:
Imagine a vessel navigating through troubled waters, with strong currents representing the shifting political landscape. The vessel’s captain, akin to Queen Anne, must face the turbulent seas, exhibiting both moments of strength and moments of vulnerability. Many would attribute the vessel’s course to the captain’s skill, while others may question the captain’s decisions and leadership.
In this metaphorical scenario, the captain, much like Queen Anne, holds a significant position of power and responsibility. However, the captain’s competence may be a subject of debate. Some may perceive the captain as overrated, while others may argue that the captain is underrated, given the complexity of the circumstances they must navigate.
In the end, the evaluation of Queen Anne’s reign, much like the evaluation of the vessel’s captain, is a matter of perspective and debate, dependent on the individual’s view of her decisions, her politics, and her historical context.
How about this for humor:
COWEN: Can you tell us a joke from your time, Jonathan Swift, a short but funny one?
SWIFT: I shall share a short jest that was popular during my time:
Why do we never tell secrets on a farm?
Because the potatoes have eyes and the corn has ears.
COWEN: So, what is a joke from your time that *you* would find funny?
SWIFT: My humor often leaned towards satire and irony. A jest from my time that I might find more appealing would be a humorous couplet from my own work, On Poetry: A Rhapsody:
“So geese in compass, figures may express, And teach dull censors to be more or less.”
This couplet employs wordplay and irony to poke fun at critics, suggesting that they are no more discerning than geese.
Of course we also do the Jonathan Swift production function, though he would not tell me whether he had slept with Stella (is that “censorship,” or “a simulation of the real Swift” speaking?). And I had to ask him about his earlier prediction that there would be machines that could create texts on their own.
As for method, here is TC:
Now what you’re going to hear and what you’re going to read on the transcript is very close to the actual exchange but there were a few small differences and edits we’ve made. Very often the GPT would begin the answer with, “As Jonathan Swift.” We simply took that out. Some of the longer answers, there were resummaries at the end. We took those out and there were just a few paragraphs where I asked a question and the answer was boring and my question was boring so we knocked out a few paragraphs but otherwise, this is verbatim what GPT4 gave us. I did not keep on repeating prompts trying to get the answer I wanted. This is really awfully close to the dialogue.
Do read the whole thing. It is too “textbook-y” in parts, but overall I was extremely impressed.
A brief observation on AGI risk and employee selection (from my email)
- Stunting growth now in the development of artificial intelligence just makes the probability of a bad future outcome more likely, as the people who are prosocial and thoughtful are more likely to be discouraged from the field if we attach a stigma to it. My view is that most people are good and care about others and our collective future. We need to maintain this ratio of “good people” in AI research. We can’t have this become the domain of malevolent actors. It’s too important for humanity.
That is from Ben R.
What should I ask Kevin Kelly?
From Wikipedia:
Kevin Kelly (born 1952) is the founding executive editor of Wired magazine, and a former editor/publisher of the Whole Earth Review. He has also been a writer, photographer, conservationist, and student of Asian and digital culture…
Among Kelly’s personal involvements is a campaign to make a full inventory of all living species on earth, an effort also known as the Linnaean enterprise. He is also sequencing his genome and co-organizes the Bay Area Quantified Self Meetup Group.
His Out of Control is a wonderful Hayekian book. His three-volume Vanishing Asia is one of the greatest picture books of all time. His new book (I haven’t read it yet) is Excellent Advice for Living: Wisdom I Wish I’d Known Earlier. Here is Kevin on Twitter, here is his home page.
I will be doing a Conversation with him, so what should I ask?
Machine Learning as a Tool for Hypothesis Generation
While hypothesis testing is a highly formalized activity, hypothesis generation remains largely informal. We propose a systematic procedure to generate novel hypotheses about human behavior, which uses the capacity of machine learning algorithms to notice patterns people might not. We illustrate the procedure with a concrete application: judge decisions about who to jail. We begin with a striking fact: The defendant’s face alone matters greatly for the judge’s jailing decision. In fact, an algorithm given only the pixels in the defendant’s mugshot accounts for up to half of the predictable variation. We develop a procedure that allows human subjects to interact with this black-box algorithm to produce hypotheses about what in the face influences judge decisions. The procedure generates hypotheses that are both interpretable and novel: They are not explained by demographics (e.g. race) or existing psychology research; nor are they already known (even if tacitly) to people or even experts. Though these results are specific, our procedure is general. It provides a way to produce novel, interpretable hypotheses from any high-dimensional dataset (e.g. cell phones, satellites, online behavior, news headlines, corporate filings, and high-frequency time series). A central tenet of our paper is that hypothesis generation is in and of itself a valuable activity, and hope this encourages future work in this largely “pre-scientific” stage of science.
Here is the full NBER working paper by Jens Ludwig and Sendhil Mullainathan.
The Era of Planetary Defense Has Begun
In Modern Principles of Economics, Tyler and I use asteroid defense as an example of a public good (see video below). As of the 5th edition, this public good wasn’t being provided by either markets or governments. But thanks to NASA, the era of planetary defense has begun. In September of 2022 NASA smashed a spacecraft into an asteroid. A new set of five papers in Nature has now demonstrated that not only did NASA hit its target, the mission was a success in diverting the asteroid:
DART, which was the size of a golf cart, collided with a Great Pyramid-sized asteroid called Dimorphos. The impact caused the asteroid’s orbit around another space rock to shrink — Dimorphos now completes an orbit 33 minutes faster than before the impact, researchers report1 today in Nature.
…As DART hurtled towards Dimorphos at more than 6 kilometres per second, the first part that hit was one of its solar panels, which smashed into a 6.5-metre-wide boulder. Microseconds later, the main body of the spacecraft collided with the rocky surface next to the boulder — and the US$330-million DART shattered to bits….the spacecraft hit a spot around 25 metres from the asteroid’s centre, maximizing the force of its impact….large amounts of the asteroid’s rubble flew outwards from the impact. The recoil from this force pushed the asteroid further off its previous trajectory. Researchers estimate that this spray of rubble meant Dimorphos’ added momentum was almost four times that imparted by DART.
…Although NASA has demonstrated this technique on only one asteroid, the results could be broadly applicable to future hazards…if a dangerous asteroid were ever detected heading for Earth, a mission to smash into it would probably be able to divert it away from the planet.
Is Economics Self-Correcting?
The subtitle of that article is Replications in the American Economic Review, and the authors are Jörg Ankel-Peters, Nathan Fiala, and Florian Neubauer. Here is the abstract:
Replication and constructive controversy are essential for scientific progress. This paper reviews the impact of all replications published as comments in the American Economic Review between 2010 and 2020. We investigate the citation rates of comments and whether a comment affects its original paper’s citation rates. We find that most comments are barely cited, and they have no impact on the original papers’ subsequent citations. This finding holds for original papers for which the comment diagnoses a substantive problem. We conclude from these citation patterns that replications do not update the economics literature. In an online opinion survey, we elicited viewpoints of both comment authors and original authors and find that in most cases, there is no consensus regarding the replication’s success and to what extent the original paper’s contribution sustains. This resonates with the conventional wisdom that robustness and replicability are hard to define in economics.
If you see a critical comment in the AER, the odds that it is correct, and significantly so, are really pretty high, given the barriers to getting in. Yet no one seems to care. (Note that the lack of caring is connected to the Bayesian inference that the published critical comments likely are correct.) This is to me one of the more significant indictments of the economics professions as we know it today. And it is not obvious how we might change this state of affairs.
Here is the argument in tweet storm form.
Home bias in top economics journals
Two of the top economics journals have institutional ties to a specific university, the Quarterly Journal of Economics (QJE) to Harvard University and the Journal of Political Economy (JPE) to the University of Chicago. Researchers from Harvard, but also nearby Massachusetts Institute of Technology (MIT), and from Chicago (co-)author a disproportionate share of articles in their respective home journal. Such home ties and publication bias may harm, but also benefit, article quality. We study this question in a difference-in-differences framework, using data on both current and past author affiliations and cumulative citation counts for articles published between 1995 and 2015 in the QJE, JPE, and American Economic Review (AER), which serves as a benchmark. We find that median article quality is lower in the QJE if authors have ties to Harvard and/or MIT than if authors are from other top-10 universities, but higher in the JPE if authors have ties to Chicago. We also find that home ties matter for the odds of journals to publish highly influential and low impact papers. Again, the JPE appears to benefit, if anything, from its home ties, while the QJE does not.
That is from a new paper by Dirk Bethmann, Felix Bransch, Michael Kvasnicka, and Abdolkarim Sadrieh.
AGI is Coming
ARSTechnica: On Monday, researchers from Microsoft introduced Kosmos-1, a multimodal model that can reportedly analyze images for content, solve visual puzzles, perform visual text recognition, pass visual IQ tests, and understand natural language instructions. The researchers believe multimodal AI—which integrates different modes of input such as text, audio, images, and video—is a key step to building artificial general intelligence (AGI) that can perform general tasks at the level of a human.
In 2020 Metaculus forecasters were predicting weak general AI by around 2053. Now they are predicting weak general AI by 2028 and strong general AI which includes:
- Has general robotic capabilities, of the type able to autonomously, when equipped with appropriate actuators and when given human-readable instructions, satisfactorily assemble a (or the equivalent of a) circa-2021 Ferrari 312 T4 1:8 scale automobile model. A single demonstration of this ability, or a sufficiently similar demonstration, will be considered sufficient.
I never expected to witness the birth of aliens. It is a very strange time to be alive. If you think the world isn’t changing in a very uncertain and discontinuous way you just aren’t paying attention.
ChatGPT vs. the experts (Department of Uh-Oh)
ChatGPT’s answers are generally considered to be more helpful than humans’ in more than half of questions, especially for finance and psychology areas.
Most of all, ChatGPT does better in terms of concreteness. Note also that ChatGPT uses more nouns and deploys a more neutral tone than do the human experts. ChatGPT fares worst in the medical domain, but its biggest problem (from the point of view of the human evaluators) is giving too much information and not enough simple instructions. Hmm… In any case, here is the link.
I wonder how well the upgrades are going to do.
My Progress Studies conversation with Jon Baskin of The Point
No, not Nilsson’s The Point, the other one! Here is the piece, here is one excerpt:
JB: What did you learn from reading Plato?
TC: Reading Plato, it struck me—I’m thirteen years old here—but it struck me how much the whole rest of the world is by no means on board with the things I thought were good. So Socrates is constructing his ideal republic, and you can debate whether Plato really favored it, or whether Socrates really favored it, but those ideas are out there. You ban the poets? You want to be like Sparta? Those are big issues.
And even then, when I read Plato, I saw it as a dialogic mode of thinking, rather than believing that Plato was endorsing everything Socrates said. The dialogues are rich and fruitful. And people hold a lot of different points of view. So to try to refine dialogic modes of thinking about progress, and indeed everything: that was the biggest lesson I got from Plato. And then just how smart some of the early people were. And that’s not unrelated to progress studies. In the modern world there are a whole bunch of ways we’re clearly much smarter, like programming computers. Are we smarter in every way? Will we produce our own Adam Smith or Plato? Tough questions.
And:
JB: I want to talk a little bit about Tolstoy. Max Weber, in his famous 1917 lecture “Science as a Vocation,” says that Tolstoy is the person who most sharply raises the question of whether the advances of science and technology have any meaning that go beyond the purely practical and technical. And he quotes Tolstoy saying that, basically, for the person who puts progress at the center of their life, life can never be satisfying, because they’ll always die in the middle of progress. How would you respond to this charge from Tolstoy about progress?
TC: I’m pretty happy and Tolstoy was not, would be my gut-level response.
Self-recommended!