GPT-3, etc.
Here is an email from a reader, I do not really have an opinion of my own yet. Please note I will not indent any further:
“I wanted to draw your attention to something. Are you familiar with “AI Dungeon,” text-based RPG “open world” game running on GPT-2 / GPT-3? Here’s the author’s discussion on medium, or you can play the GPT-2 version for free to get a sense of it directly.
But what I really want to draw your attention to is players who are using custom prompts to open up dialogs with GPT-3 about non-game things.
This result is particularly impressive: https://www.reddit.com/r/slatestarcodex/comments/hrx2id/a_collection_of_amazing_things_gpt3_has_done/fy7i7im/
( Following that string of posts is a little hampered by reddit’s format; here are the posts in order: part 1, part 2, part 3)
If the author is to be believed, they’ve had GPT-3 / “Dragon”:
1. write code
2. act as a pharmacology tutor
3. write poetry
4. translate english, french, chinese (the instruction to “balance the intent of the author with artistic liberty” is particularly interesting)
It’s hard to excerpt, I’d recommend reading the whole thing if you have time.
Here’s another user’s eloquent conversation about the experience of being an AI, using a similar mechanism (screencap images of the convo, part 1 and part 2 ), with a sample prompt if you want to converse with GPT-3 yourself via AI Dungeon.
I am increasingly convinced that Scott Alexander was right that NLP and human language might boostrap a general intelligence. A rough criteria for AGI might be something like (i) pass the Turing test, and (ii) solve general problems; the GPT-3-AI-Dungeon examples above appear to accomplish preliminary versions of both.
GPT was published in June 2018, GPT-2 in February 2019, GPT-3 in May 2020.
As best I can tell GPT -> GPT2 was ~10x increase in parameters over ~8 months, and GPT2 -> GPT3 was ~100x increase of parameters over ~14 months. Any number of naive projections puts a much more powerful release happening over the next ~1-2yrs, and I also know that GPT-3 isn’t necessarily the most powerful NLP AI (perhaps rather the most popularly known.)
When future AI textbooks are written, I could easily imagine them citing 2020 or 2021 as years when preliminary AGI first emerged,. This is very different than my own previous personal forecasts for AGI emerging in something like 20-50 years…
p.s. One of the users above notes that AI Dungeon GPT-3 (“Dragon”) is a subscription service, something like ~$6 a week. MIE.”
Mood affiliation, the police, and rising crime rates
Had a thought on the discussion of rising crime over the last few months inspired by your MR posts on mood affiliation that I wanted to pass along:
There’s been a bit of discussion lately about increased shootings in major cities in the wake of the George Floyd protests, and the two main narratives trying to explain them have been “protests fueling higher tensions” and “cops backing off and not patrolling as much or doing their jobs”. Interestingly, the latter seems to be based on a model where fewer cops and patrols results in more crime, so you might naively expect people who hold that belief would be more likely to believe that simple defunding and reduction of police presence would lead to more crime generally.
But if you believe that mood affiliation predicts opinions better than factual consistency, then it matters more that the former position sounds like “cops to blame, cops bad”, while the second sounds more like “cops are important, cops good”. And most commentators care more about the correct affect towards the police, rather than consistent models of reality, so you largely have commentators that are pro-defund police, but blame their lack of presence for crime increases, or commentators that are pro-police, think defunding would lead to increases in crime, but are less willing to entertain the idea that recent increases in crime are caused by the choices of officers.
That is from an email by Benjamin Hawley.
Saturday assorted links
New York City facts of the day
In June alone some 270 people were shot in the city, a 154 per cent increase from a year earlier. July is not looking any better. Over the recent July 4 holiday weekend, 64 people were shot. Seventeen more were shot this Monday, a day after Gardner’s death. Those shootings have contributed to a 23 per cent increase in homicides so far this year. Burglary is also soaring.
Why does America have such old leaders?
Since 1950, the average age of heads of government in the three dozen member countries of the Organization for Economic Cooperation and Development has steadily declined, from above 60 years old to around 54 today. The average O.E.C.D. national leader is now two decades younger than Mr. Trump — and almost a quarter century younger than Mr. Biden…
And it isn’t just the American presidency that’s gone gray. The average age of Congress has trended upward for decades. Nancy Pelosi, the House speaker, is 80; Mitch McConnell, the Senate majority leader, is 78. The Supreme Court’s nine justices average above 67. Mr. Trump’s cabinet averages over 60, among the oldest in the O.E.C.D.
Here is one (not the only) explanation:
The way countries select their leaders offers a third possibility. In most O.E.C.D. countries, the head of government is a prime minister chosen by fellow lawmakers in the national legislature. Because party members pick their leaders, and can recall them at will, parliamentary systems give political parties significant control over whom to elevate, said Kaare Strom, a political scientist at the University of California, San Diego. And parties often have strategic reasons to pick younger leaders: to appeal to particular constituencies or to brand themselves behind more telegenic faces.
Presidential systems, by contrast, give parties less control over who from their ranks will run — and which candidate voters will prefer. “The process is more driven by who’s out there, who’s interested, and what kind of resources they have,” said Professor Strom. “There’s not the same kind of institutional control and vetting of those candidates that you have in the European parliamentary systems.”
Accumulated wealth and fundraising connections may matter more too in the American system, also favoring seniority. Here is the full NYT story by Ian Prasad Philbrick.
The Thai coronavirus paradox
Dr. Wiput Phoolcharoen, a public health expert at Chulalongkorn University in Bangkok who is researching an outbreak of the coronavirus in Pattani in southern Thailand, noted that more than 90 percent of those who tested positive there were asymptomatic, much higher than normal.
“What we are studying now is the immune system,” he said.
Dr. Wiput said Thais and other people from this part of Southeast Asia were more susceptible to certain serious cases of dengue fever, a mosquito-borne virus, than those from other continents.
“If our immune systems against dengue are so bad, why can’t our immune system against Covid be better?” he asked.
Here is more from the NYT, good to see coverage of this. Finally we are getting somewhere.
Friday assorted links
1. Mormonism encourages multilevel marketing companies.
2. Prison population down eight percent during coronavirus outbreak.
3. The world’s cheapest hospital (India) has to get even cheaper.
4. Ultra-black fish (NYT).
5. Music and synasthesia, and Brian Wilson has hardly ever written a song in a minor key.
6. New movie about Chicago Boys in Chile? Is it even new?
7. How can you make maximum profit from a Twitter hack? And an explanatory attempt at what happened, with various questions.
What I’ve been reading
1. Brent Tarter, Virginians and Their Histories. The best book I have read on the history of Virginia, by an order of magnitude. And in turn that makes it an excellent book on race as well, and also on broader American history. If I have to spend the whole year in this state, I might as well read about it. I learned also that 21,172 Virginians have identified themselves as American Indians, and that this movement is more active than I had realized.
2. Diary of Anne Frank. It seems inappropriate to call this a “good” or even “great” book. I had not read it since high school, I will just say it deserves its enduring status, and the reread was much more rewarding and interesting than what I was expecting.
3. Howard Brotz, editor, African-American Social & Political Thought 1850-1920. A fascinating selection from the debates of the time, reprinting Douglass, Booker T., Du Bois, Marcus Garvey, Martin Delany, and others. Douglass holds up best, including his critique of colonialism. The weakest argument in the volume was “Haiti is working out fine, so Liberia will succeed as well.” Of greatest interest to me was the extent to which the African-American debates of that time overlapped with opinions about Africa and the Caribbean. Recommended, and excellent background for many of the current disputes.
4. Simone Weil: An Anthology, and Gravity and Grace. Gravity and Grace is the early work. Its ten best pages are superb, but reading it is mostly a frustrating experience, due to the diffuse nature of the presentation (to be clear, overall I consider that a relatively high reward ratio). The former collection is the best place to start, noting again there is a certain degree of diffuseness, but as with Žižek there are insights you just don’t get anywhere else. A good question for any talent selection algorithm is whether it would pick out the teenage Weil and give her a grant to pursue her writing projects. Sadly she died at age 34 in 1943.
George Mason’s critique of the American Constitution
Not long ago someone tweeted this part:
The President of the United States has the unrestrained Power of granting Pardons for Treason; which may be sometimes exercised to screen from Punishment those whom he had secretly instigated to commit the Crime, & thereby prevent a Discovery of his own Guilt.
And that led me to wish to read the whole thing. Mason of course was an anti-Federalist, and in his short piece he lays out why he opposes the proposed new constitution. Here is what I found striking:
1. He feared that the President would become a tool of the Senate or of his own cabinet.
2. He feared the Senate would not be directly accountable to the people. Of course, in due time we changed that through constitutional amendment.
3. He feared the federal judiciary would end up taking over state and local judiciaries.
4. The Senate can excessively legislate through the use of treaties — quite a contemporary objection by the way.
5. The individual states won’t be able to levy tariffs on trade across state borders.
6. Federal and state legislatures won’t be able to pass enough ex post facto laws (the strangest worry to me).
7. He made various claims that ended up being made obsolete by the adoption of a Bill of Rights.
8. The southern states would end up systematically outvoted.
9. The Vice President could end up becoming too powerful through his role in the Senate.
It is striking to me in these early writings how much people worry about the evolution of the Senate, and how little attention they pay to the Supreme Court, which at the time was viewed as not slated to be so powerful.
The problem of “Congress will toss away its legislative and war-making roles, and give up a lot of effective control of the budget” was also nowhere to be found in the words of the early critics, as far as I can tell. Nor did they have much of a notion of the rise of the administrative state.
Mason was a forceful writer, but the broad lesson is simply that the future is very difficult to predict.
Thursday assorted links
1. Markets in everything: selfie masks that look like your face.
2. Orthodox privilege, by Paul Graham.
3. Chess players make bigger mistakes when they are playing on-line (are some of those hand/cursor slips? How much is lack of immersion in the competitive atmosphere of a face to face game?).
4. Derek Lowe on T-cell immunity. And further research suggesting herd immunity sets in much sooner than we used to think (again, these results are no excuse for stupidity and complacency, the urgency to make other good decisions remains).
5. Shootings are rising more than usual and arrests are falling (NYT).
6. Wagner in the parking lot as opera returns to Germany (NYT).
7. The current accelerated Chinese testing of vaccines (NYT).
Oxford’s Jenner Institute to Prepare for Challenge Trials for COVID-19
I am one of the signatories to an open letter from 1DaySooner on challenge trials sent to Dr. Francis Collins at NIH. A major development announced with the letter is that 1Day Sooner and Oxford’s Jenner Institute are collaborating to prepare viral production for use in challenge trials.The Jenner Institute is the creator of the AstraZeneca produced vaccine, the vaccine farthest along in development.
A key goal of the letter is to encourage the NIH to start its own preparation for challenge trials:
The undersigned urge the U.S. government…its allies, international funders, and world bodies (e.g. the World Health Organization), to undertake immediate preparations for human challenge trials, including supporting safe and reliable production of the virus and any biocontainment facilities necessary to house participants.
Among the signatories are 15 Nobel prize winners including Oliver Hart and Al Roth, Molecular geneticist Mario Capecchi, professor of medicine William G. Kaelin and physician Barry Marshall (who knows a thing or two about volunteer trials.)
As I discussed earlier, since challenge trials restrict the volunteers to be young and healthy, you can’t apply their results directly to the sick and elderly (the external validity problem) but “challenge trials could help us whittle down [candidate vaccines]… to the best two to three, substantially speeding up the vaccine discovery process.” You could also use challenge trials to help figure out the right dosing which is unusually important in the current situation because if a vaccine can work with .5ml instead of 1ml that’s equivalent to doubling the available supply. The Director of the Jenner Institute, Adrian Hill, agrees writing:
We see considerable potential in the use of human challenge studies to accelerate COVID-19 vaccine development, down-select and help validate the best candidate vaccines, and optimise vaccination approaches.
You can read the whole letter here.
Self-doubt as a barrier
Nearly half of adults who responded to a national survey said self-doubt is one of the largest challenges they would face if they enrolled in a postsecondary education or training program.
Self-doubt was one of the top three challenges respondents cited, below time and above cost.
The new data are included in the findings from the latest “Public Viewpoints” report from Strada Education Network, which surveyed American adults on their motivations for pursuing more education, as well as the barriers they face.
The importance of mental barriers was one of the findings that stuck out the most, said Nichole Torpey-Saboe, director of research at Strada. It’s yet another layer colleges have to consider when trying to attract people without degrees for enrollment.
Here is the full story. This is what I call a “trivial” blog post. There is nothing startling in the content, and perhaps it is not so interesting to read. The comments on it won’t be very good either. Yet it represents a very important truth, and one that, while it is well-known, is probably not sufficiently emotionally internalized. And the gains would be high if more people understood this.
Swedish update, and which places need to fear second waves?
Your recent question intrigued me. Do you have any new info/opinions on what’s happening in Sweden? Despite no mask wearing, continued indoor dining (at least judging from recent photos on instagram), their case AND death daily counts are plummeting (looks like an inverse exponential). This would also explain excess deaths returning to normal throughout US. Bizarrely, my cursory reading of Swedish newpapers online did not result in any recent articles discussing the dramatic decline in cases there!
One theory circulating is they achieved herd immunity on the math: 10x true seroprevalence (from CDC tests in US) * 2x true immunity (from Tcell things not measured by antibody tests that I don’t fully understand) * 0.75% reported case penetration * 2x for relatively low tests per capita rate = 30% true immunity (likely much higher in densest areas where spread would be much faster resulting in maybe >70% immunity in Stockholm). This puts them r0 < 1.
The nice thing about this hypothesis is that it’s easily falsifiable. If true immunity rates are 20x reported case load (dropping last 2x factor since test rate higher in US), then Florida should have just gotten to the 1.4% necessary to trigger similar immunity in dense cities and from now on, cases per day should follow an inverse exponential.
This would also explain why NYC has not seen a resurgence despite very similar reopening as SF and LA – they achieved dense herd immunity in May and thus the subsequent decline in reported cases was driven by herd immunity rather than more strict closures or mask compliance, reversing either of those factors now doesn’t reverse immunity. To be clear, I’m not disputing that distancing or mask wearing works – they do. But so does infecting everyone quickly. No value judgements on what’s the better policy decision here, just trying to make a predictive statement.
At least, one can hope!
That is my email from Mayank Gupta. In my view, some version of this view is looking more true with each passing day. We also are not seeing second waves in hard-hit northern Italy. Still, many surprises remain and we should not leap to premature conclusions.
To be clear, I was and still am pro-lockdown (without regrets), pro-mask, pro-testing, and I believe Denmark followed a better path than did Sweden. Long-term damage (rather than death) still may be a significant risk, and furthermore many parts of the world may be far more vulnerable than the United States. Still, you need to put all of the moralizing and partisanship aside and ask what we are learning from the new data, and I think Mayank Gupta has put it (probabilistically) very well. And see this related Atlantic piece, though I would have some quibbles with it. And here is a bit more commentary on the new T-cell results.
In any case, always be prepared to revise! I believe that within a month we will have a much better sense of these questions.
Addendum: You will note these hypotheses also significantly raise the probability of much earlier animal-to-human transmission, especially in Southeast Asia. A very good baseline principle for reasoning is simply “Origins usually go back longer and earlier than what you first might think!”
Second addendum: If you go back to March, leading epidemiologist Michael Osterhalm argued: “We conservatively estimate that this could require 48 million hospitalizations, 96 million cases actually occurring, over 480,000 deaths that can occur over the next four to seven months with this situation.” Covid-19 has been terrible, and the performance of the executive branch (and many governors) absymal, but do those look like good predictions right now? (Hospitalizations for instance have yet to hit 250k.) If not, why not? How hard have you thought about this question? (Added note: one correspondent suggests that Osterhalm misspoke and in fact meant 4.8 million hospitalizations — note that still would be off by quite a large margin, almost a factor of twenty.)
Wednesday assorted links
1. Michael Rosenwald 2013 prediction of what would go wrong with Twitter. And from 2012.
2. That was then, this is now, Playboy edition (link is “safe for work,” as they say).
3. Bek Khosimov of Uzbekistan does a podcast with me; the subtitles in Uzbek are not yet ready, and here is Bek on Twitter.
4. Jaw-dropping crash in global fertility.
5. New and important and positive results on T-cell immunity. Paper here, likely to prove the biggest and best news you have heard all day. A strong and very cheering result.
6. Research paper arguing that de facto disease severity is declining due to lower viral load.
7. What we are learning about mutations and different strains.
My Conversation with Melissa Dell
Interesting throughout, here is the transcript and video and audio, here is part of the summary:
From the impact of the Mexican Revolution to the different development paths of northern and southern Vietnam, her work exploits what are often accidents of history — whether a Peruvian village was just inside or outside a mine’s catchment area, for example — to explain persistent differences in outcomes. Her work has earned numerous plaudits, including the John Bates Clark Medal earlier this year.
On the 100th episode of Conversations with Tyler, Melissa joined Tyler to discuss what’s behind Vietnam’s economic performance, why persistence isn’t predictive, the benefits and drawbacks of state capacity, the differing economic legacies of forced labor in Indonesia and Peru, whether people like her should still be called a Rhodes scholar, if SATs are useful, the joys of long-distance running, why higher temps are bad for economic growth, how her grandmother cultivated her curiosity, her next project looking to unlock huge historical datasets, and more.
And here is one excerpt:
DELL: Yeah, I’ve done some work looking at the persistence of economic development in Vietnam. The work I did, actually, was limited to what was South Vietnam because there’s also been huge events, obviously, that have happened in the past hundred years in North Vietnam, with a war that destroyed much of the country and was fought over an extended period of time.
But when you look, in general, at places in Vietnam that have a similar recent history, but going back in time, one of them was part of a much stronger, more centralized state. The other one was part of what is today Cambodia, a much weaker state, generally ruled by local lords instead of by a strong centralized state.
You see the towns that were part of the stronger, centralized state going back before colonialism, so several hundred years ago. More recently, they have better-functioning local governments. They’re richer. They’re better off, which shows that places that have a long history of governance seem better able to do that more recently.
So places going back a long time ago — they were part of the central state. They had to collect taxes locally to send up to the central state. They had to organize military conscripts. The central state mandated that they had certain laws.
More recently, those places also have more functional local governments and are also better off economically, whereas the places that never had that structure that comes from a state — it was, essentially, if you were living in that area, there’d be a warlord that you sent tribute to. But there was never any regular taxation, never any organized local government under a central state. Those places much more recently — when there were constitutional reforms in Vietnam that gave them a degree of self-government — they weren’t able to do that very effectively.
They weren’t able to keep the positions on their local city council filled. They weren’t able to provide, as effectively, local public goods, like education or health services. So really, having this long history of governance makes places more able to do that today. That’s relevant because there’s been a big push to have local governments provide an important role in providing public goods, et cetera.
If places don’t have a history of doing that, perhaps not surprisingly, they tend to have a much harder time. When the World Bank says, “Well, we want to give local autonomy to let local governments decide how to provide schools in the way that works best for them,” that’s going to work in places that have a long history of providing education. In places that don’t, they’re more likely to have a hard time.
COWEN: But if you select your cases on the basis of having similar histories, aren’t you selecting for persistence because the locales that have reversals of fortune — a big war in North Vietnam, Communism coming to North Vietnam — that’s a kind of mean reversion. It deliberately gives them a not-similar history. Do you then not overrate the degree of persistence in the dataset by just taking the sliver that is indeed continuous with its own history?
DELL: I think that you could imagine writing papers about different things. Our motivation was, we wanted to think about if the historical state could have a role at all. In order to do that, you don’t really want to compare South Korea to the Philippines — which is what most of the historical literature on this does — because they’re different in so many ways. We know that South Korea looks really different from the Philippines, but there’s so many ways that they can be different.
By looking within South Vietnam, we wanted to say, “Okay, these are places that had a much more recent modern history. Can their past history still matter?” But we’re not saying that that begins to explain everything. There’s other forces that happened more recently that we think are also important.
Certainly, there can be mean reversions, and the argument is not that things are always persistent. I think part of the literature is about understanding why sometimes things persist and sometimes they don’t. Certainly, more recent events can matter, and we’re not claiming that there’s an R-squared of one that a place’s history is its destiny, but that there are these forces.
Next up will be Nathan Nunn, and I asked him a series of related questions about persistence…