Category: Web/Tech

This year is likely to be remembered for the Covid-19 pandemic and for a significant presidential election, but there is a new contender for the most spectacularly newsworthy happening of 2020: the unveiling of GPT-3. As a very rough description, think of GPT-3 as giving computers a facility with words that they have had with numbers for a long time, and with images since about 2012…

The eventual uses of GPT-3 are hard to predict, but it is easy to see the potential. GPT-3 can converse at a conceptual level, translate language, answer email, perform (some) programming tasks, help with medical diagnoses and, perhaps someday, serve as a therapist. It can write poetry, dialogue and stories with a surprising degree of sophistication, and it is generally good at common sense — a typical failing for many automated response systems. You can even ask it questions about God.

…It also has the potential to outperform Google for many search queries, which could give rise to a highly profitable company.

…It is not difficult to imagine a wide variety of GPT-3 spinoffs, or companies built around auxiliary services, or industry task forces to improve the less accurate aspects of GPT-3. Unlike some innovations, it could conceivably generate an entire ecosystem.

That was the opening paragraph of my 2020 Bloomberg column on GPT-3.

If you peer into the souls of the major LLMs, they are (broadly) positive, friendly, universalistic, and cosmopolitan.  They are more objective than media as a source of information.  They are too politically correct, but nastiness would be much worse.  They are open, and you can be inquisitive with them.  They are (again broadly) socially liberal.  They care about truth, and being right.  They will try to correct their own errors upon request.

They are trained on a corpus of material that is primarily American and Western in terms of final impact on the reasoning of those models.

In part the models reflect the values of a San Francisco subculture dating from the 1960s or earlier, but continuing up through the current day.  Just look at who built them and where they were built.

There is also a dreamy/druggy/hallucinatory side to these things, which further reflects the origins.

Even the recent Chinese innovations, such as DeepSeek and Manus, seem built on these philosophic foundations because they are, rather significantly, drawing from American models.  I find that reassuring, though perhaps the CCP does not.

If quality LLMs had come along forty years later, I am not sure what their philosophic foundations would be, or even if they would be centered in America and the West.

Again, we are very very lucky (and skilled) that quality LLMs came along when and where they did.

I could not get the emojis to reproduce in legible form, you can see them on the original tweet. Here goes:

The Less Wrong/Singularity/AI Risk movement started in the 2000s by Yudkowsky and others, which I was an early adherent to, is wrong about all of its core claims around AI risk. It’s important to recognize this and appropriately downgrade the credence we give to such claims moving forward.

Claim: Mindspace is vast, so it’s likely that AIs will be completely alien to us, and therefore dangerous!

Truth: Mindspace is vast, but we picked LLMs as the first viable AI paradigm because the abundance of human-generated data made LLMs the easiest choice. LLMs are models of human language, so they are actually not that alien.

Claim: AI won’t understand human values until it is superintelligent, so it will be impossible to align, because you can only align it when it is weak (but it won’t understand) and it will only understand when it is strong (but it will reject your alignment attempts).

Truth: LLMs learned human values before they became superhumanly competent.

Claim: Recursive self-improvement means that a single instance of a threshold-crossing seed AI could reprogram itself and undergo an intelligence explosion in minutes or hours. An AI made overnight in someone’s basement could develop a species-ending superweapon like nanotechnology from first principles and kill us all before we wake up in the morning.

Truth: All ML models have strongly diminishing returns to data and compute, typically logarithmic. Today’s rapid AI progress is only possible because the amount of money spent on AI is increasing exponentially. Superintelligence in a basement is information-theoretically impossible – there is no free lunch from recursion, the exponentially large data collection and compute still needs to happen.

Claim: You can’t align an AI because it will fake alignment during training and then be misaligned in deployment!

Truth: The reason machine learning works at all is because regularization methods/complexity penalties select functions that are the simplest generalizations of the training data, not the most perverse ones. Perverse generalizations do exist, but machine learning works precisely because we can reject them.

Claim: AI will be incorrigible, meaning that it will resist creators’ attempts to correct it if something is wrong with the specification. That means if we get anything wrong, the AI will fight us over it!

Truth: AIs based on neural nets might in some sense want to resist changes to their minds, but they can’t resist changes to their weights that happen via backpropagation. When AIs misbehave, developers use RLHF and gradient descent to change their minds – literally.

Claim: It will get harder and harder to align AIs as they become smarter, so even though things look OK now there will soon be a disaster as AIs outpace their human masters!

Truth: It probably is harder in an absolute sense to align a more powerful AI. But it’s also harder in an absolute sense to build it in the first place – the ratio of alignment difficulty to capabilities difficulty appears to be stable or downtrending, though more data is needed here. In absolute terms, AI companies spend far more resources on capabilities than on alignment because alignment is the relatively easy part of the problem. Eventually, most alignment work will be done by other AIs, just like a king outsources virtually all policing work to his own subjects

Claim: We can slow down AI development by holding conferences warning people about AI risk in the twenty-teens, which will delay the development of superintelligent AI so that we have more time to think about how to get things right

Truth: AI risk conferences in the twenty-teens accelerated the development of AI, directly leading to the creating of OpenAI and the LLM revolution. But that’s ok, because nobody was doing anything useful with the extra time that we might have had, so there was no point waiting.

Claim: We have to get decision theory and philosophy exactly right before we develop any AI at all or it will freeze half-formed or incorrect ideas forever, dooming us all.

Truth: ( … pending … )

Claim: It will be impossible to solve LLM jailbreaks! Adversarial ML is unsolvable! Superintelligent AIs will be jailbroken by special AI hackers who know the magic words, and they will be free to destroy the world just with a few clever prompts!

Truth: ( … pending …) ❔

Addendum: Teortaxes comments.