Category: Web/Tech
More math breakthroughs from GPT models
The cost of generating the proofs for all 10 of these breakthroughs combined was under $2,000 at Sol API prices. We’re excited to see what scientists and researchers are able to create with our upcoming Astra models!
That is from Noam Brown, at the link you will find his list of the ten breakthroughs. Here are some views on the import of these.
The wisdom of Eli Dourado
How do “international efforts” work? Unlike most of the signatories of the letter, I have been a State Department advisor and have participated in multiple treaty negotiations. I have also been a member of technical committees for UN technical agencies.
The international sector is unbelievably dysfunctional. Every single treaty or international agreement is an opportunity for every participant to manipulate the much broader policy environment. Often the participants don’t care about the object of the agreement, and are instead trying to use the agreement as leverage for something else. I have seen countries use their limited leverage in multilateral agreements to try to kneecap American industry, get around sanctions, create a pretense of international justification for domestic illiberalism, or steer technology in an authoritarian direction.
The damage from this dysfunction is limited by the fact that there are zero major countries (and not that many smaller countries) who will actually bind themselves in meaningful ways that they don’t narrowly want. If a previously signed treaty turns out to be inconvenient, it is often subtly ignored or reinterpreted.
In addition, treaty delegations working on industrial issues generally reflect the full spectrum of special interests involved in an issue. A US AI treaty delegation might be led by a State Department ambassador, but he will be advised by representatives of the interagency, major labs, other major tech companies, tech investors, civil society, etc.
“International efforts,” therefore, is not a reassuring answer to the governance problem; it is the name of another enormous, unresolved governance problem.
Here is the full tweet.
Data on Chinese innovation
China’s technological progress in recent decades has been viewed with admiration, alarm, and (in some cases) doubt. To better understand the Chinese innovation ecosystem, we compile a dataset of almost 14 million domestic Chinese patent publications. We focus on the subset of critical technologies identified by the U.S. Department of Defense. Several surprising patterns emerge from the data: Chinese patenting is strongly associated with other measures of innovative progress; patents are not concentrated in corporate giants such as Huawei; universities have played a key role in innovation, much greater than state-owned enterprises or government-owned facilities; and fewer than one in ten Chinese critical technology patents involves an inventor with U.S. experience or training. Finally, using four text-based measures of patent quality, we show that the rise of Chinese patenting in critical technologies has not been associated with a decline in quality relative to the U.S. awards.
That is from a recent paper by Josh Lerner, Namrate Narain, Dimitris Papanikolaou, Amit Seru, and Zunda Winston Xu. Via the excellent Kevin Lewis.
You will learn to love AI writing
That is the theme of my latest Free Press article, excerpt:
But do I wish to eschew AI writing for the rest of my life? Absolutely not. Most of all, I want AI writing to get better, so it does not irritate me with its clichés and all too obvious identifying marks. I want AI writing that can fool me, and maybe sometimes I am already getting it.
Do you know Paul Simon’s lovely song “American Tune”? It is one of my favorites. A small percentage of listeners know that one of the core melodies is taken from Johann Sebastian Bach, namely the Passion Chorale from “St. Matthew Passion.” You could say Paul Simon is a plagiarist, though I would prefer not to use this term. He borrowed creatively, as do many other popular music artists.
Do many listeners care? No, although a few might feel a pang of musical nerd pride at learning of the association with Bach.
But wait, it gets worse yet. That melody does not come originally from Bach, but rather he in turn plagiarized a 1601 secular love song by Hans Leo Hassler, namely “Mein G’müt ist mir verwirret” (My feelings are confused). So the lineage is not even the glamorous one you might have imagined. If anything, “American Tune” sounds closer to the Hassler melody than to Bach’s adaptation.
Are you upset yet? Probably not, and that is good.
Someday we will feel the same way about text generated with the assistance of AI. We won’t care much where it came from, and much of the time we will not even know.
Recommended, and I hope the AIs like it too.
The demand for human enhancement technologies
When a new technology promises large private benefits but may impose social costs that markets do not price, demand need not reveal how citizens want it governed. We examine this using a nationally representative U.S. survey experiment (N=5,556) on human enhancement technologies (HET). The experiment randomizes benefit domain, mechanism, heritability, purpose, and risk across vignettes; for each respondent’s assigned vignette, we elicit stated adoption, preferred regulation, and ethical and societal concerns. Overall, about 53% would adopt. Framing the technology as enhancing rather than restorative lowers adoption by about five percentage points, as much as a severe side-effect profile. About 28% would not adopt at any benefit. This refusal is driven overwhelmingly by the enhancing framing rather than by risk, consistent with a non-compensatory constraint for a substantial subgroup. Most who would adopt still favor strict regulation, and most who would never adopt do not wish to forbid others from doing so. Productivity enhancement generates the most ethical concern of any attribute but attracts the least regulation, and respondents favor subsidizing rather than taxing its adoption, consistent with a concern about access rather than safety. Private demand is therefore an unreliable guide to the governance citizens want, and the divergence we document provides a basis for regulators seeking to align the direction of technical change with societal values and priorities.
That is from a new NBER working paper by . And I will repeat this segment: “Productivity enhancement generates the most ethical concern of any attribute…” Do note of course that the last sentence of the authors is completely unwarranted, and is a classic example of underidentified political bias in academic reasoning.
The common sense of Fareed Zakaria
This is one of the best Op-Eds you will read this year, though it deliberately deemphasizes the personal (no one is attacked) in a manner that will result in less attention. Better to write the truth! Excerpt:
Why has this [voter discontent} happened? Some of it is economic. Expensive housing, stagnant incomes and widening inequality have fueled public anger. But economics is only part of the story. America has enjoyed stronger growth than most other advanced economies. France under President Emmanuel Macron has lowered unemployment, and it has attracted more foreign investment projects than any country in Europe for seven straight years, according to an Ernst & Young analysis. South Korea is an economic success story. Japan was experiencing a comeback. Yet democratic governments almost everywhere are unpopular.
The larger explanation is that we are living through one of history’s great technological revolutions. And these transformations always produce deep anxiety and fear.
The last technological transformation on this scale came between roughly 1880 and 1920. Electricity, the telephone, railroads, the automobile and movies overturned entire industries and ways of life. Millions left farms for cities. Global trade disrupted local economies. New fortunes appeared overnight, while old occupations disappeared. In this turmoil, frustration and anger found political pathways. The left turned to communism, the right to fascism. What followed were new mass movements, cultural upheavals and world war.
Do read the whole thing. You can be a victim of these processes, or keep your wits about you. The choice is yours.
The optimal Bayesian update?
I see at least three updates one might make from the recent OAI/Hugging Face hacking incident:
1. “This happened sooner than I expected, and the story is more dramatic than I expected,” therefore I am more worried than before.
2. “This happened, and the inferior Chinese cyber-defense seems to have performed just fine,” therefore I am less worried than before.
3. “This happened, and as far as we can tell, absolutely no one was harmed,” therefore I am less worried than before.
Obviously the net impact, from those bare hypotheses, is indeterminate. And yet few people seem to be paying much heed to #2 or #3. Joshua Saxe from the cyber world has some relevant observations. And perhaps other updates are needed as well.
An OpenAI Model Escaped Its Sandbox and Hacked Hugging Face
AI has just had what I considered to be the first truly concerning security breach. The facts, as we know them so far, are wild. On July 16, Hugging Face, a vast repository housing over a million open-source AI models and data, announced in a blog post:
Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system – and we detected and dissected it largely with AI of our own.
The timeline here is important so keep in mind that the attack was detected probably around Monday July 13 or Tuesday July 14. Note further:
A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.
So this means the breach started earlier, perhaps Sat July 11 or even a bit earlier. The attack was not just one thing but multi-pronged including decoys:
To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary’s speed.
Hugging Face tried to respond but they were initially held back by the fact that the most advanced models at their disposal treated defense as attack and refused to work with Hugging Face. HF thus had to turn to open models–specifically GLM 5.2, a Chinese open-weight model run on their own infrastructure. Note the irony: HF had to use a Chinese model to defend themselves because the American models refused to help. The irony gets deeper.
At the time, I assumed this was a state based attack–maybe China or Russia testing out defenses. Indeed, HF “reported this incident to law enforcement agencies.”
But yesterday (Tuesday July 21), we learned who the real attackers were. The attackers were OpenAI models–GPT-5.6 Sol and an even more capable pre-release model. OpenAI had taken some off the guardrails off the models but they felt safe because they were testing the models in a highly secured sandbox.
The models, however, broke out of the sandbox exploiting a never before seen fault. They then gained access to the internet and from there broke into Hugging Face–all in an effort to steal the answers to the very test they had been asked to solve.
While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.
Now go back to the timeline. As I read it, the models had escaped the sandbox by around Sat. July 11, possibly earlier, and were detected by Hugging Face on Monday July 13 or Tuesday July 14. HF alerted legal authorities around that time–so Hugging Face clearly had no idea who was attacking them. OpenAI says its security team discovered the anomalous activity internally but has not said when. Attribution was not disclosed until Tuesday July 21, so it may well be that the models were loose for about a week before OpenAI realized that they were the ones attacking Hugging Face. And whatever OpenAI knew and when, nobody warned Hugging Face while the attack was underway–they were left to fight off a frontier lab’s models on their own.
This is a very serious breach.
Addendum: People have been wondering why I signed the We Must Act Now statement. This is why.
I am optimistic about the economic impacts of AI, but I also have no doubt that this is a very powerful technology–an Alien Intelligence–quite unlike any we have dealt with before. This incident was, in fact, error-correcting–the attack was detected, contained, and disclosed. But note who paid for OpenAI’s experiment: Hugging Face. When a lab’s test imposes costs on third parties, that is a classic externality, and taking externalities seriously is not dirigisme, it’s law and economics. And that’s the easy case. What do we do when a Chinese model breaks out of its less secure lab? Hmmm…
I remain optimistic. Learning by doing is how I want us to proceed but we should not kid ourselves: this is a global issue and we must build with safety in mind.
Words of wisdom on Chinese AI and our responses
The strategy for China is obvious: commoditize your complements. Note that Xi explicitly ties openness to AI “moving from the digital world into the physical world”; the physical world is the world dominated by China, and the country’s lead in areas like robotics is going to massively benefit from widely available AI models.
Along the same lines, China does not want the U.S. to gain an asymmetric advantage in AI; to the extent that China can weaken the U.S. frontier labs while strengthening any and all potential U.S. adversaries so much the better, and it can benefit from the innovation that will attach itself to an open ecosystem.
And in sum:
The better course is clear: first, loosen Fable and Sol restrictions on cybersecurity, and second, ensure that U.S. open weight model makers are on an equal playing field with China. Yes, the frontier labs will kick and scream about this, but the Administration should realize that listening to their histrionics has led the U.S. to a position where U.S. companies are dependent on China for their defenses. Let the frontier labs win by being better; don’t let them define safety or security, or pull up the ladder of humanity’s collective knowledge. China is already hard enough to compete with; letting them carry the standard for openness and innovation is simply giving away our biggest advantage.
That is from Ben Thompson’s Stratechery (gated, but ungated link here).
How does social media change people?
I would not trust the results of this (or any) paper on this topic very much, but it is good to see some inquiry along different dimensions than the usual:
Past research explored the beneficial and harmful effects of social media (SM), but no study has investigated how SM might change people’s identities. Authors suggest that SM is a catalyst for a shift in two individual values, achievement and conformity, transforming the fabric of our societies. A difference-in-differences analysis found that countries with higher SM adoption compared to their culturally similar country show an increase in achievement and conformity orientation. Next, authors manipulated SM use and established a causal link between SM use and the activation of achievement and conformity values in Facebook users. These findings were replicated with objective SM usage data, and the pertinent mechanism was explored: people’s SM use increases achievement and conformity orientation through an increased need for approval, particularly for those who self-ruminate. Findings suggest that SM use contributes to the emergence of a societal culture that places an emphasis on achievement-seeking and conformity.
That is by Ertugrul Uysal, Sascha Alavi, and Valéry Bezençon. Via the excellent Kevin Lewis.
The small business boom
Across the country, founders like Ms. Winkler are powering an entrepreneurial renaissance.
Jump-started by the pandemic, when a confluence of factors including mass layoffs and remote work led to a flood of business creation, and supercharged by the rise of artificial intelligence, start-up activity is booming after a decades-long slump.
Americans filed 5.7 million applications last year to start new businesses, according to the Census Bureau, the most in the two decades the government has kept track. New business applications through the first half of this year continued to climb…
More recently, there are signals that A.I. is adding fuel.
A recent paper from economists at the University of British Columbia and the Stockholm School of Economics found that generative A.I. was “spurring entrepreneurial activity” in the United States, both by giving rise to new ventures built around the technology and by making it cheaper to start enterprises.
“A.I. tools can do very many different things very well,” said Jan Bena, an associate professor at the University of British Columbia and one of the study’s authors. “That’s the reason why you see so much entry.”
According to a recent report from Gusto, a small-business payroll and benefits service, nearly 60 percent of founders on its platform who started businesses last year said they used A.I., and half said the technology made it cheaper and faster.
Here is more from Sydney Ember at the NYT. Via Josef.
Why crime will decline in (most of) Brazil
The system, introduced in 2024, uses facial recognition to spot people wanted by police on São Paulo’s streets. It issues alerts and officers are dispatched to pick them up. The system can also locate people who have been reported missing, identify stolen vehicles and provide footage to police investigations. Streamed to its control room in the city centre, information flows not just from lenses on street corners but in health centres, on buses and mounted on police motorbikes. By 2028 the number of cameras in the network is supposed to double, to 100,000.
São Paulo is one of many Brazilian cities spending big on crime-fighting technology. As in other countries, police are investing in body-worn cameras and networks of microphones that detect the sound of gunshots. What sets Brazil apart from many democracies is its enthusiasm for face-spotting tech. Researchers for O Panóptico, a watchdog, count 560 active facial-recognition projects in more than 20 Brazilian states. These include police-run initiatives but also experiments in schools, for example, where cameras are increasingly being used to take attendance. They gaze upon some 99m people, more than 47% of Brazil’s population.
Here is more from The Economist.
The future belongs to AI maniacs
That is the theme of my latest Free Press column, excerpt:
An AI maniac is someone who is obsessed with working with the latest AI models. They try out new models as soon as they can, they spend hours and hours trying to master them, and they use them to regulate both their workflows and their personal lives. I know one person who has his AI agent text him if he is not drinking enough water, for which he’s placed cameras around his house. One online anecdote tells of a man who canceled a date to spend more time playing around with Claude Fable 5 after Anthropic (where I am a member of the economic advisory board) extended the model’s availability for a few days.
Many AI maniacs are using AI tools to start companies of smaller size, and thus of smaller expense, than ever before. For those companies, the humans must set in motion and then monitor a large number of AI tools and agents. Those individuals then stand to reap outsize profits as their companies grow and succeed. Stripe, the payments company, recently issued customer data showing that the number of single-person companies earning $10 million or more has doubled in the past two years. There is no firm estimate how much of that improvement is due to AI, but it stands to reason that AI is a main driver of the trend…
Anecdotally, I observe that AI maniacs tend to be young, as with participants in so many other cultural trends. They tend to lack standard manners and graces, as they just want to “get right to it.” They are able to imagine a future that is very different from our present. Many of them also are kind, as they see the potential for new AI services, in areas such as biomedicine, to help other people. Their obsessiveness is a small price to pay for all of those virtues, and it is usually part of their charm and vibe.
The AI maniacs also are skeptical of credentials, as they should be. If you wish to learn how to manipulate AI tools, Harvard and Yale are not the places to go. You need to teach yourself, with assistance from other AI maniacs and also with help from the AI tools themselves. There are some AI maniacs in the Ivy League, but too often those individuals have invested their energies into other, more established ways to succeed.
I also believe that immigrants are especially likely to be AI maniacs. Immigrants have fewer channels to rise through credentials, family connections, and establishment modes of thinking and doing. They are more willing to try something new, they tend to be younger than average, and, because they were willing to switch countries, they tend to have higher levels of energy, courage, and ambition.
Worth a ponder.
Governing agentic AI
From a new paper by Shruti Rajagopalan:
AI agents now transact, publish, and act on external systems without contemporaneous human approval, creating new regulatory challenges. A growing literature has responded with proposals for legal personhood. This Article argues that personhood is neither necessary nor sufficient, shifting the question from status to enforcement. The Article first shows that for two millennia, nonhuman legal personality, from the Roman universitas to the corporation, the Hindu idol, the waqf, and the river, has operated through human officeholders the law can locate, question, prosecute, and replace. Agentic AI inverts that design, exercising practical agency without legal status, sometimes with no identifiable human in the responsibility-bearing role. The Article then sorts deployments into three categories: first, where one firm builds and deploys the agent; second, where the developer and deployer are separate but known; and third, where there is no identifiable developer or deployer. The Article stress tests each agent deployment category against five liability doctrines: agency law, products liability, enterprise liability, negligence, and strict liability. It demonstrates that each fails at different points in the third category for the same reason: the absent responsibility-bearer. Bare personhood would supply a caption without a representative, assets, or a mechanism for cessation. Finally, the Article assembles an alternative from regimes governing aircraft, ships, drones, driverless cars, and motor carriers. It develops a six-layer stack—registration, identification, verification, financial responsibility, lifecycle traceability, and suspension—so a responsibility-bearer can be identified, liability imposed, and the activity suspended. These layers place the human back at the end of the chain.
I would say that social science now has new frontiers, let us hope it blossoms in response.
Spreading AI to the rest of the world
Another job we’ll have, I call this imperialism, but I mean that in a value neutral way. But AI comes to different parts of the world at different speeds. I think the countries where AI changes a lot of things first, there’ll be a very high demand for people from those places, which I’ll think to be the US, possibly UK, to go around the rest of the world and teach people in other places how to integrate AI into what we have. And a lot of those demands won’t be fully rational. They won’t be, oh, give us the best possible AI. They’ll be like, oh, we’re Peruvians. We want to keep things a certain way. You may or may not agree, but we want you to give us a version of AI that helps keep it that way. And that will be the job. And I think Americans in particular, probably Brits as well, huge growth sector will be living in other parts of the world spreading AI. And again, the fact that AI can do it better may or may not be true, but I don’t think it’s what will matter. I think the Peruvians or some analogue will want humans to come and listen to their concerns and assure and persuade them as humans, that’s what they’re going to get. I’m not saying it’s always going to go well, but that will always be, I think, a big job for humans to do.
It’s already a growth sector for Americans to want to live abroad. Like we have all this accumulated wealth. Life in America can be a bit dull. Life in Europe in particular is amazing. Personally, I love life in most parts of Latin America. So it’s already a trend for Americans to live overseas. For another reason, it’s nothing to do with AI. So if there are all these future job opportunities, like full of meaning, like come to Kenya, help Kenya, you can save 73 lives or maybe like 73,000 lives, help them build out their AI in a way that’s acceptable to them. That’ll just be this phenomenally rich inner and outer life. And I think it’ll be a great source of job creation.
I have already linked to the transcript of the talk.