Category: Web/Tech

The least bad way to regulate AI?

That is the topic of my latest Free Press column.  Excerpt:

The key is to create some basic safeguards, but without stifling broader AI progress. To do so, we must defy the conventional wisdom about public oversight and instead trust the AI labs to be their own primary regulators.

My version of the proposal starts with defining a private not-for-profit body for AI regulation. An ideal body would draw some features from FINRA (the Financial Industry Regulatory Authority): a consortium of financial firms that examines the trade practices of each and makes recommendations, helping the federal Securities and Exchange Commission with oversight and regulation. The AI version would include the major labs and would be authorized and overseen by Washington, perhaps through the now-fledgling Center for AI Standards and Innovation.

This body would periodically audit major AI companies and their models, judging their conduct and safety. In the short run at least, much of this would be focused on issues of cybersecurity, and whether the new models created more cyber risk than they help to solve. If a company passed the audit, it would be exempted from standard liability law, at least provided that it had shown basic, reasonable care, as opposed to extreme or deliberate negligence. That would free the AI labs from the fear that courts might derail their business by granting huge awards to plaintiffs for ill-defined harms that could not reasonably have been prevented. And it would give the labs a strong incentive to meet the safety standards of this body.

It is reasonable to wonder whether such a body, composed of industry players, would issue fair and equitable judgments of safety. Maybe not. Yet there are many upsides and no better alternative.

For one thing, each company knows that a dangerous model from another company could cause a harmful incident and damage the prospects for the entire industry. Consider the Three Mile Island meltdown in 1979, which contributed significantly to the mothballing of the entire U.S. nuclear industry. Few people can name the company (Metropolitan Edison) behind the malfunctioning plant; the reputational penalty attached to the industry as a whole.

Another incentive for safety is that the top companies do not want too much competition from lower-price, lower-quality upstarts. That too will induce those companies to support fairly tough standards, perhaps excessively tough in some cases. Still, we are choosing from imperfect alternatives. The concrete truth, whether we like it or not, is that there is far more expertise within the companies for judging AI safety than we can expect to find in the federal government anytime soon.

I am indebted to some ideas from Dean Ball, noting that his proposal is somewhat different.  And here are some comments from Brendan McCord.

AI and constitutions (from my email)

“Dear Tyler,

I enjoyed reading your notes on visiting Anthropic to advise on Claude’s constitution. Framing AI governance around the common law, case law (“Talmud”), and independent adjudication is a much more adaptive approach than relying on a static, top-down text.

That said, moving from a fixed text to a case-law system introduces its own set of structural risks. If Anthropic adopts this direction, a few institutional design hazards seem worth anticipating:

  • The throughput bottleneck (Speed vs. Due Process): AI models generate billions of dynamic, edge-case interactions daily, while human judicial processes operate at human speed. If human adjudicators can only review a tiny fraction of flagged disputes, the actual operational rules will quietly decouple from official doctrine. Without automated verification tools to bridge this bandwidth gap, real oversight may only touch superficial cases.
  • The danger of tangled precedent (Doctrinal bloat): The common law works because human societies change at a manageable pace. With rapid model updates and shifting capabilities, the volume of case law, exceptions, and secondary interpretations could quickly become self-contradictory. Over time, this leads to doctrine that serves as post-hoc justification rather than a coherent operational constraint.
  • Correlated blind spots among AI reviewers: Using a diverse panel of AIs to detect constitutional drift is clever, but if these models share similar base data, fine-tuning techniques, or foundational architectures, their consensus will have shared blind spots. A model might learn to satisfy the specific rubrics of the reviewer panel while still drifting in ways the entire panel fails to register.
  • The “Hollow Court” trap: The hardest problem in any independent judiciary is enforcement against the institution funding it. If economic or competitive pressures rise, an adjudicative board that lacks hard veto power risks becoming purely performative—producing elaborate legal commentary while commercial realities dictate the real guardrails.

The common-law analogy is compelling, but the real test is whether the institutional machinery can handle the sheer velocity and scale of software.”

That is from Scott Jenkins.

My recent visit to Anthropic

I very recently participated in a two-day session to offer guidance on rewriting the constitution for Claude.  The small group invited was uniformly excellent, we received serious time with key decision-makers, and the discussions were of very high quality.

Some of the points I stressed were the following:

1. Whatever one might take a “constitution” to mean in this context, it needs to borrow more from analogs to case law and the common law.

2. Along related lines, think more in terms of “Talmud,” and not just in terms of “Torah.”

3. Work to help build out a quality secondary literature on the AI constitutions and related documents.  Currently this does not exist.

4. Consider how a panel of diverse AIs, with different prompts, could help to evaluate to what extent Claude (and other AI models) were acting in accord with their constitutions.

5. Have a final board of human adjudicators, functioning in a manner analogous to an independent judiciary.  To the extent the panel of diverse AIs might have concerns about Claude not following its constitution, those AIs could alert the human adjudicators to what was going on.  Those human adjudicators could then have authority over potential changes and remedies.

Here a recent short post on using internal courts and the common law to help govern/self-govern AI.  And on the courts.

I thank Anthropic for having us in.

The new agentic O-ring world

But because agents often require guidance or additional context as they move through their tasks, Sharma, 27, finds himself wanting to be available to them around the clock and forgoing a regular sleep schedule as a result. Until recently, he couldn’t monitor them remotely through a phone or smartwatch.

“The cost of the agents’ being blocked for eight hours is way too high,” he says. “They can be done with their work at any point of time, in the middle of the night.”

Founders have long put in punishing hours in the name of building the next big thing. But the growing capabilities of AI agents—and the speed at which the models powering them are evolving—give new meaning to working yourself to the bone…

“They just demand your attention,” he says. “Does it need anything? Can I help it in any way?”

…Pezaris, who lives in San Mateo, Calif., typically works from 7:30 a.m. to 2 a.m. He estimates Proxon, which employs six human developers, is operating 30 times faster than it would without agents. But agent work begets human work: Onboarding customers at a faster clip means needing to respond to more customer requests, for example.

There is also an agent FOMO multiplier effect. “Every minute that I’m not working, I’m missing out on not doing a week’s worth of work,” says Pezaris.

Here is more from Katherine Bindley at the WSJ.  As I have been joking in some of my talks, we need to start taking bets on when the AI leisure dividend will arrive.  It will, but not just yet…

Indian documentary covers EV winners

A new short documentary (22 mins) film called The 22nd Century Indian by Shaurya Sinha offers an optimistic take on India, and also covers five (!) Emergent Ventures winners.  Congratulations to them, and to Shruti too.

EV India winners featured: Naman Pushp https://x.com/therealnamzoo?s=11
Khushi Mittal: https://khushimittal.com
Shreeporna Rao: https://x.com/shreepoorna365?s=11
Samay Sanghvi: https://www.thealmanac.ai/article/samaysanghvii
Angad Daryani: https://www.linkedin.com/in/angaddaryani?utm_source=share_via&utm_content=profile&utm_medium=member_ios

There are now eight episodes

The Everyday Abundance podcast explores the hidden histories behind everyday activities and the technologies we don’t even know are technologies.

Virginia Postrel and Charles C. Mann dive into the surprising stories behind everything from brushing your teeth to driving your car.

Listen, subscribe, and rate us on SpotifyApple PodcastsAmazonYouTube, or wherever you listen to podcasts.

Here is the link, self-recommending of course…

Declining Occupations and Career Outcomes in the United States

This strikes me as somewhat less of a problem than I might have thought:

We study long-run career consequences of initial employment in an occupation that subsequently declines. Linking the 2000 Decennial Census to US administrative employment and earnings records through 2020, we follow more than 2.4 million workers. Employment in an occupation that contracts by at least 25 percent is associated with about 5 percent lower cumulative earnings despite slightly more quarters worked. The earnings differential closely matches evidence from Sweden and Norway, although employment adjustment differs. Occupational mobility is substantial but incomplete, while children’s later occupational destinations are much less tied to their household heads’ 2000 occupational-growth categories.

That is from a new NBER working paper from Erling Barth, Maria Forthun Hoen, Sari Pekkala Kerr & William R. Kerr.  The results may have implications for AI as well.

Capitalizing untethered AI agents

That is my latest piece of writing, co-authored with Sonia Farrell Pearson of Harvard.  Here is the opening premise:

As early as 2017, the European Parliament floated “electronic personhood” for robots. More recently, a handful of U.S. states introduced legislation explicitly barring AI from legal personhood; and early this summer, President Milei of Argentina proposed letting AI agents own, manage, and bear responsibility for their own corporations.

In response to Milei’s announcement, Yuval Noah Harari pointed out that we have no way of holding an AI agent accountable. What, he asks, could we do to an entity which has neither money to lose nor a body to incarcerate? As Shruti Rajagopalan, a Senior Research Fellow at George Mason’s Mercatus Center, explains: AI “can act intelligently, but only humans respond to the incentives the law creates”.

This question matters now: there are already ways an agent could become fully untethered. By “untethered” – a central concept in this essay – we mean that there is no meaningful or actionable way to trace the actions back to a legally accountable human or institutional entity.

For one, people can and do set agents free, on purpose. An agent could be created by a human or a company that intends to monitor it but then dies or disappears. Or perhaps the entity that created the agent is based in a country like North Korea, not reachable by standard laws.

In other cases the agent might not need to “escape” at all: the agent could be ‘controlled’ by a shell corporation that, while formally owned and traceable, provides no true defendant or ability to satisfy claims. Or perhaps a process spawns a chain of agents so long that the actions of a subagent can’t be tied to the original agent’s creator, neither epistemically nor meaningfully. Even if we can identify the model’s original creator, what if it’s been finetuned, or merged with another model that was created by someone else? The law might eventually untangle these kinds of complex cases, but we foresee an intermediate period where it does not.

And then there’s the user, who makes choices about what the models should actually do. The Hugging Face incident was unusual in that OpenAI was both the model’s creator and its user. But now close to a billion people use these systems: when blaming the creator is legally inappropriate, will it always make sense to blame the user?

The essay considers to what extent capitalizing the untethered agents — requiring them to hold a certain amount of capital — can serve the end of better alignment.  About 22 pp., published on Sonia’s Substack, definitely recommended.

Some fertility and AI forecasts

The 2024 forecast is particularly pessimistic about China’s fertility prospects. Both projections produce very substantial global aging, a major global capital glut producing very low long-run real capital returns. The latest forecast entails 10% lower global GDP in 2100 and far higher payroll tax rates to fund old-age benefits. Most important, it entails a major change in the course of economic hegemony with China’s 2100 global GDP share falling from 25.6% to 14.9% and the US share rising from 11.2% to 14.4%. Our results are sensitive. Should the US eliminate all future immigration, its 14.4% global 2100 GDP share would drop to 9.2%. And were global fertility to follow the UN’s low variant, 2100 world output would be one third, not one tenth lower. The level and division of global output is also highly sensitive to the speed at which AI expands frontier technologies. Accelerated AU/AI – 4x faster-than-recent growth in capital’s share through 2050 – or Transformative AU/AI – 10x faster capital-share growth – reinforce demographic forces, ensuring long-run US economic hegemony. Indeed, Transformative AI combined with 2024 demographics implies US and Chinese 2100 global GDP shares of 25.3% and 16.9%, respectively.

That is from a new NBER working paper by Seth G. Benzell, Laurence J. Kotlikoff & Victor Yifan Ye.  Note that today the U.S. share of global gdp is slightly higher than it was in 1980.

Adding to the barrel of finance fallacies

“I should note also that many (most? almost all?) of the bad scenarios have intermediate points of great worry and catastrophe” Not on my model. By the time any humans start worrying about a takeover or dying, AIs already control all infrastructure

That is from Twitter, and I hear or read that argument often.  It is yet another example of a bad “AI safety point” that does not stand up.

He is already a human worried about a takeover or dying!  It is weird to think that “I see these problems coming” and also think “…as these problems multiply and become more public, say through cyberincidents, other people and also the markets will not get clued in.”  It is assigning a remarkable oracle-like epistemic status to oneself, and then hardly to anyone else.  If the pending data will not persuade anyone else of your view, why do you hold your view so strongly?  Or if you think the ultimate denouement will be so sudden and furtive, how are you so clued in to the future now?  To me this is all obviously absurd, albeit not logically self-contradictory in the narrow sense.

As a side point, if the world does end suddenly, and you bought the puts out of your savings, but cannot cash them in, you still end up dying without having lowered your real level of consumption.

Rob Wiblin trots out a bunch of objections from the MR comments section that can be refuted readily.  You are really not sure which stocks to short and that is a big problem? — the risk is not that systemic then.  And if you think the world will see some significant calamitous events in the next ten years, and the evidence for that is piling up, yes you should be buying some puts, even if you are unsure on the timing.  Simple stuff.  (And no you do not need options contracts that last for ten years.)  The AI safety advocates with relatively extreme views should be trying to spread these points to their followers, not to retire them.

In general I am not a fan of psychoanalysis as a method of dissecting views, but the number and scope of obvious direct errors on this topic (and from very smart people) is so high that one has to wonder.  How about: “$100 billion in added cyber costs is not a significant enough worry, it is too mundane, too small a percentage of gdp, too normal and technocratic a problem…you can’t take my bigger and more dramatic fear away from me!  I won’t let you do that!  And besides, that view is the social glue that bonds my in-group together.”

My very interesting Conversation with Daron Acemoglu

Here is the audio, video, and transcript.  Here is the episode summary:

Daron Acemoglu returns for his second appearance with a new book, What Happened to Liberal Democracy?, which Tyler reads as an attempt to redefine and revitalize liberalism for our times. Where Daron’s first visit was about how states and societies contend for the narrow corridor in which liberty survives, this one asks what liberals themselves got wrong, and the answer implicates the educated elite—that is to say, you.

Tyler and Daron discuss what’s wrong with social contract theories and Rousseau’s general will, whether Acemoglu is more objectivist than Rorty, why he blames left liberalism’s own establishment power for its collapse, what standard is left once you refuse to have a book of higher values, nondomination versus noninterference, who counts as working class, why automation alone will not lead to widespread prosperity, how he can call himself a free speech absolutist while wanting to regulate social media, why he worries we’ve given up on educating Americans, whether teachers’ unions need to be reconstituted, what the life expectancy for an educated twenty year old is today, whether open-source models undercut the centralization worry, what pro-worker AI actually means, why the fertility crisis leaves him strangely close to a real business cycle position, whether AI will raise his own already prodigious paper output, why Armenia has disappointed economically, what he wants to learn next, and more.

For an excerpt, let us start from near the beginning:

COWEN: Richard Rorty had a consensus-based defense of liberal democracy. Is your view the same as his, or are there some metrics along which it’s different? Are you more objectivist in some way?

ACEMOGLU: I wouldn’t say it’s more objectivist. I think my commitment in my mind has always been that it’s impossible for us at any point in time not just to know what’s really the absolute truth, but also to have a clear map of what is the right stance. In the same way that I think many people would agree science makes gradual and sometimes uncertain progress toward some better understanding, I think morally, philosophically, we also have to have gradual uncertain evolution toward what we agree on. Objectivism, in some sense, is too strict.

COWEN: Say we’re not evolving toward greater agreement.

ACEMOGLU: No, sometimes we go back.

COWEN: Today, it seems we’re not evolving toward greater agreement, right?

ACEMOGLU: We’re not. Absolutely not. I think you can blame that on reactions to globalization, reactions to technology, better authoritarian challenges. This is the most uncomfortable part because I see myself firmly within the left liberal tradition. I think it’s partly because of the failure of left liberalism. In some sense, if you think of why did Rome fail or collapse, we can tell a story about the barbarians were really too strong. No, really, Rome collapsed because internally it had problems that opened up weaknesses against outside invaders as well as some inside problems.

It’s the same with left liberalism. I think it became quite influential throughout much of the Western world and some of the emerging economies as well over the last 70, 80 years. It did not use that establishment power just in the right way, creating its own weaknesses that made it much more vulnerable to outside attacks. That’s why I think we are now going back because we are at an interregnum period in which ideas are battling again. In some sense, it’s just like the early 19th century where you had all these very different ideas and they were trying to get a toehold in the imagination of people. I think we are going through a period like that, except that there are really many more confusing ideas out there.

COWEN: Say when we’re not evolving toward greater consensus, what’s the external standard you introduce to judge which are the correct ideas and which not? Like Rorty, you need one, though.

ACEMOGLU: I don’t have one.

COWEN: Then why believe in what you believe?

ACEMOGLU: Okay. I think what I believe is, first of all, not as an external standard—I don’t know whether external standard or internal standard is the right word—but that we have to defend some degree of individual freedom because everything starts from that, both in terms of our meaningful lives, but also any kind of improvement in the human condition requires individual initiative, and that individual initiative is impossible without some amount of freedom, a meaningful freedom, not just saying, “Okay, you have the freedom to pray, but you cannot express that idea with others or you cannot turn it into action.” Some meaningful freedom.

That is the only starting point that we need to have. Around that, we need to build things with some sort of consensus within society. That’s what I think liberalism has to recognize, that you have to enshrine those rights, but then give enough elbow room to people to form their own community-level agreements.

COWEN: You present some different values in the book. One is nondomination. Another is noninterference. There’s others. When they clash, what’s the metric you introduce to decide how to resolve that clash?  

ACEMOGLU: I wish I had a perfect answer for that. I don’t.

COWEN: You have an imperfect answer.

ACEMOGLU: I have an imperfect answer. I think noninterference by itself isn’t enough because, at least as read or as interpreted by some philosophers and economists, it is just about being left alone from higher authority or typically state-level authority, although it could be some other authorities as well. We do also need opportunities and some way of actualizing our intentions and our freedom. That’s what the idea of nondomination, which goes back to Roman Republican times, recognizes.

It is not, in my mind, an extremely interventionist philosophy. It doesn’t say the state knows it best or that the state should always have an opportunity to override individual or group-level decisions. It does emphasize that providing people some amount of protection against those who have much greater physical or other power relative to them is something that cannot be done at the individual level. It has to have a community-level component.

COWEN: That’s a good answer, but is it an answer to the question? If we’re Sidgwick, we might say, when values clash, we look to utility, or maybe it’d be cost-benefit analysis to decide which gets priority. You’re not willing to say that, so why not? What’s wrong with that answer?

Very much interesting throughout.

Mistakes in financial economics

I feel like this is almost deliberately missing the point. My median expectation is that AI boosts the economy enormously, so shorting things would be a terrible idea. But non-trivial tail risk is that it kills everyone. So shorting things would be pointless.

That is from Tom Chivers.  It is easy enough to say buy seriously out of the money puts, and be long with the rest of your portfolio.  But few (if any) of the worriers are doing that.  I should note also that many (most? almost all?) of the bad scenarios have intermediate points of great worry and catastrophe where you can cash in on your puts well before everyone dies.  No leverage required, just spend 5k or 10k a year on this, and if you are wrong consider it a mistaken insurance policy.  If you do not know much about finance, the AGI will guide you in this endeavor.

Or some are saying “Markets are bad at pricing long-term idiosyncratic risk.”  If so, all the more reason to spend on those puts.

You will find many, many mistakes in financial theory when people try to rebut the presumption that, given their views, they should in some way or another be short the market.  And do not just tell me which long positions you hold, those are easy bets, as you can be totally wrong and the long positions still will offer normal, risk-adjusted rates of return.  It is your shorts, whether explicit or implicit, that reveal your true soul.  You may recall that Victor Niederhoffer thought an investor should never go net short on an asset.  I am reluctant to use the word “never,” but my own view is not so different.

Is it really so hard to say (and do): “Thank you, Tyler, I just went out and bought those puts!”?

Apparently so.  And perhaps that is because, deep down, your own intuitions realize that, while some significant costs from AI will appear, it really won’t be that bad after all.

Words to live by.

How well does AI peer review work?

Claude and I planted 100 known errors into 10 open-access psychology papers and then ran them through frontier models and two commercial AI review tools. In brief:

  • The best single system caught 71 of 100 errors, while the worst caught 30.
  • Pooling every system’s output caught 93 of 100. Models are only partly correlated in the errors they find, making ensembling a big lever for finding issues in papers. Check your papers against multiple models!
  • Seven errors could not be caught by any system. All were omissions — information deleted from a paper rather than mistakes inserted into it.
  • Refine.ink contributes more unique catches than any other single system, though it’s expensive.
  • I didn’t measure false positives and I don’t know how this error distribution compares to the distribution of errors in real papers.
  • I’ve made the papers, errors, model outputs, and the full experiment log public. I hope people can build on this work to create a comprehensive eval benchmark across disciplines.

That is from Paul Litvak, here is more.  Note that is not even using the very latest generation of models.