Category: Web/Tech

It’s amazing how a theory can take over a brain. Consider the idea that people believe what serves their interests. As heuristics go, it’s a good one. I use it all the time. Yet when Dario Amodei says AI is dangerous, perhaps even an extinction risk, some people conclude he must be running a marketing campaign. That is stupid. Which is more likely, that a useful heuristic sometimes misfires or that “our product might kill you” is a clever way to sell it? Death threats are a poor marketing strategy.

We also have plenty of evidence that the fears of AI experts are sincere. Amodei, Altman and Musk were all publicly warning about AI risk long before they had AI companies to promote. The worry runs well beyond the executive suite; rank and file researchers share it. And it extends outside the industry altogether, to computer scientists with no product to sell, among them Nobel laureate Geoffrey Hinton. Hinton left a high-paying job at Google precisely so he could speak out and Hinton is not in a Berkeley polycule with Eliezer Yudkowsky, at least as far as I know. Whatever else you may say about the belief that AI presents a serious risk, plenty of AI researchers believe it sincerely.

Similarly, when Amodei recently proposed to slow the pace and install independent safety teams at AI companies many people jumped to the conclusion that this was regulatory capture. Sorry, but no, that theory doesn’t make sense. To see why, we should review the theory of regulatory capture.

Regulatory capture came out of the political science literature especially Marver Bernstein’s 1955 classic, Regulating Business by Independent Commission. Bernstein argues for a regulatory life cycle: Gestation, Youth, Maturity, Old Age. A scandal brings a bureaucracy into existence—or gives an existing one new powers. The public’s attention, like Sauron’s eye, fixes on the issue of the day: something must be done. The thalidomide scandal, for example, helped establish the modern FDA.

Gestation gives way to a youthful burst of reform and the do-gooders come to Washington ready to battle the industry. Inevitably, however, the public’s eye looks elsewhere. But the industry never looks away. It lobbies Congress, hires former regulators, trains future ones, and supplies much of the information the agency needs. As the agency matures, accommodation replaces confrontation. By old age, the regulator has become the industry’s protector. The classic example is the ICC, created to regulate the railroads but it eventually came to shield them from competition from the trucking industry.

Notice that classic regulatory capture takes time, it’s a process of erosion rather than a battle, it happens in the shadows, in the backrooms, away from the public’s eye. As Culpepper argues in Quiet Politics and Business Power, business power goes down as political salience goes up. Regulatory capture and lobbying does a good job explaining why roasting coffee beans was defined as “domestic manufacturing”, thereby lowering Starbuck’s tax rate by 2%. It does less well at explaining big cross-industry issues the public cares about such as environmental regulation or race and gender discrimination regulation. Finally, don’t confuse capture with firms making the best of a bad situation. Philip Morris supported the 2009 Tobacco Control Act not because FDA regulation was Philip Morris’s unconstrained ideal but because it knew regulation was coming and it wanted a seat at the table to nudge the rules in its favor. That’s ordinary political bargaining—or rent-seeking—not evidence that the regulator has been captured.

Now let’s evaluate Amodei’s call for regulation in light of regulatory capture theory. AI regulation is in gestation. Public attention is fixed on the industry, and much of that attention is hostile. The big profits in AI lie in automating work, and job loss is a much more salient fear than extinction. AI politics is now loud–precisely the environment in which Culpepper predicts business power will be weakest. A mature industry can bend regulation to its purposes through revolving doors, longstanding relationships and obscure rulemaking. An industry under Sauron’s eye has much less power and faces much greater risk that politics will bend regulation to its purposes. Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups (ala Peltzman).

Regulation will reduce AI profits. That doesn’t prove that every rule Amodei favors is innocent of self-interest, an absurd proposition. I suspect that Amodei’s ideal may be something like a single regulated AI monopoly—safe and reasonable profitable, like the old AT&T. But that’s not the profit maximizing outcome. If transformative AI can capture even a fraction of the enormous labor market–the world’s biggest market–laissez-faire would mean vastly greater profits. Amodei may simply prefer a smaller fortune and a safer world. That is perfectly consistent with self-interest playing a role; it is not consistent with the bastardized theory that profit maximization is the only thing that matters or that “capture” is universal.

Go ahead: argue that Amodei and other AI experts are wrong about AI risk. Ask whether his proposals favor Anthropic. But calling “our product might kill you” a clever marketing and regulatory-capture strategy isn’t sophisticated analysis. The facts don’t fit regulatory capture theory and trying to make them fit requires epistemically painful Ptolemaic epicycles. Even a dull Ockham’s razor cuts through that story to the obvious alternative: Amodei actually believes what he’s saying.