Month: April 2023

From the comments, on AI safety

by  Tyler Cowen in

This is from Richard Ngo, who works on the governance team at OpenAI:

A few points:
1. I agree that the alignment community has generally been remiss in not trying hard enough to clarify the arguments in more formal papers.
2. The only peer-reviewed paper making the case for AI risk that I know of is: https://onlinelibrary.wiley.com/doi/10.1002/aaai.12064. Though note that my paper (the second you linked) is currently under review at a top ML conference.
3. I don’t think that a formal model would shed much light here. My goal in writing my paper was to establish misaligned power-seeking AGI as a credible scientific hypothesis; I think that most who think it’s credible would then agree that investigating it further should be a key priority, whether or not their credences are more like 10% or more like 90%.

From this batch of comments.  Here is Richard on Twitter.

Tuesday assorted links

by  Tyler Cowen

1. Review of the new Philip Wallach book on Congress (Rep. Katie Porter’s book too).

2. Good Ding vs. Nepo coverage.

3. On properly translating Macron (having dealt with French diplomats, both through translation and not, I agree with the general points about context).  That said, the whole world has to receive the proper message, as a matter of common knowledge, and arguably he failed in that regard.

4. How AI differs in warfare.  And something about “BabyAGI.”  Self-improving AI making its debut?  And lots of discussion.

5. The roots of our military recruiting crisis, good and interesting piece.

6. Is Tupperware toast? And upscale compost (WSJ).

“Date me” docs

by  Tyler Cowen

Here is one from Katja Grace, by the way I know her a bit and very much like her and find her very smart (NB: not interested in comments mocking Katja, put them somewhere else, I will delete them).  But that is not my main point today.  Do such documents work?  I have been hearing of them more often lately.  And how should we model them?

Should we think of them as batch auctions of a sort, namely wanting to get in a lot of bids at once rather than sequentially?  Which kinds of people should prefer such a batch auction?  (Btw, is there any paper in Science or Nature on this?  Should there be?)

Are they better suited for polyamory than monogamy?  Are batch auctions better suited for polyamory?  Because the process is more like assembling a portfolio?

Is this all somehow better suited for San Francisco and other “Woke” cultures, where perhaps asking someone out on a date counts as a microaggression?  I suppose the Date Me document gives you permission to reach out?  I am curious to read or hear some serious takes on this phenomenon.

Anarchy in South Africa

by  Alex Tabarrok in

Public services such as police, fire, and traffic control in South Africa are breaking down. Private firms are stepping in to take some of the burden. Twenty two percent of Johannesburg’s fire engines are owned and operated by private firms.

Fire Ops employs more than 60 firefighters across seven fire stations in Johannesburg and owns two fire engines—including one now sporting the same shade of blue Discovery uses for its logo and much of its branding—as well as six smaller high-pressure-pump response vehicles.

Discovery says the blue firetruck responded to 172 building fires between Fire Force’s launch through the end of January.

Mr. Ossip said the Discovery-branded truck promotes the insurer’s brand and lowers damages, including to multimillion-dollar homes in some of Johannesburg’s toniest areas. “You need to just save one or two of those a year and it is substantial savings,” he said.

The service helps alleviate a shortage of operational fire engines in Johannesburg, a spread-out city of more than 5.5 million residents, in situations where minutes can make the difference between a blaze limited to a couple of rooms and one that destroys an entire house or spreads to neighboring homes.

Robert Mulaudzi, a spokesman for the City of Johannesburg Emergency Management Services, said the city currently has about seven operational fire engines across 30 fire stations.

…Fire Ops, which invoices buildings’ owners for fire services, says that while it responds to all calls, it will give priority to clients, including Discovery policyholders, when simultaneous fires break out. Other insurers usually pick up the bill when the company puts out a fire in a home not insured by Discovery, said De Wet Engelbrecht, Fire Ops’s chief executive.

In 19th century Great Britain prosecution assocations and insurance firms were responsible for much of the policing (see Stephen Davies in The Voluntary City.) In Lessons from Gurgaon, India’s Private City (working paper) Shruti Rajagopolan and I discuss private police and fire services in modern day Gurgaon. In general, the private firms provide excellent service relative to their public counterparts but, as in Gurgaon, there are limits to how much the private firms can do without large economies of scale:

…Fire Ops also has to navigate public infrastructure that doesn’t always work, including traffic lights, fire hydrants and municipal water supplies….In September, both Fire Ops and the city’s fire department responded to a blaze at Little Forest Centre, a private special-needs school in Johannesburg, but a water shortage in the area meant all fire hydrants were empty, said Kate More, the school’s owner and principal, who isn’t a Discovery policyholder.

Despite Fire Ops sourcing water from a neighbor’s pool, the school burned down.

Addendum: In unrelated news, just one year after its grand opening Whole Foods is closing its downtown San Francisco store because they can’t ensure the safety of their employees.

This GPT-4 answer speaks for itself

by  Tyler Cowen in

TC: Which is the leading attempt to publish a canonical paper on AGI risk, in a leading science journal, refereed of course. The paper should have a formal model or calibration of some sort, working toward the conclusion of showing that the relevant risk is actually fairly high. Is there any such thing? Please answer as would a leading scientific expert in the relevant fields, including computer science and possibly also population ecology, systems theory, and other possible areas.

GPT-4: As a leading scientific expert, I must clarify that while there are numerous papers discussing various aspects of AGI (Artificial General Intelligence) risk, no single paper can be labeled as the “canonical” paper on the subject. It is worth noting that the field of AGI risk is interdisciplinary, involving computer science, artificial intelligence, ethics, population ecology, systems theory, and other domains. Consequently, the body of research on AGI risk is diverse, with various approaches and models, some of which may involve formal calibration.

One paper that may be of interest to you is “The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation” by Brundage et al., published in 2018. This paper, co-authored by experts from various institutions, including OpenAI, the University of Oxford, and the Center for a New American Security, discusses potential risks from the malicious use of AI and offers recommendations for researchers, policymakers, and industry practitioners. While this paper does not provide a formal calibration model, it does shed light on the potential risks associated with AGI.

Another important work in the field is Nick Bostrom’s book “Superintelligence: Paths, Dangers, Strategies,” which discusses the risks associated with the development of AGI and offers possible strategies to mitigate those risks. Bostrom’s work has been influential in shaping the discourse on AGI risk, but it is not a refereed journal article….[and some concluding GPT blather]

Perhaps I am too quick to the draw here, but I don’t see this answer as far from “nothing, there is no such paper.”  Comments are open, of course, if you can surpass GPT-4, which perhaps is better at evolving into a world-destroying entity than it is at finding citations.  Further prods did not change the basic answer, and if anything GPT models tend to confabulate or hallucinate entries, not deny them.  Or perhaps in this case it is hiding the refereed articles and deceiving us?

And maybe I’ve missed it, but I’ve also never seen Scott Alexander or Zvi point to such a paper, or even a good example of a rejected paper aiming in this direction.  Nor have I seen them make a big stink about the absence of such a paper, though in virtually any other area they will hit you with a fire hose of citations and links to published models in referred journals.

I’ve also asked a whole bunch of “people who ought to know” and not received a single concrete answer, one such individual responding immediately with the answer “zero.”

In part, I would like to encourage those fascinated with AGI risk to try to create and publish such a paper, or perhaps to fund it or otherwise encourage it.  Something more systematically fleshed out than “10 reasons why lists of 10 reasons might be a winning strategy.”  It would go a long way to giving the idea more credibility in the scientific community, not to mention with yours truly.  How about Nature?  Science?  Somewhere else?  I know top journals can be closed or unfair, but at the very least you can put the paper and the associated referee reports on-line for the rest of us to judge.  And then try it in a lesser journal, it still will get traction and you will get valuable feedback, of a very different kind than from on-line forums.

If the chance of existential risk from AGI is 99 percent, or 80 percent, or even 30 percent, surely some kind of modeled demonstration of the basic mechanics and interlocking pieces is possible.  Indeed a certain kind of clarity should be evident, at least conditional on the more extreme views being correct.  In general, I am not a fan of the “you should work on this!’ strategy, but if you think the whole future of the entire world is at stake…shouldn’t you be obsessed with working on such a thing, if only to convince the rest of us?  And in as many different formats as possible, including the methods most commonly recognized by the scientific community?

In the meantime, if you are a young person interested in this issue, and you observe such a paucity of refereed, published model-based papers in the area — consider any area just to get your mind off the fraught and emotional topic of AGI existential risk — what would you infer from that absence?

And what if said community of commentators almost universally insisted they were the most extreme of rationalists?

Now none of this means the claims about extreme risk are wrong.  But you can think of it as a kind of propaedeutic to reading the literature and current debates.

Addendum: I have looked at papers such as these:

https://arxiv.org/abs/2206.13353, https://arxiv.org/abs/2209.00626, https://arxiv.org/abs/2109.13916

Whatever you think of them, they are not close to counting for my search.

A new Operation Warp Speed for better vaccines

by  Tyler Cowen in

The Biden administration is launching a $5 billion-plus program to accelerate development of new coronavirus vaccines and treatments, seeking to better protect against a still-mutating virus, as well as other coronaviruses that might threaten us in the future.

“Project Next Gen” — the long-anticipated follow-up to “Operation Warp Speed,” the Trump-era program that sped coronavirus vaccines to patients in 2020 — would take a similar approach to partnering with private-sector companies to expedite development of vaccines and therapies. Scientists, public heath experts and politicians have called for the initiative, warning that existing therapies have steadily lost their effectiveness and that new ones are needed…

Jha and others said the new effort will focus on three goals: creating long-lasting monoclonal antibodies, after an evolving virus rendered many current treatments ineffective; accelerating development of vaccines that produce mucosal immunity, which is thought to reduce transmission and infection risks; and speeding efforts to develop pan-coronavirus vaccines to guard against new SARS-CoV-2 variants, as well as other coronaviruses.

Here is the WaPo article, here is commentary from Eric Topol.

Monday assorted links

by  Tyler Cowen

1. Benevolent sexism.  And should single American women look for suitors abroad?

2. Does summer learning loss replicate? And on Covid learning loss recovery: “On average, we find that 20% of test score losses are recovered in English language arts (ELA) by 2022, compared to 37% in math.”

3. Balenciega beige unicorn sneakers.  And, via Yana, MEN’S TRASH BAG LARGE POUCH IN BLACK.

4. Does Ozempic improve impulse control? (speculative)

5. No adverse labor supply effects from the expanded child tax credit (short-term only, though).

6. “Apparently, tsunami survivors were inclined to assume greater financial risk in the short-term while rebuilding their lives after the disaster.”  Link here.

The case for nurse practitioners

by  Tyler Cowen in

Many states have recently changed their scope of practice laws and granted full practice authority to nurse practitioners, allowing them to practice without oversight from physicians. Physician groups have argued against this change, citing patient safety concerns. In this paper, we use a ratio-in-ratio approach to evaluate whether the transition to full practice authority results in harm to patients as proxied by rates of malpractice payouts and adverse action reports against nurse practitioners. We find no evidence of such harm, and instead find that physicians may benefit from the law change in terms of reduced malpractice payouts against them.

That is from a new NBER working paper by Sara Markowitz and Andrew J.D. Smith.

Science is proceeding

by  Tyler Cowen

In a few days, a £1.4bn probe will be blasted into space on an eight-year mission to find signs of life on other worlds in our solar system. The spacecraft will not head to local destinations such as the planet Mars, however. Instead, it will fly into deep space and survey the icy moons of distant Jupiter. In doing so, it will open up a new chapter in the hunt for extraterrestrial life.

The Jupiter Icy Moons Explorer – or Juice – will exploit an unexpected feature of our solar system. The greatest reserves of water turn out to exist on worlds very far from Earth, in deep space, and in orbit around the giant planets Jupiter and Saturn. Juice is the first mission to be launched specifically to explore these remote worlds.

Here is the full story, via mdschultz.

Where is the best place to live if a cataclysm comes?

by  Tyler Cowen

My counterintuitive answer is northern Virginia, or at least the general DC area, putting LDS options aside.  I’m talking about asteroids, super-volcanos, and nuclear exchanges, not AGI risk.  Here is a Bloomberg column on that topic:

I have a counterintuitive answer: If you live in a dense urban area, stay put — especially if, like me, you live in the suburbs of Washington, D.C.

The biggest advantage of the Washington region is that, in the case of a real catastrophe, it would receive a lot of direct aid. It’s not just that Congress and the White House are nearby — so are the Pentagon, the FBI, the CIA and hundreds if not thousands of government agencies. Insofar as there might be an emergency response to a cataclysmic event, the Washington area will be prioritized.

The region also has plenty of hospitals and doctors, and a wide variety of law-enforcement units — including the various federal agencies as well as police from Maryland, Virginia and D.C. If you care about order being restored, Washington will be better than most places.

Of course, a counterargument is that Washington is more likely than most places to be hit by a cataclysmic event, especially if it involves a nuclear exchange or some other weapon of mass destruction. But there’s “good news,” scare-quotes intended: If a foreign enemy is truly intent on targeting America’s capital, the conflict may be so extreme that it won’t matter where you go. (If I were a foreign power attacking the US, Washington would not be my first choice as a target, as it would virtually guarantee the complete destruction of my own country.)

I consider — and reject — New Zealand and the American West as alternate options.  New Zealand might not even let you in.

Saudi fact of the day

by  Tyler Cowen in

In line with its ambitions to diversify its economy away from oil and to become a video gaming powerhouse, Saudi Arabia will be investing $38 billion in the local online gaming industry in Riyadh.

According to a report by Bloomberg on Monday, Savvy Gaming Group, a subsidiary of the kingdom’s sovereign Public Investment Fund (PIF), is seeking not only game projects to acquire, but also to develop and publish its own.

Here is the full story.  Remember all those stories years ago, about how Saudi stability was at its end and the Kingdom soon would be bankrupt?  Or maybe taken over by terrorists?  It seems they were wrong.

Via Anecdotal.

Some doubts about Chinese industrial policy

by  Tyler Cowen in

Each year governments worldwide spend an enormous amount of money subsidising businesses. This column investigates the relationship between the allocation of government subsidies and total productivity for Chinese listed firms. The authors find little evidence that the Chinese government consistently ‘picks winners’. Firms’ ex-ante productivity is negatively correlated with subsidies received by firms, and subsidies appear to have a negative impact on firms’ ex-post productivity growth.

And this:

Our sample includes all firms listed on the Shanghai and Shenzhen stock exchanges from 2007 to 2018 except financial services firms, for which computation of productivity presents a number of challenges.

That is from a new study by Lee Branstetter, Guangwei Li, and Mengjia Ren.  Such estimations are typically fraught with questionable assumptions, but at the very least the results do not come up positive.

EA, AI, and the rationality community

by  Tyler Cowen

More broadly, I think AI Alignment ideas/the EA community/the rationality community played a pretty substantial role in the founding of the three leading AGI labs (Deepmind, OpenAI, Anthropic), and man, I sure would feel better about a world where none of these would exist, though I also feel quite uncertain here. But it does sure feel like we had a quite large counterfactual effect on AI timelines.

That is from “habryka, Ben Pace” on LessWrong blog.  As you might expect, I would give those comments a different valence, nonetheless they are insightful.  Here are my points:

1. It is truly remarkable how much influence the cited movements have had.  Whether or not you agree in full (or at all), this should be recognized and respected.  Kudos to them!  And remember, so often ideas lie behind technology.

2. Anthropic has announced a raise of $5 billion and is promoting its intention to compete with Open AI and indeed outdo them.  The concept “Solve for the equilibrium” should rise in status.

3. You cannot separate “interest in funding AI safety” (which I am all for) from “AI progress.”  That by now should be obvious.  No progress, no real interest in safety issues.

4. To this day, the Doomsters are de facto the greatest accelerationists.  Have you noticed how the Democrats (or Republicans) “own” certain political issues?  For instance, voters trust the Democrats more with Social Security, and the mere mention of the topic helps them, even if a Republican has a good point to make.  Well, the national security establishment “owns” the ideas of existential risk and risk from foreign powers.  The more you talk about doomsday issues, the more AI risk gets slotted into their purview, for better or worse.  And they ain’t Brussels (thank goodness).  To the extent the Doomsters have impact, their net effect will be to place the national security types in charge, or at least to raise their influence.  And how do they think that is going to work out (on their own terms)?  Perhaps they would do better to focus on mundane copyright and libel issues with LLMs, but that is not their nature.