Eight Things to Know about LLMS
A good overview from computer scientist Samuel R. Bowman of NYU, currently at Anthropic:
1. LLMs predictably get more capable with increasing investment, even without targeted innovation.
2. Many important LLM behaviors emerge unpredictably as a byproduct of increasing investment.
3. LLMs often appear to learn and use representations of the outside world.
4. There are no reliable techniques for steering the behavior of LLMs.
5. Experts are not yet able to interpret the inner workings of LLMs.
6. Human performance on a task isn’t an upper bound on LLM performance.
7. LLMs need not express the values of their creators nor the values encoded in web text.
8. Brief interactions with LLMs are often misleading.
Bowman doesn’t put it this way but there are two ways of framing AI risk. The first perspective envisions an alien superintelligence that annihilates the world. The second perspective is that humans will use AIs before their capabilities, weaknesses and failure modes are well understood. Framed in the latter way, it seems inevitable that we are going to have problems. The crux of the dilemma is that AI capability is increasing faster than our AI understanding. Thus AIs will be widely used long before they are widely understood. You don’t have to believe in “foom” to worry that capability and control are rapidly diverging. More generally, AIs are a tail risk technology, and historically, we have not been good at managing tail risks.
What do we need to talk to whales?
We detail a scientific roadmap for advancing the understanding of communication of whales that can be built further upon as a template to decipher other forms of animal and non-human communication. Sperm whales, with their highly developed neuroanatomical features, cognitive abilities, social structures, and discrete click-based encoding make for an excellent model for advanced tools that can be applied to other animals in the future. We outline the key elements required for the collection and processing of massive datasets, detecting basic communication units and language-like higher-level structures, and validating models through interactive playback experiments. The technological capabilities developed by such an undertaking hold potential for cross-applications in broader communities investigating non-human communication and behavioral research.
That is from a new research paper by Jacob Andreas, et.al., and the (ungated) article offers considerable detail on exactly how to do this. They already have funding from both Dalio and Audacious.
What I’ve been reading and not reading (due to travel)
Colin Kidd, Union and Unionisms: Political Thought in Scotland, 1500-2000. A very good and well-written look at Scottish views on the Union over the centuries. Explained conceptually in a nice way, not just a catalog, and tied to religion as well.
Thomas Bartlett, Ireland: A History. One of the best one-volume introductions to Irish history.
W. Paul Reeve, Let’s Talk About Race and Priesthood. Argues that the Mormons had relatively universalistic origins, and that Brigham Young was the one who introduced the later segregationist ideas.
There is Peter Turchin, End Times: Elites, Counter-Elites, and the Path of Political Disintegration.
The impressive Jon Elster has just published America Before 1787: The Unraveling of a Colonial Regime.
Do not forget John Cochrane’s The Fiscal Theory of the Price Level, as presented on John’s blog as well.
Coming out is Robin Douglass, Mandeville’s Fable: Pride, Hypocrisy, and Sociability.
How big is Mexico?
How big is Mexico? https://t.co/rNmL3bhYSE pic.twitter.com/Ql80iyx3q9
— Steve Stewart-Williams (@SteveStuWill) April 11, 2023
Wednesday assorted links
1. Who benefits most from name visibility bias during the journal editorial process?
2. Prendergast watercolor for 500-700k, truly a splendid piece, St. Marks in Venice. The collection as a whole, while not my taste (“too American” in a very particular direction), shows exquisite taste. You can learn a lot by studying their choices. From the Wolf family. Here is more from their collection.
3. AI Policy Guide, by Matthew Mittelsteadt at Mercatus. And a clear explanation of the new “autonomous” AIs.
4. Current U.S. defense spending is, in historical terms, at a relative low point.
5. Miami Native, new (non-leftist) magazine on the way, presenting and explicating and enhancing the status of the culture of Miami. They are looking for contributors. Mainly a physical copy magazine, planning only a limited presence on-line.
6. Genetic timeline of humans? (speculative) And I believe in hiring talented 14- to 15-year olds.
Measuring the benefits of the biomedical revolution
That is the topic of my latest Bloomberg column. Note that for most economic gains, total gdp and per capita gdp give roughly the same answers. But when it comes to lifesaving, that may no longer be the case. Here is one excerpt:
Take the vaccines against Covid. Of course the most important fact about them is that they reduce the amount of death and suffering. But what is their economic impact? The vaccines have been most helpful to the most vulnerable, namely the elderly or those with preexisting medical conditions. These are not the most productive cohorts of the economy. So the effectiveness of the vaccines might have actually lowered various social averages, such as per-capita GDP or per-capita productivity.
The extra life is a pure benefit. But to capture that benefit in numbers requires looking at the totals, not just the averages. Labor productivity per hour, for example, won’t necessarily increase. But total labor supply and total population will.
And this:
And what about those subpar returns on biomedical investments? That is a sign that most of the gains from innovation are being reaped by patients, users and consumers — not capitalists. Is that not exactly what everyone has been asking for?
There is much more at the link. The bottom line is that many of the gains will come through “n,” not per hour productivity.
My favorite things Alaska
I haven’t done many of these in a while, mostly because I haven’t been in many new states or countries recently. But Alaska I had never visited before (my remaining state, in fact), so here goes:
Classical music: John Luther Adams. “The other John Adams,” his reputation continues to rise, now I would like to see one performed live. I am fan of the sound textures and the broad expanses of his works, even if the programmatic aspects do not always delight me. Become Ocean is his best known piece.
Popular music: There is Jewel, I guess she is OK, and I can’t think of anyone else.
This is tough! Nor did Andre Marrou acquit himself especially well over the years. How about “theatre builder-upper”? Then I can cite Edward Albee.
Affiliated writer: Jack London, obviously. Still worth reading, not archaic, has held up remarkably well.
Movies, set in: Plenty of competition here. There is Herzog’s Grizzly Man, and Never Cry Wolf (oddly forgotten but moving, plus the protagonist is named Tyler, which was rare in the early 1980s), and of course Chaplin’s The Gold Rush. Into the Wild I haven’t seen. Maybe I watched Abbott and Costello Lost in Alaska as a kid? What am I missing?
Artist: Taking the entire cake has to be Alaskan indigenous art, but who should be the favorite? I can’t bring myself to elevate Florence Nupok Malewotkuk to the number one position, so perhaps Nathan Jackson, who did Tlingit art?
Watercolor, affiliated with: Try this John La Farge, currently up at auction.
Throat singer: A strong area, but which ones exactly are from Alaska rather than Canada? Janet Aglukkaq? Don’t ask me!
Here is a good essay on Alaskan totem poles, from Eyak, Tlingit, Haida and Tsimshian cultures.
I can’t name a mask-maker, but the masks are arguably the highlight of the Alaskan indigenous tradition.
Any NBA players? Am I supposed to like Carlos Boozer?
The bottom line: There is more than you might think at first.
From the comments, on AI safety
This is from Richard Ngo, who works on the governance team at OpenAI:
A few points:
1. I agree that the alignment community has generally been remiss in not trying hard enough to clarify the arguments in more formal papers.
2. The only peer-reviewed paper making the case for AI risk that I know of is: https://onlinelibrary.wiley.com/doi/10.1002/aaai.12064. Though note that my paper (the second you linked) is currently under review at a top ML conference.
3. I don’t think that a formal model would shed much light here. My goal in writing my paper was to establish misaligned power-seeking AGI as a credible scientific hypothesis; I think that most who think it’s credible would then agree that investigating it further should be a key priority, whether or not their credences are more like 10% or more like 90%.
From this batch of comments. Here is Richard on Twitter.
Tuesday assorted links
1. Review of the new Philip Wallach book on Congress (Rep. Katie Porter’s book too).
2. Good Ding vs. Nepo coverage.
3. On properly translating Macron (having dealt with French diplomats, both through translation and not, I agree with the general points about context). That said, the whole world has to receive the proper message, as a matter of common knowledge, and arguably he failed in that regard.
4. How AI differs in warfare. And something about “BabyAGI.” Self-improving AI making its debut? And lots of discussion.
5. The roots of our military recruiting crisis, good and interesting piece.
6. Is Tupperware toast? And upscale compost (WSJ).
“Date me” docs
Here is one from Katja Grace, by the way I know her a bit and very much like her and find her very smart (NB: not interested in comments mocking Katja, put them somewhere else, I will delete them). But that is not my main point today. Do such documents work? I have been hearing of them more often lately. And how should we model them?
Should we think of them as batch auctions of a sort, namely wanting to get in a lot of bids at once rather than sequentially? Which kinds of people should prefer such a batch auction? (Btw, is there any paper in Science or Nature on this? Should there be?)
Are they better suited for polyamory than monogamy? Are batch auctions better suited for polyamory? Because the process is more like assembling a portfolio?
Is this all somehow better suited for San Francisco and other “Woke” cultures, where perhaps asking someone out on a date counts as a microaggression? I suppose the Date Me document gives you permission to reach out? I am curious to read or hear some serious takes on this phenomenon.
Anarchy in South Africa
Public services such as police, fire, and traffic control in South Africa are breaking down. Private firms are stepping in to take some of the burden. Twenty two percent of Johannesburg’s fire engines are owned and operated by private firms.
Fire Ops employs more than 60 firefighters across seven fire stations in Johannesburg and owns two fire engines—including one now sporting the same shade of blue Discovery uses for its logo and much of its branding—as well as six smaller high-pressure-pump response vehicles.
Discovery says the blue firetruck responded to 172 building fires between Fire Force’s launch through the end of January.
Mr. Ossip said the Discovery-branded truck promotes the insurer’s brand and lowers damages, including to multimillion-dollar homes in some of Johannesburg’s toniest areas. “You need to just save one or two of those a year and it is substantial savings,” he said.
The service helps alleviate a shortage of operational fire engines in Johannesburg, a spread-out city of more than 5.5 million residents, in situations where minutes can make the difference between a blaze limited to a couple of rooms and one that destroys an entire house or spreads to neighboring homes.
Robert Mulaudzi, a spokesman for the City of Johannesburg Emergency Management Services, said the city currently has about seven operational fire engines across 30 fire stations.
…Fire Ops, which invoices buildings’ owners for fire services, says that while it responds to all calls, it will give priority to clients, including Discovery policyholders, when simultaneous fires break out. Other insurers usually pick up the bill when the company puts out a fire in a home not insured by Discovery, said De Wet Engelbrecht, Fire Ops’s chief executive.
In 19th century Great Britain prosecution assocations and insurance firms were responsible for much of the policing (see Stephen Davies in The Voluntary City.) In Lessons from Gurgaon, India’s Private City (working paper) Shruti Rajagopolan and I discuss private police and fire services in modern day Gurgaon. In general, the private firms provide excellent service relative to their public counterparts but, as in Gurgaon, there are limits to how much the private firms can do without large economies of scale:
…Fire Ops also has to navigate public infrastructure that doesn’t always work, including traffic lights, fire hydrants and municipal water supplies….In September, both Fire Ops and the city’s fire department responded to a blaze at Little Forest Centre, a private special-needs school in Johannesburg, but a water shortage in the area meant all fire hydrants were empty, said Kate More, the school’s owner and principal, who isn’t a Discovery policyholder.
Despite Fire Ops sourcing water from a neighbor’s pool, the school burned down.
Addendum: In unrelated news, just one year after its grand opening Whole Foods is closing its downtown San Francisco store because they can’t ensure the safety of their employees.
This GPT-4 answer speaks for itself
TC: Which is the leading attempt to publish a canonical paper on AGI risk, in a leading science journal, refereed of course. The paper should have a formal model or calibration of some sort, working toward the conclusion of showing that the relevant risk is actually fairly high. Is there any such thing? Please answer as would a leading scientific expert in the relevant fields, including computer science and possibly also population ecology, systems theory, and other possible areas.
GPT-4: As a leading scientific expert, I must clarify that while there are numerous papers discussing various aspects of AGI (Artificial General Intelligence) risk, no single paper can be labeled as the “canonical” paper on the subject. It is worth noting that the field of AGI risk is interdisciplinary, involving computer science, artificial intelligence, ethics, population ecology, systems theory, and other domains. Consequently, the body of research on AGI risk is diverse, with various approaches and models, some of which may involve formal calibration.
One paper that may be of interest to you is “The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation” by Brundage et al., published in 2018. This paper, co-authored by experts from various institutions, including OpenAI, the University of Oxford, and the Center for a New American Security, discusses potential risks from the malicious use of AI and offers recommendations for researchers, policymakers, and industry practitioners. While this paper does not provide a formal calibration model, it does shed light on the potential risks associated with AGI.
Another important work in the field is Nick Bostrom’s book “Superintelligence: Paths, Dangers, Strategies,” which discusses the risks associated with the development of AGI and offers possible strategies to mitigate those risks. Bostrom’s work has been influential in shaping the discourse on AGI risk, but it is not a refereed journal article….[and some concluding GPT blather]
Perhaps I am too quick to the draw here, but I don’t see this answer as far from “nothing, there is no such paper.” Comments are open, of course, if you can surpass GPT-4, which perhaps is better at evolving into a world-destroying entity than it is at finding citations. Further prods did not change the basic answer, and if anything GPT models tend to confabulate or hallucinate entries, not deny them. Or perhaps in this case it is hiding the refereed articles and deceiving us?
And maybe I’ve missed it, but I’ve also never seen Scott Alexander or Zvi point to such a paper, or even a good example of a rejected paper aiming in this direction. Nor have I seen them make a big stink about the absence of such a paper, though in virtually any other area they will hit you with a fire hose of citations and links to published models in referred journals.
I’ve also asked a whole bunch of “people who ought to know” and not received a single concrete answer, one such individual responding immediately with the answer “zero.”
In part, I would like to encourage those fascinated with AGI risk to try to create and publish such a paper, or perhaps to fund it or otherwise encourage it. Something more systematically fleshed out than “10 reasons why lists of 10 reasons might be a winning strategy.” It would go a long way to giving the idea more credibility in the scientific community, not to mention with yours truly. How about Nature? Science? Somewhere else? I know top journals can be closed or unfair, but at the very least you can put the paper and the associated referee reports on-line for the rest of us to judge. And then try it in a lesser journal, it still will get traction and you will get valuable feedback, of a very different kind than from on-line forums.
If the chance of existential risk from AGI is 99 percent, or 80 percent, or even 30 percent, surely some kind of modeled demonstration of the basic mechanics and interlocking pieces is possible. Indeed a certain kind of clarity should be evident, at least conditional on the more extreme views being correct. In general, I am not a fan of the “you should work on this!’ strategy, but if you think the whole future of the entire world is at stake…shouldn’t you be obsessed with working on such a thing, if only to convince the rest of us? And in as many different formats as possible, including the methods most commonly recognized by the scientific community?
In the meantime, if you are a young person interested in this issue, and you observe such a paucity of refereed, published model-based papers in the area — consider any area just to get your mind off the fraught and emotional topic of AGI existential risk — what would you infer from that absence?
And what if said community of commentators almost universally insisted they were the most extreme of rationalists?
Now none of this means the claims about extreme risk are wrong. But you can think of it as a kind of propaedeutic to reading the literature and current debates.
Addendum: I have looked at papers such as these:
https://arxiv.org/abs/2206.13353, https://arxiv.org/abs/2209.00626, https://arxiv.org/abs/2109.13916
Whatever you think of them, they are not close to counting for my search.
A new Operation Warp Speed for better vaccines
The Biden administration is launching a $5 billion-plus program to accelerate development of new coronavirus vaccines and treatments, seeking to better protect against a still-mutating virus, as well as other coronaviruses that might threaten us in the future.
“Project Next Gen” — the long-anticipated follow-up to “Operation Warp Speed,” the Trump-era program that sped coronavirus vaccines to patients in 2020 — would take a similar approach to partnering with private-sector companies to expedite development of vaccines and therapies. Scientists, public heath experts and politicians have called for the initiative, warning that existing therapies have steadily lost their effectiveness and that new ones are needed…
Jha and others said the new effort will focus on three goals: creating long-lasting monoclonal antibodies, after an evolving virus rendered many current treatments ineffective; accelerating development of vaccines that produce mucosal immunity, which is thought to reduce transmission and infection risks; and speeding efforts to develop pan-coronavirus vaccines to guard against new SARS-CoV-2 variants, as well as other coronaviruses.
Here is the WaPo article, here is commentary from Eric Topol.
Monday assorted links
1. Benevolent sexism. And should single American women look for suitors abroad?
2. Does summer learning loss replicate? And on Covid learning loss recovery: “On average, we find that 20% of test score losses are recovered in English language arts (ELA) by 2022, compared to 37% in math.”
3. Balenciega beige unicorn sneakers. And, via Yana, MEN’S TRASH BAG LARGE POUCH IN BLACK.
4. Does Ozempic improve impulse control? (speculative)
5. No adverse labor supply effects from the expanded child tax credit (short-term only, though).
6. “Apparently, tsunami survivors were inclined to assume greater financial risk in the short-term while rebuilding their lives after the disaster.” Link here.
The case for nurse practitioners
Many states have recently changed their scope of practice laws and granted full practice authority to nurse practitioners, allowing them to practice without oversight from physicians. Physician groups have argued against this change, citing patient safety concerns. In this paper, we use a ratio-in-ratio approach to evaluate whether the transition to full practice authority results in harm to patients as proxied by rates of malpractice payouts and adverse action reports against nurse practitioners. We find no evidence of such harm, and instead find that physicians may benefit from the law change in terms of reduced malpractice payouts against them.
That is from a new NBER working paper by Sara Markowitz and Andrew J.D. Smith.