GPT4 passed the medical licensure exam but the critics want to know how does it perform in the real world? Zak Kohane, pediatric endocrinologist, data scientist, and chair of the Harvard Chair of the Department of Biomedical Informatics at Harvard Medical School has apparently been working with GPT4 for about 6 months. He has a forthcoming book (with Peter Lee and Carey Goldberg). He writes:
“How well does the AI perform clinically? And my answer is, I’m stunned to say: Better than many doctors I’ve observed.”—Isaac Kohane MD
That’s from a review of the book by Eric Topol. Not much more information to be had in the review but if you think about it, this bit is hilarious:
I’ve thought it would be pretty darn difficult to see machines express empathy, but there are many interactions that suggest this is not only achievable but can even be used to coach clinicians to be more sensitive and empathic with their communication to patients.
Beginning with the Trump administration, and accelerating under the Biden administration, U.S. trade and industrial policy has prioritized relocating manufacturing production back to the United States. For all their differences, both administrations disregarded other countries in this pursuit. Both also attacked international trade and investment as harmful to U.S. economic and national security, even though the rules for that very system were established by the United States and serve its interests. Along with members of Congress from both parties, the Biden administration has sought to take away production from others in a zero-sum way—explicitly from China and a bit more courteously from others.
This policy approach, while having considerable popular appeal at home, is based on four profound analytic fallacies: that self-dealing is smart; that self-sufficiency is attainable; that more subsidies are better; and that local production is what matters. Each of these assumptions is contradicted by more than two centuries of well-researched history of foreign economic policies and their effects.
The US has benefitted from leading a rules based system of global trade but it is throwing the rules away to go after individual countries on a one-on-one basis.
In big-league sports, the best job is to be league commissioner. As commissioner, you make money whichever team wins or loses on a given day, you are welcome at every stadium (even if occasionally booed), and you can ultimately decide the big questions of how the game is played and who is allowed to own a team. If you instead become identified with a single team, sometimes you win, sometimes you lose, but most importantly, others have an interest in your losing. You might even get repeatedly punished for cheating, instead of being the one to decide who is cheating.
Buy American doesn’t work.
The idea of “Buy American” has broad populist appeal. It connotes an economy that is self- sufficient, producing all it needs, and “putting American workers first.” Yet detailed research has repeatedly shown that policies aimed at maximizing domestic manufacturing employment rather than the development and adoption of new technologies are not only doomed to fail but crowd out the very industrial and trade policies that contribute the most to innovation, national security, and decarbonization.
The US should bet on rules and growth.
At its core, a successful U.S. industrial policy is one that promotes the widespread diffusion and adoption of the best technologies, even if that means the United States purchasing them from production located abroad. Innovation and technical progress are accelerated by having common standards at global scale, not by politically captured industries with barriers to entry. This approach is especially necessary for decarbonization but also to increase supply chain resilience and the ability of other countries to stand up to Chinese threats.
Fifty years ago today, on March 23, 1973, Alexander P. Butterfield, the Administrator of the Federal Aviation Administration, issued a rule that remains one of the most destructive acts of industrial vandalism in history.
“No person may operate a civil aircraft at a true flight mach number greater than 1 except in compliance with conditions and limitations in an authorization to exceed mach 1 issued to the operator under Appendix B of this part.”
This text was slightly modified in 1989 and again in 2021, but the upshot remains the same. The rule imposed a speed limit on US airspace. Not a noise standard, which would make sense. A speed limit.
This speed limit has naturally distorted the development of civil aircraft. For fifty years, the aviation industry has worked to improve subsonic aviation. Commercial passenger aircraft are safer and more economical today than they were in 1973, but they are no faster.
If we had propagated the rate of growth in commercial transatlantic aircraft speeds that existed from 1939 to the mid-1970s, we would have Mach-4 airliners by now. But the overland ban put an end to all that. It made small supersonic aircraft, which need to fly shorter overland routes, essentially illegal, closing off the iteration cycle that could drive progress in the industry.
That’s Eli Dourado who notes that modern designs greatly reduce sonic boom. I would also add the following. In 2019 there were 811 million passengers on US domestic flights and 241 million passengers on US international flights. The average duration of a domestic flight is about 2.5 hours and an international fight about 7.3 hours so Americans spend about 3.7 billion hours every year on airplanes. If we could cut even 20% of that time that’s a saving of 757 million hours which has to be weighed against a few people experiencing sonic booms near airports. Indeed, since the people on the airplane are subjected to a lot of the noise the total amount of noise experienced could easily go down with faster aircraft!
The central claim of our work is that GPT-4 attains a form of general intelligence, indeed showing sparks of artificial general intelligence. This is demonstrated by its core mental capabilities (such as reasoning, creativity, and deduction), its range of topics on which it has gained expertise (such as literature, medicine, and coding), and the variety of tasks it is able to perform (e.g., playing games, using tools, explaining itself…). A lot remains to be done to create a system that could qualify as a complete AGI.
In The New Madness of Crowds I argued that SVB failed because “Greater transparency and lower transaction costs have intensified the madness of the masses and expanded their reach.” A piece by Miao, Zuckerman and Eisen in the WSJ now adds to to the other side of the problem. Depositors were working on twitter time, the regulatory apparatus was not.
Depositors were draining their accounts via smartphone apps and telling their startup networks to do the same. But inside Silicon Valley Bank, executives were trying to navigate the U.S. banking system’s creaky apparatus for emergency lending and to persuade its custodian bank to stay open late to handle a multibillion-dollar transfer.
Instead of hearing a rumor at the coffee shop and running down to the bank branch to wait on line to withdraw your money, now you can hear a rumor on Twitter or the group chat and use an app to withdraw money instantly. A tech-friendly bank with a highly digitally connected set of depositors can lose 25% of its deposits in hours, which did not seem conceivable in previous eras of bank runs.
But the other part of the problem is that, while depositors can panic faster and banks can give them their money faster, the lender-of-last-resort system on which all of this relies is still stuck in a slower, more leisurely era. “When the user interface improves faster than the core system, it means customers can act faster than the bank can react,” wrote Byrne Hobart. You can panic in an instant and withdraw your money with an app, but the bank can’t get more money without a series of phone calls and test trades that can only happen during regular business hours.
It’s not obvious whether the right thing to do is slow down depositors, at least in some circumstances, or speed up regulators but the two systems can’t work well at different speeds.
When interest rates go up, the price of bonds goes down. As Tyler and I discuss in Modern Principles, the inverse relationship between interest rates and prices holds for any asset that pays out over time. In particular, as Patrick McKenzie points out, when interest rates go up, the value of a loan goes down. McKenzie suggests that you can use this fact to buy back your mortgage from a bank when interest rates rise.
For example, suppose you get a 500k 30-year fixed rate mortgage when interest rates are 3%–that loan obligates you to pay $2108 per month for 30 years. Now suppose that interest rates go to 6%, now that same stream of payments is only worth, in present value, about $358k. Thus, the bank should be willing to let you buy your mortgage for $358k–that is, after all, what the market would pay for such a stream of payments if your mortgage was securitized.
I am skeptical that I could find the right person at the right bank to actually authorize a deal like this but it turns out that the Danish mortgage system is built to allow this relatively easily. The Danish mortgage system is built on the match principle:
JYSKE Bank: The match-funding principle entails that for every loan made by the mortgage bank, a new bond is issued with matching cash-flow properties. This eliminates mismatches in cash-flows and refinancing risk for the mortgage bank, which also secures payments for the bondholder. In the Danish mortgage system the mortgage bank functions as an intermediary between the investor and borrower. Mortgage banks fund loans on a current basis, meaning that the bond must be sold before the loan can be given. This also entails that the market price of the bond determines the loan rate. The loan is therefore equal to the investment, which passes through the mortgage bank.
In essence, in the Danish system, mortgage banks are more like a futures clearinghouse or a platform (ala Airbnb) than a lender–they take on some credit risk but not interest rate risk.
Thus, if a Danish borrower takes out a 500k mortgage at 3% interest and then rates rise to 6%, the value of that mortgage falls to $358k and the borrower could go to the market, buy their own mortgage, deliver it to the bank, and, in this way, extinguish the loan. Since the value of homes also falls as interest rates rise this is also a neat bit of insurance. Remarkable!
The Danish mortgage market appears to be very successful and so may be a model for American reform:
JYSKE Bank: The Danish Mortgage Bond Market is one of the oldest and most stable in the world, tracing its roots all the way back to 1797 with no records of defaults since inception. Furthermore, the market value of the Danish Mortgage Bond Market is approx. EUR 402bn, making it the largest mortgage bond market in Europe.
Here’s an interesting idea it wouldn’t have occured to me to ask. What is the length of time described in the average 250 words of narration and how has this changed over time? Most famously James Joyce’s “Ulysses” is a long novel about single day with many pages describing brief experiences in minute detail. In contrast, Olaf Stapledon’s Last and First Men covers 2 billion years in fewer words than Joyce uses to cover a single day.
Using human readers grading 1000 passages, Underwood et al. (2018) finds that the average length of time described in a typical passage has declined substantially since the 1700s, from a day to about an hour so a decline by a factor of 24. Writers have become much more focused on describing individual experiences than events.
In a blog post, Underwood revisits his earlier work but this time comparing human readers with various statistical techniques and also GPT. To use GPT he prompts
Read the following passage of fiction. Then do five things. 1: Briefly summarize the passage. 2: Reason step by step to decide how much time is described in the passage. If the passage doesn’t include any explicit reference to time, you can guess how much time the events described would have taken. Even description can imply the passage of time by describing the earlier history of people or buildings. But characters’ references to the past or future in spoken dialogue should not count as time that passed in the scene. Report the time using units of years, weeks, days, hours, or minutes. Do not say zero or N/A. 3: If you described a range of possible times in step 2 take the midpoint of the range. Then multiply to convert the units into minutes. 4: Report only the number of minutes elapsed, which should match the number in step 3. Do not reply N/A. 5: Given the amount of speculation required in step 2, describe your certainty about the estimate–either high, moderate, or low. The passage follows: <omitting this to save space>
Here’s an example of the analysis:
ABRIDGED PASSAGE FROM Jane Eyre: And signs, for aught we know, may be but the sympathies of Nature with man. When I was a little girl, only six years old, I one night heard Bessie Leaven say to Martha Abbot that she had been dreaming about a little child … Of late I had often recalled this saying and this incident; for during the past week scarcely a night had gone over my couch that had not brought with it a dream of an infant, which I sometimes hushed in my arms, sometimes dandled on my knee, sometimes watched playing with daisies on a lawn, or again, dabbling its hands in running water. It was a wailing child this night, and a laughing one the next: now it nestled close to me, and now it ran from me; but whatever mood the apparition evinced, whatever aspect it wore, it failed not for seven successive nights to meet me the moment I entered the land of slumber.
ASSISTANT REPLIES.
1: The narrator recalls a saying about dreaming of children being a sign of trouble and remembers an incident from her childhood. She then describes her recent dreams of infants over the past week.
2: The passage mentions the narrator’s childhood memory, which happened when she was six years old, and the past week of dreams. The time elapsed in the passage is the week of dreams.
3: 1 week has elapsed. At seven days a week that’s 7 days. 24 hours in a day, so 168 hours. 60 minutes in an hour, so 10080 minutes.
4: 10080 minutes.
5: Moderate confidence, as the childhood memory is not counted as time passing in the scene.
Overall, Underwood finds that GPT-4 is more accurate than previous statistical techniques in that its judgments correlate with those of human readers nearly as much as human readers correlate with themselves.
USDC and USDT are two well-known stablecoins. USDC is fully backing by safe, liquid assets, which are verified monthly by a major U.S. accounting firm under the scrutiny of U.S. state regulators. USDT (Tether) is an unregulated stablecoin with questionable asset backing and opaque operations, founded by an actor from the Mighty Ducks and supported by a bank established by one of the creators of Inspector Gadget.
Yet, when Silicon Valley Bank (SVB) went into crisis, USDC broke the peg, and people fled to the nutty, opaque, unregulated Inspector Gadget backed coin.
(USDC is in blue and measured on the right axis and spiked below par, USDT is in red and measured on the left axis and spiked over par.)
Now, this is in some sense “explainable”. USDC kept some money at SVB and Tether (probably) did not. Matthew Zeitlin, channeling Matt Levine, put it this way:
One problem with being transparently and fully backed is that sometimes your investors can transparently see how much of your assets are in a bank that went bottom up, Tether does not have this problem.
SVB’s troubles stemmed from its investments in long-term government bonds, which dropped in value as interest rates rose. However, the bank’s fundamentals were not that dire. If no one had panicked, SVB could probably have paid off all its depositors in the ordinary course of business. The problem happened because some investors saw information they thought others might interpret negatively, prompting them to withdraw their funds. This led others to believe the information was indeed bad, validating the initial belief and causing a massive $42 billion withdrawal in a single day. Had transparency been less and transaction costs more, this wouldn’t have happened and, quite possibly, everything would have been fine.
Indeed, in the past, banks probably become insolvent on a mark-to-market basis but few people noticed. Today, a bank dips below the line and depositors are heading to the door.
SVB’s fundamentals may have been worse than I believe, poor management undoubtedly played a role. But fundamentals aren’t driving the boat; the boat is being driven by sunspots, memes, and vibes. Tether’s fundamentals are much worse than SVBs ever were. And USDC was even less imperiled than SVB, yet people ran to Tether. Why? Because there wasn’t a Tether sunspot. But be careful. Tether’s stability doesn’t mean that its fundamentals are strong. Not even close. Stability doesn’t mean good fundamentals and instability doesn’t mean bad fundamentals. The mad crowd is capricious. Tether’s time is coming, but no one knows what will spark the fire.
Greater transparency and lower transaction costs have intensified the madness of the masses and expanded their reach. From finance to politics and culture, no domain remains untouched by the new madness of crowds.
PricewaterhouseCoopers (PWC), the global business services firm, has signed a deal with OpenAI for access to “Harvey”, OpenAI’s Chatbot for legal services.
Reuters: PricewaterhouseCoopers said Wednesday that it will give 4,000 of its legal professionals access to an artificial intelligence platform, becoming the latest firm to introduce generative AI technology for legal work.
PwC said it partnered with AI startup Harvey for an initial 12-month contract, which the accounting and consulting firm said will help lawyers with contract analysis, regulatory compliance work, due diligence and other legal advisory and consulting services.
PwC said it will also determine ways for tax professionals to use the technology.
IBM’s Watson was a failure so we will see but, yeah I will say it, this time feels different. For one, lawyers deal with text where GPTs excel. Second, GPTs have already revolutionized software coding and unlike Watson I am using GPTs every day for writing and researching and it works. The entire world of white collar work is going to be transformed over the next year. See also my paper with Tyler, How to Learn and Teach Economics with Large Language Models, Including GPT.
GPTs, such as ChatGPT and Bing Chat, are capable of answering economics questions, solving specific economic models, creating exams, assisting with research, generating ideas, and enhancing writing, among other tasks. This paper highlights how these innovative tools differ from prior software and necessitate novel methods of interaction. By providing examples, tips, and guidance, we aim to optimize the use of GPTs and LLMs for learning and teaching economics effectively.
Most of the paper is about how to use GPTs effectively but we also make some substantive points that many people are missing:
GPTs are not simply a chatty front end to the internet. Some GPTs like ChatGPT have no ability to search the internet. Others, like Bing Chat, can search the internet and might do so to aid in answering a question, but that is not fundamentally how they work. It is possible to ask a GPT questions that no one has ever asked before. For example, we asked how Fred Flintstone was like Hamlet, and ChatGPT responded (in part):
Fred Flintstone and Hamlet are two vastly different characters from different time periods, cultures, and mediums of storytelling. It is difficult to draw direct comparisons between the two.
However, one possible point of similarity is that both characters face existential dilemmas and struggles with their sense of purpose and identity. Hamlet is plagued by doubts about his ability to avenge his father’s murder, and his own worthiness as a human being. Similarly, Fred Flintstone often grapples with his place in society and his ability to provide for his family and live up to his own expectations.
Not a bad answer for a silly question and one that (as far as we can tell) cannot be found on the internet.
GPTs have “read” or “absorbed” a great amount of text but that text isn’t stored in a database; instead the text was used to weight the billions of parameters in the neural net. It is thus possible to run a GPT on a powerful home computer. It would be very slow, since computing each word requires billions of calculations, but unlike storing the internet on your home computer, it is feasible to run a GPT on a home computer or even (fairly soon) on a mobile device.
GPTs work by predicting the next word in a sequence. If you hear the phrase “the Star-Spangled”, for example, you and a GPT might predict that the word “Banner” is likely to come next. This is what GPTs are doing but it would be a mistake to conclude that GPTs are simply “autocompletes” or even autocompletes on steroids.
Autocompletes are primarily statistical guesses based on previously asked questions. GPTs in contrast have some understanding (recall the as if modifier) of the meaning of words. Thus GPTs understand that Red, Green, and Blue are related concepts that King, Queen, Man and Woman are related in a specific way such that a woman cannot be a King. It also understands that fast and slow are related concepts, such that a car cannot be going fast and slow at the same time but can be fast and red and so forth. Thus GPTs are able to “autocomplete” sentences which have never been written before, as we described earlier.2 More generally, it seems likely that GPTs are building internal models to help them predict the next word in a sentence (e.g. Li et al. 2023).
The paper is a work in progress so comments are welcome.
If the United States and, say, Great Britain had drug-approval reciprocity, then drugs approved in Britain would gain immediate approval in the United States, and drugs approved in the United States would gain immediate approval in Great Britain. Some countries such as Australia and New Zealand already take into account U.S. approvals when making their own approval decisions. The U.S. government should establish reciprocity with countries that have a proven record of approving safe drugs—including most west European countries, Canada, Japan, and Australia. Such an arrangement would reduce delay and eliminate duplication and wasted resources. By relieving itself of having to review drugs already approved in partner countries, the FDA could review and investigate NDAs more quickly and thoroughly.
Well, it’s happening! After Brexit, there were concerns that drugs would take longer to get approved in the UK because the EU was a much larger market. To address this, the UK introduced the “reliance procedure” which recognized the EU as a stringent regulator and guaranteed approval in the UK within 67 days for any drug approved in the EU. The Reliance Procedure essentially kept the UK in the pre-Brexit situation, and was supposed to be temporary. However, recognizing the logic of recognizing the EU, the UK is now saying that it will recognize other countries.
Our aim is to extend the countries whose assessments we will take account of, increasing routes to market in the UK. We will communicate who these additional regulators are and publish detailed guidance about this new framework in due course, including any transition arrangements for applications received under existing frameworks.
The UK is already participating in a mutual recognition agreement with the FDA over some cancer drugs. Therefore, it seems likely that the FDA will be among the regulatory authorities that the UK recognizes. If the UK does recognize the FDA, then we only need the FDA to recognize the UK for my scenario from more than 20 years ago to be fulfilled.
Reciprocity Ensures Streamlined Use of Lifesaving Treatments Act (S. 2388), or the RESULT Act,” which would amend the Food, Drug and Cosmetic Act to allow for reciprocal approval of drugs.
We don’t yet know the impact that AI will have on productivity but some evidence is starting to come in. Peng et al. (2023) hired programmers on Upwork to write an HTTP server in Javascript; half of the programmers got access to CoPilot (this was before CoPilot was widely available) half did not.
Conditioning on completing the task, the average completion time from the treated group is 71.17 minutes and 160.89 minutes for the control group. This represents a 55.8% reduction in completion time. The p-value for the t-test is 0.0017, and a 95% confidence interval for the improvement is between [21%, 89%]. There are four outliers with time to completion above 300 min. All outliers are in the control group, however our results remain robust if these outliers are dropped. This result suggests that Copilot increases average productivity significantly in our experiment population. We also find that the treated group’s success rate is 7 percentage points higher than the control group, but the estimate is not statistically significant, with a 95% confidence interval of [-0.11, 0.25].
The authors extrapolate wildly:
In 2021, over 4.6 million people in the United States worked in computer and mathematical occupations,1 a Bureau of Labor Statistics category that includes computer programmers, data scientists, and statisticians. These workers earned $464.8 billion or roughly 2% of US GDP. If the results of this study were to be extrapolated to the population level, a 55.8% increase in productivity would imply a significant amount of cost savings in the economy and have a notable impact on GDP growth.
I talk with entreprenreur Niklas Anzinger on the Stranded Technlogies Podcast. Niklas summarizes some of the discussion:
This episode is an intellectual journey that discovers insights that can be used by entrepreneurs and city developers. We talk about the Baumol effect that Alex uses to explain the now infamous price chart.
Alex’s recommendation to new city or governance startups like Prospera, Ciudad Morazan or the Catawba DEZ is to think of city development as a “dance between centralization and decentralization”.
Economists have developed concepts that are waiting to be commercialized, e.g. prediction markets. In this episode, we talk about dominant assurance contracts and how they could be used in new city developments and fundraising.
We have innovations like Uber and Airbnb and many others only because entrepreneurs didn’t have to ask for permission. Had we put these ideas to the vote they would have been defeated. Allow almost anyone with a car to drive customers around town? Stranger danger! Let any house be turned into a hotel? Not in my neighborhood! Once the innovations were brought into existence, the masses saw the benefits but they would not have seen those benefits if the idea had been put to a vote. Demonstration is more powerful than imagination.
More and more, however, the sphere of individual action shrinks and that of collective action grows.
A small but sadly amusing case in point is building in San Francisco. A plan was proposed to build the apartments at right. Love it, hate it. I don’t care. But it shouldn’t be up for collective action. Instead, what we have, however, is a planning process in which the President of the SF Planning Commission, Myrna Melgar, can opposed the plan because:
….I have to just state that I hate the design. Nothing against the architect, I think that the big windows, to me, are a statement of class and privilege. …having that building, with all of those windows it’s such a statement of, to me, class privilege because you know, poor people don’t do that, they don’t you know, like, win- you know, have everything out on the street. It just, so it just, it really rankles me the wrong way. So I just have to say it is a design issue. To me, design guidelines for what’s going to come are going be really important because I do think it does say to the community – is this still our community, what are we building for?
The building was proposed as a replacement for an auto shop (!) in 2014! Building didn’t start until 2022 and as of January 2023 it still wasn’t complete, although it looks like they got most of the windows approved.
Indeed, in the past, banks probably become insolvent on a mark-to-market basis but few people noticed. Today, a bank dips below the line and depositors are heading to the door.:no_upscale()/cdn.vox-cdn.com/uploads/chorus_asset/file/8429171/1900mission5.jpg)
A small but sadly amusing case in point is building in San Francisco. A plan was proposed to build the apartments at right. Love it, hate it. I don’t care. But it shouldn’t be up for collective action. Instead, what we have, however, is a planning process in which the President of the SF Planning Commission, Myrna Melgar, can opposed the plan because: