Category: Web/Tech
A new RCT on banning smartphones in the classroom
Widespread smartphone bans are being implemented in classrooms worldwide, yet their causal effects on student outcomes remain unclear. In a randomized controlled trial involving nearly 17,000 students, we find that mandatory in-class phone collection led to higher grades — particularly among lower-performing, first-year, and non-STEM students — with an average increase of 0.086 standard deviations. Importantly, students exposed to the ban were substantially more supportive of phone-use restrictions, perceiving greater benefits from these policies and displaying reduced preferences for unrestricted access. This enhanced student receptivity to restrictive digital policies may create a self-reinforcing cycle, where positive firsthand experiences strengthen support for continued implementation. Despite a mild rise in reported fear of missing out, there were no significant changes in overall student well-being, academic motivation, digital usage, or experiences of online harassment. Random classroom spot checks revealed fewer instances of student chatter and disruptive behaviors, along with reduced phone usage and increased engagement among teachers in phone-ban classrooms, suggesting a classroom environment more conducive to learning. Spot checks also revealed that students appear more distracted, possibly due to withdrawal from habitual phone checking, yet, students did not report being more distracted. These results suggest that in-class phone bans represent a low-cost, effective policy to modestly improve academic outcomes, especially for vulnerable student groups, while enhancing student receptivity to digital policy interventions.
That is from a recent paper by Alp Sungu, Pradeep Kumar Choudhury, and Andreas Bjerre-Nielsen. Note with grades there is “an average increase of 0.086 standard deviations.” I have no problem with these policies, but it mystifies me why anyone would put them in their top five hundred priorities, or is that five thousand? Here is my earlier post on Norwegian smart phone bans, with comparable results.
My Hope Axis podcast with Anna Gát
Here is the YouTube, here is transcript access, here is their episode summary:
The brilliant @tylercowen joins @TheAnnaGat for a lively, wide-ranging conversation exploring hope from the perspective of insiders and outsiders, the obsessed and the competitive, immigrants and hard workers. They talk about talent and luck, what makes America unique, whether the dream of Internet Utopia has ended, and how Gen-Z might rebel. Along the way: Jack Nicholson, John Stuart Mill, road trips through Eastern Europe, the Enlightenment of AI, and why courage shapes the future.
Excerpt:
Tyler Cowen: But the top players I’ve met, like Anand or Magnus Carlsen or Kasparov, they truly hate losing with every bone in their body. They do not approach it philosophically. They can become very miserable as a result. And that’s very far from my attitudes. It shaped my life in a significant way.
Anna Gát: I was so surprised. I was like, what? But actually, what? In Maggie Smith-high RP—what? This never occurred to me that losing can be approached philosophically.
Tyler Cowen: And I think always keeping my equanimity has been good for me, getting these compound returns over long periods of time. But if you’re doing a thing like chess or math or sports that really favors the young, you don’t have all those decades of compound returns. You’ve got to motivate yourself to the maximum extent right now. And then hating losing is super useful. But that’s just—those are not the things I’ve done. The people who hate losing should do things that are youth-weighted, and the people who have equanimity should do things that are maturity and age-weighted with compounding returns.
Excellent discussion, lots of fresh material. Here is the Hope Axis podcast more generally. Here is Anna’s Interintellect project, worthy of media attention. Most of all it is intellectual discourse, but it also seems to be the most successful “dating service” I am aware of.
How to think about AI progress
The Zvi has a good survey post on what is going on with the actual evidence. I have a more general point to make, which I am drawing from my background in Austrian capital theory.
There are easy projects, and there are hard projects. You might also say short-term vs. long-term investments.
The easier, shorter-term projects get done first. For instance, the best LLMs now have near-perfect answers for a wide range of queries. Those answers will not be getting much better, though they may be integrated into different services in higher productivity ways.
Those improvements will yield an ongoing stream of benefits, but you will not see much incremental progress in the underlying models themselves. Ten years from now, the word “strawberry” still will have three r’s, and the LLMs still will tell us that. There are other questions, such as “what is the meaning of life?” where the AI answers also will not get much better. I do not mean that statement as AI pessimism, rather the answers can only get so good because the question is not ideally specified in the first place.
Then there are the very difficult concrete problems, such as in the biosciences or with math olympiad problems, and so on. Progress in these areas seems quite steady and I would call it impressive. But it will take quite a few years before that progress is turned into improvements in daily life. Again, that does not have to be AI pessimism. Just look at how we run our clinical trials, or how long the FDA approval process takes for new drugs, or how many people are reluctant to accept beneficial vaccines. I predict that AI will not speed up those processes nearly as much as it ideally might.
So the AI world before us is rather rapidly being bifurcated into two sectors:
a) progress already is extreme, and is hard to improve upon, and
b) progress is ongoing, but will take a long time to be visible to actual users and consumers
And so people will complain that AI progress is failing us, but mostly they will be wrong. They will be the victim of cognitive error and biases. The reality is that progress is continuing apace, but it swallows up and renders ordinary some of its more visible successes. What is left behind for future progress can be pretty slow.
AI-led job interviews
We study the impact of replacing human recruiters with AI voice agents to conduct job interviews. Partnering with a recruitment firm, we conducted a natural field experiment in which 70,000 applicants were randomly assigned to be interviewed by human recruiters, AI voice agents, or given a choice between the two. In all three conditions, human recruiters evaluated interviews and made hiring decisions based on applicants’ performance in the interview and a standardized test. Contrary to the forecasts of professional recruiters, we find that AI-led interviews increase job offers by 12%, job starts by 18%, and 30-day retention by 17% among all applicants. Applicants accept job offers with a similar likelihood and rate interview, as well as recruiter quality, similarly in a customer experience survey. When offered the choice, 78% of applicants choose the AI recruiter, and we find evidence that applicants with lower test scores are more likely to choose AI. Analyzing interview transcripts reveals that AI-led interviews elicit more hiring-relevant information from applicants compared to human-led interviews. Recruiters score the interview performance of AI-interviewed applicants higher, but place greater weight on standardized tests in their hiring decisions. Overall, we provide evidence that AI can match human recruiters in conducting job interviews while preserving applicants’ satisfaction and firm operations.
That is from a new paper by Brian Jabarian and Luca Henkel.
The “Marvel Universe” of faith
In a recent video posted to the AI Bible’s Youtube channel, buildings crumble and terrified-looking people claw their way through the rubble. Horns blare, and an angel appears floating above the chaos. Then come monsters, including a seven-headed dragon that looks like something out of a Dungeons and Dragons rulebook.
The visuals in this eight-minute video, which depicts a section of the Book of Revelation, are entirely generated by artificial intelligence tools. At times it feels like a high-budget Hollywood movie, at times more like a scene from a video game, and at times like fantasy art. Despite the somewhat muddled visual styles, viewers seem to like what they see – it has racked up over 750,000 views in the two months since it was posted.
The viewers are mostly under 30, and skew male.
Here is the full story, via Ari Armstrong.
Pathbreaking paper on AI simulations of human behavior
By Benjamin Manning and John Horton:
Useful social science theories predict behavior across settings. However, applying a theory to make predictions in new settings is challenging: rarely can it be done without ad hoc modifications to account for setting-specific factors. We argue that AI agents put in simulations of those novel settings offer an alternative for applying theory, requiring minimal or no modifications. We present an approach for building such “general” agents that use theory-grounded natural language instructions, existing empirical data, and knowledge acquired by the underlying AI during training. To demonstrate the approach in settings where no data from that data-generating process exists—as is often the case in applied prediction problems—we design a highly heterogeneous population of 883,320 novel games. AI agents are constructed using human data from a small set of conceptually related, but structurally distinct “seed” games. In preregistered experiments, on average, agents predict human play better than (i) game-theoretic equilibria and (ii) out-of-the-box agents in a random sample of 1,500 games from the population. For a small set of separate novel games, these simulations predict responses from a new sample of human subjects better even than the most plausibly relevant published human data.
Here is a good Twitter thread. A broader AI lesson here is that you often have to put in a lot of work to get the best from your LLMs. And these results ought to have implications for the methods of psychology and some of the other social sciences as well.
Where are the trillion dollar biotech companies?
In today’s market, even companies with multiple approved drugs can trade below their cash balances. Given this, it is truly perplexing to see AI-biotechs raise mega-rounds at the preclinical stage – Xaira with a billion-dollar seed, Isomorphic with $600M, EvolutionaryScale with $142M, and InceptiveBio with $100M, to name a few. The scale and stage of these rounds reflect some investors’ belief that AI-biology pairing can bend the drug discovery economics I described before.
To me, the question of whether AI will be helpful in drug discovery is not as interesting as the question of whether AI can turn a 2-billion-dollar drug development into a 200-million-dollar drug development, or whether 10 years to approve a drug can become 5 years to approve a drug. AI will be used to assist drug discovery in the same way software has been used for decades, and, given enough time, we know it will change everything [4]. But is “enough time” 3 years or half a century?
One number that is worth appreciating is that 80% of all costs associated with bringing a drug to market come from clinical-stage work. That is, if we ever get to molecules designed and preclinically validated in under 1 year, we’ll be impacting only a small fraction of what makes drug discovery hard. This productivity gain cap is especially striking given that the majority of the data we can use to train models today is still preclinical, and, in most cases, even pre-animal. A perfect model predictive of in vitro tox saves you time on running in vitro tox (which is less than a few weeks anyway!), doesn’t bridge the in vitro to animal translation gap, and especially does not affect the dreaded animal-to-human jump. As such, perfecting predictive validity for preclinical work is the current best-case scenario for the industry. Though we don’t have a sufficient amount and types of data to solve even that.
Here is the full and very interesting essay, from the excellent Lada Nuzhna.
I podcast with Jacob Watson-Howland
He is a young and very smart British photographer. He sent me this:
Links to the episode:
Youtube: https://youtu.be/4nMg0Qg7KRI
Spotify: https://open.spotify.com/episode/2wDyCaXhGN5ruN1SNkcaBt?si=3c054eb0396f4787
Apple: https://podcasts.apple.com/gb/podcast/watson-howland/id1813625992?i=1000724051310
Jacob’s episode summary is at the first link.
Those new service sector jobs
Poets with Mercor, for $150 an hour. And that is just a start.
Discrimination on #EconTwitter
This paper documents discrimination in the formation of professional networks among academic economists. We created 80 bot accounts that claim to be PhD students differing in three characteristics: gender (male or female), race (Black or White), and university affiliation (top- or lower-ranked). The bots randomly followed 6,920 users in the #EconTwitter community. Follow-back rates were 12 percent higher for White students compared to Black students, 21 percent higher for students from top-ranked universities compared to those from lower-ranked institutions, and 25 percent higher for female compared to male students. Notably, the racial gap persists even among students from top-ranked institutions.
That is from a new AERInsights paper by Nicolás Ajzenman, Bruno Ferman, and Pedro C. Sant’Anna. Here is a useful picture from the paper. Being at a top school, or at least pretending to be, is what really matters?
How Retrainable are AI-Exposed Workers?
We document the extent to which workers in AI-exposed occupations can successfully retrain for AI-intensive work. We assemble a new workforce development dataset spanning over 1.6 million job training participation spells from all US Workforce Investment and Opportunity Act programs from 2012–2023 linked with occupational measures of AI exposure. Using earnings records observed before and after training, we compare high AI exposure trainees to a matched sample of similar workers who only received job search assistance. We find that AI-exposed workers have high earnings returns from training that are only 25% lower than the returns for low AI exposure workers. However, training participants who target AI-intensive occupations face a penalty for doing so, with 29% lower returns than AI-exposed workers pursuing more general training. We estimate that between 25% to 40% of occupations are “AI retrainable” as measured by its workers receiving higher pay for moving to more AI-intensive occupations—a large magnitude given the relatively low-income sample of displaced workers. Positive earnings returns in all groups are driven by the most recent years when labor markets were tightest, suggesting training programs may have stronger signal value when firms reach deeper into the skill market.
That is from a new NBER working paper by
Dean Ball on state-level AI laws
He is now out of government and has resumed writing his Substack. Here is one excerpt from his latest:
Several states have banned (see also “regulated,” “put guardrails on” for the polite phraseology) the use of AI for mental health services. Nevada, for example, passed a law (AB 406) that bans schools from “[using] artificial intelligence to perform the functions and duties of a school counselor, school psychologist, or school social worker,” though it indicates that such human employees are free to use AI in the performance of their work provided that they comply with school policies for the use of AI. Some school districts, no doubt, will end up making policies that effectively ban any AI use at all by those employees. If the law stopped here, I’d be fine with it; not supportive, not hopeful about the likely outcomes, but fine nonetheless.
But the Nevada law, and a similar law passed in Illinois, goes further than that. They also impose regulations on AI developers, stating that it is illegal for them to explicitly or implicitly claim of their models that (quoting from the Nevada law):
(a) The artificial intelligence system is capable of providing professional mental or behavioral health care;
(b) A user of the artificial intelligence system may interact with any feature of the artificial intelligence system which simulates human conversation in order to obtain professional mental or behavioral health care; or
(c) The artificial intelligence system, or any component, feature, avatar or embodiment of the artificial intelligence system is a provider of mental or behavioral health care, a therapist, a clinical therapist, a counselor, a psychiatrist, a doctor or any other term commonly used to refer to a provider of professional mental health or behavioral health care.
First there is the fact that the law uses an extremely broad definition of AI that covers a huge swath of modern software. This means that it may become trickier to market older machine learning-based systems that have been used in the provision of mental healthcare, for instance in the detection psychological stress, dementia, intoxication, epilepsy, intellectual disability, or substance abuse (all conditions explicitly included in Nevada’s statutory definition of mental health).
But there is something deeper here, too. Nevada AB 406, and its similar companion in Illinois, deal with AI in mental healthcare by simply pretending it does not exist. “Sure, AI may be a useful tool for organizing information,” these legislators seem to be saying, “but only a human could ever do mental healthcare.”
And then there are hundreds of thousands, if not millions, of Americans who use chatbots for something that resembles mental healthcare every day. Should those people be using language models in this way? If they cannot afford a therapist, is it better that they talk to a low-cost chatbot, or no one at all? Up to what point of mental distress? What should or could the developers of language models do to ensure that their products do the right thing in mental health-related contexts? What is the right thing to do?
The State of Nevada would prefer not to think about such issues. Instead, they want to deny that they are issues in the first place and instead insist that school employees and occupationally licensed human professionals are the only parties capable of providing mental healthcare services (I wonder what interest groups drove the passage of this law?).
AI-engaged economics papers are growing rapidly
…share of economics papers that is ABOUT or USES AI increased 10X to 5% in 5 years and growth is basically vertical.
Be there or be square!
Here is the tweet, here is the underlying paper by Eamon Duede, et.al. Other science are considered as well, I do not need to tell you the results, they consider philosophy too.
Profile of Joe Liemandt and Alpha School
The one thing Liemandt will talk about for hours on end is Alpha School: the teacherless, homeworkless, K-12 private school in Austin, Texas, where students have been testing in the top 0.1% nationally by self-directing coursework with AI tutoring apps for two hours a day. Alpha students are incentivized to complete coursework to “mastery-level” (i.e., scoring over 90%) in only two hours via a mix of various material and immaterial rewards, including the right to spend the other four hours of the school day in “workshops,” learning things like how to run an Airbnb or food truck, manage a brokerage account or Broadway production, or build a business or drone.
Since the explosive debut of Generative AI in 2022, Liemandt has taken $1 billion out of Trilogy/ESW in order to fund and incubate proprietary AI software products at Alpha School, where he has also served quietly as “product guy,” dean of parents, and principal. After collecting a three-year data stream in these roles, while also working in a nearby stealth lab, Liemandt believes he now has “the single best product I’ve ever built, in four decades, by far.” The product is called Timeback, and its purpose, in essence, is to scale Alpha School’s concepts and results—learn 2x in 2 hours, test in the 99th percentile, and then give students the rest of their childhood back—to a billion kids.
Here is the full story by Jeremy Stern.
The AI polity that is Albania?
While the rest of Europe bickers over the safety and scope of artificial intelligence, Albania is tapping it to accelerate its EU accession.
It’s even mulling an AI-run ministry.
Prime Minister Edi Rama mentioned AI last month as a tool to stamp out corruption and increase transparency, saying the technology could soon become the most efficient member of the Albanian government.
“One day, we might even have a ministry run entirely by AI,” Rama said at a July press conference while discussing digitalization. “That way, there would be no nepotism or conflicts of interest,” he argued.
Local developers could even work toward creating an AI model to elect as minister, which could lead the country to “be the first to have an entire government with AI ministers and a prime minister,” Rama added.
While no formal steps have been taken and Rama’s job is not yet officially up for grabs, the prime minister said the idea should be seriously considered…
AI is already being used in the administration to manage the thorny matter of public procurement, an area the EU has asked the government to shore up, as well as to analyze tax and customs transactions in real time, identifying irregularities.
Here is the whole Politico story, via Holger.