The Impact of AI on Productivity

We don’t yet know the impact that AI will have on productivity but some evidence is starting to come in. Peng et al. (2023) hired programmers on Upwork to write an HTTP server in Javascript; half of the programmers got access to CoPilot (this was before CoPilot was widely available) half did not.

Conditioning on completing the task, the average completion time from the treated group is 71.17 minutes and 160.89 minutes for the control group. This represents a 55.8% reduction in completion time. The p-value for the t-test is 0.0017, and a 95% confidence interval for the improvement is between [21%, 89%]. There are four outliers with time to completion above 300 min. All outliers are in the control group, however our results remain robust if these outliers are dropped. This result suggests that Copilot increases average productivity significantly in our experiment population. We also find that the treated group’s success rate is 7 percentage points higher than the control group, but the estimate is not statistically significant, with a 95% confidence interval of [-0.11, 0.25].

The authors extrapolate wildly:

In 2021, over 4.6 million people in the United States worked in computer and mathematical occupations,1 a Bureau of Labor Statistics category that includes computer programmers, data scientists, and statisticians. These workers earned $464.8 billion or roughly 2% of US GDP. If the results of this study were to be extrapolated to the population level, a 55.8% increase in productivity would imply a significant amount of cost savings in the economy and have a notable impact on GDP growth.

Still, worth thinking about.