Checking the AI audit of Saez and Zucman’s research.
We checked the anonymous AI audit of Emmanuel Saez and Gabriel Zucman’s research that Marginal Revolution featured on 28 Oct: all 147 problems it calls serious enough to change a headline number, against the papers, their sources and the authors’ own data. The audit’s facts hold up: for 131 of the 147, every fact we could check was right, and none had its core facts wrong. 21 of the 147 change a headline number; 98 depend on a disputed choice of method; 20 change no number; 8 we could not check on that question. A second pass graded the 99 flags we first counted as depending on a disputed choice of method: 11 standard, 70 defensible, 11 questionable, 6 we could not grade, and 1 contested. It found one, No. 54, that does change a headline number, which is why the counts are now 21 and 98. Saez and Zucman were sent our findings on 29 Oct and did not reply by 6 Nov.
Here is the audit of the audit, open code and data, kudos to the authors. Here was the original AI audit. Here is the October 30 revision of that audit.