Jim Manzi’s *Uncontrolled*

by  Tyler Cowen in

The subtitle is The Surprising Payoff of Trial-and-Error for Business, Politics, and Society, with an emphasis on RCT.

This is a truly stimulating book, about how methods of controlled experimentation will bring a new wave of business and social innovation.  Here is an Eric Posner review.  Here is a Kirkus review.  There will be more.  Kevin Drum offers good remarks.

Comments

Daniel

Posner's review of the charter school idea isn't realistic, in my opinion. There already exists implicit top-down testing standards from Washington. Clearly, a freer system for parents with the restraint of testing standards would be superior, in free market theory, to a more closed system with the restraint of testing standards. Furthermore, I'm not convinced that what we should be teaching students up through highschool is such a controversy - certainly, there are a few big fights (evolution), but those issues can be settled in compromise while we still require that the core curriculum of language and quantitative skills be taught.

Posner does reasonably point out the larger issues with RFT, however. And the overall idea of federalism still has big problems, such as the undesirable scenario of neighboring states having drastically different drug laws.

Jim Manzi

I think your comment about the relationship between "tight" measurement and "loose" control is spot-on. I discuss the problem you raise at length in the book (and will respond to Posner's comments, made in the context of a review that I found to be thoughtful and positive, as soon as I find the time).

Comments for this post are closed

Comments for this post are closed

Ray Lopez

The book is about the Hawthorne Effect, pretty freaky-o-nomic. I wonder if a huge Keynesian stimulus would be, from a Rational Expectations point of view, credible as a Hawthorne Effect positive way of increasing demand? P. Krugman might think so. Sorry if I'm mixing my metaphors.

RL

http://en.wikipedia.org/wiki/Hawthorne_effect

Research on the demand effect also suggests that people might take on pleasing the experimenter as a goal, at least if it does not conflict with any other motive,[14] but also, improving their performance by improving their skill will be dependent on getting feedback on their performance, and an experiment may give them this for the first time. So you often will not see any Hawthorne effect—only when it turns out that with the attention came either usable feedback or a change in motivation.

Adair (1984) warns of gross factual inaccuracy in most secondary publications on Hawthorne effect and that many studies failed to find it. He argues that it should be viewed as a variant of Orne's (1973) experimental demand effect. So for Adair, the issue is that an experimental effect depends on the participants' interpretation of the situation; this is why manipulation checks are important in social sciences experiments. So he thinks it is not awareness per se, nor special attention per se, but participants' interpretation that must be investigated in order to discover if/how the experimental conditions interact with the participants' goals. This can affect whether participants believe something, if they act on it or do not see it as in their interest, etc.

In a 2011 paper, economists Steven Levitt and John A. List claim that in the illumination experiments the variance in productivity is partly accounted for by other factors such as the weekly cycle of work or the seasonal temperature, and so the original conclusions were overstated.[16] If so, this confirms the analysis of SRG Jones's 1992 article examining the relay experiments.[17][18]

Jim Manzi

You probably want to read the book before you declare what it is "about.".

I discuss the problem of external validity, threats to external validity, and methods for measuring and alleviating these threats at great length in the book, beginning in the Introduction and continuing to the final chapter.

Comments for this post are closed

Comments for this post are closed

Martin Ringo

"This is a truly stimulating book, about how methods of controlled experimentation will bring a new wave of business and social innovation."

I like Mr. Manzi: a most interesting and perspective guy. And his book may be a blessing for much of society. But as I read the Kindle sample, I felt he was preaching to the choir. Heck, I was brought up on the virtues of the Salk field trial to answer the question of "will it work?" (Note it -- the test vaccine -- reduced the rate a bit over 50%, not near 100%. So statistical issues remained. But I digress.

What is it about the book that makes it a worthwhile read for the choir? For instance is there any discussion of things like Regression Discontinuity versus, say, quasi random field experiments?

Comments for this post are closed

Comments for this post are closed