Oomph v. Statistical Significance

For those after the full debate, see:

  1. McCloskey and Ziliak, "The Standard Error of Regressions," Journal of Economic Literature 1996.
  2. Ziliak and McCloskey, "Size Matters: The Standard Error of Regressions in the American Economic Review," Journal of Socio-Economics 2004.
  3. Hoover and Siegler, "Sound and Fury: McCloskey and Significance Testing in Economics," Journal of Economic Methodology, 2008.
  4. McCloskey and Ziliak, "Signifying Nothing: Reply to Hoover and Siegler."

McCloskey has been making this point for some time, and a longer list of papers is available here, including several shorter (yet unsurprisingly persuasive) pieces.

Comments

You know, if you have a high liquidity preference, that $0.02 of your would be worth more like $0.06 in today's market

Don't you think that the last response from McCloskey and Ziliak very emotional rather than rational?

And what is their view and foundation on time series? Why they take a time series as one observation?

Bruce is right in his assessment of many people taking the test as a "deus ex SAS," as I've often said. What they also don't realize is that you can exhaust a dataset. That is, the power of the significance test declines with each run of the programs. Roughly, this means that if you start with 10 degrees of freedom for one's first analysis, say a mean, then when you run a regression you really only have about 9 degrees remaining.

And, as an aside, the number of degrees of freedom for the significance test depends not only on the number of observations, but on the survey design as well. It may be that this additional complication is what drives the retreat from significance testing.

It appears that much work in economics has at bottom a desire for the researcher to "prove" some preconceived notion.

This leads to all sorts of mischief. The most important distortions come from the selective use of data. Applying questionable statistical techniques is just the icing on the cake.

Perhaps things are getting better (if they really are) because the feedback mechanism of critical evaluation is working better. In the past debates would stretch over years as one published paper was countered by another. Now papers are put online and available much more quickly. In addition the audience is larger and the rise of the blogs give many others to comment as well.

This discussion thread is a perfect example of the evolution.

There is also (at least in the US) much more push back to the prevailing macro-economic ideas, especially those having to do with free-market ideology and marginal tax rates. The increasing amount of criticism these schools of thought are receiving may have altered the behavior of those publishing in this field. Either they may be publishing less, or they may have trimmed their sails.

Eric:
Yes, just consult any of the web sites that track blog traffic. You'll find that those critical of the status quo (which generally means left leaning) have been growing rapidly over the past four or five years, while those which take the opposite point of view are doing less well.

Then there are all the conservatives who have started to speak out against their own party and some of its economic assumptions.

OK, Eric here goes (my definitions, your mileage may vary).
The "conservative" view has historically been connected to support of the status quo. Since the status quo usually favors the privileged the effect is that conservatives are the party of the rich. The specifics of what policies this entails changes over time and place.

The "liberal" view favors the interests of the common man. The few times the "liberals" have gotten into power have usually been as a result of a revolution, and their control has usually been short-lived. Class solidarity from outside forces usually works towards restoration of the status quo ante.

The "libertarian" view is of fairly recent origin and mostly exists in the US. It has a focus on personal liberty which implies less government.

At this moment in time, in the US, I see the following:
Social conservatives - concerned with issues of morality, usually as determined within a religious framework.
Foreign policy conservatives - the so-called neo-cons. They are interested in promoting US power abroad, their motives are in dispute.

Social liberals - concerned with issues of personal liberty, which sometimes overlaps the libertarian position. They also think that government should play a role in regulating business.

Foreign policy liberals - A, mostly theoretical category. Those who object to our current foreign policy failures are mostly concerned with poor implementation, not misplaced goals. True peaceniks are few and far between.

Libertarians - The focus is on freedom of action especially as it concerns property and risk. They claim they want smaller government, but require a strong legal/police structure to enforce property rights. Since this model has never existed in a functioning government they are always in opposition to the existing administration in some way or other.

This is a libertarian site. Judging by some of your comments I assume you agree with most of the precepts of the owners.

I used to have a quip to the effect that conservatives want to tell people what to do while liberals want to tell institutions what to do. Or, to put it another way, conservatives don't trust people and liberals don't trust institutions.

Libertarians don't fit into the left-right model. In the past 40 years they have been allied with the conservatives because of the stated claim that the conservatives were in favor of "smaller government". Now that they have seen that this hasn't been true in practice I think they are somewhat adrift.

As for political parties, I don't see too much difference between them. The GOP will favor less regulation and lower taxes while the Dems will favor more social spending and more regulation of business. This is just a matter of degree, despite all the noise they generate. Both parties believe in the applicability of capitalism to all circumstances and they also believe in a strong military and police structure.

Given the institutional inertia present in the government there isn't much change in overall spending patterns when the majority party changes. If you look at the historic CBO data you'll see what I mean:
http://cbo.gov/budget/historical.pdf

Dear Barkley, and some others who have complained about the sharp tone in Ziliak's and my reply to Hoover and Siegler,

I plead innocent on grounds of self-defense. The "personalistic" tone was introduced by Hoover and Siegler. Their use of a French masculine gender to describe me is the least of their personalistic assaults on our integrity, intelligence, and character.

But surely, dear, you are not going to decide a scientific question by who is nicest, are you? That would be a big scientific mistake, wouldn't it? (In any case your vote on that score would have to go to sweet little me!) Perhaps you don't realize how important the issue is. 90 percent of the empirical papers in economics depend on it. As our epidemiologist colleague notes (and as many others in many other fields note, as we record in our book The Cult of Statistical Significance), most economists think Fisherian "significance" means "important," and that "insignificant" means "droppable, unimportant." Neither is true---and remarkably Hoover and Siegler say they agree. Understanding the point would mean that all econometric testing-without-loss-functions would have to stop, today. Big issue. Hot tone.

Regards,

Deirdre

Dear Barkley, and some others who have complained about the sharp tone in Ziliak's and my reply to Hoover and Siegler,

I plead innocent on grounds of self-defense. The "personalistic" tone was introduced by Hoover and Siegler. Their use of a French masculine gender to describe me is the least of their personalistic assaults on our integrity, intelligence, and character.

But surely, dear, you are not going to decide a scientific question by who is nicest, are you? That would be a big scientific mistake, wouldn't it? (In any case your vote on that score would have to go to sweet little me!) Perhaps you don't realize how important the issue is. 90 percent of the empirical papers in economics depend on it. As our epidemiologist colleague notes (and as many others in many other fields note, as we record in our book The Cult of Statistical Significance), most economists think Fisherian "significance" means "important," and that "insignificant" means "droppable, unimportant." Neither is true---and remarkably Hoover and Siegler say they agree. Understanding the point would mean that all econometric testing-without-loss-functions would have to stop, today. Big issue. Hot tone.

Regards,

Deirdre

i like it

Comments for this post are closed