Fetishizing p-Values - The Cult of Statistical Significance

en.wikipedia.org William Sealy Gosset

Fetishizing p-Values; Tom Leinster - The n-Category Cafe

Recovering the insight of "Student" Gosset from the over-simplification of Ronald A. Fisher

http://golem.ph.utexas.edu/category/2010/09/fetishizing_pvalues.html

Leinster: Now there’s a whole book making the same point: The Cult of Statistical Significance, by two economists, Stephen T. Ziliak and Deirdre N. McCloskey. You can see their argument in this 15-page paper with the same title. Just because they’re economists doesn’t mean their prose is sober: according to one subheading, ‘Precision is Nice but Oomph is the Bomb’.

http://www.statlit.org/pdf/2009ZiliakMcCloskeyASA.pdf

Leinster: it is true that p-value does not measure the magnitude of the effect (but then, anyone who has taken at least one course in statistics should know that)

I think Jost, Ziliak and McCloskey would completely agree that anyone who has taken at least one course in statistics should know that. They’re pointing out, open-mouthed, that this incredibly basic mistake is being made on a massive scale, including by many people who should know much, much better. Bane used the term ‘collective self-deception’; one might go further and say ‘mass delusion’. It’s a situation where a fundamental mistake has become so ingrained in how science is done that it’s hard to get your paper accepted if you don’t perpetuate that mistake.

That last statement is probably putting it too strongly, but as I understand it, the point they’re making is along those lines.

From the 15-page paper "The Cult of Statistical Significance":

In 1937 Gosset, the inventor and original calculator of “Student’s” t-table told Egon, then editor of Biometrika, that a significant finding is by itself “nearly valueless”:

...obviously the important thing in such is to have a low real error, not to have a "significant" result at a particular station. The latter seems to me to be nearly valueless in itself. . . . Experiments at a single station [that is, tests of statistical significance on a single set of data] are almost valueless. . . . What you really want is a low real error. You want to be able to say not only "We have significant evidence that if farmers in general do this they will make money by it", but also "we have found it so in nineteen cases out of twenty and we are finding out why it doesn't work in the twentieth.” To do that you have to be as sure as possible which is the 20th—your real error must be small...

Gosset to E. S. Pearson 1937, in Pearson 1939, p. 244.

Gosset, we have noted, is unknown to most users of statistics, including economists. Yet he was proposing and using in his own work at Guinness a characteristically economic way of looking at the acquisition of knowledge and the meaning of “error.” The inventor of small sample econometrics focused on the opportunity cost of each observation; he tried to minimize random and non-random errors, real errors.

Edit 11/12/10

A very nice write-up here, along same lines: Significance Tests in Climate Science -- Maarten H. P. Ambaum -- http://www.met.reading.ac.uk/~sws97mha/Publications/jclim_ambaum_rev2.pdf

Consider a scientist who is interested in measuring some effect and who does an experiment in the lab. Now consider the following thought process that the scientist goes through:

1. My measurement stands out from the noise.
2. So my measurement is not likely to be caused by noise.
3. It is therefore unlikely that what I am seeing is noise.
4. The measurement is therefore positive evidence that there is really something happening.
5. This provides evidence for my theory.

This apparently innocuous train of thought contains a serious logical fallacy, and it appears at a spot where not many people notice it.

To the surprise of most, the logical fallacy occurs between step 2 and step 3. Step 2 says that there is a low probability of finding our specific measurement if our system would just produce noise. Step 3 says that there is a low probability that the system just produces noise. These sound the same but they are entirely different.

This can be compactly described using Bayesian statistics...

This comes from a summary of the paper: How significance tests are misused in climate science -- Guest post by Dr Maarten H. P. Ambaum from the Department of Meteorology, University of Reading, U.K. -- http://www.skepticalscience.com/news.php?n=456#

Edit 11/21/10

Significance Tests, frequentist vs. bayesian

Open Mind | tamino.wordpress.com | much-ado-about-something

Dikran Marsupial | November 14, 2010 at 10:49 am

When we perform a test of statistical significance test, what we
would really like to ask is “what is the probability that the
alternative hypothesis is true?”. A frequentist analysis
fundamentally cannot give a direct answer to that question, as
they cannot meaningfully talk of the probability of a hypothesis
being true – it is not a random variable, it is either true or
false and has no “long run frequency”. Instead, the frequentists
gives a rather indirect answer to the question by telling you the
likelihood of the observations assuming the null hypothesis is
true and leaving it up to you to decide what to conclude from
that. A Bayesian on the other hand can answer the question
directly as the Bayesian definition of probability is not based
on long run frequencies but on the state of knowledge of the
truth of a proposition. The problem with frequentist statistical
test is that there is a tendency to interpret the result as if it
were the result of a Bayesian test, which is natural as that is
the form of answer we generally want, but still wrong.

The frequentist approach avoids the “subjectivity” of the
Bayesian approach (although the extent of that “subjectivity” is
debatable), but this is only achieved at the expense of not
answering the question we would most like to ask. It could be
argued that the frequentist approach merely shifts the
subjectivity from the analysis to the interpretation (what should
we conclude based on our p-value). Which form of analysis you
should use depends on whether you find the “subjectivity” of the
Bayesian approach or the “indirectness” of the frequentist
approach most abhorrent! ;o)

At the end of the day, as long as the interpretation is
consistent with the formulation, there is no problem and both
forms of analysis are useful.

This was my favorite comment, the whole sub-thread underneath is interesting. The original Open Mind | tamino.wordpress.com article has good qualifications to Dr Maarten H. P. Ambaum's Skeptical Science post.

Andrew Gelman said something significant, above my head

Image via Wikipedia

A new post by Andrew Gelman, with a quite wordy title

The single most useful piece of advice I can give you, along with a theory as to why it isn't better known, all embedded in some comments on a recent article that appeared in the Journal of the American College of Cardiology

http://www.stat.columbia.edu/~cook/movabletype/archives/2010/03/the_single_most.html

I would summarize, but I am embarrassed to say I understand very little of it.  In a comment, I made an attempt:

Hello Prof. Gelman,

Are you saying "model building" will naturally lead to applying fruitful transformations that will lead to statistics that do more than only prove "a formally statistically significant difference for a trivial effect"?

(By "model building" you mean the scientist taking responsibility for an abstraction that goes beyond statistics, i.e. causality and value judgments about what is more than a trivial effect.)

I am having trouble translating your description into something I can understand, so I would appreciate your help if I made a hash of things with my little summary.

I will add edits to this as I learn more.

I feel Pearl's causality graphs (directed acyclic graphs, to be specific) are the appropriate format to present any model.  If you want to allow the possibility of "no true zeros", then use multiple models, and "collapse" all the points where you wish to use statistics to show the possibility of "no true zeros", maybe even "collapsing" everything into a single point!  The multiple models you then have will now compete in different uses - based on predictive power, accuracy, ability to calculate meaningful error ranges, cost of collecting data, cost of computation, cost of comparison, ability to predict outcomes from interventions, cost of understanding, etc.

"No true zeros" - see Andrew Gelman's Review Essay "Causality and Statistical Learning" Section Heading: "There are (almost) no true zeroes: difficulties with the research program of learning causal structure" http://www.stat.columbia.edu/~cook/movabletype/archives/2010/03/causality_and_s.html

I also have a hard time understanding all this in isolation from a model of a rational being working under a motivating sense of responsibility to make a decision about an action (or remaining inactive).  Especially statistical analysis divorced from utility in making a decision.  Comprehension is nice, but comprehension that cannot play a part in any morally motivated decision is valueless.