Statistical Methods & Medical Research
Sunday, March 4th, 2007People born under the astrological sign of Leo are 15% more likely to be admitted to hospital with gastric bleeding than those born under the other 11 signs. Sagittarians are 38% more likely than others to land up there because of a broken arm. Those are the conclusions that many medical researchers would be forced to make from a set of data presented by Peter Austin — if they applied the lax statistical methods of their own work.
Austin’s point was to shock medical researchers into using better statistics.
The confusion arises because each result is tested separately to see how likely, in statistical terms, it was to have happened by chance. If that likelihood is below a certain threshold, typically 5%, then the convention is that an effect is “real.” And that is fine if only one hypothesis is being tested. But if, say, 20 are being tested at the same time, then on average one of them will be accepted as provisionally true.
In his study, Dr Austin tested 24 hypotheses, two for each astrological sign. He was looking for instances in which a certain sign “caused” an increased risk of a particular ailment. The hypotheses were less than 5% likely to have come about by chance. However, when he modified his statistical methods to take into account the fact that he was testing 24 hypotheses, not one, the boundary of significance dropped dramatically. At that point, none of the astrological associations remained.
Unfortunately, many researchers looking for risk factors for diseases are not aware that they need to modify their statistics when they test multiple hypotheses. According to work by John Ioannidis, observational health studies that trawl through databases, rather than relying on controlled experiments, are likely to be completely correct only 20% of the time.
“Signs of the times,” The Economist

