Critics come down hard on fatally flawed happiness study
A high-profile 2013 study that concluded that different kinds of happiness are associated with dramatically different patterns of gene activity is fatally flawed, according to an analysis which tore into its target with language rarely seen in science journals.
The paper, published like the first in Proceedings of the National Academy of Sciences, faults the research for "dubious analyses" and "erroneous methodology" and says it "conjured nonexistent effects out of thin air".
In the 2013 study, researchers had adults answer 14 questions meant to sort them into two groups: interested in hedonic well-being (fun and selfish pleasure) or eudaimonic well-being (leading a meaningful life).
The two groups, researchers led by psychologist Barbara Frederickson of the University of North Carolina reported, had different patterns of activity in 53 genes. Hedonists had DNA activity akin to people suffering from chronic, illness-inducing stress.
Hedonists, it seemed, were headed for a disease-ridden existence and an early grave.
The claim caught the eye of Nick Brown, a British IT worker who has become a persistent amateur critic of what he sees as shoddy statistical analysis in psychology research.
As he and colleagues scrutinised the 2013 paper, they saw that the authors failed to rule out that people with specific gene-activity patterns in the immune system might be under the weather when tested.
More crucially, Brown said, the questionnaire was flawed. People who scored high on three items meant to identify hedonists scored equally highly on 11 items meant to identify people who seek eudaimonic well-being.
"The two constructs are essentially measuring the same thing," Brown said, so putting people in one category rather than another was "meaningless".
When Brown grouped the items randomly, calling those who scored high on questions one, seven and eight (or any of 8,191 other combinations) one kind of person and those who scored high on others a second type, even with meaningless groupings there were patterns of gene activity seemingly characteristic of each group.
Statistics professor Andrew Gelman of New York's Columbia University called Brown's critique "reasonable".
Flawed statistics have become such a serious problem for journals that many of the world's top titles are adding extra levels of statistical checks in the peer-review process. Deputy executive editor Daniel Salsbury said PNAS was not changing its longstanding practice, which is "to work within our review process to ensure the work is sound in all aspects".