Damned lies or statistics? Use your brain
Mark Twain single-handedly helped discredit an entire mathematical discipline in the popular mind by saying: 'There are three kinds of lies: lies, damned lies and statistics.' There is little doubt that data is frequently manipulated by politicians, lobbyists and commercial interests for their own ends; and the media often accepts their claims uncritically.
Ironically, it was precisely during Twain's lifetime that modern statistics emerged, in the west at least, as a revolutionary science that laid the foundations for empirical observations in the hard sciences and transformed social philosophy into modern social sciences.
The new science of uncertainty teaches practitioners to correct for errors, eliminate biases, detect fraud and apply mathematical rigour to observational data. It is the opposite of what Twain had in mind. Ken Alder, a US historian, argues it helped create the modern notion of professional scientists, as opposed to gentleman-savants.
It is interesting in this context that a heated debate has broken out in the exciting new field of social neuroscience that turns on its statistical validity. These days, it is impossible to avoid reading in the news the latest findings from the field. By using a type of brain scan called functional magnetic resonance imaging (fMRI), its practitioners claim to capture images of brain activities that correlate with various emotions, conscious states, and mental activities, from memorising a poem to solving an equation.
Researchers have claimed to find mental differences between those who vote for Democrats and Republicans in US elections; and between men and women in experiencing jealousy, handling social rejection, and reacting to grief and anger. Areas in the brain associated with imagining will light up when we tell a lie; others when we make moral decisions. Social neuroscience promises much and seems, on the surface, to deliver much. Alas, MIT graduate student Edward Vul and his supervisors have recently dropped a bombshell, raising questions about the goals and methods of the new science; how the public can learn or be misled by its claims; and the use and abuse of statistics in science in general.
The controversy hinges on what those brain correlations really mean, if they exist. The resolution of digital images is defined by pixels; that of fMRI is by voxels, which cover a tiny volume of brain tissue. The brain scans 'light up' by the amount of haemoglobin in the blood, whose flows indicate mental activities. They are, if social neuroscience is right, objective data of ephemeral subjective feelings or mental states.
Mr Vul and his associates argue there is a statistical limit to accuracies that the scientists can observe in correlating the scans to brain activities, yet many studies make claims that are statistically impossible to such a high degree of accuracy. The Vul camp argues the high correlations are likely to come from bias being introduced by selecting data from a first statistical test that already presumes a correlation. Those who rebut Mr Vul say they are well aware of the pitfalls involving 'the second non-independent statistical test' and have avoided them.
In any scientistic field, some researchers are careful while others are sloppy. So Mr Vul is probably right about average and below-average scientists. But, unsurprisingly, some of the more established neuroscientists have rebutted Mr Vul, whom they accuse, probably rightly, of being unfair to those who have been scrupulous. Still, the debate is healthy and should force researchers to be more careful about their claims, and will help general readers be more sceptical of reports of other fMRI discoveries.
As a parent, I see promise in fMRI as an educational tool. Having observed how my children tune in and out during study, a brain scan that can determine their moods and behaviour would help decide when to teach what subjects, or avoid them. It would save so much time and so many headaches.
Alex Lo is a senior writer at the Post