The ritual dance of the annual company survey
Human beings have, on average, fewer than two legs. This may seem like a startling thing to read in a newspaper, but it says more about the fallibility of statistical analysis than it does about the distribution of limbs on human beings. Because there is at least one person who actually has fewer than two legs, and no people with more than two, the average number is less than two. Not much less, but less.
This is one of the great things about statistics - accurate and interesting remarks that are completely meaningless.
Once every year or so, our bank illustrates this point by conducting an employee survey. A consultant is hired to find out things about morale and how pleased everyone is to be working there. We answer multiple-choice questions about things like whether 'the bank respects me as an individual' or 'the management of the bank understands my concerns as an employee'.
Our answers are given in shades of agreement between 'strongly disagree' and 'strongly agree', including the enticing option of 'no opinion'. At the end of the process, the consultant will present results that show that the staff are generally happy and think that management are doing a good job.
There are three reasons that the results always come out this way. First, people who can't stand their jobs and think management are a bunch of monkeys generally don't bother filling in surveys. Second, despite assurances to the contrary, there is always a lingering suspicion that our answers might not be completely anonymous, so we tend to give favourable responses. And the final and most important reason is statistics itself.
Statistics is a very complicated field, one which most people don't realise that they don't understand. The ways in which the results of a survey can be misinterpreted or even misrepresented through statistical analysis are endless.
For example, if in last year's survey only 12 per cent of our staff said they had confidence in management, and this year that number went up to 17 per cent, it could be interpreted as a five percentage point rise in confidence. Doesn't sound much good, a 5 percentage point increase. But it also could, and probably would, also be described as a 42 per cent increase, because five over 12 is 0.42. And that sounds pretty good.
Another very popular statistical misdirection is subgroup analysis. If the overall results don't show the trend you are looking for, you search for a smaller group within these results that does. Then you just talk about that group.
So if on the whole 60 per cent of staff say that they disagree or strongly disagree with management's response to the financial crisis, there might be a smaller group who were more positive. If you find that staff who joined less than two years ago were the most approving of management's bumbling, then you say: 'Recent joiners rate management more highly than average.' Meaningless, but accurate.
Another very popular error is to mistake correlation for causation and make announcements like 'Bank's green initiatives increase productivity'. A statement like this, based simply on a statistical study, erroneously suggests that the introduction of recycling bins caused people to work harder, when all you can say really is that both things happened at the same time.
Another common mistake is to talk about average when you mean mean, or mean when you mean median, or median when you mean average.
I could go on, but I have to admit that there is some actual value to employee surveys. And this is because a survey can influence the behaviour it is intended to measure. This phenomenon is known as the Hawthorne effect, after an experiment conducted in the 1920s at the Hawthorne Works manufacturing plant in Chicago.
The goal of the experiment was to determine whether productivity of the plant improved if the lights were turned up. The results were unsurprising. When the lights were turned up productivity increased.
However, to everyone's surprise, productivity also increased when the lights were turned down. After much head-scratching, it was determined that what was actually happening was that productivity increased while the survey was taking place.
It did not matter what was being tested. Because the employees knew they were being assessed, they worked harder. The same results that were achieved when the lights were turned up would have also been achieved had the researchers pretended to turn the lights up. An industrial placebo effect.
So although the results of the bank's survey will be inaccurate, misleading and misinterpreted; and although there will be much discussion afterwards about how to improve staff morale even though the answer can easily be summarised in a single word - bonus - it's not all pointless. Just being asked how we feel will make us all feel a little better.
