A Terminator of American jobs went unnoticed on Trump’s judgment day: artificial intelligence
MogIA algorithm that correctly called the election against army of human experts portends a bigger threat to human employment than any politician
If anyone needs to be convinced that robots have triumphed over humans, look at the US president-elect Donald Trump. Not that I’m suggesting the winner of the 2016 presidential election is a Terminator sent from the future. But the results of the race give us another sober warning about the advent of artificial intelligence, which will impact our lives in a far more profound way than any individual can.
President Trump: How did the polls get U.S. election so wrong?
The news of the week is not just Trump winning the race. It is the surprise of his victory. For weeks and months, mainstream media and pollsters had been telling us this would be an easy victory for Hillary Clinton. On the eve of the election, Reuters put the chance of a Clinton presidency at 90 per cent. When the results came in, the world was stunned. For those rooting for Clinton, the psychological blow was harder to bear because of rosy expectations.
Buried under an avalanche of polling reports, a lone voice correctly predicted two weeks in advance that Trump would win. And it was no fluke. MogIA, an algorithm created by Indian entrepreneur Sanjiv Rai in 2004, has now correctly predicted four out of four US presidential elections.
Defying pundits and pollsters, Donald Trump is the next US president. Here’s how he did it
Its predictions are based on 20 million data points collected from social media platforms such as Google, Facebook, Twitter and YouTube. It uses this data to evaluate public engagement with information relating to each candidate. It found Trump’s engagement rate was 25 per cent higher than Barack Obama’s at his peak in 2008, indicating a high chance of him winning.
Unfortunately, our digital sibyl’s voice did not fit the mainstream media narrative, so received little attention.
The practice of using public polls to measure the pulse of the electorate was always a popular but precarious business. Before this week, the most memorable upset in a US presidential election was in 1948, when Harry Truman defeated Republican candidate Thomas Dewey. All the surveys suggested a clear victory for Dewey. So confident of his success, the Chicago Tribune went to press on its early edition with the headline “Dewey defeats Truman”. The photo of a defiant Truman holding a copy of the Tribune and shouting “Ain’t the way I heard it” has become part of American folklore.
Watch: Obama and Trump’s first White House meeting
The most cited poll in that election was by Gallup. Founded by George Gallup in 1935, the company was famous for its scientific quota sampling system. The Gallup Poll had correctly predicted the winner of the 1936, 1940 and 1944 elections and made its name by knocking the highbrow Literary Digest off its lofty perch.
The Literary Digest was a highly respected magazine of the time and built its reputation on accurate election forecasts. In the 1936 race between Alfred Landon and Franklin Roosevelt, it predicted that Landon would get 57 per cent of the votes against Roosevelt’s 43 per cent. The actual results were 62 per cent for Roosevelt. The sample error of the Literary Digest poll was a whopping 19 per cent – the highest on record.
Even more embarrassing was that the Literary Digest poll was the most extensive and expensive one in history. It had a sample size of 2.4 million respondents. The magazine mailed mock ballots to 10 million people across the States. This was the most ambitious exercise to gauge public opinion. Yet in the end, George Gallup made a correct prediction with a sample size of 50,000. Shortly after, the Literary Digest went out of business.
The Literary Digest’s failure became a textbook case for statisticians. Two basic findings were made: a badly chosen large sample is much worse than a well-chosen small sample; and selection bias and non-response bias are the evil twins pollsters should guard against.
Selection bias occurs when pollsters unconsciously reach out to those of similar social status. Those on the Literary Digest mailing list were mostly its subscribers, names listed on telephone directories or on club membership rolls. They were urban and middle class. The largest segment of the population was never reached. Of the 10 million people it mailed the mock ballots to, only 2.4 million responded. While the figure looked impressive, the actual response rate was low – just under a quarter of people responded. Many chose not to answer because they did not want to make their political views public. Today, when the response rate is low, a survey is considered to have non-response bias.
Eighty years have passed since then. Surely pollsters today are much smarter. How could we explain what happened this week? Some blame it on “shy Trumpers” – people who are reluctant to declare support for the controversial billionaire but secretly admire him. Quartz ran an article before the election headlined, “The rich and educated don’t like to admit they’re voting for Trump”. Yet days later, digital polling firm Morning Consult conducted a survey and declared the “shy Trumpers” a mirage.
Trump’s shock victory a reversal of fortune for Obama
We will read analyses in the days ahead of why the pollsters failed so miserably. A bigger question is whether they are still needed at all. The success of computer algorithms like MogIA, with no emotional attachment or unconscious bias, can do a much better job for a lot less.
This year has been called the year of elections but maybe it is also the year of artificial intelligence. Earlier, AlphaGo demonstrated that machines could beat human masters in the most subtle and creative game, Go. This week, we saw how a cleverly written algorithm outperformed teams of human professionals with ease. Before long, we will see AI overtaking humans in more fields and industries. It is said that those who have been left behind in the American economic recovery are the key to the rise of Trump. Looking ahead, the rise of the robots could leave millions more behind.
Chow Chung-yan is executive editor of the South China Morning Post, overseeing daily print and digital operations
