Pneumatically numerate

Nate Silver's fame took off during the 2012 United States presidential elections when he predicted the outcome of voting in all 50 states. It was a stellar record given that big elections involve a lot of moving parts, such as voter turnout and last-minute political surprises, and because the polling underpinning such analysis is notoriously unreliable, with data routinely showing radically different results.

Silver comes from a political family. His mother, Sally, was a community activist, and his father was chair of the political science department at Michigan State University. But he is drawn to politics because the analysis is so woolly and partisan. Politicians and their handlers are in the business of "reality distortion", as Silver puts it, and too often journalists get co-opted into that process. It is the perfect arena for a clear-headed statistician who has no other agenda but accuracy, to make an impact.

"Pundits can be completely wrong forecast after forecast, spouting BS and still get invited back to the talk shows every week," said Silver while in Hong Kong for the CLSA Investor Forum. "My approach calls for more accountability. It's more scientific … I wish journalism were more enamoured of the scientific method and not what are very odd constructs about journalistic objectivity."

Silver wrote political analysis for The New York Times and, unsurprisingly, his disdain for traditional journalism annoyed some. When he ended his relationship with the paper in July, the public editor wrote in a column that "a number of traditional and well-respected Times journalists disliked his work". She added: "I don't think Nate Silver ever really fit into The Times culture and I think he was aware of that. He was, in a word, disruptive."

Silver for his part says some of his less numerate peers at The Times were quietly undermining, perhaps because they resented his ability to draw a wide readership and capture so many column inches of the paper.

"Journalists have strong views and opinions on the world. But also are by and large not Type A personalities. They are very good observers, so that can lead to more of a passive-aggressive environment," said Silver.

But Silver's disruptive impact at the paper is part of a pattern. He uses his skills as a data analyst to exploit markets shot through with bad information. He was part of the Moneyball revolution captured in the 2003 book written by Michael Lewis of that name, which told the story of Oakland Athletics' manager Billy Beane.

Beane built one of the best teams in baseball despite having the third worst budget in the league. His insight was to use statistics to track player performance. An objective, hard-headed analysis of data often led Beane to make decisions that were at odds with talent scouts, who used gut instinct. But Beane's methods were effective and most teams now follow his methods.

"Billy Beane was kind of running circles around teams and spending half their budget. It was that big of a mistake that teams had to catch on," said Silver.

Silver at the time was a writer for Baseball Prospectus, which does statistical analysis of baseball data, and which served as scaffolding and source code for the theories applied so successfully by Beane.

Fundamentally Silver, Beane and others involved in Moneyball saw how businesses such as baseball that involve a lot of money, may nonetheless be driven by individuals making subjective calls, with little accountability for their mistakes. The Moneyballers see their advantage. They aim to overtake such people with objective, data-driven analysis. They want their performance measured and compared with that of their subjective peers, and they invite accountability.

"Most predications are poor. Most people are poor at understanding the future. Most people do not understand risks that might not have cropped up in the past and then appear," said Silver. "A lot of this comes down to people overrating their intelligence. When you look at the track record of people making predictions in a way that you could measure, it's a poor track record."

While at Baseball Prospectus Silver got involved in professional poker, mostly active in 2004-08. It was a boom time for poker. The game had become fashionable, drawing in a lot of mediocre players, and people like Silver used their number skills to beat them.

In 2006 the US Congress passed a law that effectively outlawed online poker, fuelling his interest in politics. He started writing on Daily Kos - a blog for politics junkies - under the pseudonym "Poblano", winning followers for his advanced techniques.

Silver began blogging about the 2008 presidential election, when he predicted results in 49 of 50 states, launching him as a political analyst.

"The coverage in politics was still light years behind in terms of empirical method and accountability in things people were saying. So that made me think there was a market opportunity," said Silver of his debut as a political writer.

In March 2008, Silver started his political blog FiveThirtyEight, launching with a 6,000-word screed in which Silver made his "statement of methodology". In 2010, The New York Times made a deal to license Silver's content giving him a platform, and fame, and in July this year he sold FiveThirtyEight to the sports network ESPN, where he will continue to write about politics and, of course, sports. He is building a team of journalists who will follow Silver's template of provocative, data-led writing.

It's telling that Silver through his career has avoided more data-efficient arenas such as finance or sports gambling. Such activities are too competitive for Silver to get a meaningful advantage, and he dropped out of professional poker after the players became too good.

But journalism and politics are rife with lazy analysis, and political journalism especially so. As such, for the next phase of his life, Silver is going for the easy advantage, which is to be a media entrepreneur.