How information mining helps impose order on chaos, for a while

Andrew Sheng says the challenge is to use these tools to create rules for unpredictability

PUBLISHED : Saturday, 10 November, 2012, 12:00am
UPDATED : Wednesday, 28 September, 2016, 8:44am


Related topics

More on this story

Travelling to Traverse City, Michigan, in cold, rainy October, serendipity struck in funny ways. In a bookshop, I picked up a 2010 book by the Serbian physicist Vlatko Vedral, Decoding Reality, which sees the world through the lens of quantum information. I read it in Boston, in the midst of Hurricane Sandy as it tore along the US East Coast, demonstrating the frightening power of nature.

At the MIT bookshop I picked up more books on information theory. It blew my mind how modern quantum physics has begun to integrate the world in a new theory of information, almost as a theory of everything.

The ancient library of Alexandria had perhaps 500,000 scrolls. Its existence spread the knowledge of maths from the Arabs to the Greeks and the Romans, and this knowledge was used by the elite to control empires. If the Industrial Revolution was a revolution created by science and technology, the New Industrial Revolution is a revolution of information. The internet made knowledge accessible to all, and smartphones in the hands of millions created network power, so recently demonstrated by social media's impact on the Arab spring revolutions.

Physicists like Vedral are now pushing the envelope by explaining our perception of reality in terms of quantum information theory, since, today we process images of reality in terms of digitised information. All theories are really compressions of large amounts of uncertain and complex information into simple laws or principles, with which we try to predict the future.

Reality is today translated into digitised information of bits and bytes, which computers then process into knowledge using complex maths. A simple picture can be reduced to 30 kilobytes of information, but a top-rated camera is now capable of taking the same picture with 20 megapixels. That has information content of 30 or more megabytes, or 1,000 times more information. It means that if you have a top-range camera, you don't need an optical zoom (telescope) to see a distant object, you simply blow up your picture with a digital zoom.

In 1997, one of the pioneers of the internet, the late MIT professor Michael Dertouzos, conceived the web as an "information marketplace", where people and computers trade information and information services. The information marketplace actually describes all markets, because, today, almost all information is stored and processed by computers, human and machine.

As all bureaucrats know, information is power. Businesses also understand that having good information is not only very powerful, but very valuable. Social media use means we can check up on almost anyone by simply googling for information. It's now possible to measure how a supermarket is doing by using Google Maps to see how many cars are parked outside. And since smartphones like the iPhone can be tracked, you can map your movements on a daily basis. So traffic movements can be mapped and predictions made about jams by simply tracking the movement of phones and tablets. A group of researchers using Google Trends data was able to predict gross domestic product trends by the amount of searches for information using Google.

Big Data, the ability to process and extract value from masses of data that is collected every day, is the flavour of the month. This year, the US government launched a Big Data Research and Development Initiative to extract knowledge and insight from the data the US government collects. Companies are already beginning to look carefully into their own databases to map out consumer behaviour patterns to predict the next product or buying trends.

Information rules are therefore theories of how reality can be understood. Maths and theory is all about how to make unknowns known. But how do we write a rule for unknown unknowns?

The world is going through political turmoil because we are all debating what caused the current crisis and how we should write new laws to prevent a new one. The new laws are getting more complicated, because everyone understands that new laws change the status quo and may hurt vested interests. All crises reveal the need for loss allocation, which is why vested interests reject such reforms that make them pay for the losses.

The trouble is that every new law or rule divides up gains and losses in unpredictable ways. This is the "law" of unintended consequences. Any new law is therefore writing rules on unknown unknowns.

Furthermore, the more complex the law, the more unknowable the unknown, because the market has the ability to game the new rules, changing behaviour in unknown ways.

No wonder the world is getting more complex.

In any situation of order, disorder wins, but in a situation of disorder, order wins. The world is no longer simple, but we need simplicity to manage complexity.

Andrew Sheng is president of the Fung Global Institute