The promise of big data: bringing technology and the economy together
With more open source and lower cost computing platforms and software available, small and medium enterprises can enjoy the benefits of big data
The subject of big data has been getting increasing attention over the years. Big data now dominates both market and technology trends. New developments and innovative efforts on big data in different sectors have been reported in the media everyday.
On the Chinese mainland, the most recent news is that the province of Guizhou, with support from the central government, is establishing itself as a pioneer and leader in the big data industry. A big data exchange began operation in 2015, the first of its kind on the Chinese mainland.
In Hong Kong, each month there are workshops and forums on big data organised by the government, industry and academics. Almost all the local universities have set up centres and labs for big data research and applications.
As time goes on, we have gained a better understanding about big data, both its meaning and the benefits. Now we know that the key concept of big data is not only about the volume of data, but more importantly, the type of data.
Different sources of data in larger volumes and greater variety come not only from devices and sensors but also from human activities. It is not only about the product itself but also how the product is used.
The increase in real-time data from sensors, web pages, click streams, and social media allows for improved measurement and monitoring, and consequently seeing things with larger scope, and with more details and greater clarity, in a way we couldn’t see before.
With several orders of magnitude more fine-grained and from different angles, big data makes the invisible visible, enabling us to see things that were impossible to see before, to formulate new questions and discover hidden knowledge.
We also have experienced more about the promise of big data, the benefits and insights from and innovations driven by big data. Big data offers us a better opportunity to capture a better and clearer view of business.
Before, efficiency had been the major concern but now, with big data, cutting cost, boosting revenues, streamlining operations, and optimising decisions have been added to executives’ agenda list.
Data guides operations and strategy, so a company can decide where to put more investment and manpower, and determine how the design of a product can be improved to increase user engagement. Data is also used to guide and make better decisions in business, in comparison with those decisions made based on experience and intuition.
A survey conducted by Lorin Hitt from Wharton School of University of Pennsylvania and Heekyung Kim from MIT found that companies which adopted data-driven decision making achieved productivity that was over 5 per cent higher than what could be explained by other factors.
Big data also drives innovations, enabling us to discover new problems and find new ways to solve a problem. Taking health care as an example, data analytics can reveal unforeseen adverse effects of drugs and enable a better understanding of highly complex health issues such as Alzheimer’s disease and dementia.
“Doctors said data analytics opens a window in a dark room for medical research,” said Danny Chen, a professor from University of Notre Dame, who has been working on computational medicine over the past 10 years, designing algorithms to analyse large numbers of medical images to determine and predict medical conditions of patients.
He reported his findings at ICOT 2015, a conference on health and happiness technology held in December last year at The Hong Kong Polytechnic University.
The payoff comes from insights gleaned from collecting large amounts of various kinds of data and analysing them to uncover hidden patterns, correlations and other insights. Machine learning software can drill down into the data to discover and analyse factors determining the profit and loss for a product, supplier, and their customers.
We can also see into the future, making better predictions and decisions. The result is that “quantitative change becomes qualitative”, as described by Steve Lohr in his best selling book Data-Is m”.
Big data brings technology and the economy together. The benefits of a data-driven economy are obvious enough that we should all embrace the concept. For both technology companies and business firms, the market is just around the corner.
Google and Alibaba both claim that, rather than being in the internet or e-commerce business, they are big data companies. More and more organisations and enterprises in multiple sectors are recognising its enormous value and ramping up the pace of their data strategy.
Today, big data is not just for large corporations. With more open source and lower cost computing platforms and software available, small and medium enterprises and start-ups can also enjoy the benefits of big data in a cost effective and efficient way to capture opportunities and grow in their business.
They can deploy their own data collection using Wi-fi, mobiles, or access data on the open web and reach out to potential customers with specific behaviours and preferences, all with a limited advertising budget.
There are still uncertainties and challenges. We are still suffering from a poverty of good and valuable data, and context of data. We need to continuously develop high-performance big data processing platforms and supporting algorithms and software tools.
In addition to technical challenges, data analytics requires close collaboration between domain experts, data scientists and computer scientists in order to build application domain models, data management and analytic methods, and parallel/distributed processing algorithms and systems.
Jiannong Cao is chair professor of distributed and mobile computing and head of the Department of Computing at The Hong Kong Polytechnic University