In Efforts to Derive Meaningful Insights from Big Data, Mathematics Reigns Supreme

The last decade or so has seen the rise of new business tools that are making it possible to get new insights into customers, their habits and their expectations. Among other things, these tools are allowing business leaders to make better products and services or change their internal business process for better outputs.

Top in the list of tools is Big Data analytics and artificial intelligence (A.I.), or as researchers prefer to call it, machine learning. While the insights that Big Data and A.I. bring have been tremendous, fault lines are emerging. A.I. for instance is becoming a challenge to companies and researchers alike when they find information that it knows, but cannot tell how it has acquired it. In a recent article on The New York Times, Cliff Kuang wonders whether A.I. can be taught to explain itself – tell researchers how it acquired the information it has.

Big Data analytics on the other hand have not given as good insights as industry watchers had predicted. Overwhelmed by mountains of data, most corporations never really analyze it enough to bring noteworthy transformation to their businesses. This, in part, explains why these companies are turning to mathematicians. By taking a math approach to big data analytics, organizations are getting new insights that promise to transform the marketplace in coming years. ‘Gathering copious amount of data alone will not deliver any beneficial results for the business. It is what you do with your data and how you process it that leads to higher profits and efficiency. Correct approach to data analytics and handling your Big Data is crucial for any organization in any industry’ says Mike Lammers, Tricension CEO, one of the leading Kansas-based software development companies.

When Dr. Pek Lum, a biologist took a set of data that had been collected 12 years before on breast cancer tumors, she noticed some clues that would later help design more effective treatments for breast cancer. While this data had existed for long with no change made to it for over a decade, the reason Dr. Lum saw clues researchers had missed before was solely because of new view of the data that a new software program had made possible using topology. Topology is a branch in mathematics that creates shapes that researchers can probe and manipulate using compressed relationships that exist in complex data sets.

Doctor Pek Lum’s scenario is increasingly becoming commonplace, particularly in the IT industry, where Big Data is being looked at through the eyes of mathematics. As a result, companies are hiring mathematicians to help create new insights in big data using mathematical formulae and principles. Long gone are the days when a business’ entire data could be hosted in one spreadsheet. The complexity of Big Data demands the help of mathematicians if any meaningful insights will be derived.

While it is clear to all that to process any kind of data, mathematics would be involved, there is a level of understanding and analyzing of complex data sets that goes beyond the common basics in math like probability, linear algebra and calculus. It is this mathematical thinking that is creating opportunities for mathematicians in companies like IBM and UST Global. This mathematical thinking includes quantifying the data to be measured and understanding how your quantification will work in math terms. The challenge is often in figuring out the math you need to do rather than doing the math.In 2016, the European Union passed a law that will come into force in 2018 where companies will be fined if their algorithms and machines make decisions that cannot be readily explained. So, instead of using machine learning alone to process big data and make decisions, corporations now more than ever need the input of mathematicians to figure out how their machines and algorithms arrived at the decisions they made. Mathematicians bring logic and simplicity to crunching the numbers, making it easy to see huge sets of data in a more straightforward way.