The rising importance of managing big data is a top priority of CEOs from all around the globe as they prepare for the deluge of data generated by the emerging ecosystem of the internet of things. The growth of big data is predicted to reach $187 billion by 2019, according to a new Worldwide Semiannual Big Data and Analytics Spending Guide from research firm International Data Corporation (IDC). The report identifies that the fastest spending growth will occur in utilities, resource industries, healthcare and banking. Big data promises to provide many benefits, such as predictive modelling and real-time streaming insights allowing companies to curate personalized offers to each customer leveraging past behaviors. However, with such blatant benefits, it’s quite surprising that today only 43 percent of companies collecting data obtain some tangible benefit from the information, and a startling 23 percent derive no benefit whatsoever, according to a Pricewaterhouse Cooper and Iron Mountain report entitled “How organizations can unlock value and insight from the information they hold.”
Although much attention is given to how and where to invest in big data mining and insights, the reality is that big data cannot accurately predict purchase decisions and behavior now, or in the foreseeable future. This humbling fact is based on one human-centric fact CIOs tend to forget: that consumers are irrational and act emotionally. We are impulsive and whimsical, and although we can be influenced, we also come with biases that are difficult for us to overcome. Although big data and artificial intelligence offer a prediction on the likelihood of someone buying a given product, these predictions go out the window depending on the mood of customers. This can come down to traffic, a good or bad day at work, or a previously held notion about something that seems at complete odds with logic.
If we use the past U.S. Presidential elections as a case in point, big data clearly failed to predict the future President of the United States. Since the U.S. elections are big business, an extensive amount of research was conducted by some of the brightest minds in voter qualitative and quantitative research. The big data had Donald Trump trailing a distant second to Hillary Clinton in pretty much every region of the United States. What big data failed to identify was the emotional mindset of voters who perhaps did not want to be seen supporting a less than marketable candidate. However, voters were clearly desiring a significant change of control in government. Many had a strong bias against the Clinton brand, and would simply refuse to vote for anyone with this name, regardless of whether their policy preferences were more aligned with this candidate. Some voters were engaged by the idea of keeping immigrants out in order to protect jobs, in spite of the fact that most job losses have been due to automation rather than immigration. Ultimately, Americans cast their votes based on emotion, not by logic or reason, leading to a significant upset that shocked the world.
Descartes’ Error, a book by professor of neuroscience Antonio Damasio, has validated what we have always believed: emotions create preferences which drive our purchase decision. Functional magnetic resonance imaging has shown that when making decisions on which brand to purchase, consumers rely primarily on emotions rather than information. The challenge for companies mining big data is how to go beyond past behaviors and truly understand how we feel in the moment has a much bigger impact on our decisions than what we have done out of habit. Amazon has discovered that there is a place for big data to predict behavior, but this is currently limited to our routine and mundane repetitive purchases. This is where I see big data providing the greatest benefit, making repetition obsolete. As it relates to predicting with accuracy what car we will buy, the type of clothes we will wear, and the food we will eat, big data will need to capture our emotions in that split second we are making that purchase decision.
