Sunday, August 25, 2013

The problem with Big Data


I am NOT a BIG Proponent of BIG data for business use especially in identifying consumer behavior. It has a lot flaws that are being neglected in the present big data narrative.

The interesting part that most of us neglect is "How is it useful?" It tells us what people are doing. Inferring why they are doing is up to the skill of the researcher and that is where most people fail.

To give a short example it was only a few economists including the now famous Nouriel Roubini who predicted the 2008 recession. While all economists had access to the same data, most of them failed. There were cases where the predictions were almost divergent.

Another important part of big data is "what is it that we are collecting?" When we have truck loads of data to collect, it is important to collect relevant information. What that relevant information is depends again on the researcher's skill. When you have a lot of dots, your inference depends on which dots you choose to connect.

Finally where big data fails miserably is that it only tells us what people disclose. In an age where most of us are curating an online identity (and concealing the real self) do you think you can figure out accurately what is going on in their mind. I don't think so!

Agree with this? Share your thoughts below.