First of all, what do people mean by this newly overused term ‘big data?’
I have encountered three explanations that I like.
The first comes from an MIT Sloane School May 2013 conference during which they said: “Only about 20% of available data is structured so it can be analysed. Big data includes the large quantities of currently unstructured data.”
This fits with Wikipedia which says: “The term big data from software engineering and computer science describes datasets that grow so large that they become awkward to work.”
And the triangulating definition is Gartner’s ‘3V’ version: “Big data are high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimisation."
So, when the Tsarnaev brothers set off bombs at the Boston Marathon, authorities sifted through untold mountains of unstructured data, sorting and analysing photos, security video, eyewitness reports and whatever else was available. In three days, they identified two men who started with – but ended without - large, heavy backpacks, were together at times and also in the two bombing locations at the opportune moments. Stateside, we (particularly we Bostonians) were incredibly grateful for the analytical techniques that led to their identification and capture within four days. Data was the saviour against the evil brothers.
However, just about six weeks later, when Edward Snowden reported that vast quantities of telephone data were being analysed for on-going identification of potential terrorist plots, many Stateside folks became outraged. They want the bad guys tracked, but not the innocents. Ahh, the scourge of this technology!
My favourite retail data story is from Charles Duhigg’s The Power of Habit. Apparently, big box retailer Target Corporation’s data analysis digs so deep that they can tell from one’s purchases whether a woman is pregnant, and how far along she it. Target uses its data to send coupons to lure customers to purchase items they are likely to want next. The father of a teen receiving such coupons called Target became irate because he felt they were encouraging his daughter to have a baby. How could they do such a thing! The man called back to apologise a few months later. It turned out that his daughter was already pregnant, and Target knew before he did. Scourge or saviour?
We are not likely to have it both ways. We will want personalised medicine and tailored treatment plans, but must accept that those techniques must be built on the backs of millions of patient records. If individuals today don’t share information, the next generations will not be likely to enjoy better cures. Likewise, new transportation methods, routes, and vehicles require analysis of what and how we currently use transportation.
George Orwell’s 1949 classic Nineteen Eighty-Four apparently is selling again at an incredible clip. The connected homes of his vision have a lot in common with the video opportunities of today’s networked world. We don’t want Orwell’s Big Brother, but we want technology to root out evil and find solutions for what ails us.
The scourge of Big Data is the fear of intrusiveness and the loss of the concept older generations call ‘privacy’. The salvation is both when the good guys catch the bad guys, and, for our industry, when that incredible quantity of data is required to conduct business, medicine, education, government, etc. When analyses face high volume, velocity, and variety of data, our carriers’ networks, colos, clouds, managed services, et. al. become increasingly valuable. So, grow data, grow.
