17 August, 2016
In case you have been living under a proverbial rock, there’s a good chance you’ve heard about big data. It’s in, it’s new and it’s been branded as the latest hype. But what is it, and how can we make the most of it? And more importantly, does today’s data have the potential to lead us to a smarter future?
There’s been a lot of hubbub about big data. This concept gained momentum in the early 2000’s when industry analyst Doug Laney coined the term, in an attempt to define large volumes of structured and unstructured data stored for future analysis.
Laney defined big data according to the three V’s:
Volume - Businesses collect data from a variety of sources. This can be anything from raw sensor data, transactions or customer feedback. Storage of data was a significant problem in the past, but recent advancements such as Hadoop offer the much needed horsepower to analyse extremely large data sets.
Velocity - With improved storage capability and fast networks, data can be streamed in near-real time from sensors, smart meters and RFID tags. Increased speed of data, brings the need to deal with information in a timely and efficient manner, or face the consequences of missing out.
Variety - Whether it’s video, text or the latest stock trading prices. It’s safe to say that data comes in a variety of formats that is both structured (numeric) and unstructured (text).
Companies are increasingly finding themselves overwhelmed with data, analysing just a fraction of the information they gather. Needless to say, the amount of data being gathered is growing at a relentless pace.
According to the IDC, the amount of data shared worldwide exceeded 2.8 zettabytes in 2012. Storing one zettabyte requires one billion 1 terabyte sized hard drives, which is enough to store 250 billion DVD’s. If that seems a lot, the IDC projects there will be 40 zettabytes of data produced annually by 2020.
It’s often said that data is the holy grail, but without asking the right questions, potential nuggets of gold can end up getting lost. Big data may elude itself to quantity but it’s a shift towards quality that’s going to take us one step closer towards a smarter future.
From raw data to actionable insights
So what can a business do if it finds itself data rich but information poor? How can an organisation go about squeezing the maximum value from all the bytes gathered?
The diagram below represents a data value pyramid which ideally progresses from meaningless archives to actionable insight. Value is created by making sense of data.
(The secret of course, lies in asking the right questions)
So how can organisations make better use of information that flows into their business every day? To gain insight we must find answers to what we don’t know we don’t know. This is best explained through the levels of knowledge below.
Getting value from data
Affirms what we know.
It tells us the number of fridges we have.
Helps us answer questions about what we don’t know we don’t know.
It tells us how many fridges are failing.
Helps us discover what we don’t know we don’t know.
It tells us how many fridges are failing, why, and what we can do to fix the problem.
By identifying the source of the issue we can take steps to remedy the problem. The value of big data translates to cost and time reductions, enhanced offerings and ultimately smart decisions.
The levels of knowledge reveal that a single data set is not comprehensive enough to be a source of transformational insight. If that’s the case, how many data sets do we need to extract valuable information? Is it one, one-hundred perhaps?
The power of three
When it comes to knowledge discovery, the biggest return on investment can be achieved by combining two subject areas. Exploring the combinatorial space of three subjects, provides maximum impact with least amount of effort required for discovery. Insights often come from unrelated and disparate data sources, which can be explained with an example in advertising.
If we’re trying to assess the impact of an advertising campaign, a good starting point would be to gather campaign specific data, in combination with social data to judge how the market responds. These two separate subject areas can be supplemented with another data set to maximise our return on investment. Insight can be enhanced by adding a third data source which could relate to specific ad clicks or requests.
By combining three subject areas across marketing, psychology and computer science we can demystify consumer behaviour and find compelling answers.
Why this is effective
The rule of three is powerful because it’s the smallest number required to make a pattern. This three-element pattern helps us to communicate complex ideas with greater ease. Big data may seem daunting, and the ever-changing technology can at times be distracting. If we want to ensure a smarter future we need to keep things simple, by asking better questions that help us uncover valuable nuggets of information.
About Karina Maksimiuk (Founder of Hubebub)
Founded in 2014, Hubebub is a data company which uses a network of wireless sensors in commercial buildings able to identify opportunities to save energy. By placing smart sensors throughout a building, Hubebub measures energy consumption and translates this data into energy management insights – such as suggesting heating be adjusted based on the current weather or the number of occupants using a room.