From: BusinessCloud9

Posted by Katy Ring

Big Data is certainly rising up the list of strategic priorities for many CIOs but is Big Data a required investment status yet for most organisations? Later this year K2 Advisory will be examining this question in more depth, but in the meantime, a few issues are worth flagging.

Good structured data management is a given in many organisations, and a valued aspiration in many more, but the reason for the noise about Big Data is, of course, the rise of unstructured data. Data has, in this century, become truly digital which means that analytics can be applied more comprehensively because there are less paper-only transactions, interactions and records to create digital information gaps. There is also the rapid growth in the generation of data from sensors, and the rise in use of pictures and video to capture information. This creates the famed three V’s of Big Data: volume, variety and velocity.

However, one of the challenges emerging so far around Big Data is that it is being promoted in our industry as a technology problem, rather than as a business enabler. This is of course no big surprise in that the IT industry loves to market a technical solution and still struggles to address the business issues. To date a lot of discussion focuses on the data operating system Apache Hadoop.

While there are technical aspects to Hadoop that your data management team needs to keep on top of (such as security and MapReduce skills), you need to be considering other factors regarding your organisation and Big Data.

Data is being seen as the raw material of business, enabling a different type of decision-making. To date most IT departments are comfortable with business asking regular questions and IT providing the data to answer them by capturing what is needed using structured methods such as SQL. However, Big Data analytics is a very different animal because you are capturing vast amounts of data to explore for determining questions worth answering.

This is quite a philosophical leap for most businesses (some of which are developing internal data science teams to enable that leap) and it creates an immediate headache for IT because ALL data must be captured regularly and stored, which is why open source Hadoop is such a blessing because there are no upfront licensing costs and it has hugely scalable storage and processing capacity.

But from an IT perspective using a system such as Hadoop is only part of the technical challenge as the issue then moves to the requirement for a unified data architecture managing structured, semi-structured and unstructured data. And then a “discovery” platform needs to be sourced that can let the organisation sift the data for patterns to question using analytics.

However, the real issue that organisations are struggling with is the lack of data discovery skills available in the market to help create value from the data. Existing business analysts are typically time poor and do not have the time to go fishing in the Big Data ocean for the organisation. At the moment our industry is focusing on the technology and not addressing the value creation challenge properly. It is this rather than  Hadoop which is the real elephant in the room.