Big Data & Data Visualisation

When talking to people about data related projects I have started noticing that it doesn’t take long to get on to the topic of Big Data. It’s a subject that is getting a lot of press at the moment so that’s not too surprising.

The thing that I am finding surprising is that few of those I have been talking to make the connection between Big Data and Data Visualisation. To them the talk is all about the “Three V’s” (Velocity, Variety and Volume) that define big data. My question, once we have covered the ground of how do we cater for and handle the three V’s, is what about the fourth V, Visualisation?

From where I stand it’s all well and good being able to stuff data into a cleverly designed warehouse at an ever increasing rate. But what is the point in that if all you do is pull it out in static, flat one dimensional reports? To make the data really useful and come alive you really need to make it tell a story in a dynamic and visually appealing way. The message you are trying to convey should jump off the screen at the user and at the same time lead them to zero in on the the good AND the bad news.

These days there are an increasing number of tools that allow you to this with varying degrees of flexibility, complexity, functionality and naturally cost. All you have to do is think about how to make it easy for people to get at the information they need through the overwhelming volumes of data (to them) that is available.

A new player bringing Hadoop and Big Data to the massess

The Microsoft SQL Server team is not the only group looking to ​bring Big Data to the masses. Datameer has a desk top version of its eponymously titled 2.0 release that is shipping for $299 and a workgroup server for $2999. Unlike the MS solution this one doesn’t use Hive as the connector to the Hadoop MapReduce interface.

This now looks like Hadoop can scale in any direction. For more details check out the Big Data blog on ZDNet:

http://www.zdnet.com/blog/big-data/hadoop-comes-to-the-desktop-with-datameer-20/522?tag=mantle_skin;content

Blending Conventional and Big Data Access

In theory combining data access methods in a horses for courses concept is a great idea. It should allow data to be accessed in the manner most suitable for the task at hand. The one concern I have with it though is the increase in complexity it introduces.

If the approach taken by Hadapt helps to reduce that complexity as implied in this ZDNet Blog Post then it may be an approach that has appeal to business. If Microsoft are successful in bringing a similar concept to SS Management studio then business uptake will be rapid as it will provide that single pane of glass that IT departments (development and support) all crave.

It’s just data! – An alternative view of Big Data.

Here’s an alternate way of looking at “Big Data”… You want the data. You can’t handle the data. Well, most of you won’t be able to at the moment unless you are amongst the few who are implementing “Big Data”. But in the end it is all just data

http://www.simple-talk.com/sql/database-administration/big-data-is-just-a-fad/?utm_source=simpletalk&utm_medium=email-main&utm_content=bigdatafad-20120430&utm_campaign=SQL

Big Data – What is it and where is it heading

Gartner describes Big Data as being all about three V’s, Velocity, Variety and Volume and lots of each. Big Data is high velocity, high volume data that comes in many different forms from a diverse set of sources in a range of forms and formats (variety). The data tends to be unstructured or semi structured in nature making it difficult to handle through normal relational database structures and methodologies.

Traditionally much of this data has been discarded due to the cost of storage. This is a restriction that is coming to an end, and now that 20gb of Ram now costs less than a 20gb drive did 10 years ago the cost of keeping large data sets in memory is becoming a possibility.

Here’s a blog courtesy of ZDNet that aims to demystify Big Data and look at the trends and developments that are emerging.

http://www.zdnet.com/blog/big-data