‘Big Data’ is a term straight out of a Hollywood blockbuster and has been one of the most trended terms on Twitter and LinkedIn for the last 18 months.

However, has it run its course? Was it merely a fad? Or has the term merely been hijacked by seemingly well meaning people who simply did not understand what the term referred to and the benefits of ‘Big Data’? Alternatively, is Big Data the greatest demonstration of under utilisation in IT history, with endless possibilities for greatness?

The question that is appearing more and more on IT leadership forums is:

“What is your Big Data strategy?”

There's a gap between what big data means on paper and what it really means to a business. Big data is at a crossroads. On one hand, big data is dead, the term having been used so often that it's been stripped of tangible value. On the other hand, big data has never been so alive, as more companies than ever are trying to improve so-called big data analytics. How can such a dichotomy exist? The answer can be found in the enormous gap between what big data means by definition and what it really means in the important practice of data management.

Big data by definition

The term big data -- by the most commonly-used definition -- refers to data sets that are too large and complex to manage within traditional systems. This data is generally unstructured or semi-structured and requires investment in new tools, technologies, skillsets, and team members to manage it. This is big data as it exists on paper, the end product of a meteoric hype cycle. This is not big data as it exists in practice, however. Analyst research and customer experiences suggest that in practice, what organizations and IT personnel refer to as big data is actually just data.

Big data in practice

In a recent TDWI research report, 88% of organizations cited structured, relational data as their primary big data type. This is traditional transactional and analytical data living in relational databases. Additional recent research from Enterprise Management Associates (EMA) shows that nearly half of organizations' big data projects are based on Oracle and/or SQL Server -- the traditional systems supposedly incapable of managing big data.

"The same research shows that only 28% of organizations are concerned that their current systems cannot scale to meet the demands of their big data projects."

My own experience in working with customers tells a similar story. Traditional relational databases and the data volumes housed in them serve as the bedrock of a surprisingly large number of big data projects. So despite the hype, in practice, many projects tagged with the big data label (and therefore supposedly too large and too complex for traditional systems) are actually built around data that is most often neither bigger nor more complex than what we've been working with for years, and that can most often be managed within the same systems we've been using for years.

Pitfalls of the hype

That's not to say the continued hype surrounding big data is all bad. It's certainly a bad thing for those companies that get too caught up in it. The pitfall of any hype cycle is that it drives companies to do things that don't actually solve their fundamental business problems. For example, I often hear from customers who invest in Hadoop, load their data, and then say "now what?" They've invested in a new technology -- because the hype led them to believe that's what they're supposed to do -- without knowing if it can actually help them. EMA's recent survey found that stakeholder support and business strategy issues are the top two barriers preventing organizations from succeeding with their big data projects. Technology and infrastructure concerns were significantly smaller issues, according to the survey. This suggests that big data projects often start from the bottom up tied to a desire for IT innovation, rather than from the top down, tied to a desire to tackle significant business challenges. When big data projects run into roadblocks, it's usually because business objectives aren't clear or the right people haven't been granted access to the right data. In other words, projects are not derailed because IT doesn't have the right technology, they're derailed because the company isn't aligned on what it's trying to accomplish in the first place, and that can only be rectified from the top down. In reality, big data is generally not a technical challenge. Maybe Yahoo and Google needed to reinvent their infrastructure well beyond traditional capabilities, but most companies do not. Not yet, anyway.

Refocusing on a fundamental need

For companies that don't get misdirected by big data, however, the hype is a great thing. The big data trend has reawakened many organizations to the longstanding and fundamental need to become more data-driven. When the end goals are to solve a business problem and make better use of data inside and outside the organization, companies can often do so without investing in expensive new platforms or costly data scientists. Make no mistake: Many things have to happen for organizations to get the most out of their data. You must outline clear business objectives, IT and business leaders must be committed to collaborating, and the right people need to be granted access to the right data. It's just that most of those things have little or nothing to do with the mainstream definition of big data.

"Awakened by the hype, but not caught up in it, smart companies are making sure data is in the right place at the right time, is shared by systems, and is available in reports that can be analyzed in ad-hoc fashion by business teams."

The hyped-up definition of big data is dead, but the pursuit of making data the lifeblood of a business is more alive than ever. ARTICLE AND IMAGE SOURCE: Courtesy of informationweek.com and emergingedtech.com

REFERENCES AND FURTHER READING

Pro-Big Data:

  • IT as a business outsider: 'undervalued, underinvested and underutilized' - fiercebigdata.com

The end of Big Data?

CONNECTING AND CONTRIBUTING

Jenrick IT invite you to participate in further discussions on these topics:

  • If you wish to contribute as a guest (or ghost) writer for our Digest newsletter, please email us