Making sense of big data
New sources of data equate to new business opportunities, but with the exponential growth of data there’s a new challenge for companies – how to store, manage and extract value from it in a cost effective way
Published Sunday, 27 May 2012
By Keri Allan
“Data is no longer a by-product of running a business, it is the raw material needed to stay in business and compete effectively,” explains Jason Bath, head of business analytics, database and technology, SAP MENA.
“Businesses want to get their hands on as much detailed data as possible in order to extract information, get insight, and make actionable business decisions. Detailed customer behaviour across multiple touch points is captured and analysed,” he adds. “Enterprise applications are able to generate extremely detailed data; for example, mobile operators typically generate billions of call detail records in any given month.
“Emerging technologies, changing behaviour and the competitive business landscape are all forces that are coming together to produce the perfect data storm.”
New sources of data equate to new business opportunities, but with the exponential growth of data there’s a new challenge for companies – how to store, manage and extract value from it in a cost effective way.
Bath advises companies to consider relating the storage tier to the value of the data, use advanced database compression features that will save on data centre real estate and energy costs and avoid creating multiple copies of data.
“New applications are driving the demand for extreme performance and real-time access to information while, at the same time, increasing compliance regulations are driving the requirement to retain historical data. Therefore the need to match the storage technology to the value and access patterns of data becomes critical in achieving a balance between performance and cost,” he explains.
“For example, a real-time social media feed of customer sentiments following a major product launch needs to be analysed frequently and may be best stored in a high performance in-memory database, while rarely accessed archival data retained for compliance purposes may be more economically stored in traditional high density disk drives.
“However, with the rapid decline in the cost of memory, which today already offers lower cost-per-performance compared to traditional disk drives, it is becoming more cost effective to store databases entirely in memory. By looking at the rate at which the cost of memory is declining, it is predicted that by 2017, it will cost the same per capacity as disk drives.”
Many companies, including IBM, Oracle and SAP, are providing solutions that incorporate these kinds of strategies, which they hope organisations will turn to as they try to make the most out of big data in an economical fashion.
“Our position with respect to technology investment is that it has to be supportive of a company’s business objectives and must demonstrate an overall lower total cost of ownership,” says Ismael Hassa, sales director, Oracle. “Hence, a modular, scalable approach to next generation analytics is the recommended way forward,” he adds.
“The departure point for any company embarking on the quest to gain control of big data is to work with experienced people and proven technology with the outcome of firstly understanding the requirement and its impact on the business, conduct capacity planning with the goal to grow as needed and ensure investment protection along the way.”
It is these new technologies that allow organisations to perform meaningful analysis of large amounts of data, in turn providing business value.
“Big data technologies describe a new generation of technologies and architectures, designed so organisations can economically extract value from very large volumes of a wide variety of data by enabling high velocity capture, discovery, and/or analysis,” says Philip Roy, director, data computing division, EMC.
“This world of big data requires a shift in computing architecture so that companies can handle both the data storage requirements and the heavy server processing required to analyse large volumes of data economically. New ‘information taming’ technologies such as deduplication, compression and analysis tools are driving down the cost of creating, capturing, managing, and storing information to one-sixth the cost in 2011 in comparison to 2005,” he adds.
“New emerging technologies can perform powerful analytical computing for analysing data at rest, or analysing data in real time with micro-latency. Rather than gathering large quantities of data, manipulating the data, storing it on disk and then analysing it (analytics on data at rest), other platforms allows us to apply analytics on the data in motion,” continues John Banks, director of IBM Software Group, Gulf Business Machines.
“With the ability to handle big data effectively, we would have the ability to manipulate the data and in-flight analysis is performed on the data. This analysis can trigger events to enable business to leverage just-in-time intelligence to perform in real time, yielding better results for the business.”
But what skill sets and capabilities do businesses need to have in place in order to make the most out of their big data? “The starting point of any big data project should be people and processes,” says Sid Deshpande, senior research analyst, Gartner.
“The key first steps to be taken include hiring qualified data scientists and big data professionals, whose numbers are disproportionately low to the high interest in the market and are, therefore, in high demand. Also, evaluating the business case and specific outcomes of the proposed big data project, before thinking of re-architecting internal IT from a people/processes/data management standpoint to avoid investing in tactical projects and to make it more strategic to the business.”
Indeed, the emerging role of the data scientist is becoming a key role for companies dealing with big data. They possess analytic, technical and business skills, which allow them to ‘get their hands dirty’ with big data, and to extract relevant and significant business insight.
“If you imagine a Formula One race, these are the guys who sit behind a computer during the race, constantly analysing data being streamed from the race cars on the track, doing complex real-time analytics and advising the driver via his headset on what to do at every turn and every stretch of the track,” says Bath. “They analyse huge amounts of data quickly and provide information that the driver can act on in order to win the race. These are exactly the kind of skills companies need to develop in order to leverage big data.”
Big data analytics is an emerging discipline in the Middle East and many vendors believe that the region still lacks the experience, skills and technologies necessary to leverage it, though they believe this is changing quickly.
Gartner, however, predicts that through 2015, 85% of Fortune 500 organisations will be unable to exploit big data for competitive advantage. This may be because they don’t expect companies to integrate their big data strategies very well.
“There is a small but emerging and rapidly growing segment of big data projects that serve as an extension to on-premises enterprise business intelligence (BI) projects, but they often deliver incremental value rather than being completely integrated into the organisation-wide BI strategy,” Deshpande explains.
“Gartner has observed that despite significant amount of interest among large enterprises, the early deployments were either tactical, one-off projects or extensions of traditional BI strategies with additional analytics and data processing. A completely integrated big data strategy is almost nonexistent among enterprise users, and Gartner expects this situation to continue until viable solutions and proof points emerge.”
Clearly views are mixed as to the future success of big data, but for the Middle East, early adopters are appearing and so the seeds have been sown.
“Many of the enterprises we meet with have strong regional or even global ambitions. These ambitious enterprises are very focused on investing in sustainable competitive advantages, and thus we think they’re very likely to investigate big data opportunities in the near future,” says Roy.
“I wouldn’t want to hazard a specific timeframe – it’s more of a general tendency. I think with each year we’ll be able to point at progressively more Middle Eastern enterprises that have started their journey towards achieving big data proficiency,” he concludes.