Middle East facing big data challenge
Skills and infrastructure in demand for big data projects
The Middle East is facing a challenge from the demands of big data projects, which is putting pressure on both skills and physical infrastructure, as the race to unlock value from massive, exponentially increasing datasets heats up.
Market research firm IDC predicts that the global big data market will grow 40% per year, which is around seven times as fast as the rest of the IT industry.
According to IDC, most of that cost will come from infrastructure-investment-calibre storage projects that are set to drive spending in the storage market to growth rates above 61% through 2015.
With big data sets growing by an average of 60% per year or more, based on IDC figures, business research specialists Aberdeen Group suggest that many companies will have to double the volume of their data storage every 2.5 years just to keep up.
IDC's 2012 Digital Universe Study, which was sponsored by data storage service and product giant EMC, estimates that the digital universe will reach 40 zettabytes (ZB) by 2020. The amount exceeds previous forecasts by 5ZBs, resulting in a 50-fold growth from the beginning of 2010. Machine-generated data is a key driver in the growth of the world's data, which is projected to increase 15x by 2020.
Big data and analytics is also drawing investment in the region. According to IDC, most of the activity to date has been focused on business intelligence (BI), with 35% of Middle Eastern CIOs having invested in BI in 2013 and 41% scoping for 2013. On the analytics front only 11% had invested in 2012 but demand is growing with 42% planning or scoping in 2013.
However the market has reached a turning point and confusion is changing into action with vendors and enterprises working closely on education around big data and its benefits, and although still fairly limited, experimental and pilot big data projects are taking place across the region.
The focus is now on fostering the skills necessary to make the most of big data. Companies need to invest in the creation of the right skill sets not just to analyse the data, but also to gain significant insights into what kind of data is valuable to enhance revenue growth and significantly impact business performance.
"As big data increasingly becomes part of corporate IT strategies and infrastructure, organisations will be on the lookout for specialist analytics skills to unlock business value from it," notes Shane Fernandes, ECEMEA business intelligence leader, Oracle. "Technical skills around Hadoop, MapReduce, Predictive, Statistical Analytics and NoSQL along with Oracle's commercial big data frameworks like Engineered Systems are in high demand," he says.
There are two main areas of skills that are required, Hadoop technical skills and data scientists. Hadoop is a new technology and as such, the skills are in short supply. For the data scientist, this role is still evolving, requiring a mix of skills and business awareness. Data scientists need a variety of skills across multiple domains including computer science, mathematics, data mining, and business analytics to rapidly explore and discover insights in data.
Gartner predicts that by 2015, big data demand will reach 4.4 million jobs globally but only one third of those jobs will be filled due to a lack of qualified candidates.
"The mix of roles in the IT workforce is changing rapidly with information management professionals who bridge IT and business expanding and pure IT specialists shrinking. Roles such as information leader, data steward, chief data officer, data scientist and information architect are the most likely additions during the next two to three years but these are difficult positions to fill due to skills shortages," says Gartner research director Arun Chandrasekaran.
"This is one of the biggest challenges and opportunities that big data presents," continues Margaret Adam, programme director, Services MEA at IDC. "To be effective, any big data project requires a combination of both IT and analytics skills - and it's not limited to this - there also needs to be new management skills in place in order to be able to effectively make decisions on the data or insight provided. IT people that are business savvy and able to combine technology with their industry expertise and business acumen have a lot to gain as the big data analytics market grows."
Chandrasekaran believes that IT managers can start preparing in several ways. He advises that they now begin to retrain senior staff in advanced statistical analysis, information management and visualisation principles. Even if they are not assigned to big data initiatives at present, being skilled in these concepts will be fundamental to success in the coming years.
He believes that IT managers should also start to determine what new roles will be needed, and begin to train or acquire the information skills that will be necessary to leverage big data strategically.
With any emerging technologies, lack of skills could cause a slow down in growth. In regards to big data, resources are growing, although they are still scarce. Fernandes has a positive outlook, believing the skills shortage isn't as big an issue as some make it out to be.
"The good news is that it doesn't necessarily take a long time to develop the skills required," Fernandes says. "Once businesses recognise the importance of big data to their future and start to initiate the necessary programs, the technical skills themselves can be developed reasonably quickly."
Indeed its importance is being recognised, as industry and academia are already working together to develop programs to deliver the skills sets the region will need.
"Many of the MENA universities are offering courses that focus and upskill around the big data challenge, ensuring a young workforce will also be available and skilled in big data," says SAP's Paul Devlin, director of Business Analytics and Database and Technology, MENA. "SAP's own young professional programme will equip the next generation with the skills required to drive big data initiatives," he adds.
SAP is not alone in developing education initiatives to help bridge the gap between the demand and supply of skills, as other companies are providing support services of their own.
With focused education initiatives and certifications programmes, there's clearly a lot going on to help develop the ‘next big thing' in IT skills. However, it's also worth noting one technology that may help IT managers in the interim.
"Through cloud based solutions, a company can effectively outsource their big data requirements and then bring this back ‘in house' when they feel better equipped," Devlin notes.
With all that's going on in this fast-moving sector, isn't it time you took a closer look at bringing big data into your business?