Starting small with big data
Small is beautiful when it comes to getting started with analytics
One of the lessons that is becoming clear from the early adoption of analytics is that big data projects don't have to be ‘big' - in fact, small can be beautiful when it comes to getting started on the road to data-driven decision making.
Compared to traditional large scale IT projects, such as a new ERP or digital services, many organisations in the public and private sectors are discovering that it is better to start with a smaller, inhouse working group, usually comprised of a few talented IT personnel, who are tasked with exploring the technology and understanding how to get value from it, instead of jumping into a big big data project.
This small scale approach in part builds on lessons of decades of major IT projects - big data analytic tools and approaches are still being defined, there is no handbook or best practices to ensure successful solutions in the same way that there are with ERP projects. The aims of analytics projects are not always clear or easily defined, and with the shortage of expertise, organisations are not going to be able to parachute in an expert or two to turnaround a project that hits problems.
The skills problem is a major issue with big data analytics. The technology is new, and even the types of skills required for success are not fully clear as of yet. What experts there are available are expensive, and most organisations cannot afford to spare their staff for training that might not be appropriate for their organisation, or could lead to them getting approached by another employer.
Big data projects can start small, however. While the traditional foundations of data analysis, such as data warehousing, are certainly not the cheapest solutions, a popular alternative is the adoption of open source solutions, particularly Hadoop.
There are many examples of organisations taking a ‘back room' approach to big data, and starting with small teams experimenting with Hadoop to see what can be done. Bahrain's Gulf Air, for example, developed a social media sentiment analysis solution to gauge what was being said about the airline on Arabic-language social media channels. No such solution existed, so with a small team and open source tools, Gulf Air built its own at a fraction of the cost and time of a full-blown project.
By creating exploratory teams of existing staff, usually with some BI or analytics experience, and with some understanding of the data sets that are available to the organisation, these pioneering teams are able to get up to speed with big data solutions and work out how to make the analytics tools work for them, and create some quick, cost-effective solutions. Hold ups or failures won't break the IT department or the budget, but early wins can become proof points for bigger projects, and can be used to demonstrate to other functions within the organisation what can be done with the tools, and encourage them to come up with their own ideas. The journey to big data can start with small steps, and can provide an excellent showcase for the IT function to showcase innovation and leadership.