Discovering value from government big data
Big data is coming into focus as government organisations look to the power of analytics technologies to find value and develop new services from diverse sources of data
Big data has always been the domain of governments. From census data covering a country’s whole population, to economic and trade statistics, through to major strategic and research projects, governments have always had big sets of data that have driven planning and decision-making. With the development of new technologies however, the potential value of data has grown exponentially.
Advances in storage, real-time analytics, in-memory processing and analytical tools allows for faster, more complex analysis that empowers users to extract insight from data, while sensors, connectivity and Internet of Things solutions are increasing the volume and variety of data that can be collected.
Dr Taha Khedro, PwC Partner, Advisory Technology commented: “Data analytics has become a powerful force for change, one that can be used to benefit individuals, businesses, and government. Big Data has risen to the forefront, in part, because it has become progressively less expensive to collect, store, and analyse information. At the same time, new sources of data — sensors, cameras, and mobile devices, to name but a few — continue to infiltrate business operations and personal lives and are generating an unprecedented volume of information.”
The new generation of big data solutions differs from older approaches to statistical data analysis in several key ways, often referenced as the three ‘V’s — volume, velocity, and variety — although two more factors for big data success — veracity and value — are now commonly included. Today’s analytics solutions have been designed to cope with a massive volume of data, in a wide variety of formats, which is arriving at unprecedented velocity. Assuming the veracity of the data — quality and timeliness — then solutions should be capable of extracting value from the information to support decisions.
The solutions to address these five Vs are coming to market, but there are still some fundamental aspects of the technology that need to be considered. Yigit Karabag, Information Management & Analytics Practice Manager, Middle East, Turkey & Africa, SAS explained: “First of all, big data is big! This means [users] need to find a cost-effective way of collecting and storing massive amounts of a wide variety of data that keeps changing at faster speeds than ever. Traditional database systems are not fit for this purpose from a feasibility perspective.”
Platforms such as the open source Hadoop database are helping to address new requirements, and while there are specific challenges to these new platforms, growing adoption of Hadoop suggests that it will be a short term challenge, Karabag said.
“The second challenge is getting value out of big data. Storing big data alone will not automatically give any value. The key here is analytics, in its most advanced and powerful form. If big data is a treasure chest, it can only be opened by the power of advanced analytics which can become a challenge if the organisations are still in the traditional business intelligence (BI) mindset with descriptive, historical reporting practices. New sets of analytical tools and skilled resources needs to be available,” he added.
In many countries, analytics are already being brought to bear in sectors as diverse as traffic and transport, healthcare, law enforcement, public safety, tax collection, fraud detection and many others.
Asif Javed, managing director and technology growth platform lead for Accenture Middle East and North Africa said that analytics is becoming a vital part of digital transformation of government, and that projects are delivering practical results and significant value. The ability to quickly perform complex analysis means that “previously impossible statistical analysis is now routine”, and that data analytics will have an impact in many areas.
“A vast majority of users (89%) believe big data will revolutionise the way business is done in the same way the Internet did. Over the next five years, users believe big data will have the biggest impact on customer relationship (63%), product development (58%) and changing operations,” he said.
Big data solutions are already delivering value to the public sector, both in solutions focused on specific issues and functions, and in more general improvements to efficiency, and the returns are often as ‘big’ as the data. Through better collection of taxes — mainly using analytics to detect underpayment, the State of California has raised an extra $1bn in tax revenue annually since 2014. A McKinsey Global Institute report identified some 250m euros ($284m) value to European public sector administration from big data deployments, with 100m euros of that coming from general operational efficiencies.
While the technology platforms are evolving, there are still challenges that have to be addressed before organisations can start crunching data and extracting value. Teradata, a long-standing provider of analytics solutions said that shifting to a true big data model from conventional analytics can be costly and complicated process.
Eric Joulie, vice president, Western Europe, Southern Europe, Middle East, Africa, (SEMEA), Teradata Corporation, commented: “As agencies attempt to create these capabilities in-house, they are often confronted with many challenges, including transitioning from conventional analytic methods from data silos to incorporating change management and data integration models necessary to truly adopt a Big Data operational enterprise.”
Data is often widely dispersed between departments, locations, systems and even departmental and policy boundaries, and utilising data can require technology solutions and a willingness to share and collaborate, Joulie said.
“Big data analytics does not come free. Even open source or GOTS software comes with a price. Hardware is required, networks, power, space, & cooling and people — people to manage the systems and the new breed of skilled professionals that are capable of leveraging the available tools, the data scientist.
“Quite often, their biggest challenge is how to get started in a way that leverages rapid and effective analysis that produces real results with sometimes limited spending capabilities and recruiting hard to find technical expertise,” he added.
One of the biggest hurdles for successful adoption of big data is skills. Joao Tapadinhas, research director, Gartner, said that governments in the GCC are leading in terms of commitment to analytics, but are still lagging somewhat in terms of deployment. There is an opportunity for the GCC to leap-frog to the latest generation of analytics technology, he said, but this still leaves the question of skills and experience.
“In the GCC region, I don’t really see the same level of adoption in analytics, there hasn’t been, over the past 20 years, the same level of adoption, so we don’t get to big data in an evolutionary way,” Tapadinhas said. “In other regions we see government and the private sector working with BI and analytics since at least the 1990s, so they have had time to experiment with different technologies, to learn and create analytics maturity in their organisations, and slowly evolve to more complex projects.”
Even without organic growth of expertise, government commitment to big data is a positive step, Tapadinhas said, and coupled with hiring, training, management and the surrounding IT ecosystem, the GCC can still deliver big data projects successfully.
There is a global shortage of talent in data analytics, however, and although there are many vendor-led initiatives, government training programs and academic programs to increase the availability of skilled personnel, the best skills and disciplines for these ‘data scientists’ are still being identified, and the successful adoption of analytics is often perceived as requiring a shift in culture as much as it is the adoption of new skills or tools.
Rajeev Lalwani, partner and Technology Consulting Leader for Deloitte in the Middle East, said that putting data at the heart of an organisation, and treating it as an asset — “data as the new currency” as the World Economic Forum described it — requires a strong vision and oversight for data, which is leading to the emergence of the chief data officer role and the development of data departments. The chief data officer is separate from the CIO function, overseeing all of an organisations data and being accountable for the use and organisation of data as a resource.
Lalwani explained: “Government requires a visionary leader to drive this change as traditional CIOs focus on systems and infrastructure with no accountability for data. Governments needs to brace up for change as the promise of data spurs the organisation into action. Data enables informed decision making, improved customer experience and drives operational efficiencies.
“However government CxOs these days overemphasis on technology rather than focus on organisational culture, people and processes. Existing C-level leaders will feel threatened by this organisation and it will require cultural realignment from the top. However organisations that embark on this journey will reap the reward. Gartner predicts that organisations that integrate high value, diverse data into a coherent information management infrastructure will outperform their peers financially by 20%.”
These new data-driven leaders also need to be able to influence the whole culture of their organisations, to deliver the full impact of analytics. Organisations not only need to be able to extract value and meaning from data, but they have to be able to apply the data and adapt their strategy to accommodate the insight gained.
“Purchasing the technology is only part of achieving analytics success. Organisations also need to educate their people, develop processes and create an organisational culture of business analytics in order to be successful. This second part can often present continual challenges,” said Ken Habson, regional sales leader, IBM Analytics, Gulf & Levant.
“It is important to measure how ready an organisation is to apply insight to strategy and tactics; how quickly an organisation can re-allocate resources and re-orient people to make better decisions; and how effectively they can act, based on how well they know past performance, current results and future possibilities,” he added.
Although there are some large-scale government approaches to big data, such as the Dubai Data Establishment, which seek to unify big data and analytics efforts across agencies, many organisations are also finding success with small scale projects that use existing resources to explore their big data potential. One of the key enablers of this low-key approach is open source analytics solutions such as the Hadoop storage and processing framework, or the R programming language, which have become common tools among organisations who want to experiment with analytics tools without a large outlay on software. Small projects, taking an experimental approach and that can afford to fail, are helping organisations to determine where to focus their larger efforts and to determine where to get the best return, without drawing heavily on already tight government budgets.
Tapadinhas commented: “There is one problem with big data projects, which is the fact that it is not always possible to determine the return on investment or how successful those initiatives will be, so the traditional approach of just having a large budget and a large team, and working on a specific topic for long periods doesn’t necessarily deliver results.
“We need a new way of addressing big data projects — we need small projects, we need to fail fast, we need to try, experiment, see what works, and if it doesn’t, try a different approach. I would recommend a continuous program where both projects with small budgets, and project managers that are not afraid to fail will try, and eventually succeed, and after understanding what they need to do, after running the proof of concept, then go for the bigger project with solid budget supporting it, to deploy big data.”
He gives the example of a healthcare ministry in Latin America that set up a big data team with only two experts, using open source solutions, who were able to run a big data project that identified hidden costs for the ministry and created multi-million dollar savings in under two months.
Another example of how a focused big data project can deliver results is the UK’s National Health Service’s Data Analytics Learning Lab. Established with Oracle, this facility was set up to use analytics to learn more from the data held by the Health Service.
Samina Rizwan, senior director, Big Data Business Development at Oracle, explained: “Within three months of starting operation, [the Learning Lab] reworked processes for European Health Insurance Card applications to prevent fraud, used anomaly detection to find fraudulent activity, analysed text to measure employee satisfaction and engagement, linking to sick leave. By showing value in a relatively short time, they proved the project to management and received support for expansion. They have a long term strategic goal of saving £1 billion ($1.56 billion) over five years.”
Along with the technology challenges and lack of experienced staff, the main issue facing organisations that are embarking on big data projects are around data governance, privacy, and security. Although governments are already responsible for managing a wide range of sensitive data, the need for data to be available for analytics solutions and the moves towards sharing of data between agencies is adding extra dimensions in governance.
“As IT executives know all too well, managing big data involves far more than just dealing with storage and retrieval challenges — it requires addressing a variety of privacy and security issues as well,” said Rizwan. “Though many organisations use big data for collecting non-personal information, potential pitfalls in data collection include ubiquitous and indiscriminate data collection from a wide range of devices, unexpected uses of collected data, especially without customer consent, unintended data breach risks with larger consequences. Governments are responsible for protecting consumers, so ‘big responsibility’ comes with big data. What this means for organisations is that they must secure the life cycle of their big data environments and ensure no breach of security happens.”
Governments around the world are developing the governance framework for big data projects, including privacy and transparency, and development of legislation frameworks is becoming a priority, but most governance is still in the early stages. Deloitte’s Lalwani said that in the Middle East, the UAE is leading such developments with initiatives such as the Dubai Data Law and Dubai Data Establishment.
As the laws are developed, governments are also considering appropriate use of big data, particularly with regard to privacy. While big data opens up new possibilities through the ability to process huge volumes of data, using sources of data such as a camera feeds or location tracking to create new services, there is an ethical question of what public or private sector organisations can or should do with the data sources they have available.
Ken Habson said that there is a need for governments to find a good balance between protecting citizens’ privacy and monitoring/predicting security threats: “Governments will also need to allow better advanced services to be developed with access to adequate data, but to control abuse of this data and security breaches. Our discussions with government clients show that this subject is of utmost importance and the right priority is being set for adopting the right policy,” he said.
Joao Tapadinhas noted that outside of public safety and security, there are other industries where the indiscriminate use of big data analytics without paying proper attention to ethics is a problem. While best practices and legislation are catching up to the technology, all organisations need to be wary of how they are using data to avoid a backlash from consumers.
“We should always run big data projects with a high levels of ethics in mind, making sure that data is used in the proper way and taking into consideration what citizens and customers consider right and wrong, not invading their privacy and not misusing their information,” he said. “Because this is an emergent area of technology, we are still learning what is wrong and what is right. We will get to that point where best practice will become law, but for now it is about self-discipline in making sure that organisations understand that the wrong use of information will backfire and hurt them in the end.”