Best laid plans

Increasing threats to business continuity in the Middle East have spawned a rejuvenated determination among CIOs to secure company data. In the second instalment of ACN’s disaster recovery report, Duncan MacRae discovers how businesses are preparing their defences.

  • E-Mail
By  Duncan MacRae Published  May 20, 2007

|~|200-Amin-of-EMC,-left,-with.jpg|~|EMC’s Amin and Roshan’s Kohja have built a successful defence against disaster.|~|While the issue of data loss was once an afterthought, it now appears that businesses are growing ever-more aware that an appropriate spend on disaster recovery solutions could reap massive rewards in the long-run.

According to a recent survey undertaken by GCR Custom Research for HP, the issue has become more of a priority with large chunks of the IT budget being made available to develop IT infrastructures that will keep businesses ticking over in the most extreme circumstances.

Mohamed Nasreddine, MIS engineer at Al Ameen Financial Brokerage in Dubai feels that while the main reasons for data loss can vary from country to country, it must be top of the agenda for businesses everywhere.

“In Lebanon, Iraq and Palestine, the loss of power is often due to power shortage during wars,” says Nasreddine, “as the main sources of power are sometimes destroyed, or the available sources are scarce.

“As for the Gulf region, I believe the main reasons are overheating, especially during the summer season. Preparing for disaster is a must for companies. They simply must avoid losing critical data by installing emergency sources of power such as UPS, batteries and a generator.”

The survey results suggest the majority of IT managers are now on the same wavelength as Nesreddine. Around 80% of IT decision-makers at both large and medium-sized businesses view business continuity and availability as an increased priority for 2007 and key to sharpening their competitive business edge.

Overall, the results mark a shift in strategy for many companies, which have historically taken a reactive approach to recovering from unplanned downtime or disasters, to longer term business continuity planning.

Many businesses in the Middle East have been learning the hard way, while many more have been looking and learning. The Middle East is home to some of the most extreme working conditions that businesses could possibly have to endure, so there are companies in countries such as Lebanon, Iran and Afghanistan that have had no choice but to invest heavily in disaster recovery solutions.

Roshan, a leading telecom operator in Afghanistan, has not just had to prepare for disaster – it has had to live with it.
The telecom firm has a presence in over 160 villages and cities in the country, as well as a roaming service with ten other operators around the world, and offers GPRS in Kabul along with a mobile banking service.

The company’s importance to the Afghan nation as a whole cannot be overestimated.

By the end of 2006 Roshan had paid US$75 million to the government of Afghanistan, making it the biggest taxpayer. There are roughly 300,000 people in the country employed directly or indirectly by the company and to date it has invested $280 million in Afghanistan.

The company’s growth in recent years has been phenomenal, which has caused further problems in its disaster recovery plans.

“Two years ago it wasn’t important because we could just restart the computer and do a cold start,” explains Karim Khoja, CEO at Roshan. “Today we have 1.3 million customers. We have a customer registration information system and we’ve got CRM. If you lose any data you’re basically writing off real money.

“The Afhgan people are not very forgiving. A few weeks ago in Dubai, Etisalat’s network was down for about two hours. You couldn’t get through and you couldn’t send text messages. If that was the case in Kabul our own customers would have stormed our shops, thrown stones and wrecked the shops. So business continuity is not just about keeping your business alive, it’s sometimes about keeping yourself alive,” Khoja adds, with tongue firmly in cheek. “But we can’t afford to have a million angry customers queuing outside our door saying that they are not happy.”

There are many factors that come into play when planning for business continuity in Afghanistan, as opposed to the Gulf. It is an exceptionally harsh environment. There is no mains electricity so Roshan must generate its own electricity in every single site throughout the country. Right now it has around 3,000 generators, as well as fuel tanks.

It can be all too easy to run out of power, with diesel getting corrupted or stolen, the generator failing, or the UPS not kicking in.||**|||~|200-MTC.jpg|~|MTC Touch had its infrastructure targeted during the 2006 war in Lebanon.|~|It is even more desert-like than the Gulf so there’s a huge amount of dust and dirt. This means any machinery can fail and that can also lead to problems and data loss.

The country, like other parts of the region, experiences huge extremes of temperature. It can be minus 40 at night and go up to 30 degrees during the day. Most equipment is not built to withstand such drastic changes, so Roshan had to find a vendor that offered equipment that could cope.

The company chose EMC partner application software – Symmetrix Remote Data Facility (SRDF) – which Khoja says offers full redundancy on all its data. All the customer registration information is stored with this, along with billing information and prepaid customer details – the majority of the customers are prepaid.

The CRM system works off that data so the system is absolutely vital. Along with this, the company stores Outlook information and ERP details, which are not time critical, on tapes.

Mohammed Amin, general manager at EMC MENA, has clearly been impressed with the company and says: “They’re building a disaster recovery infrastructure with a network and technology, but what they are really doing that differentiates them from other companies is implementing a high delivery process and high standards. It's very impressive."

So the data protection does not stop there for Roshan. “We have to be very careful from a security standpoint,” Khoja insists. “We spend about $10 million on security every year.

“We employ more than 2,500 guards to manage and protect our sites. We are not going to be an international peacekeeping force – that’s not our business. And we’re not going to be a terrorist organisation.

“However, we focus on the people of Afghanistan and fortunately or unfortunately, the people of Afghanistan will be represented by every faction that’s there whether it be the Taliban, government officials, or corrupt government officials.

“We can only change the world so much,” he insists. “Our job is to provide a high quality communications network that paying people, who follow the rules of the land – in other words give me a registration document and ID – can use. That doesn’t mitigate people lying or cheating – we can’t police that and it’s not our job.”

Khoja needs to prepare thoroughly to make sure that if one of his buildings is attacked the company will still function. Roshan has a committee board which the management team has to present disaster recovery plans and strategies to. “Tomorrow a mortar bomb could take out one of our buildings so we have to think of bombs, terrorist attacks, suicide attackers, other issues like electricity, what happens if we lose continuity,” says Khoja.

“How would we get from Kabul to Iran if there are no roads and no planes? We even have our own plane so that we can move data, customer information, SIM cards, scratch cards, spare parts and all other products to every part of the country. “Literally no part of Afghanistan should be more than four hours away from us.”

There are also dangers of earthquakes, which hit the country regularly and can reach up to six on the Richter scale. Only recently, Roshan endured such a natural disaster.

“Our network control centre is in the centre of Kabul,” Khoja explains. “It’s on the third floor of a very strong building. We had the building surveyed before we put our equipment in and it was earthquake-proof but that doesn’t mean it won’t shake.

“The man who came and fitted the cabling hadn’t done it to earthquake standards so the cabling length was short. During the earthquake the building started to shake and so did our machine. The cable stretched, came undone and sparked, causing a fire. That led to some data loss. We take this very seriously so we have full redundancy and automatic switchover and that’s exactly what happened on this occasion.”

There will be plenty of businesses throughout the region feeling Khoja’s pain right now. One company in particular that can easily empathise with Roshan is MTC Touch, one of the two licensees in the Lebanese telecommunications market.
For MTC Touch chief technical officer, Rula Abu Daher, the 2006 war was a testing time and the company’s resilience was pushed to its limits.

“It was entirely unexpected,” explains Daher. “It dealt a major blow to our efforts in the market. Some of MTC Touch’s facilities were specifically targeted during bombing raids on Lebanon. The effect at one stage was that 25% of our network capabilities were incapacitated during the height of the conflict.”

The resulting major communication difficulties hit the bulk of Lebanon’s key industries, and crucially the emergency services attempting to help the injured during the air strikes as well as the ordinary citizens looking to verify the safety of friends and family in the aftermath of an attack. ||**|||~|200-Sharma,-Vipin-----TRIPP.jpg|~|Sharma: Businesses need to build a solid power infrastructure that can give them availability.|~|“Our endeavours during this time were as much motivated by the national need as they were by commercial interests,” says Daher. “We were acutely aware that one phonecall could potentially save a life.

“When the conflict began it was clear that the country’s lines of communication were being targeted,.Our main objective at this time was to provide the continuity of our operations."

The extent of the damage stretched the length and breadth of Lebanon with bombs raining down on many major MTC Touch hub sites. At one stage there were literally hundreds of sites down at the same time across the north and south of the country.

“MTC Touch’s first priority was to formulate a strategy for our emergency response team to implement on-the-spot reparatory work in affected areas,” says Daher.

“The process involved lateral thinking with a dynamic approach. Many of our wireless hubs were badly damaged and all subsequent lines of communications connected to it were down as well. We responded to this by using fibre-optic cables to attach the affected wireless sites to the other hubs across the north of Lebanon.

“In the long term, operating in the Lebanese market requires a dynamic and versatile approach as it is often governed by this security situation,” claims Daher.

“This approach involves us cooperating with our competitors to circumvent the operational difficulties dictated by the ongoing security situation. For instance during the war we were forced to share a lot of equipment with our ‘rivals’ as many of the ports of entry were controlled by Israeli forces.

“Another strategy to meet these challenges is to charter flights for the express purposes of importing the equipment necessary to maintain and update our network,” says Daher.
For MTC Touch it has been more about a rebuilding process rather than disaster recovery but it is important to remember how crucial it is for all companies to be prepared, whether based on earthquake fault lines, in the heart of war zones, or not.

For Al Ameen Financial Brokerage, and hundreds of other businesses like it, the dangers of power and data loss might not be so blatant or frequent, but they still exist.
As far as the brokerage’s Nasreddine is concerned, planning for disaster is now a necessity for all.

“In our case, data loss is more than damaging it would be a disaster – a catastrophe,” explains Nasreddine. “This is because we are a financial company and we are speaking about valuable customers’ accounts and financial transactions. If for whatever reason loss of data happens, the cost will be a lot higher than if we spend money beforehand, installing a system to prevent data loss.”

He believes that the most important thing is to prevent data loss rather than waste time dealing with the consequences. “Data recovery costs both time and money, and sometimes data recovery isn’t even possible anyway, in which case the cost of damages would be incalculable.

“So protecting data is a must for our company and all companies involved in banking and finance. Loss of data is simply not acceptable in our industry regardless of the reason.
In order to protect its system, Al Ameen has installed Tripp Lite UPS System-3 nodules. During a power outage, the system will provide the brokerage with sufficient power to keep its servers running and allow employees enough time to save any work and shut down equipment.

Vipin Sharma, VP of EMEA sales at Tripp Lite, says: “When we try to design a power infrastructure for a sensitive or critical environment, whether it’s in the financial sector, healthcare, real estate, airport or railways, it’s important that people understand that power might not always be there. They need to build a solid power infrastructure that can give them availability – an infrastructure that can buy them time to make repairs and carry on working.”

Along with Al Ameen Financial Brokerage’s Tripp Lite solutions, a Sun blade backup system has been implemented and data is also saved onto backup tapes and DVDs and stored in a safe.

“This is not good. If there was a fire this could be a big problem,” Nasreddine says.
As with most companies though, the location of backup tools is simply down to financial restraints and, as Roshan’s Karim Khoja says: “You’re damned if you do and you’re damned if you don’t.”

He can see that spending on disaster recovery is crucial, but perhaps only up to a certain point, with 100% business continuity very difficult to achieve.

“You can spend an amount of money to get to 80% and then you can spend so much money to get the last 20% that it’s not worth the money. Right now I think we're at 70%," he adds.||**||

Add a Comment

Your display name This field is mandatory

Your e-mail address This field is mandatory (Your e-mail address won't be published)

Security code