When disaster strikes, telecoms giant AT&T has a Plan B

Elite engineers re-establish lines of communication across the globe in times of crisis

PUBLISHED : Thursday, 04 December, 2014, 4:12pm
UPDATED : Thursday, 04 December, 2014, 4:12pm

We live in an age when the ability to communicate is everything. Keeping us all calling, texting, tweeting and instant messaging are huge international telecoms networks that constitute the global internet. But what happens when they get overloaded or damaged?

From an all-out terrorist attack to unexpected solar storms, deadly hurricanes and tidal waves, most big telecoms companies have a back-up plan for wherever and whenever disaster strikes. Where that plan is carried out from, however, might surprise; the south of England.

In leafy surroundings, a secluded warehouse doubles as the Global Network Disaster Centre for telecoms giant AT&T. When a network node goes down anywhere in Europe, the Middle East or the Asia-Pacific region, 14 cable-heavy, server-packed trailers are deployed instantly to plug the gap.

The stakes are high because AT&T's network is responsible for a big slice of global telecoms and internet traffic. This US$140 billion network relies on 155 sites in 60 countries and contains more than 1.5 million kilometres of network fibre. Its 116 million wireless customers, 649,000 Wi-fi hotspots and 38 internet data centres transmit a staggering 67 petabytes of data every day.

The trailers parked in the warehouse are just 5 per cent of AT&T's global rescue fleet, though most of them are US-based, so not as accessible to Asia in terms of time zones and flight schedules; England is much better placed. Some of the trailers are on wheels, others are designed for 747 cargo holds, but together they create a network node capable of handling 15 terabits per second.

After an earthquake, quickly restoring the network with a pre-configured drive-in solution like this can be a life-saver. "Before we deploy we forward the configurations of the lost site," says Justin Williams, Network Disaster Recovery (NDR) - International, AT&T Network Emergency Management, Preparedness and Response.

"While it's on the network it inherits that configuration and leaves this warehouse as a replica of the lost site."

His nightmare scenario - but one he and his team are well prepared for - is known as the "smoking hole". "We obviously monitor breaking news, but with a disaster such as the World Trade Centre you can see the call volumes in the area spike and you know instantly that there's been some major incident. That's when we deploy - it's so sudden."

There is often some warning; if a government raises its threat level then that triggers activity.

If the smoking hole of Ground Zero was a wake-up call for the telecoms industry in 2001, it was Hurricane Katrina in 2005 that provided a timely reminder for it to get a Plan B in place permanently.

"We deployed 3,600 generators, established 30 lines for emergency response teams in just 30 minutes, and provided communications for an emergency pop-up prison," Williams says.

A key deployment for the team was the 2010 earthquake in Santiago, Chile. "The country's mobile telecoms infrastructure suffered quite badly, and that's not uncommon," Williams says. "Even in parts of the US, when storms come through, a lot of mobile masts are destroyed and we need to respond quickly and put the capacity back in."

In the US - AT&T's home - its NDR team supports local and federal events where cellphone coverage isn't good; it put a pop-up mobile phone network around a campsite after a landslide to help out emergency response agencies, and a mobile cell into the middle of a forest to help a manhunt in an area without cellphone coverage.

However ready and well prepared it is to resuscitate disaster-struck networks and replace flattened mobile masts, AT&T's NDR team rarely gets on the first plane into disaster zones.

"The airport was closed to anyone other than those on a humanitarian mission, but we have groups that work with governments and we were on the first non-humanitarian flight into Santiago," Williams says.

There really is only one team in a crisis, and telecoms do tend to reach out to support each other if they can
Justin Williams, AT&T Network Emergency Management, Preparedness and Response

If mobile masts are down, AT&T gets to work. Its hardware effectively replaces the network node that's been struck off by an "event", but there's more to NDR than simply protecting company assets or brand image. "There's a feeling of mutual aid," says Williams, who admits he's helped disaster response teams and even competitors while in disaster zones. "There really is only one team in a crisis, and telecoms companies do tend to reach out to support each other if they can - we're all engineers trying to fix a problem, so we work together."

However, NDR isn't just about getting dial tones in disaster zones. Away from earthquakes, tidal waves and hurricanes, AT&T also plans for impacts on specific nodes of its network during social and political events when demand is high from the raised level of texting, tweeting, phone calls and internet activity.

"Anything that can have an impact on the function of the network, from the protests in Hong Kong, to bird flu and the unrest in Venezuela," Williams says. "I have responsibility for business continuity planning, so we were reaching out to our teams in Hong Kong to assess the impact on the ground, and working out how we can move around and get access to buildings to figure out if we can fix problems on the network."

Some threats are known about, and get assessed long before they could occur.

"We put a plan in place for major events," Williams says. "We look at lot of political and sporting events globally, particularly mass-transit of people that could create an issue," he says, name-checking the recent Nato summit in South Wales, the World Cup, the Olympic Games, as well as predicted protests.

Every trailer is dripping with fibre-optic cables. "We sort the equipment in the trucks and then we flood it with fibre, so that when we deploy we don't have to spend a lot of time cabling things up."

This is where Williams believes that AT&T is ahead of the curve.

"If you do have a smoking hole scenario and you've lost everything, you're starting from scratch and you've got to find somewhere to put your equipment," he says. "Then you've got to rack it, stack it, power it, cable it, air condition the environment and design it - and for a typical PoP site that would take from 12 to 18 months," he says.

"All that is pre-done here, so it takes us only a few days. There is still a lot of work that needs to be done, but it gives us a good shot at recovery."

The pre-cabling is astonishing; stand inside one of the mobile PoP trucks and the cables are neatly stacked overhead and even below. They're all packed with the latest Cisco routers capable of high-speed packet switching across the network, each capable of handling 140GB per second.

The NDR team is used to working in all kinds of temperatures, but inside the trucks it's always cold. Each of the routers needs a cubic metre of cool air every second just to keep from overheating.

No wonder, then, that the NDR takes all of its power with it; there is just no way it can afford to rely on local electricity grids, some or all of which may be down after a disaster.

"The objective is to recreate what was there before and lost," Williams says.

"It's not about creating emergency communications."

He describes it as an onerous task, though he is sitting on a US$16 million collection of gear that's already set up and can be driven or flown in.

"If you're talking about a smoking hole scenario," he says, "this is the only game in town."