All at sea
Wilfred Kwan Tai-lai, president and chief technology officer at undersea cable operator Asia Netcom, looked forward to a pleasant dinner at a fine restaurant on Boxing Day.
Andrew Kwok Wing-pong, vice-president of international business for Hutchison Global Communications (HGC), was keen to take some rest after a seven-hour road trip from Manila to spend his holiday at the mountain resort community of Baguio in the northern Philippines.
The two executives were soon to be involved in a regional hi-tech disaster that quickly had global implications, spending the rest of their holidays scrambling to help prevent a modern-day telecommunications breakdown of unprecedented scale as business and financial firms in Hong Kong and the region faced crippling internet hold-ups.
At about 8.30pm on December 26, an earthquake measuring 7.1 on the Richter scale occurred off the southern coast of Taiwan. The quake and the strong aftershocks knocked out power, damaged buildings and caused a few casualties and dozens of injuries in southern Taiwan.
The tremors were felt across Taiwan and in the Pearl River Delta region, including Hong Kong, Macau, Guangdong and Fujan .
Mother nature's rumble - reportedly the strongest earthquake in 100 years to hit the Luzon Strait which links the South China Sea and the Philippine Sea - also damaged six regional submarine cable systems and disrupted telecommunications services across Asia.
Mr Kwan managed to finish his dinner that evening, but only after making several phone calls to the operations team of Asia Netcom, which runs the 19,800km East Asia Crossing (EAC) fibre-optic undersea cable system that links Hong Kong, the mainland, Taiwan, the Philippines, Japan, Korea and Singapore.
'We prayed that if nothing happens in the next 12 hours, we should be OK,' Mr Kwan said of reports coming to him at the time. 'Unless there was a really massive earthquake, the cables on the seabed would normally stay intact.'
Not this time. At about 2am on December 27, Mr Kwan received the news that Asia Netcom's operations team needed to scramble after an incident - an interruption of services - was reported. Asia Netcom has about 40 customers, including telecommunications carriers, internet service providers and various commercial enterprises.
'Things were back to normal by 2.45 that morning,' Mr Kwan said, noting that Asia Netcom's operations team of more than 20 people performed needed software reconfiguration and rerouting work. No physical work was needed on the EAC system, which lies about 4km under the sea surface.
But at about 11am that day, the cable operator's worst fears were realised. Its EAC system was cut, a direct result of the Taiwan earthquake.
In Baguio, the two handset batteries Mr Kwok carried were all used up making several conference calls on Boxing Day evening and during the trip back to Manila the next morning. 'By the time I reached my office in Manila it was too late. The message I received from my people was to the point: 'cable break',' Mr Kwok said, noting that what happened next was the gradual system outage of several strategic international communications links for Hong Kong.
The full extent of the damage and disruption to business and commerce is only now fully understood and repairs are now under way in trying conditions in the sea off Taiwan. Mr Kwan said the deep-sea restoration work, covering an area 300km by 150km, involved dealing with submarine cables 'spread out like spaghetti strands' on the ocean floor. There are about five cable-repair ships deployed in the area now sorting through that pile.
'Getting hold of the correct damaged cable takes a few runs, but there are cases when 50 to 100 tries are made,' Mr Kwan said. Weather permitting, the repair ships must pick up the two ends of the severed cable. All repair work and testing is done on board. Once patched up, the cable is returned to seabed 4km below.
'It's tough out there. Hopefully, our boys will go home in about three to four weeks,' Mr Kwan said.
The key communications links broken included the Asia-Pacific Cable Network (APCN), an old 12,000km undersea cable system that handles small traffic due to its transmission capacity of only 5 gigabits per second. Another failure was the 19,000km APCN-2, which handles plenty of traffic because of its bandwidth capacity of 2.56 terabits per second. The Flag North Asia Loop and Reach North Asia Loop cable systems, which together total more than 10,000km in length and provide multi-terabit capacity, also failed.
Also down was the China-US Cable Network, which covers a distance of 30,000km and has a capacity osupplies about ; the C2C cable network, which is about 17,000km long and provides 7.56Tbps; the Flag Europe-Asia cable system, which covers about 28,000km; and the EAC cable system.
To make matters worse, the Southeast 39,000km Asia-Middle East-Western Europe 3 system had gone down two weeks before the earthquake.
Across Asia, people struggled to come to terms with the extent of the damage wrought on telecommunications network operators and their users. External telecommunications services in Hong Kong, including internet access to overseas websites, international direct dial and cellular roaming calls were badly affected.
Assistant director (regulatory) of the Office of the Telecommunications Authority (Ofta), Chan Tze-yee, said it was the 'most serious damage' Hong Kong had seen in overseas communications and the impact was 'unimaginable'.
According to reports submitted to the Ofta by network operators on December 27, telecommunications users faced severe congestion in external services.
PCCW said its capacity was down by 50 per cent at the peak of the crisis, while i-Cable Communications said its capacity was down to 10 per cent.
The Securities and Futures Commission said some licensed intermediaries - brokers and fund managers - experienced difficulties in communicating with their clients or executing orders.
'By 9.30am, a huge number of clients phoned in to ask us what the problem was,' said Tai Fook Securities Group spokesman George Lau Chor-yuen of the reaction by businesses on the Wednesday. He said the worst affected clients were overseas customers from North America, Singapore and Australia, all of whom were unable to access the trading house's website to view trading information.
Trading and clearing systems for company shares and futures were not affected at Hong Kong Exchanges and Clearing.
While some banks reported significant disruptions to their services, the Hong Kong Monetary Authority said the monetary system was functioning smoothly. The authority had received reports from some banks only of some slower than usual response times for some internet banking services, a spokesman said. HSBC said many customers' connections to some of the bank's internet banking services had been affected. This included personal internet banking, business internet banking and HSBCnet.
Investment banking business was less affected because its operational platform was based outside of China.
The effects were widely felt on the mainland, where internet access to all overseas websites ending in .com was severed. In an effort to ease congestion, the Ofta urged internet users to 'minimise non-essential downloading of large files from overseas websites and non-essential activities that demand large bandwidth over international connections'.
Still, some criticism was levelled at the authority because of a statement made on December 27 by Ofta director-general Au Man-ho. 'It is expected that some of the submarine cables will take at least five days to repair,' he said before the full extent of the damage was known.
Deputy director-general Ha Yung-kuen said the remarks were based on the input of local operators, who had analysed the situation based on their past experience. 'Of course, it has turned out to be much longer than that,' Mr Ha said.
Ofta said repairs to the damaged system were expected to be completed progressively by the end of this month or the beginning of February, if the environment and the weather permitted.
Over the past week there has been continued improvement in internet access. Last Thursday Ofta said the major ISPs had recovered about 80 per cent of their international connection capacity. IDD and roaming services have almost resumed normal.
Fixed-line network operator Hong Kong Broadband Network (HKBN) claimed it was one of the first service providers to restore overseas telecommunications service. On Sunday it took the opportunity to take a swipe at local competitors, launching a programme called 'bbAid' to attract subscribers of rival PCCW's Netvigator service.
But rather than spark competition, the incident has brought calls for increased co-ordination between rival local players and their counterparts in other markets.
Mr Ha said Ofta last week met the highly competitive undersea cable system operators and local carriers in a bid to push for greater co-operation and dialogue in the industry.
'We are kicking off a discussion between working groups formed by these providers,' Mr Ha said. 'We are looking to improve the reporting mechanism and pursue other action items to ensure greater reliability in gathering data and providing information to the public.'
Charles Mok Nai-kwong, president of user group Internet Society Hong Kong Chapter, said that move stemmed from the earthquake experience when 'nobody had a handle of the whole picture'.
There ought to be a code of practice set up so the government can ensure 'correct and up-to-date information', Mr Mok said, adding that increased investment in new capacity and backup networks was needed.
'We dodged a bullet this time because the disruption happened during the holidays,' said Terence Ronson, managing director at specialist hotel systems integrator Pertlink.
One of the potential action plans would involve direct dialogue between the Hong Kong government and other governments, especially those across Asia, to ease the handling of another regional telecommunications infrastructure outage.
Information technology sector lawmaker Sin Chung-kai said there was not much Ofta could do about the disruption. 'Hong Kong by itself cannot build up an emergency co-ordinating system; we are talking about something beyond Hong Kong, it's an inter-regional matter,' Mr Sin said.
Such a co-operation is also supposed to extend to alternative carriers of telecommunications traffic, such as the satellite industry.
Simon Twiston Davies, chief executive of the Cable & Satellite Broadcasting Association of Asia, said the vulnerability of undersea cables meant that companies needed to seriously consider satellite backup options.
Mr Kwok confirmed that development, but said fibre-optic networks provided the most bandwidth capacity and direct access points for critical information to be sent and delivered by internet users. 'The internet is a major challenge because it is a capacity-intense application, unlike voice.'
'What more operators should do before any crisis strikes is to plan for new routes - just like what airlines and logistics service providers do - and invest in cable diversification. I think Hong Kong operators should also get more involved in regional cable infrastructure planning, pushing for more active dialogue with the submarine cable system providers.'