Rogue computers acted alone to cause Hong Kong rail failure and MTR Corp’s top engineer doesn’t know why
Rail operator at loss to explain uncontrolled transfer of large amounts of information that brought chaos to city on Tuesday morning
The worst breakdown to ever hit Hong Kong’s railway was caused by rogue computers operating without orders, the MTR Corporation’s chief engineer said on Wednesday.
Initial investigations have found an uncontrolled transfer of large amounts of information between computers was responsible for the six-hour long disruption, which caused chaos for the city’s commuters on Tuesday morning.
But Tony Lee Kar-yun, the rail operator’s top engineer, said he still did not know why the system failure had happened.
“It was done by the computers,” he said. “No one gave [the machines] an order.”
On Tuesday, the signalling system on four major lines – Island, Tsuen Wan, Kwun Tong and Tseung Kwan O – stopped working shortly before rush hour.
Lee said a preliminary investigation by MTR technicians, and the supplier of the signalling system, showed the abnormal data transfer started on the Tsuen Wan line, where the test of a new signalling system had been carried out the night before.
But, Lee said they “couldn’t see any correlation between the signal failures and the testing of new signalling system [on the Tsuen Wan line]”.
Speaking on a radio programme, Lee said: “Analysis of the existing data connected by our technicians and the signalling system supplier show that, at the time [of signal failures], computers controlling different sections of the rail lines were transferring large amount of data among themselves.
“They were doing something we called ‘synchronisation’. It occupied large amounts of computing resources and made the computers unstable. We believe this was related to [problems in] programmes and setting.
“We have never seen this kind of data synchronisation before. By 4.30pm on Tuesday, we had successfully switched back to the existing system.”
According to Lee, the MTR Corp will send all the relevant data to the supplier’s headquarters in France for, “more in-depth analysis”.
The railway operator has also unlinked the systems controlling the Tsuen Wan, Kwun Tong, and Island lines after the disruption.
“Trains running across the lines might be affected, but it won’t be a big problem,” Lee said.
Alan Cheng Kwan-hing, deputy chief of operating, said MTR staff had found out about the signal failures at about 5.30am on Tuesday, and tried to conduct emergency repairs before the first train was scheduled to run at 6am.
“But the situation was still pessimistic at around 5.50am,” Cheng said. “So we reported to the Transport Department immediately, and issued a code red alert at around 6.20am.”
Cheng did not explain why it took the corporation 30 minutes to issue the alert, but promised the company would be reviewing its procedures.
As for complaints from passengers over the length of delays, compared with what MTR had estimated, Cheng admitted some trains came “a little bit later than predicted”.
“Our estimation was an average number, and trains needed time to pick up passengers and close their doors,” Cheng said.
Tanya Chan, a Civic Party lawmaker sitting in the Legislative Council’s railway subcommittee, said the MTR Corp should suspend overnight testing of the new signalling system until it knew what had caused Tuesday’s chaos.
“Can the MTR Corp tell Legco clearly how to guarantee a smooth switch between the two systems during the test period?” Chan asked on the same radio programme. “Moreover, can we rely on the new system after it is fully in place, or we will have to keep the old system on stand by?”
Chan Han-pan, deputy chairman of the subcommittee, said the body planned to discuss the signal failure by the end of October.
“We were never told the signalling systems of the lines were connected,” Chan Han-pan said. “And between 4.30am, when they finished the overnight test, and 6am [on Tuesday], I didn’t see that any emergency response plan had been activated.”
Both Tanya Chan, and Chan Han-pan, said the mechanism for fining the MTR Corp for service delay and disruption should be reviewed, and stepped up.
“The MTR Corp was only fined HK$2 million for more than 10 hours of delays on the Kwun Tong line last year, because only the 83 minutes’ delay suffered by the most seriously affected train was counted according to the mechanism,” Tanya Chan said.
“Over all these years, fines given to MTR for service delays haven’t reached HK$100 million.”
According to the service performance agreement between the MTR Corp and the government, introduced in 2013, the railway operator can be subject to a fine for service disruptions that last for more than 31 minutes.
In 2017, the MTR Corp also agreed to raise the maximum penalty for each incident from HK$15 million, to HK$25 million.
Chan Han-pan agreed that there were “loopholes” in the mechanism.
“I once suggested we amend the mechanism by taking the total hours of limited service into account, while lowering the fine for each hour of delay,” he said. “But every time we brought these issues to the MTR Corp and the government, they would come up with a lot of excuses, and eventually dismissed any change.”
Speaking separately on Wednesday, Secretary for Transport and Housing Frank Chan Fan said the government would “discuss the penalty system with the MTR Corp to see what we can do to help”.