Neville Ray (pictured), president of technology at T-Mobile US, which is largely controlled by Deutsche Telekom, admitted in a note of apology to customers that “the trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast”.
Worse, the company’s back-up services also failed, admitted Ray. “This redundancy failed us and resulted in an overload situation that was then compounded by other factors,” he said.
Ray did not identify any of the fibre companies involved in the outage, which particularly affected voice over LTE (VoLTE) connections.
“This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls,” he said. T-Mobile US has worked with suppliers “to add permanent additional safeguards to prevent this from happening again and we’re continuing to work on determining the cause of the initial overload failure.”
Meanwhile T-Mobile US will have to face the wrath of the Federal Communications Commission (FCC), which said the outage was “unacceptable”.
Chairman Ajit Pai said on Twitter: “The T-Mobile network outage is unacceptable. The FCC is launching an investigation. We’re demanding answers – and so are American consumers.”
Reports pointed out that T-Mobile paid a US$17.5 million fine for two nationwide outages six years ago, together lasting three hours. This week’s breakdown lasted around six hours.
Ray said that during the outage “data connections continued to work, as did our non-VoLTE calling for many customers and services like FaceTime, iMessage, Google Meet, Google Duo, Zoom, Skype and others allowed our customers to stay in touch”.
He added that “many customers were able to use circuit-switched voice connections and customers on the Sprint network were unaffected”. T-Mobile is in the process of integrating the Sprint network, which it took over earlier this year.
Meanwhile SoftBank, which owned Sprint before the merger, appears ready to sell $20 billion of its remaining shares in the company. The likeliest buyer is Deutsche Telekom, which has 40% of the combined T-Mobile/Sprint business but has effective control as SoftBank assigned management of its shares to the German company.
SoftBank said in March that it plans to sell off assets worth $41 billion to buy back shares and cut debt. SoftBank lost $13 billion last year.
