It was one of three root causes for the outage that affected business, hospitals, governments and other organizations worldwide.
Microsoft has revealed more details about the outage that took services like Outlook, Teams and Copilot 365 offline, affecting businesses, hospitals, law enforcement and other organizations around the world. Charles Lamanna, the company’s EVP for Copilot and AI Agents, explained on X that there were three root causes for the outage.
He said that Microsoft typically maintains two network gateway systems that connect separate systems and allow data to pass between them. One of those gateways wasn’t working for Microsoft Teams, and it overwhelmed the primary gateway. It also just so happened that nearly 14 percent of the capacity for that gateway was out of service due to an unrelated issue that happened the previous day. Those two factors caused that gateway to encounter a latent bug that’s been in Microsoft’s systems since July.
Together, the higher load that overwhelmed the gateway, the reduced server capacity and the latent bug led to a build up in traffic requests until Microsoft’s servers gave out and stopped functioning. This all happened on a Thursday afternoon, during peak hours of use in the US.
The outage lasted for almost a full day. Microsoft worked on fixing the issue overnight, but the bug was months-old at that point, and the company was unable to simply do a rollback in order to quickly undo the changes it had caused. In addition, its servers were overwhelmed by waiting traffic after returning to service and became unresponsive again.
“We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations,” Lamanna wrote on his post. These details are from Microsoft’s preliminary post-incident report, and the company said it will release more details in the final report to be released later this week.