A Hidden Software Bug From July: Microsoft Finally Reveals What Triggered Global Outlook Outage

Teams, OneDrive and SharePoint could route around the failure, but Outlook's email backbone could not, which is why mailboxes stayed down the longest, Microsoft said.

Advertisement
Read Time: 3 mins
Microsoft says a bug that lay dormant for months, a surge in traffic and missing server capacity combined to knock out Outlook worldwide.
Unsplash

Microsoft has said a software defect rolled out in July, combined with a spike in traffic and reduced server capacity, caused the global outage that hit Outlook and other Microsoft 365 services on November 12.

Charles Lamanna, executive vice president of Copilot, Agents and Platform at Microsoft, laid out the cause in a post on X. He said the three problems together brought down the networking gateway that sits in front of several Microsoft 365 services.

Advertisement

According to Microsoft, the defect caused servers to become unresponsive under certain load conditions, due to lock contention in an internal process. It was part of a fix for the company's Microsoft 365 networking gateway servers, which began rolling out in July and was deployed worldwide over several weeks. The company said the flaw caused no noticeable problems for more than three months and was not caught during routine load testing.

ALSO READ | Microsoft 365 Users Targeted By Fake Outage Notifications In Phishing Campaign

Microsoft said it runs two independent gateway systems to limit disruptions. About 15 hours before the outage, Teams moved from the alternative gateway to the primary one as part of regular failover operations. This raised the load on the primary gateway and changed its traffic patterns.

Advertisement

At the same time, roughly 14 percent of the gateway's server capacity was unavailable because of an unrelated issue from the previous day, the company said.

Requests then began to build up on the affected servers until they stopped responding. Microsoft said its traffic management systems redirected users to other servers, which added to the strain. At the height of the outage, more than 80 percent of the gateway servers were unresponsive.

Advertisement

ALSO READ | Microsoft Outage: Services Partly Restored As Microsoft Restarts Affected Infrastructure

The gateway is divided into thousands of capacity units organised by region to keep failures contained. Microsoft said a separate issue stopped those safeguards from working, allowing the problem to spread globally.

Outlook Slowest To Recover

Teams, OneDrive, SharePoint and Office apps were able to switch away from the failing gateway. Exchange Online, which powers Outlook email, could not, Microsoft said, which made restoring mailboxes harder. Servers that were brought back were also overwhelmed by queued requests and went down again.

Fix Rolled Out In Stages

Once the defect was identified, engineers returned the affected code to its earlier working state. Microsoft said it took servers out of service in groups, applied the update, restarted them and returned them gradually while monitoring stability.

Lamanna said the company's final post-incident review, with further details, is expected later this week.

ALSO READ | Microsoft Outage Disrupts Outlook, Teams, Copilot: Seven Things You Should Do Right Now

Essential Business Intelligence, Sharp Market Insights, Practical Personal Finance Advice, Daily Fuel, Gold and Silver Prices and Latest Stories — On NDTV Profit.


Loading...