A latent bug is a flaw that sits in production code without causing harm until an unusual condition runs it. Microsoft said one was behind its 12 to 13 November Microsoft 365 outage, and Amazon Web Services and Google Cloud have described similar sequences in their own post-mortems of earlier outages.
Microsoft’s final service-health entry gives a start of 16:20 UTC on 12 November and an end of 16:10 UTC on 13 November, 23 hours 50 minutes by our arithmetic. Across three precedent incidents, from AWS in December 2021 and October 2025 and Google Cloud in June 2025, the median duration in each vendor’s own report was 11 hours 10 minutes, so Microsoft’s recorded window was about 2.1 times that median, by our arithmetic.
Business Pill · Embedded Dependency
A short explainer on embedded dependency: what sits inside the systems you buy. It teaches the general idea only and does not discuss Microsoft or the outage.
The key insight: As we read the four reports, the number that matters is the tail: in the October 2025 AWS event, 11 hours 55 minutes passed between fixing the DNS fault and the stated end, by our arithmetic, so a cascade’s length depends less on the bug than on how fast queues drain and systems return.
Microsoft’s preliminary root cause for incident MO1550821, in its final service-health entry of 14 November as republished by the University of Pennsylvania and Cloud Admin Hub, says “traffic shifts during standard network maintenance and upgrades exposed a latent bug in which our systems were unable to complete new connection requests, causing a rapidly increasing number of requests to queue for processing”.
The entry continues that “the affected servers exhausted their available processing capacity and could no longer accept new connections, resulting in widespread service access failures”. The failure sat in “the systems responsible for establishing secure connections to the service”, Microsoft wrote. As we read it, that is four stages: a latent bug, a routine trigger, a queue that grows faster than it drains, and capacity exhaustion. Our earlier root-cause piece covers the full timeline.
Hours from each incident’s stated start to the vendor’s main recovery point and to the last recovery its own report states. Microsoft’s first point is its recorded end; its second is its 14 November X post. Our arithmetic from vendor times.
Stage 1: The Latent Bug
In its summary of the 7 December 2021 us-east-1 event, AWS said its networking clients had back-off behaviour designed for congestion, but “a latent issue prevented these clients from adequately backing off during this event”. AWS added: “This code path has been in production for many years but the automated scaling activity triggered a previously unobserved behavior.”
For the 19 to 20 October 2025 DynamoDB disruption, AWS wrote that “The incident was triggered by a latent defect within the service’s automated DNS management system that caused endpoint resolution failures for DynamoDB”, and called the root cause “a latent race condition in the DynamoDB DNS management system”. Google’s June 2025 incident report describes a new feature whose “code path that failed was never exercised during this rollout due to needing a policy change that would trigger the code”.
Stage 2: A Routine Trigger
As we read the reports, the condition that ran the flaw was ordinary work each time. Microsoft named traffic shifts during standard maintenance. AWS said that in 2021 “an automated activity to scale capacity of one of the AWS services hosted in the main AWS network triggered an unexpected behavior from a large number of clients inside the internal network”. Google said a policy change with “unintended blank fields” was replicated globally within seconds and sent its Service Control binaries into a crash loop.
Stage 3: Queues and Capacity Exhaustion
The bug alone does not explain the scale. In Microsoft’s case, requests queued until servers ran out of processing capacity. An update at 19:30 UTC on 12 November said “the elevated load isn’t driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure”, per the PennO365 copy of the record.
AWS described the same loop in 2021 as “a large surge of connection activity that overwhelmed the networking devices” between its internal and main networks. In October 2025, AWS wrote that one EC2 subsystem queued more work to retry leases until it “had entered a state of congestive collapse and was unable to make forward progress”. Google said restarting tasks in large regions “created a herd effect on the underlying infrastructure it depends on”, and that “Service Control did not have the appropriate randomized exponential backoff implemented to avoid this”.
Stage 4: The Long Recovery Tail
Fixing the trigger did not end any of these outages. Microsoft reported at 00:00 UTC on 13 November that servers returned to service “can become unresponsive again once they begin receiving traffic”, and its final entry says “Given the scale of demand, we restored services gradually, validating stability at each stage”. It extended the email timeout window to 48 hours to protect delayed messages, an earlier update said.
AWS resolved the DynamoDB DNS issue at 2:25 AM PDT, 2 hours 37 minutes into the event, but the event ran until 2:20 PM PDT, 11 hours 55 minutes later, as EC2 and load-balancer systems caught up, by our arithmetic from its summary. In 2021, AWS network devices fully recovered at 2:22 PM PST, and EventBridge had elevated delivery latency until 6:40 PM PST while it processed a backlog of events.
How the Durations Compare
We measured each precedent from its stated start to the end its vendor states for the event or, where no end is given, to the last recovery time in its report. On that basis the three ran about 7 hours 33 minutes (Google, June 2025, to Vertex AI Online Prediction’s recovery at 18:18 PDT), 11 hours 10 minutes (AWS, December 2021, to EventBridge’s backlog clearing at 6:40 PM PST) and 14 hours 32 minutes (AWS, October 2025, the event window AWS states; some Redshift clusters were restored later, by 4:05 AM PDT on 21 October, AWS wrote).
The median is 11 hours 10 minutes, by our arithmetic. Microsoft’s stated window of 23 hours 50 minutes is about 2.1 times that; to Microsoft’s X post on 14 November that it had restored the remaining customers, 37 hours 49 minutes, it is about 3.4 times.
CrowdStrike’s July 2024 incident follows a different shape and is not in that median. Its preliminary post incident review says that “Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data”, and that the resulting exception “could not be gracefully handled, resulting in a Windows operating system crash (BSOD)”. The update was live from 04:09 to 05:27 UTC on 19 July 2024, 78 minutes; the review gives no end time for customer recovery, so its window does not measure the same thing.
The Structural Read
As we read Microsoft’s updates, its outage followed the same stages as the AWS and Google reports: servers returned to service could fail again once traffic arrived, so recovery was staged.
As we read the comparison, the 2.1 times figure says Microsoft’s stated window was long against these precedents; it does not say why, which the post-incident report Microsoft has promised may explain.
AWS post-event summary, 10 December 2021
“This code path has been in production for many years but the automated scaling activity triggered a previously unobserved behavior.”
Three Implications
FOR ENGINEERING TEAMSGoogle said it will audit and ensure its systems employ randomized exponential backoff; AWS said in 2021 that a latent issue stopped its clients backing off.
FOR IT BUYERSMicrosoft said it would provide a preliminary post-incident report within two business days and a final one within five business days.
FOR RISK TEAMS The three precedents ran from about 7 hours 33 minutes to 14 hours 32 minutes, by our arithmetic, against Microsoft’s stated 23 hours 50 minutes.
The framework says: “The value proposition shifts from features to integration depth.”
As we read it, integration depth is also why a fault in one shared layer, such as the connection systems Microsoft described, reaches so many services at once. The framework predates the outage and does not discuss it.
What Is Not Established
Microsoft’s account is preliminary. It said it would provide a preliminary post-incident report within two business days and a final one within five; by our count that points to about 17 and 20 November, and the full root cause may differ from the summary quoted here. The four windows are each vendor’s own measure and are not defined the same way: Microsoft’s and the October 2025 AWS figure are the start and end each vendor states, while the December 2021 AWS and June 2025 Google figures run to the last recovery time each report gives.
Microsoft’s record header also shows an earlier start time, 10:05 UTC, than the 16:20 UTC in the entry’s text; we use the text. Three precedents are a small sample, and the median is ours.
This Week In The AI Supercycle
AI Lost Its Central Banker, the Lab Oligopoly, and the Future of Open Source
This week’s issue: the Nvidia leadership change, read as the AI build-out losing its central banker: who now finances the GPU landlords, backstops the data centres and sponsors open models, and why two labs are becoming the bank.
Microsoft and AWS have described outages that began with a latent bug, and Google one that began with a code path never exercised in rollout; in each, routine work ran the flaw, and queues, retries or restarts grew the impact, as the vendors told it. Microsoft’s stated 23 hours 50 minutes was about 2.1 times the 11-hour-10-minute median of three precedents, by our arithmetic. Who the outage reached is in our tally of 84 named organisations, what uptime promises allow is in SLA Uptime Explained, and the wider week is in 3 Outages, 46 Hours in Two Days.
95,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.
A note on sourcing. Microsoft’s root cause and timeline come from its service-health record for MO1550821, read through the University of Pennsylvania’s PennO365 copy and Cloud Admin Hub. The precedents come from each vendor’s own report: AWS’s post-event summaries for December 2021 and October 2025, Google Cloud’s June 2025 incident report and CrowdStrike’s preliminary post incident review. Durations, the median and the ratios are our arithmetic from the times those reports give. Microsoft has not yet published its post-incident report.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
