Prometheus training interruptions per 8K GPUs per day: nearly 1,000 at the mid-2025 peak, about 10 after a 99% reduction (our arithmetic), and Meta's goal of 2

Meta Says Its Prometheus Cluster Will Grow to 1 Million GPUs

Meta said on 19 October 2026 that the first phase of Prometheus, its gigawatt-scale supercluster in New Albany, Ohio, is complete and that it is “working toward scaling it to comprise ~1 million GPUs,” in a post on its Engineering at Meta blog. The company says the first phase was used to train its Muse foundational models, including the Muse Wonder models.

Meta writes that Prometheus today “comprises an array of hundreds of thousands of GPUs across multiple interconnected buildings” and that “At full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep.” The post comes two days before Meta reports third-quarter results, which its investor relations notice puts after market close on Wednesday, 21 October.

Business Pill · GPUS

A short explainer on why AI runs on GPUs. It teaches the general idea only and says nothing about any company in this story.

The key insight: As we read it, Meta’s post treats reliability as a limit on scale: it reports a 99% cut in training interruptions as it moves Prometheus toward roughly 1 million GPUs.

What Meta Disclosed

The post says Prometheus “currently deploys two NVIDIA GPU platforms, with a third (based on NVIDIA’s Vera Rubin NVL72) arriving soon.” The first, which Meta calls Catalina, uses the NVIDIA GB200 NVL72; the second, Clemente, uses the GB300 NVL72; the third, San Miguel, will use Vera Rubin NVL72.

Meta lists Catalina at “~120 kW total system power across the two-rack pair”, Clemente at “~135 kW rack power” and San Miguel at “over 170 kW per rack”. It says Clemente gives a “5-10% performance improvement” over the GB200 NVL72, and San Miguel a “10-20% improvement in performance per watt” over the GB300 NVL72.

Meta says Prometheus spans “traditional and AI-optimized data center buildings and rapid deployment structures,” and that it integrates “leased capacity from strategic partners in addition to our own buildings”.

Prometheus training interruptions per 8K GPUs per day: nearly 1,000 at the mid-2025 peak, about 10 after a 99% reduction (our arithmetic), and Meta's goal of 2
Figures from Meta’s Engineering at Meta post “Inside Prometheus”, 19 October 2026. The 1.4 GW figure for GPU power alone is our arithmetic (roughly 1 million GPUs at the 1,400W Meta cites per GPU), not a Meta figure.

How the Cluster Is Wired

At the building level, Meta writes, “each Ethernet-based fabric interconnects up to 20,736 GPUs”. A separate aggregation network, which Meta calls BAG, connects the building fabrics “spanning 2 km to over 100 km” and is built with Arista 7800-series chassis running Broadcom Jericho3 silicon.

The post puts the whole network at “millions of miles of fiber strand, tens of thousands of switches, and millions of optics”. It says pre-training over those distances required “a dedicated optimization effort with Meta Superintelligence Labs”, given speed-of-light limits.

Keeping the Training Jobs Running

Meta says that “During the initial phase of Prometheus in mid-2025, our interruption rates reached a peak of nearly 1,000 events per 8K GPUs daily,” and that a year of engineering brought “a 99% reduction in interruptions”. Its stated goal is “two interruptions per 8K GPUs per day, which is ~100k GPU hours MTBI.”

For the Muse Wonder training runs, the post reports “more than 85% goodput for pre-training runs and more than 90% goodput for reinforcement learning (RL) runs” and a mean time between interruptions of “~47k GPU hours”.

Each 72-GPU rack domain, Meta writes, “runs a 64-GPU slice and keeps its remaining eight GPUs as in-rack spares.” It says firmware fixes can now reach “more than 100k GPUs within a few hours”.

Prometheus training interruptions per 8K GPUs per day: nearly 1,000 at the mid-2025 peak, about 10 after a 99% reduction (our arithmetic), Meta goal of 2
Training interruptions per 8K GPUs per day on Prometheus, from Meta’s 19 October 2026 post: a mid-2025 peak of nearly 1,000, a reported 99% reduction since, and Meta’s goal of two. The figure of about 10 is our arithmetic.

Power Swings and Cooling

Synchronised training, the post says, triggers “sudden load fluctuations of 30-40% within the cluster”, and “Initiating or pausing a major training job can shift loads of more than tens of megawatts in just seconds.” Meta says a power-management system it calls Dynamo Dimmer adjusts GPU power limits, backed by rack batteries, capacitor trays and facility battery storage.

Meta writes that it has “long partnered with local utilities to ensure we are paying the full cost of energy for our site” in New Albany. It says the cluster runs on a mix of air-assisted liquid cooling and facility liquid cooling, and that its newest AI buildings use closed-loop liquid cooling.

The Arithmetic

The post cites “the 1,400W load of individual GPUs” in its power testing. If each of roughly 1 million GPUs drew 1,400W, the GPUs alone would draw about 1.4 GW, before CPUs, networking and cooling. That is our arithmetic, not a Meta figure, and GPU power varies by generation.

A 99% cut from nearly 1,000 interruptions per 8K GPUs a day leaves about 10, by our arithmetic, against Meta’s goal of two. Keeping 8 of every 72 GPUs as spares sets aside about 11% of each rack, also our arithmetic.

Where It Sits in Meta’s Spending

Meta’s second-quarter release said: “We anticipate 2026 capital expenditures, including principal payments on finance leases, to be in the range of $130-145 billion, narrowed from our prior outlook of $125-145 billion.” The Prometheus post gives no cost for the cluster.

The post says Hyperion, Meta’s planned supercluster in Richland Parish, Louisiana, is being designed “from the ground up with insights from Prometheus in mind.”

Update, 19 October: China’s Sugon Puts a 100,000-Card Cluster on Show

China’s Sugon showed the physical system of Sugon 8000, which Global Times described as China’s first AI supercluster “capable of supporting over 100,000 domestically developed computing cards”, at the China National Annual Conference on High Performance Computing, Global Times reported on 18 October, citing CCTV News. CGTN placed the conference in Shenzhen, called the system Dawning 8000 and said it was “completed earlier this year”. Pandaily reported in July that it had already “made its public debut in physical form” at the World Artificial Intelligence Conference in Shanghai.

At the conference, Sugon presented results from 10 research applications run on the system, Global Times said, and “more than 300 key applications have been extensively optimized for Sugon 8000, spanning more than 30 research and industrial fields, with over 20 running on the full system.” Neither 18 October report we read gives the system’s chip model, total computing power or power draw.

As we read it, the two systems are counted differently: Meta describes Prometheus as “hundreds of thousands of GPUs” today on the way to “roughly 1 million distributed GPUs”, while Sugon’s figure is a 100,000-card capacity, and a card count says nothing about how much each card can do.

The Structural Read

As we read it, the rack has become the unit Meta plans around: each 72-GPU domain trains on 64 GPUs and holds eight as spares, about 11% of each rack by our arithmetic.

As we read it, power is the other limit the post returns to: Meta says starting or pausing a major training job can shift tens of megawatts in seconds, and it caps GPU power with a system it calls Dynamo Dimmer.

As we read it, the post gives a capacity figure two days before results but no cost: Meta’s 2026 capital expenditure outlook is $130-145 billion, and the blog does not say how much of that Prometheus takes.

Engineering at Meta, 19 October 2026

“At full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep.”

Three Implications

SPARES BUILT INTO EVERY RACK Meta says each 72-GPU domain runs a 64-GPU slice and keeps eight GPUs as in-rack spares.

POWER SWINGS AT CLUSTER SCALE Meta says synchronised training causes load swings of 30-40% across the cluster.

THREE NVIDIA GENERATIONS IN ONE CLUSTER Meta says Prometheus runs GB200 and GB300 NVL72 racks, with Vera Rubin NVL72 arriving soon.

The Business Engineer Lens

This story maps onto the Business Engineer framework The State of AI Data Centers.

The framework puts it this way: “The structural bottleneck isn’t computing, it’s electrons.”

As we read it, Prometheus shows that problem inside a single cluster: Meta reports load swings of tens of megawatts in seconds and says it pays the full cost of energy for its New Albany site. The framework was published in December 2025 and does not discuss Prometheus.

What Is Not Established

The post does not say what Prometheus costs, how many GPUs are installed today beyond “hundreds of thousands”, when the cluster will reach full scale, or how much of it is leased. It does not give the cluster’s total power draw; “gigawatt-scale” is Meta’s description. The performance and reliability figures are Meta’s own and we have not verified them.

Business Engineer Framework

The State of AI Data Centers

A Business Engineer framework on why power, not chips, sets the pace of the AI build-out.

Read the Map of AI →

The Bottom Line

Meta said on 19 October that the first phase of Prometheus, in New Albany, Ohio, is complete and trained its Muse models, and that it is scaling the cluster toward roughly 1 million Nvidia GPUs across GB200, GB300 and, soon, Vera Rubin racks. It reports a 99% cut in training interruptions since mid-2025.

95,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.

A note on sourcing. This piece is based on Meta’s Engineering at Meta post “Inside Prometheus” of 19 October 2026, Meta’s second-quarter 2026 results release and its investor relations notice of 12 October 2026, all read in full on 19 October 2026. The Sugon update draws on Global Times and CGTN (18 October 2026) and Pandaily (17 July 2026), read in full; Sugon’s figures are its own and those outlets’, which we have not verified. The performance and reliability figures are Meta’s own, which we have not verified. The 1.4 GW, interruption and spare-GPU figures are our arithmetic. Nothing here is a forecast, and nothing here is financial or investment advice.

Sources: Engineering at Meta, Inside Prometheus (19 Oct 2026) · Meta IR, Q3 2026 results notice · Meta investor relations (Q2 2026 results release) · Global Times: China unveils first 100,000-card AI supercluster hardware (18 Oct 2026) · CGTN: China’s 100,000-card supercluster (18 Oct 2026) · Pandaily: Sugon Dawn 8000 debuts at WAIC 2026 (17 Jul 2026)

Scroll to Top

Discover more from FourWeekMBA

Subscribe now to keep reading and get access to the full archive.

Continue reading

FourWeekMBA