Meta's 1M-GPU Prometheus Cluster Cuts AI Training Interruptions by 99% in Ohio

Meta has published technical details regarding its Prometheus supercluster, a massive computing facility located in New Albany, Ohio, which was used to train the company’s Muse family of foundational models. The engineering blog post outlines the scale and operational challenges of this infrastructure, which is designed to support large-scale artificial intelligence development.

Root cause

The primary challenge in operating a cluster of this magnitude is maintaining signal integrity across hundreds of thousands of connections. Meta reported that signal quality issues on over 600,000 NVLink connections between graphics processing units were the biggest early problem. As the company scaled up to newer hardware platforms, memory faults also became a significant operational hurdle, requiring extensive engineering efforts to resolve.

Attack path

While not a security breach, the failure mode involves hardware faults halting entire training runs. A single machine failure can stop a whole process, leading to thousands of interruption events daily in mid-2025. Meta stated that a year of engineering work cut these interruptions by 99%. The cluster now runs about 54,000 GPU hours between interruptions, up from an average of 47,000, with a target of 100,000 hours.

Affected versions

The infrastructure currently operates two Nvidia platforms: Catalina, using GB200 NVL72 racks, and Clemente, using the newer GB300 NVL72 racks. A third platform, San Miguel, will add Nvidia’s Vera Rubin NVL72 racks. Each rack contains 72 GPUs acting as one accelerator, with eight GPUs kept as spares and approximately 2% of racks held in reserve to maintain reliability.

Mitigation

Meta implemented several measures to improve stability and manage power. Firmware fixes increased the mean time between interruptions. The company uses a software tool called Dynamo Dimmer to cap GPU power gradually, preventing sudden shifts in energy draw. This is backed by batteries at the rack, facility, and grid levels. Meta also matches its energy consumption in New Albany with new clean and renewable energy sources.

Why it matters

Large-scale AI infrastructure requires robust engineering to maintain operational continuity. Teams usually monitor hardware telemetry closely to identify signal degradation before it causes widespread failures. Check whether your own systems have adequate spare capacity and automated recovery procedures. Ensure that power management protocols are in place to handle rapid load changes. Regular firmware updates and rigorous testing of interconnects are essential for maintaining high availability in dense computing environments.

What to do and how to stay safe: Meta

• Monitor hardware logs for signs of signal degradation or memory faults in high-density GPU setups.
• Verify that power management systems can handle rapid fluctuations in energy demand without tripping safeguards.
• Ensure that spare hardware and automated recovery protocols are tested and ready for deployment.
• Review energy sourcing strategies to align with sustainability goals and grid stability requirements.

Step-by-step guide: Password Policies Examples That Reduce Password Fatigue

General security guidance from the Vector Bulletin newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

How many GPUs does the Prometheus supercluster plan to support?

Meta plans for the Prometheus cluster to comprise roughly 1 million distributed GPUs at full scale.

What was the main cause of interruptions in the first phase?

The biggest early problem was signal quality on more than 600,000 NVLink connections between GPUs.

How does Meta manage the power draw of the cluster?

Meta uses software called Dynamo Dimmer to cap GPU power gradually, backed by batteries at multiple levels.