Network Outage in Worcester

In this RCA we want to share our analysis of a network connectivity incident that affected virtual servers on 20 May 2026 in our Worcester datacenter. During this period, LANs associated with a subset of your virtual machines became unavailable due to a network configuration issue in the underlying cluster. We have documented the sequence of events, the technical root cause, and the measures we are implementing to prevent recurrence.

What happened?

On 20 May 2026 at approximately 11:19 UTC, LAN connectivity was lost for virtual machines hosted on a subset of physical servers in an affected cluster. The impact manifested as a complete loss of LAN connectivity for the virtual machines running on those servers.

Engineering was alerted and began investigation. The root cause was identified, and the affected virtual machines were migrated to alternative physical servers with compatible interface speeds. LAN connectivity was fully restored by 14:27 UTC.

How was that possible? (Root Cause)

The affected cluster contained a mix of physical servers operating at different interface speeds: a subset of servers were running at 100Gbps, while the remainder of the cluster had been upgraded to 200Gbps. This mixed-speed configuration was present prior to the incident.

Multicast groups - which underpin LAN connectivity between virtual machines - form at the interface speed of the highest-speed server participating at the time the group is created. By platform design, a server may join an existing group only if its interface speed is equal to or greater than the group's speed. A faster server can therefore join a slower group, and its presence does not raise the group's speed, but a slower server cannot join a faster group.

Before the incident, the affected groups had been created at 100Gbps. The 200Gbps servers were able to participate in them because their speed exceeded the group speed, and their presence did not change it. As a result, LANs spanning both 100Gbps and 200Gbps hosts operated normally.

On 20 May, a platform-level liveboot rollout of a gateway component caused multicast groups to be recreated across the cluster. During recreation, each group's speed was re-derived from the highest-speed server then participating. Any group spanning both 100Gbps and 200Gbps hosts therefore reformed at 200Gbps. The 100Gbps servers were now slower than the group and were rejected on rejoin. All multicast join requests from the affected 100Gbps servers failed, and the LANs hosted on those servers went down.

The gateway liveboot was the operational trigger that caused multicast groups to be rebuilt. The underlying condition that made those rebuilds produce a connectivity failure was the presence of servers with 100Gbps interface speeds in an otherwise 200Gbps cluster. This disparity had not been detected prior to the event.

What we are doing to prevent recurrence

Completed

• The affected virtual machines were migrated to physical servers operating at 200Gbps, restoring LAN connectivity. The 100Gbps servers have been removed from active customer workloads.

In progress

• Interface upgrades: The affected physical servers are being upgraded to 200Gbps network interface cards, bringing their interface speed in line with the rest of the cluster. This eliminates the mixed-speed condition entirely. (ETA: Q3 2026)
• Cluster speed monitoring: We are implementing automated monitoring to detect physical servers operating at non-standard interface speeds within a cluster. This will ensure that any future speed disparity is identified and flagged before it can affect customer workloads during operational events such as gateway liveboots. (ETA: Q3 2026)

Structural review

We are reviewing our gateway liveboot and rollout procedures to assess whether multicast group recreation can be performed in a way that is resilient to mixed-speed cluster configurations, adding a safeguard independent of monitoring coverage.

Closing remarks

The loss of LAN connectivity, even for a bounded period, disrupts operational continuity, and we recognize the impact this incident had on your environment. The root condition - a speed disparity between servers in the same cluster - should have been identified before it could be exposed by a routine operational event. We have addressed the immediate impact, and the NIC upgrades and monitoring improvements underway will close the gap that allowed this configuration to persist undetected.