Network packet loss
Incident
Report for IONOS Cloud
Resolved
This incident has been resolved.
We have prepared the following preliminary Root Cause Analysis:
What happened:
On 29 October 2025 at 12:30 UTC an unintended route advertisement occurred within the IONOS Cloud network at site FR7 (metro region DE/FRA), causing traffic destined for external networks to be black‑holed for 27 minutes.
This was possible because two events coincided:
- A Border Gateway Protocol (BGP) session flap on a compute server triggered the unintended advertisement.
- A bug in the upgrade algorithm of the latest software‑defined‑networking (SDN) package caused the route‑suppression rule in the node to be omitted.
What we are doing to prevent this from happening again:
- Harden compute nodes to avoid recurrence (done).
- Extending the QA and rollout tests for the SDN software. (ETA 3th November 2025)
- Fixing the upgrade algorithm in the current SDN package. (ETA End of December 2025)
- Harden existing route filters on the route‑reflector layer to reject any default‑route advertisement from hypervisors. (ETA 7th November 2025)
We have prepared the following preliminary Root Cause Analysis:
What happened:
On 29 October 2025 at 12:30 UTC an unintended route advertisement occurred within the IONOS Cloud network at site FR7 (metro region DE/FRA), causing traffic destined for external networks to be black‑holed for 27 minutes.
This was possible because two events coincided:
- A Border Gateway Protocol (BGP) session flap on a compute server triggered the unintended advertisement.
- A bug in the upgrade algorithm of the latest software‑defined‑networking (SDN) package caused the route‑suppression rule in the node to be omitted.
What we are doing to prevent this from happening again:
- Harden compute nodes to avoid recurrence (done).
- Extending the QA and rollout tests for the SDN software. (ETA 3th November 2025)
- Fixing the upgrade algorithm in the current SDN package. (ETA End of December 2025)
- Harden existing route filters on the route‑reflector layer to reject any default‑route advertisement from hypervisors. (ETA 7th November 2025)
Monitoring
Network traffic has been restored to normal operations. The teams are monitoring.
Identified
The cause of the network outage has been identified and mitigated. Networking teams are working to normalize network traffic most customers should be back to normal operations.
Investigating
We are writing to inform you that we have been experiencing connection issues and substantial delays on packet delivery.
Network technicians have started working on the occurrence immediately after detection and will isolate the problem and solve the issue as quick as possible. However, it is possible that there will be a certain degradation in connection quality affecting individual virtual resources.
We will inform you as soon as the functionality has been restored.
Network technicians have started working on the occurrence immediately after detection and will isolate the problem and solve the issue as quick as possible. However, it is possible that there will be a certain degradation in connection quality affecting individual virtual resources.
We will inform you as soon as the functionality has been restored.
This incident affected: Location DE/FRA (Network, Managed Kubernetes, Provisioning).