Preliminary Root Cause Analysis

This Root Cause Analysis is preliminary as research is still being conducted to determine the technical root cause of the incident.

What happened?

On August 31, 2026, between 15:00 UTC and 15:43 ITC, and again between 16:32 UTC and 16:35 UTC, customers were unable to reach IONOS Cloud's Identity and Access Management (IAM) service and the Data Center Designer (DCD), which is dependent on this service. The disruption affected direct logins, partner and reseller portal access, and associated management consoles. The total customer-facing impact lasted approximately 48 minutes across both intervals.

Cloud APIs (api.ionos.com) remained fully operational throughout the incident. The underlying application services were healthy at all times - the failure was confined to the network edge layer.

How was this possible? (Root Cause)

During a scheduled maintenance window on August 31, 2026, a network configuration update was applied to the edge network infrastructure with the intent of optimizing routing filters. The configuration was verified as correct prior to and during application. Following the rollout, a routing propagation anomaly emerged on one edge network switch, causing asymmetric routing behavior: incoming TCP connection requests from clients were silently dropped (black-holed) at the network edge before reaching the application cluster.

Because the application services themselves remained up and healthy, this failure mode was not immediately visible through internal health checks - the services were isolated from receiving inbound public internet traffic rather than failing.

The root cause of why this specific switch exhibited asymmetric routing behavior following an otherwise valid configuration change remains under active investigation. Whether this was triggered by a switch platform behavior or a software version-specific bug is being determined through staging environment reproduction.

What are we doing to prevent recurrence?

Immediate Actions

• Configuration Rollback: Upon identifying the routing anomaly, a full rollback of the network configuration was executed across all affected edge switches. Routing announcements and TCP reachability of the public production IP addresses were verified following the rollback. All affected services - DCD, Partner Portal, Reseller Portal, and IAM - were confirmed fully operational by 16:35 UTC. (DONE)

Short-term

• Staging Environment Replication: Detailed test cases are being executed in the staging environment to reproduce the exact routing propagation behavior under the same switch configuration conditions. The goal is to determine whether the anomaly is attributable to a specific software version bug or a switch platform behavior, so that the precise failure condition can be isolated and addressed before any future rollout. ETA: Within two weeks

Mid-term

• Revised Rollout Strategy: Based on the findings from staging, a revised rollout approach will be designed to ensure that any future application of these routing filter optimizations can be performed with greater stability guarantees - including more granular validation checkpoints between switch-level changes. ETA: October 2026

Closing remarks

We recognize that loss of access to IAM and DCD carries real operational impact. The fact that the underlying services were healthy throughout is not a mitigation of that impact.

We are committed to ensuring that the root cause is fully understood before any re-attempt of the original change, and that the revised rollout strategy addresses the conditions that led to the asymmetric routing behavior.

We thank you for your patience while we conclude the investigation.