Network Service Degradation in TXL
Incident
Report for IONOS Cloud
Resolved
The Provisioning and DBaaS services have been restored to full operational status. The backlog has been cleared and no further service degradation is expected.
Our Network Team is currently preparing a comprehensive Root Cause Analysis (RCA) of this incident, which we will publish on this page. We anticipate releasing the complete analysis by tomorrow.
Thank you for your patience.
Our Network Team is currently preparing a comprehensive Root Cause Analysis (RCA) of this incident, which we will publish on this page. We anticipate releasing the complete analysis by tomorrow.
Thank you for your patience.
Monitoring
We are setting the DCD frontend back to 'Operational' now that the dependency on the provisioning side has been resolved. Customers should now be able to use the DCD via the web frontend again.
We are setting the incident to 'Monitoring' status and will ensure the successful recovery of the still affected services. Customers might still experience a performance impact on provisioning-related activities, such as modifying infrastructure resources in the DCD.
We recommend to postpone non-mission critical/urgent changes until the incident is marked as "Resolved"
We are setting the incident to 'Monitoring' status and will ensure the successful recovery of the still affected services. Customers might still experience a performance impact on provisioning-related activities, such as modifying infrastructure resources in the DCD.
We recommend to postpone non-mission critical/urgent changes until the incident is marked as "Resolved"
Update
We are still waiting for the provisioning backlog to be processed. Customers might face extended provisioning times. DCD web-frontend availability and DBaaS services remain affected due to their dependency on provisioning assignments. We will continue to update the status page, though likely at a reduced cadence as we monitor backlog consumption.
In parallel, a Root Cause Analysis (RCA) has been initiated for the triggering incident. We will share the RCA here as soon as it becomes available.
In parallel, a Root Cause Analysis (RCA) has been initiated for the triggering incident. We will share the RCA here as soon as it becomes available.
Update
We are setting AI Model Hub back to "operational"
Update
We are currently working on re-establishing connectivity for the supporting systems for the DCD to restore the service, as well.
Update
We see that AI Model Hub service is recovering. We are closely monitoring request processing and reducing the severity of the impact for this service.
Update
Network connectivity is improving. We will re-enable provisioning and monitor job execution as well as the rest of the affected services.
Update
Our network team is still diagnosing ongoing connectivity issues that affect the referenced services.
Update
We are upgrading the impact for the AI Model Hub.
Update
We are temporarily pausing provisioning in a cluster currently experiencing a backlog of queued jobs. While running services remain unaffected, updates to existing entities will not be processed until provisioning is reactivated.
Update
We have added DBaaS in the list of affected services due to increased error count. The team is aware and actively working on the service.
Identified
We are seeing issues again on DCD frontend, as well as AI Model Hub. Customers may still see connectivity issues with these services. Our Services Team are investigating.
Monitoring
We are seeing services recovering. We will monitor the progress over the next few minutes and update the status of the affected services.
Update
We have added DCD and AI Modelhub to the list of affected services. Customers might experience connectivity issues for these services currently.
Identified
We have identified an issue related to the ongoing maintenance: https://status.ionos.cloud/incidents/zg4mpk9x724t
Our network team is currently working on restoring service to the affected components. Customers might experience intermittent network service degradation or outages.
We are upgrading the severity of the incident and will keep you informed on the progress of the recovery.
Our network team is currently working on restoring service to the affected components. Customers might experience intermittent network service degradation or outages.
We are upgrading the severity of the incident and will keep you informed on the progress of the recovery.
Investigating
We are currently investigating monitoring alerts in TXL. We will keep you updated on our investigation.
This incident affected: APIs and Frontends (Data Center Designer (DCD)), Global Services (Database as a Service (DBaaS), AI Model Hub), and Location DE/TXL (Network, Provisioning).