Provisioning: Degraded Performance in Job Processing
Incident
Report for IONOS CLOUD
Resolved
This incident has been resolved. We are working on the RCA and will publish it here asap
Monitoring
Our Provisioning Team has been closely tracking queue performance over the last few hours and has noted steady improvements. We are moving this incident to Monitoring status.
We will continue to monitor queue processing closely and will update this page on any changes.
We will continue to monitor queue processing closely and will update this page on any changes.
Update
Still, the global provisioning system is not working properly.
Therefore, all other dependencies, like K8s and managed products, are affected, too.
We know this is affecting customer setups, and that urgent changes are not processed properly, and we apologize for the inconvenience.
We identified possible distractors, but fixing this is a complex task, with need to involve and coordinate several technical teams.
We are working with highest prio to get the provisioning back working,
and we will let you know asap on any progress.
Therefore, all other dependencies, like K8s and managed products, are affected, too.
We know this is affecting customer setups, and that urgent changes are not processed properly, and we apologize for the inconvenience.
We identified possible distractors, but fixing this is a complex task, with need to involve and coordinate several technical teams.
We are working with highest prio to get the provisioning back working,
and we will let you know asap on any progress.
Identified
We are currently investigating instances of performance degradation in job processing. Our Provisioning Team is currently working on fixing the root cause of these events.
This incident affected: APIs and Frontends (Cloud API), Location FR/PAR (Managed Kubernetes, Provisioning), Location US/MCI (Managed Kubernetes, Provisioning), Location DE/TXL (Managed Kubernetes, Provisioning), Location DE/FRA (Managed Kubernetes, Provisioning), Location DE/FKB (Managed Kubernetes, Provisioning), Location GB/LHR (Managed Kubernetes, Provisioning), Location US/EWR (Managed Kubernetes, Provisioning), Location US/LAS (Managed Kubernetes, Provisioning), Location ES/VIT (Managed Kubernetes, Provisioning), Location GB/BHX (Managed Kubernetes, Provisioning), and Location DE/FRA/2 (Managed Kubernetes, Provisioning).