MK8s: Partial Connectivity Degradation to Control Planes
Incident
Report for IONOS Cloud
Subscribe to Updates
Monitoring
Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress
Update
Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress
Update
The changes applied to some control plane clusters have had positive effects. The team is continuing the rollout to other affected clusters.
Update
We are marking DBaaS as recovered.
Our Kubernetes Team is currently working on stabilizing the Kubernetes Control Plane. We are focusing on mitigating recurring load spikes influencing stability.
Our Kubernetes Team is currently working on stabilizing the Kubernetes Control Plane. We are focusing on mitigating recurring load spikes influencing stability.
Update
We are marking the Container Registry Service as recovered.
Update
We are closing the incident for the AI Model Hub. All metrics have recovered and the service should be up and running again normally.
Update
We are adding the Container Registry as an affected Service. Customers may currently experience issues pulling and pushing images from the Registry.
Update
Our Kubernetes Team has deployed a fix for the affected AI Model Hub Database Services. We currently see metrics improving and monitoring the situation closely.
Update
We are expanding the scope of this incident to include DBaaS and AI Model Hub. We have observed an increased error count originating from PostgresDB on Kubernetes. Additionally, to improve transparency, the previously reported separate incident regarding the AI Model Hub (https://status.ionos.cloud/incidents/rmgs845klm32) is being merged into this primary incident.
Identified
The team has identified the root cause as a resource constraint within the etcd database. Mitigation efforts are currently underway.
Investigating
Some customers may experience connection problems to the control plane and degraded functionality of kubernetes.
Our teams are investigating and working on a resolution.
Our teams are investigating and working on a resolution.
This incident affects: Global Services (Database as a Service (DBaaS), AI Model Hub, Container Registry) and Location DE/FRA (Managed Kubernetes).