Database connectivity issues
Incident Report for MarketMan
Postmortem

Azure issue:

Between 07:20 and 22:03 UTC on 29 Oct 2020, a subset of customers may have experienced intermittent issues or encountered error notifications when attempting to connect to SQL Database/SQL Managed Instance resources in East US.  Connectivity to these resources was impacted due to an issue with the load balancing service serving traffic to the databases.

 

The load balancing service in Azure is composed from two components, the data path that routes traffic to virtual machines and the control plane that determines the health of the VMs behind a virtual endpoint. In this incident the network layer that supports the control plane saturated which resulted in the load balancer incorrectly marking VMs out of service. If all the VMs behind a virtual endpoint are marked down traffic to the service is interrupted.  Detection of the issue was delayed because saturation in the networking layer also caused interruption to the monitoring data flowing from the control plane.  To mitigate the issue, the source of the traffic that was impacting the control plane was identified and removed.

Posted Nov 19, 2020 - 14:44 UTC

Resolved
Could not connect to database
Posted Oct 29, 2020 - 20:30 UTC