Recap:

At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers.

The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production.

By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring.

By 15:32 CET the final store was back to full production.

All functionality is back.

The issue was an internal issue in one of our database servers, it was isolated and resolved without any data lose.

We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2.