At 18:59 CET, a virtual instance hosting one of our databases suddenly went down/dark.
An alert went out and the pager duty team identified the issue within minutes through our monitoring.
At 19:08 CET all stores on the affected database were placed in “maintenance” to prevent workers from attempting to access the database. This stabilized our services for all customers not located on the affected database server.
Due to the issue being on our cloud provider (AWS) side, it took until 19:22 CET for the database to come back up.
As the server came back up, the DBMS replayed all pending transactions and found 2 tables had “notes” (less than warnings) in the log. These were manually investigated, after which all stores were taken out of maintenance mode.
At 19:23 CET all services were returned to normal.
The issue is fully resolved and our API has full performance and stability back.