A bad scheduling of data processing resulted in us exceeding our storage bandwidth resulting in average response times growing from the typical 30-40ms to 1s.
This occurred 08:08 UTC, was identified at 08:16 UTC and was finally resolved at 08:20 UTC leading to a total of 12 minutes with slow API response times.
This was a one of issue.
We have increased our storage bandwith to cope with such corner cases and are working on making our scheduler smarter and handle storage bandwidth as well.
Posted Jun 16, 2016 - 10:41 CEST
Monitoring
We are back and fully operational.
It looks like a queue overflow caused extra stress on our storage service but we are still investigating the issue.
Posted Jun 16, 2016 - 10:22 CEST
Investigating
We are investigating the issue and will report back shortly.