API Slowdown
Incident Report for Clerk.io
Postmortem

Between 19:18 UTC and 19:23 UTC the Clerk.io API experienced a minor outage causing slow response times.

At 19:18 UTC a series of commands were issued to our datastore slowing down our response times and thus caused a domino effect slowing all other queries. The domino effect were to big to be resolved automatically by our autonomous reliability systems.

At 19:19 UTC our team was alerted and at 19:21 UTC we had identified the problem and started our tools to cooldown the system.

At 19:23 UTC API response times were back to normal.

Though we are still investigating the initial cause we will now start our evaluation process of our preventive measures that could have avoided this incident.

A good thing to take from this outage is that our alerting, monitoring and recovery tools worked exactly as planned which meant that the issues was resolved as fast as humanly possible.

We know the reliability of our service is crucial for your business and we always works seriously on expanding Clerk.io’s stability and reliability.

--

Hans-Kristian Bjerregaard

CEO, Clerk.io

Posted 4 months ago. Sep 13, 2018 - 21:01 CEST

Resolved
This incident has been resolved.
Posted 4 months ago. Sep 13, 2018 - 20:37 CEST
Monitoring
We are back to normal performance.

Our team was able immediately to resolve the issue through our tools build for such events. We are now investigating what caused the spike in data store load.
Posted 4 months ago. Sep 13, 2018 - 20:31 CEST
Investigating
We are seeing unusual high load on our data storage leading to slower than usual response times. We are investigating the issue now and will repost back here shortly.
Posted 4 months ago. Sep 13, 2018 - 20:28 CEST
This incident affected: API.