Today, between 13:30 and 14:00 we had a issue with very long response times and timeouts interrupting the stability of the service.
This was caused by internal routing errors in the datacenter holding our main database increasing latency of internal messages from less than 1ms to 250-300ms.
Since the issue were slowdowns but the affected servers were still running and answering our normal emergency mode did not kick in. This system would normally let the service run uninterrupted.
As soon as the issue were resolved we started working on improvements to our setup to prevent this. We have already rolled out a series of improvements and will continue to do so based on our new learnings.
Most central is the ability for automatic detection of network bottlenecks.
At Clerk.io we know we provide services that are critical to your business and your relationship with your customers.
We are truly sorry for any inconvenience this have caused and thank you for your understanding.
If you have any questions please reach out directly to me at hkb@clerk.io.
Your sincerely
Hans-Kristian
Founder & CEO