10-15 customers had non-optimal recommendations during the night between Saturday and Sunday due to infrastructure rollout
Incident Report for Clerk.io
Resolved
This weekend we rolled out one of the largest infrastructure changes to Clerk.io resulting in greatly improved stability to the overall service and greatly speeded up reporting and analytics.

The rollout was successfully accomplished with zero downtime.

During the rollout, we had paused all data processing but due to a manual error when re-enabling it again it ran on the old infrastructure for the first few minutes. This means that a small subset of we exposed to non-optimal recommendations during some hours the night between Saturday and Sunday (CET).

After identifying the problem we have issued a full re-analyzation of all customers to ensure that everything is running optimally.
Posted about 1 year ago. Sep 17, 2017 - 10:08 CEST