Possible Service Interruptions

Incident Report for Clerk.io

Postmortem

A summary of all the updates:

We got a message this morning from one of our hosting providers that many of our servers would be taken down for emergency maintenance a couple of hours later.

Through a fantastic effort from our team we have managed to get through the day without any interruptions for end consumers.

Posted Dec 14, 2015 - 23:41 CET

Resolved

Everything went fine without any interruptions.

Or strictly speaking we had 42 seconds in total over 3 periods in a 12 hour window so nothing noticeable to end consumers.

Everything is up and running again. Data processing will take some hours to run through the backlog from the day.

Posted Dec 14, 2015 - 23:32 CET

Update

We are almost trough the maintenance window and everything have been running smoothly.

Access to my.clerk.io is open again.

Data sync will be re-enabled within the hour.

Posted Dec 14, 2015 - 22:27 CET

Update

my.clerk.io and data processing have been shut down for the maintenance window (18:00 - 23:00).

Posted Dec 14, 2015 - 18:07 CET

Update

We will shut down my.clerk.io and all data processing operations between 18:00 - 23:00 CET.

This is solely to prevent loss of any changes to templates, data configuration etc. in the transition window.

The main service will not be affected in any way. We only shut down the admin interface on my.clerk.io and background updates.

No end consumer should notice anything.

Posted Dec 14, 2015 - 17:23 CET

Monitoring

We just received a critical scheduled maintenance notice from one of our hosting providers.

Normally this is standard procedure and we work around these without any interruption to our service.

This time how ever we have been given less than 10 hours notice where we typically are given 1-2 weeks notice.

We will still work around the maintenance without downtime but due to the short notice we will have to follow a different, more crude, procedure.

This means that during the day you might expect small interruptions of 1-2 minutes or some delayed data processing. Hopefully you won't experience any interruptions at all.

We sincerely apologize for this in the middle of the Christmas sales season!

We know that we provide a extremely business critical service and high uptime and stability is of vital importance. Thus it has already been decided that we will move away from this provider first thing next year.

We will update this incident report during the day if there are any important information and bring a post mortem around midnight CET.

Posted Dec 14, 2015 - 11:48 CET