Incidents reported on status page for Clerk.io https://status.clerk.io/ https://d1lppblt9t2x15.cloudfront.net/logos/624b434589872a01643163827954e073.png https://status.clerk.io/ en https://status.clerk.io/ Wed, 17 Sep 2025 12:10:25 +0000 https://status.clerk.io/#b90dca92853966b3587c9b1b9be8b34e09ad4bc028602bd288d7afd6c6d0e215 clerk.io recovered https://status.clerk.io/ Wed, 17 Sep 2025 12:09:38 +0000 https://status.clerk.io/#b90dca92853966b3587c9b1b9be8b34e09ad4bc028602bd288d7afd6c6d0e215 clerk.io went down https://status.clerk.io/incident/722367 Thu, 11 Sep 2025 14:07:00 -0000 https://status.clerk.io/incident/722367#063e84b5692f92a8fbe57f7df2c2113393e7a6b0239bd1aede1819a63a429f4c # Post Mortem - 2025-09-11 - Intermittent API failures All times are CEST. ## Overview Earlier today logging, search, and recommendation API endpoints began exhibiting intermittent failures, not returning any results. We’re sorry for the disruption this caused to you and your customers. ### Impact - Affected services: logging, search (v2), and recommendation (v2) APIs - Symptoms: intermittent errors responses and missing results - Windows: 08:52–09:25 CEST and ~12:20 (brief) ### Timeline - 08:19 - A slow increase in incoming live updates (and CRUD requests) begin. The increase is masked by the rise of requests that happen every morning. - 08:52 - Our inter-service messaging and event system reaches its memory limit, causing the first failed requests. As messages are consumed, capacity is freed up, leading to the intermittent nature of the issue. - 09:02 - The issue is confirmed in monitoring and SRE begins investigating. - 09:10 - Engineering confirms the issue in the messaging and event system and begins raising resources to increase event processing. - 09:25 - Issue is resolved and intermittent failures cease. - 12:20 - Another severe spike in incoming live updates cause another few minutes of intermittent request failures. ## Root Cause Analysis The root cause was the lack of available resources in the messaging and event system. A transient increase in load from live-updates led to it quite simply hitting a hardware limit and stopped accepting new events. Parts of the API rely on this system to perform session-, message-, user activity-, and usage-logging. With the system refusing new messages and events these parts of the API failed. ## Remediation Several avenues of remediation are being pursued, most have already been implemented: 1. Completed: The parts of the API that rely on being able to hand over events to the messaging and event system have been modified to 'degrade gracefully'. That means that in the case our infrastructure has issues, you will still get search results and recommendations back from us, instead we disable the tracking of usage and other logging events. 2. Completed: Increased resources for the messaging and event system. We have increased the amount of resources in the system eight-fold (8x), allowing for the absorption of much larger transient loads. 3. Planned for tonight: Separation of specific event types. We will be separating some of the event types out to another more resilient system, isolating it from large spikes in live updates and increasing its durability from hardware errors. We sincerely apologize for the inconvenience this has caused not just to you, but your customers and want to reassure you that we are not done looking into ways we can further strengthen our systems. https://status.clerk.io/incident/722367 Thu, 11 Sep 2025 14:07:00 -0000 https://status.clerk.io/incident/722367#063e84b5692f92a8fbe57f7df2c2113393e7a6b0239bd1aede1819a63a429f4c # Post Mortem - 2025-09-11 - Intermittent API failures All times are CEST. ## Overview Earlier today logging, search, and recommendation API endpoints began exhibiting intermittent failures, not returning any results. We’re sorry for the disruption this caused to you and your customers. ### Impact - Affected services: logging, search (v2), and recommendation (v2) APIs - Symptoms: intermittent errors responses and missing results - Windows: 08:52–09:25 CEST and ~12:20 (brief) ### Timeline - 08:19 - A slow increase in incoming live updates (and CRUD requests) begin. The increase is masked by the rise of requests that happen every morning. - 08:52 - Our inter-service messaging and event system reaches its memory limit, causing the first failed requests. As messages are consumed, capacity is freed up, leading to the intermittent nature of the issue. - 09:02 - The issue is confirmed in monitoring and SRE begins investigating. - 09:10 - Engineering confirms the issue in the messaging and event system and begins raising resources to increase event processing. - 09:25 - Issue is resolved and intermittent failures cease. - 12:20 - Another severe spike in incoming live updates cause another few minutes of intermittent request failures. ## Root Cause Analysis The root cause was the lack of available resources in the messaging and event system. A transient increase in load from live-updates led to it quite simply hitting a hardware limit and stopped accepting new events. Parts of the API rely on this system to perform session-, message-, user activity-, and usage-logging. With the system refusing new messages and events these parts of the API failed. ## Remediation Several avenues of remediation are being pursued, most have already been implemented: 1. Completed: The parts of the API that rely on being able to hand over events to the messaging and event system have been modified to 'degrade gracefully'. That means that in the case our infrastructure has issues, you will still get search results and recommendations back from us, instead we disable the tracking of usage and other logging events. 2. Completed: Increased resources for the messaging and event system. We have increased the amount of resources in the system eight-fold (8x), allowing for the absorption of much larger transient loads. 3. Planned for tonight: Separation of specific event types. We will be separating some of the event types out to another more resilient system, isolating it from large spikes in live updates and increasing its durability from hardware errors. We sincerely apologize for the inconvenience this has caused not just to you, but your customers and want to reassure you that we are not done looking into ways we can further strengthen our systems. https://status.clerk.io/incident/722367 Thu, 11 Sep 2025 14:07:00 -0000 https://status.clerk.io/incident/722367#063e84b5692f92a8fbe57f7df2c2113393e7a6b0239bd1aede1819a63a429f4c # Post Mortem - 2025-09-11 - Intermittent API failures All times are CEST. ## Overview Earlier today logging, search, and recommendation API endpoints began exhibiting intermittent failures, not returning any results. We’re sorry for the disruption this caused to you and your customers. ### Impact - Affected services: logging, search (v2), and recommendation (v2) APIs - Symptoms: intermittent errors responses and missing results - Windows: 08:52–09:25 CEST and ~12:20 (brief) ### Timeline - 08:19 - A slow increase in incoming live updates (and CRUD requests) begin. The increase is masked by the rise of requests that happen every morning. - 08:52 - Our inter-service messaging and event system reaches its memory limit, causing the first failed requests. As messages are consumed, capacity is freed up, leading to the intermittent nature of the issue. - 09:02 - The issue is confirmed in monitoring and SRE begins investigating. - 09:10 - Engineering confirms the issue in the messaging and event system and begins raising resources to increase event processing. - 09:25 - Issue is resolved and intermittent failures cease. - 12:20 - Another severe spike in incoming live updates cause another few minutes of intermittent request failures. ## Root Cause Analysis The root cause was the lack of available resources in the messaging and event system. A transient increase in load from live-updates led to it quite simply hitting a hardware limit and stopped accepting new events. Parts of the API rely on this system to perform session-, message-, user activity-, and usage-logging. With the system refusing new messages and events these parts of the API failed. ## Remediation Several avenues of remediation are being pursued, most have already been implemented: 1. Completed: The parts of the API that rely on being able to hand over events to the messaging and event system have been modified to 'degrade gracefully'. That means that in the case our infrastructure has issues, you will still get search results and recommendations back from us, instead we disable the tracking of usage and other logging events. 2. Completed: Increased resources for the messaging and event system. We have increased the amount of resources in the system eight-fold (8x), allowing for the absorption of much larger transient loads. 3. Planned for tonight: Separation of specific event types. We will be separating some of the event types out to another more resilient system, isolating it from large spikes in live updates and increasing its durability from hardware errors. We sincerely apologize for the inconvenience this has caused not just to you, but your customers and want to reassure you that we are not done looking into ways we can further strengthen our systems. https://status.clerk.io/ Thu, 11 Sep 2025 10:36:33 +0000 https://status.clerk.io/#d00abda16a892c76e8a027977a7854b7fcfa8e45a34b792a6afbfe0e32bbd55e api.clerk.io recovered https://status.clerk.io/ Thu, 11 Sep 2025 10:29:34 +0000 https://status.clerk.io/#d00abda16a892c76e8a027977a7854b7fcfa8e45a34b792a6afbfe0e32bbd55e api.clerk.io went down https://status.clerk.io/ Thu, 11 Sep 2025 10:22:38 +0000 https://status.clerk.io/#03b4a7070a63832ebac6d3009a209001b3dad79e21c695bdcf6b50a269d28ab8 api.clerk.io recovered https://status.clerk.io/ Thu, 11 Sep 2025 10:22:04 +0000 https://status.clerk.io/#03b4a7070a63832ebac6d3009a209001b3dad79e21c695bdcf6b50a269d28ab8 api.clerk.io went down https://status.clerk.io/incident/722367 Thu, 11 Sep 2025 07:29:00 -0000 https://status.clerk.io/incident/722367#292dee8ea585b5f2f2533f073864bd23bc9a9b5e34cd953e5de5a91755debfe1 The issue has been resolved. We are continuing close monitoring and working on a root cause analysis. Complete post mortem will follow. https://status.clerk.io/incident/722367 Thu, 11 Sep 2025 07:29:00 -0000 https://status.clerk.io/incident/722367#292dee8ea585b5f2f2533f073864bd23bc9a9b5e34cd953e5de5a91755debfe1 The issue has been resolved. We are continuing close monitoring and working on a root cause analysis. Complete post mortem will follow. https://status.clerk.io/incident/722367 Thu, 11 Sep 2025 07:29:00 -0000 https://status.clerk.io/incident/722367#292dee8ea585b5f2f2533f073864bd23bc9a9b5e34cd953e5de5a91755debfe1 The issue has been resolved. We are continuing close monitoring and working on a root cause analysis. Complete post mortem will follow. https://status.clerk.io/ Thu, 11 Sep 2025 07:19:31 +0000 https://status.clerk.io/#c87982082837111253923dc7ebc389f3f82cacd1e84ed85ef4e5cf5d416770e4 api.clerk.io recovered https://status.clerk.io/incident/722367 Thu, 11 Sep 2025 07:03:00 -0000 https://status.clerk.io/incident/722367#8e6a14aa1fc593bd23de25ec1cc571a822467029121277e0647e9b86cfeda62e We are currently experiencing intermittent failures to requests for search and recommendations. Engineering is working to restore regular service. More information and RCA will follow. https://status.clerk.io/incident/722367 Thu, 11 Sep 2025 07:03:00 -0000 https://status.clerk.io/incident/722367#8e6a14aa1fc593bd23de25ec1cc571a822467029121277e0647e9b86cfeda62e We are currently experiencing intermittent failures to requests for search and recommendations. Engineering is working to restore regular service. More information and RCA will follow. https://status.clerk.io/incident/722367 Thu, 11 Sep 2025 07:03:00 -0000 https://status.clerk.io/incident/722367#8e6a14aa1fc593bd23de25ec1cc571a822467029121277e0647e9b86cfeda62e We are currently experiencing intermittent failures to requests for search and recommendations. Engineering is working to restore regular service. More information and RCA will follow. https://status.clerk.io/ Thu, 11 Sep 2025 07:02:44 +0000 https://status.clerk.io/#c87982082837111253923dc7ebc389f3f82cacd1e84ed85ef4e5cf5d416770e4 api.clerk.io went down https://status.clerk.io/ Fri, 05 Sep 2025 08:00:56 +0000 https://status.clerk.io/#48d15610994bac7d8bd2fd31d9a45cf01a28ab648a923b639673ee6325c60f5b clerk.io recovered https://status.clerk.io/ Fri, 05 Sep 2025 08:00:24 +0000 https://status.clerk.io/#48d15610994bac7d8bd2fd31d9a45cf01a28ab648a923b639673ee6325c60f5b clerk.io went down https://status.clerk.io/incident/714768 Fri, 29 Aug 2025 11:14:00 -0000 https://status.clerk.io/incident/714768#efde7c23a12903c5bfad2cbebfc4eee5f159e0fd2f214f353a47432ad98d6338 Postmortem - 2025-08-29 Overview Earlier today, the search, recommendation, and chat services at `api.clerk.io` began malfunctioning, returning an `InternalError` with the message: "An internal error occurred. If you need assistance, please contact our support team." Timeline: All times CEST. - 09:29 - A dependency update from our APM provider (Application Performance Monitoring) was staged for test and deployment. It contained no behavior changes on our part; dependency updates like these are routinely performed without issue. - 09:31 - The update entered the testing pipeline and underwent unit and integration tests. - 09:43 - The full testing pipeline was completed with no issues, and a blue/green deployment was started in production. A full set of servers was spun up with the new APM dependency. - 09:44 - All new instances reported ready and began taking fractional traffic. - 09:49 - No errors were detected in the new instances, and the old instances began spinning down, increasing traffic to the patched instances. - 09:50 - An internal Redis server handling data queues showed a rapid—but not critical—rise in memory usage. - 09:51 - To avoid crashing, the Redis instance began refusing new write commands. This led to several services that rely on this queue system malfunctioning in unexpected ways. - Due to the structure of our internal and external monitoring, these issues occurred in a "gap" between automatic monitors and were not reported. - External monitors use an endpoint that does not rely on visitor or message logging, so they did not report the error. - Internal monitoring observed Redis closing connections but not out-of-memory conditions, and did not alert. - ~10:20 - The issue was reported, and internal escalation began. - 10:34 - The issue was identified as stemming from either of two updates, and both were staged for rollback. - 10:47 - All code changes were rolled back, and normal operation was restored. Remediation: Several measures have been and will be taken to prevent similar issues in the future: - The endpoint used for external monitoring will be modified to be more thorough and to "fail" earlier. - External monitoring will be strengthened with additional requests that actively exercise different features, allowing us to detect issues in specific subsystems. - Internal policy and training for error reporting and escalation will be updated to ensure faster responses to errors reported outside our established monitoring. - The issue cannot yet be reproduced in development; we will work with the APM vendor to reproduce and isolate the behavior in a controlled environment and will update this document with a confirmed root cause. https://status.clerk.io/incident/714768 Fri, 29 Aug 2025 11:14:00 -0000 https://status.clerk.io/incident/714768#efde7c23a12903c5bfad2cbebfc4eee5f159e0fd2f214f353a47432ad98d6338 Postmortem - 2025-08-29 Overview Earlier today, the search, recommendation, and chat services at `api.clerk.io` began malfunctioning, returning an `InternalError` with the message: "An internal error occurred. If you need assistance, please contact our support team." Timeline: All times CEST. - 09:29 - A dependency update from our APM provider (Application Performance Monitoring) was staged for test and deployment. It contained no behavior changes on our part; dependency updates like these are routinely performed without issue. - 09:31 - The update entered the testing pipeline and underwent unit and integration tests. - 09:43 - The full testing pipeline was completed with no issues, and a blue/green deployment was started in production. A full set of servers was spun up with the new APM dependency. - 09:44 - All new instances reported ready and began taking fractional traffic. - 09:49 - No errors were detected in the new instances, and the old instances began spinning down, increasing traffic to the patched instances. - 09:50 - An internal Redis server handling data queues showed a rapid—but not critical—rise in memory usage. - 09:51 - To avoid crashing, the Redis instance began refusing new write commands. This led to several services that rely on this queue system malfunctioning in unexpected ways. - Due to the structure of our internal and external monitoring, these issues occurred in a "gap" between automatic monitors and were not reported. - External monitors use an endpoint that does not rely on visitor or message logging, so they did not report the error. - Internal monitoring observed Redis closing connections but not out-of-memory conditions, and did not alert. - ~10:20 - The issue was reported, and internal escalation began. - 10:34 - The issue was identified as stemming from either of two updates, and both were staged for rollback. - 10:47 - All code changes were rolled back, and normal operation was restored. Remediation: Several measures have been and will be taken to prevent similar issues in the future: - The endpoint used for external monitoring will be modified to be more thorough and to "fail" earlier. - External monitoring will be strengthened with additional requests that actively exercise different features, allowing us to detect issues in specific subsystems. - Internal policy and training for error reporting and escalation will be updated to ensure faster responses to errors reported outside our established monitoring. - The issue cannot yet be reproduced in development; we will work with the APM vendor to reproduce and isolate the behavior in a controlled environment and will update this document with a confirmed root cause. https://status.clerk.io/incident/714768 Fri, 29 Aug 2025 11:14:00 -0000 https://status.clerk.io/incident/714768#efde7c23a12903c5bfad2cbebfc4eee5f159e0fd2f214f353a47432ad98d6338 Postmortem - 2025-08-29 Overview Earlier today, the search, recommendation, and chat services at `api.clerk.io` began malfunctioning, returning an `InternalError` with the message: "An internal error occurred. If you need assistance, please contact our support team." Timeline: All times CEST. - 09:29 - A dependency update from our APM provider (Application Performance Monitoring) was staged for test and deployment. It contained no behavior changes on our part; dependency updates like these are routinely performed without issue. - 09:31 - The update entered the testing pipeline and underwent unit and integration tests. - 09:43 - The full testing pipeline was completed with no issues, and a blue/green deployment was started in production. A full set of servers was spun up with the new APM dependency. - 09:44 - All new instances reported ready and began taking fractional traffic. - 09:49 - No errors were detected in the new instances, and the old instances began spinning down, increasing traffic to the patched instances. - 09:50 - An internal Redis server handling data queues showed a rapid—but not critical—rise in memory usage. - 09:51 - To avoid crashing, the Redis instance began refusing new write commands. This led to several services that rely on this queue system malfunctioning in unexpected ways. - Due to the structure of our internal and external monitoring, these issues occurred in a "gap" between automatic monitors and were not reported. - External monitors use an endpoint that does not rely on visitor or message logging, so they did not report the error. - Internal monitoring observed Redis closing connections but not out-of-memory conditions, and did not alert. - ~10:20 - The issue was reported, and internal escalation began. - 10:34 - The issue was identified as stemming from either of two updates, and both were staged for rollback. - 10:47 - All code changes were rolled back, and normal operation was restored. Remediation: Several measures have been and will be taken to prevent similar issues in the future: - The endpoint used for external monitoring will be modified to be more thorough and to "fail" earlier. - External monitoring will be strengthened with additional requests that actively exercise different features, allowing us to detect issues in specific subsystems. - Internal policy and training for error reporting and escalation will be updated to ensure faster responses to errors reported outside our established monitoring. - The issue cannot yet be reproduced in development; we will work with the APM vendor to reproduce and isolate the behavior in a controlled environment and will update this document with a confirmed root cause. https://status.clerk.io/incident/714768 Fri, 29 Aug 2025 08:47:00 -0000 https://status.clerk.io/incident/714768#4796d7313c39a2673ff0ed83b08a66cc6a5c2594a68662bea8763391fea0e2dd We did a rollback and will investigate further. https://status.clerk.io/incident/714768 Fri, 29 Aug 2025 08:47:00 -0000 https://status.clerk.io/incident/714768#4796d7313c39a2673ff0ed83b08a66cc6a5c2594a68662bea8763391fea0e2dd We did a rollback and will investigate further. https://status.clerk.io/incident/714768 Fri, 29 Aug 2025 08:47:00 -0000 https://status.clerk.io/incident/714768#4796d7313c39a2673ff0ed83b08a66cc6a5c2594a68662bea8763391fea0e2dd We did a rollback and will investigate further. https://status.clerk.io/incident/714768 Fri, 29 Aug 2025 08:01:00 -0000 https://status.clerk.io/incident/714768#352456d37e22e9621958467c12b0dc16f2c7832b08d2ed4439748921d6848276 Doing internal updates of dependencies, we encountered a problem that affected some parts of our search and recommendations. We are very sorry for the inconvenience https://status.clerk.io/incident/714768 Fri, 29 Aug 2025 08:01:00 -0000 https://status.clerk.io/incident/714768#352456d37e22e9621958467c12b0dc16f2c7832b08d2ed4439748921d6848276 Doing internal updates of dependencies, we encountered a problem that affected some parts of our search and recommendations. We are very sorry for the inconvenience https://status.clerk.io/incident/714768 Fri, 29 Aug 2025 08:01:00 -0000 https://status.clerk.io/incident/714768#352456d37e22e9621958467c12b0dc16f2c7832b08d2ed4439748921d6848276 Doing internal updates of dependencies, we encountered a problem that affected some parts of our search and recommendations. We are very sorry for the inconvenience https://status.clerk.io/incident/714004 Thu, 28 Aug 2025 08:01:00 -0000 https://status.clerk.io/incident/714004#044ee36d08d3fea959dd023a86e45bcbc482037d3bb11dacab2de6ce00c48fda We did a rollback and will investigate further. https://status.clerk.io/incident/714004 Thu, 28 Aug 2025 08:01:00 -0000 https://status.clerk.io/incident/714004#044ee36d08d3fea959dd023a86e45bcbc482037d3bb11dacab2de6ce00c48fda We did a rollback and will investigate further. https://status.clerk.io/incident/714004 Thu, 28 Aug 2025 08:01:00 -0000 https://status.clerk.io/incident/714004#044ee36d08d3fea959dd023a86e45bcbc482037d3bb11dacab2de6ce00c48fda We did a rollback and will investigate further. https://status.clerk.io/incident/714004 Thu, 28 Aug 2025 08:01:00 -0000 https://status.clerk.io/incident/714004#044ee36d08d3fea959dd023a86e45bcbc482037d3bb11dacab2de6ce00c48fda We did a rollback and will investigate further. https://status.clerk.io/incident/714004 Thu, 28 Aug 2025 07:56:00 -0000 https://status.clerk.io/incident/714004#ab9335d2287d4b0086ae9faf4e8710a083d7494a24d3cebc564d78d6d6b03da3 Some internal updates led to downtime for our Search, Recommendations, and Chat. https://status.clerk.io/incident/714004 Thu, 28 Aug 2025 07:56:00 -0000 https://status.clerk.io/incident/714004#ab9335d2287d4b0086ae9faf4e8710a083d7494a24d3cebc564d78d6d6b03da3 Some internal updates led to downtime for our Search, Recommendations, and Chat. https://status.clerk.io/incident/714004 Thu, 28 Aug 2025 07:56:00 -0000 https://status.clerk.io/incident/714004#ab9335d2287d4b0086ae9faf4e8710a083d7494a24d3cebc564d78d6d6b03da3 Some internal updates led to downtime for our Search, Recommendations, and Chat. https://status.clerk.io/incident/714004 Thu, 28 Aug 2025 07:56:00 -0000 https://status.clerk.io/incident/714004#ab9335d2287d4b0086ae9faf4e8710a083d7494a24d3cebc564d78d6d6b03da3 Some internal updates led to downtime for our Search, Recommendations, and Chat. https://status.clerk.io/ Thu, 07 Aug 2025 08:53:31 +0000 https://status.clerk.io/#db237bc0cc0a0f30f577ccd50444f630dc3c1f54eb8894f41ea5400a0e977e9c clerk.io recovered https://status.clerk.io/ Thu, 07 Aug 2025 08:52:24 +0000 https://status.clerk.io/#db237bc0cc0a0f30f577ccd50444f630dc3c1f54eb8894f41ea5400a0e977e9c clerk.io went down https://status.clerk.io/ Tue, 05 Aug 2025 08:16:32 +0000 https://status.clerk.io/#dd4c7f93bfd80235b65abb33e84baedbba3e93c1efaf3be108c074c158ae5796 docs.clerk.io recovered https://status.clerk.io/ Tue, 05 Aug 2025 07:56:52 +0000 https://status.clerk.io/#dd4c7f93bfd80235b65abb33e84baedbba3e93c1efaf3be108c074c158ae5796 docs.clerk.io went down https://status.clerk.io/ Tue, 22 Jul 2025 23:26:38 +0000 https://status.clerk.io/#1fbf2b39b465840071a56154e6670f0ff0ab6afcb2bca92ba5f17880d9e17b95 api.clerk.io recovered https://status.clerk.io/ Tue, 22 Jul 2025 23:15:39 +0000 https://status.clerk.io/#1fbf2b39b465840071a56154e6670f0ff0ab6afcb2bca92ba5f17880d9e17b95 api.clerk.io went down https://status.clerk.io/ Tue, 15 Jul 2025 07:56:57 +0000 https://status.clerk.io/#85059b4510e921488d358e990ed144546b9677a8aabf574921d0752027347b8e api.clerk.io recovered https://status.clerk.io/ Tue, 15 Jul 2025 07:56:26 +0000 https://status.clerk.io/#85059b4510e921488d358e990ed144546b9677a8aabf574921d0752027347b8e api.clerk.io went down https://status.clerk.io/ Fri, 11 Jul 2025 13:15:50 +0000 https://status.clerk.io/#80fd1e2f09b6f3f7d929e0b951262b9cecafaf5394075ba5ea961d593f65d065 api.clerk.io recovered https://status.clerk.io/ Fri, 11 Jul 2025 13:15:20 +0000 https://status.clerk.io/#80fd1e2f09b6f3f7d929e0b951262b9cecafaf5394075ba5ea961d593f65d065 api.clerk.io went down https://status.clerk.io/ Wed, 09 Jul 2025 09:42:38 +0000 https://status.clerk.io/#4b16dc1809932fccda68eda563882d16fa3b1d41c595e539072dd8d1519073da api.clerk.io recovered https://status.clerk.io/ Wed, 09 Jul 2025 09:41:39 +0000 https://status.clerk.io/#4b16dc1809932fccda68eda563882d16fa3b1d41c595e539072dd8d1519073da api.clerk.io went down https://status.clerk.io/incident/615997 Tue, 08 Jul 2025 07:30:00 -0000 https://status.clerk.io/incident/615997#b0e27cb737e1286e0f014d677d275fd72c233dca31e535a2a964bc93f25ecf63 The issue is resolved and we keep investigating further. https://status.clerk.io/incident/615997 Tue, 08 Jul 2025 07:05:00 -0000 https://status.clerk.io/incident/615997#9556a22a65aa0ff7a89ddd527233d932a2086a20ded58bc963515d8eae0e4250 We have experienced a slowdown in our API performance. It's solved, and we are actively working to investigate what caused the issue. We appreciate your patience. https://status.clerk.io/ Mon, 07 Jul 2025 23:35:27 +0000 https://status.clerk.io/#3a051156fbd0ca470f650e9863f984207edbf814c27987d5807ec3bb94099635 clerk.io recovered https://status.clerk.io/ Mon, 07 Jul 2025 23:34:58 +0000 https://status.clerk.io/#3a051156fbd0ca470f650e9863f984207edbf814c27987d5807ec3bb94099635 clerk.io went down https://status.clerk.io/ Mon, 07 Jul 2025 08:16:42 +0000 https://status.clerk.io/#a495b8557857d78daa4d78fed4faddaedc89bee5ec3cae40170abf8c26a81b67 clerk.io recovered https://status.clerk.io/ Mon, 07 Jul 2025 08:16:13 +0000 https://status.clerk.io/#a495b8557857d78daa4d78fed4faddaedc89bee5ec3cae40170abf8c26a81b67 clerk.io went down https://status.clerk.io/ Mon, 07 Jul 2025 08:10:41 +0000 https://status.clerk.io/#13395bb699d05bc9a1055436044cbe6f6a8f6482998a0be052188f6171c2ee55 clerk.io recovered https://status.clerk.io/ Mon, 07 Jul 2025 08:10:03 +0000 https://status.clerk.io/#13395bb699d05bc9a1055436044cbe6f6a8f6482998a0be052188f6171c2ee55 clerk.io went down https://status.clerk.io/ Tue, 24 Jun 2025 14:10:52 +0000 https://status.clerk.io/#537edc0029a21268cca26988aeaba33035321b11949da5fdf3e64bd813114b1c docs.clerk.io recovered https://status.clerk.io/ Tue, 24 Jun 2025 13:28:53 +0000 https://status.clerk.io/#537edc0029a21268cca26988aeaba33035321b11949da5fdf3e64bd813114b1c docs.clerk.io went down https://status.clerk.io/ Tue, 24 Jun 2025 11:21:06 +0000 https://status.clerk.io/#6a3b0a0c5ad42222cd2fc0baa0aac956e4af4a911f5472c392ac8b3cbf9457be docs.clerk.io recovered https://status.clerk.io/ Tue, 24 Jun 2025 10:32:49 +0000 https://status.clerk.io/#6a3b0a0c5ad42222cd2fc0baa0aac956e4af4a911f5472c392ac8b3cbf9457be docs.clerk.io went down https://status.clerk.io/ Mon, 23 Jun 2025 13:46:55 +0000 https://status.clerk.io/#93473f977f30c74aeb92152e65b7d8304493e05ccddec036a386265e2ae7b4ab docs.clerk.io recovered https://status.clerk.io/ Mon, 23 Jun 2025 13:26:26 +0000 https://status.clerk.io/#93473f977f30c74aeb92152e65b7d8304493e05ccddec036a386265e2ae7b4ab docs.clerk.io went down https://status.clerk.io/incident/605178 Tue, 17 Jun 2025 09:22:00 -0000 https://status.clerk.io/incident/605178#c85bcb0411fb795030b0f20d0519370cdaaef37168b3e303fcfdbc528a12b26e The issue was fully resolved. https://status.clerk.io/incident/605178 Tue, 17 Jun 2025 09:22:00 -0000 https://status.clerk.io/incident/605178#c85bcb0411fb795030b0f20d0519370cdaaef37168b3e303fcfdbc528a12b26e The issue was fully resolved. https://status.clerk.io/incident/605178 Tue, 17 Jun 2025 09:02:00 -0000 https://status.clerk.io/incident/605178#1effa90e6eec28b0e85d7a3330be62f734d5cce1e8da3e4fd0036e1d353db96f A release of Clerk.js at 10:43 AM CET introduced an error that mistakenly converted limit values from integers to strings. This caused issues for some customers where search and recommendation features were impacted, depending on the integration method used. We were alerted to the issue at 11:02 AM CET and had it fully resolved by 11:22 AM CET. Timeline: • 10:43 AM CET: Faulty release deployed • 11:02 AM CET: Issue identified and investigation started • 11:22 AM CET: Issue fully resolved We continuously analyze incidents like this to understand the root cause and improve our systems to prevent similar issues in the future. We are sorry for the inconvenience https://status.clerk.io/incident/605178 Tue, 17 Jun 2025 09:02:00 -0000 https://status.clerk.io/incident/605178#1effa90e6eec28b0e85d7a3330be62f734d5cce1e8da3e4fd0036e1d353db96f A release of Clerk.js at 10:43 AM CET introduced an error that mistakenly converted limit values from integers to strings. This caused issues for some customers where search and recommendation features were impacted, depending on the integration method used. We were alerted to the issue at 11:02 AM CET and had it fully resolved by 11:22 AM CET. Timeline: • 10:43 AM CET: Faulty release deployed • 11:02 AM CET: Issue identified and investigation started • 11:22 AM CET: Issue fully resolved We continuously analyze incidents like this to understand the root cause and improve our systems to prevent similar issues in the future. We are sorry for the inconvenience https://status.clerk.io/incident/530142 Fri, 04 Apr 2025 01:00:00 -0000 https://status.clerk.io/incident/530142#493f987009e0deea2450d621cc8d0df0021aa83930c990735da8904836113b45 Maintenance completed https://status.clerk.io/incident/530142 Fri, 04 Apr 2025 01:00:00 -0000 https://status.clerk.io/incident/530142#493f987009e0deea2450d621cc8d0df0021aa83930c990735da8904836113b45 Maintenance completed https://status.clerk.io/incident/530142 Fri, 04 Apr 2025 01:00:00 -0000 https://status.clerk.io/incident/530142#493f987009e0deea2450d621cc8d0df0021aa83930c990735da8904836113b45 Maintenance completed https://status.clerk.io/incident/530142 Fri, 04 Apr 2025 01:00:00 -0000 https://status.clerk.io/incident/530142#493f987009e0deea2450d621cc8d0df0021aa83930c990735da8904836113b45 Maintenance completed https://status.clerk.io/incident/530142 Fri, 04 Apr 2025 01:00:00 -0000 https://status.clerk.io/incident/530142#493f987009e0deea2450d621cc8d0df0021aa83930c990735da8904836113b45 Maintenance completed https://status.clerk.io/incident/530142 Fri, 04 Apr 2025 01:00:00 -0000 https://status.clerk.io/incident/530142#493f987009e0deea2450d621cc8d0df0021aa83930c990735da8904836113b45 Maintenance completed https://status.clerk.io/incident/530142 Fri, 04 Apr 2025 01:00:00 -0000 https://status.clerk.io/incident/530142#493f987009e0deea2450d621cc8d0df0021aa83930c990735da8904836113b45 Maintenance completed https://status.clerk.io/incident/530142 Fri, 04 Apr 2025 01:00:00 -0000 https://status.clerk.io/incident/530142#493f987009e0deea2450d621cc8d0df0021aa83930c990735da8904836113b45 Maintenance completed https://status.clerk.io/incident/530142 Thu, 03 Apr 2025 23:00:00 -0000 https://status.clerk.io/incident/530142#c21ddbb7030ab889d52334810a4c6fc003690d55b539f5dc2bd5a3156a1d38a0 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530142 Thu, 03 Apr 2025 23:00:00 -0000 https://status.clerk.io/incident/530142#c21ddbb7030ab889d52334810a4c6fc003690d55b539f5dc2bd5a3156a1d38a0 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530142 Thu, 03 Apr 2025 23:00:00 -0000 https://status.clerk.io/incident/530142#c21ddbb7030ab889d52334810a4c6fc003690d55b539f5dc2bd5a3156a1d38a0 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530142 Thu, 03 Apr 2025 23:00:00 -0000 https://status.clerk.io/incident/530142#c21ddbb7030ab889d52334810a4c6fc003690d55b539f5dc2bd5a3156a1d38a0 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530142 Thu, 03 Apr 2025 23:00:00 -0000 https://status.clerk.io/incident/530142#c21ddbb7030ab889d52334810a4c6fc003690d55b539f5dc2bd5a3156a1d38a0 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530142 Thu, 03 Apr 2025 23:00:00 -0000 https://status.clerk.io/incident/530142#c21ddbb7030ab889d52334810a4c6fc003690d55b539f5dc2bd5a3156a1d38a0 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530142 Thu, 03 Apr 2025 23:00:00 -0000 https://status.clerk.io/incident/530142#c21ddbb7030ab889d52334810a4c6fc003690d55b539f5dc2bd5a3156a1d38a0 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530142 Thu, 03 Apr 2025 23:00:00 -0000 https://status.clerk.io/incident/530142#c21ddbb7030ab889d52334810a4c6fc003690d55b539f5dc2bd5a3156a1d38a0 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530140 Thu, 03 Apr 2025 01:00:59 -0000 https://status.clerk.io/incident/530140#75c36836b400044fcbe27ee9b1f4e3acdcd8e3c5c94317dba7e74067c8540b43 Maintenance completed https://status.clerk.io/incident/530140 Thu, 03 Apr 2025 01:00:59 -0000 https://status.clerk.io/incident/530140#75c36836b400044fcbe27ee9b1f4e3acdcd8e3c5c94317dba7e74067c8540b43 Maintenance completed https://status.clerk.io/incident/530140 Thu, 03 Apr 2025 01:00:59 -0000 https://status.clerk.io/incident/530140#75c36836b400044fcbe27ee9b1f4e3acdcd8e3c5c94317dba7e74067c8540b43 Maintenance completed https://status.clerk.io/incident/530140 Thu, 03 Apr 2025 01:00:59 -0000 https://status.clerk.io/incident/530140#75c36836b400044fcbe27ee9b1f4e3acdcd8e3c5c94317dba7e74067c8540b43 Maintenance completed https://status.clerk.io/incident/530140 Thu, 03 Apr 2025 01:00:59 -0000 https://status.clerk.io/incident/530140#75c36836b400044fcbe27ee9b1f4e3acdcd8e3c5c94317dba7e74067c8540b43 Maintenance completed https://status.clerk.io/incident/530140 Thu, 03 Apr 2025 01:00:59 -0000 https://status.clerk.io/incident/530140#75c36836b400044fcbe27ee9b1f4e3acdcd8e3c5c94317dba7e74067c8540b43 Maintenance completed https://status.clerk.io/incident/530140 Thu, 03 Apr 2025 01:00:59 -0000 https://status.clerk.io/incident/530140#75c36836b400044fcbe27ee9b1f4e3acdcd8e3c5c94317dba7e74067c8540b43 Maintenance completed https://status.clerk.io/incident/530140 Thu, 03 Apr 2025 01:00:59 -0000 https://status.clerk.io/incident/530140#75c36836b400044fcbe27ee9b1f4e3acdcd8e3c5c94317dba7e74067c8540b43 Maintenance completed https://status.clerk.io/incident/530140 Wed, 02 Apr 2025 23:00:59 -0000 https://status.clerk.io/incident/530140#4e85c322c9a5b2ae6275696a8b9b319e78bd218ca0ba405c6bd629c4e2eb8f47 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530140 Wed, 02 Apr 2025 23:00:59 -0000 https://status.clerk.io/incident/530140#4e85c322c9a5b2ae6275696a8b9b319e78bd218ca0ba405c6bd629c4e2eb8f47 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530140 Wed, 02 Apr 2025 23:00:59 -0000 https://status.clerk.io/incident/530140#4e85c322c9a5b2ae6275696a8b9b319e78bd218ca0ba405c6bd629c4e2eb8f47 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530140 Wed, 02 Apr 2025 23:00:59 -0000 https://status.clerk.io/incident/530140#4e85c322c9a5b2ae6275696a8b9b319e78bd218ca0ba405c6bd629c4e2eb8f47 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530140 Wed, 02 Apr 2025 23:00:59 -0000 https://status.clerk.io/incident/530140#4e85c322c9a5b2ae6275696a8b9b319e78bd218ca0ba405c6bd629c4e2eb8f47 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530140 Wed, 02 Apr 2025 23:00:59 -0000 https://status.clerk.io/incident/530140#4e85c322c9a5b2ae6275696a8b9b319e78bd218ca0ba405c6bd629c4e2eb8f47 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530140 Wed, 02 Apr 2025 23:00:59 -0000 https://status.clerk.io/incident/530140#4e85c322c9a5b2ae6275696a8b9b319e78bd218ca0ba405c6bd629c4e2eb8f47 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530140 Wed, 02 Apr 2025 23:00:59 -0000 https://status.clerk.io/incident/530140#4e85c322c9a5b2ae6275696a8b9b319e78bd218ca0ba405c6bd629c4e2eb8f47 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530137 Wed, 02 Apr 2025 01:00:16 -0000 https://status.clerk.io/incident/530137#f90902111df8f7d6f350fd073f53ebe8784275461de4839b51d70a855076e2cf Maintenance completed https://status.clerk.io/incident/530137 Wed, 02 Apr 2025 01:00:16 -0000 https://status.clerk.io/incident/530137#f90902111df8f7d6f350fd073f53ebe8784275461de4839b51d70a855076e2cf Maintenance completed https://status.clerk.io/incident/530137 Wed, 02 Apr 2025 01:00:16 -0000 https://status.clerk.io/incident/530137#f90902111df8f7d6f350fd073f53ebe8784275461de4839b51d70a855076e2cf Maintenance completed https://status.clerk.io/incident/530137 Wed, 02 Apr 2025 01:00:16 -0000 https://status.clerk.io/incident/530137#f90902111df8f7d6f350fd073f53ebe8784275461de4839b51d70a855076e2cf Maintenance completed https://status.clerk.io/incident/530137 Wed, 02 Apr 2025 01:00:16 -0000 https://status.clerk.io/incident/530137#f90902111df8f7d6f350fd073f53ebe8784275461de4839b51d70a855076e2cf Maintenance completed https://status.clerk.io/incident/530137 Wed, 02 Apr 2025 01:00:16 -0000 https://status.clerk.io/incident/530137#f90902111df8f7d6f350fd073f53ebe8784275461de4839b51d70a855076e2cf Maintenance completed https://status.clerk.io/incident/530137 Wed, 02 Apr 2025 01:00:16 -0000 https://status.clerk.io/incident/530137#f90902111df8f7d6f350fd073f53ebe8784275461de4839b51d70a855076e2cf Maintenance completed https://status.clerk.io/incident/530137 Wed, 02 Apr 2025 01:00:16 -0000 https://status.clerk.io/incident/530137#f90902111df8f7d6f350fd073f53ebe8784275461de4839b51d70a855076e2cf Maintenance completed https://status.clerk.io/incident/530137 Tue, 01 Apr 2025 23:00:16 -0000 https://status.clerk.io/incident/530137#8587063575e318c3a8ca0d5cf9289bdfdefa9df1d42ff7204d6e65cd580488a2 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530137 Tue, 01 Apr 2025 23:00:16 -0000 https://status.clerk.io/incident/530137#8587063575e318c3a8ca0d5cf9289bdfdefa9df1d42ff7204d6e65cd580488a2 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530137 Tue, 01 Apr 2025 23:00:16 -0000 https://status.clerk.io/incident/530137#8587063575e318c3a8ca0d5cf9289bdfdefa9df1d42ff7204d6e65cd580488a2 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530137 Tue, 01 Apr 2025 23:00:16 -0000 https://status.clerk.io/incident/530137#8587063575e318c3a8ca0d5cf9289bdfdefa9df1d42ff7204d6e65cd580488a2 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530137 Tue, 01 Apr 2025 23:00:16 -0000 https://status.clerk.io/incident/530137#8587063575e318c3a8ca0d5cf9289bdfdefa9df1d42ff7204d6e65cd580488a2 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530137 Tue, 01 Apr 2025 23:00:16 -0000 https://status.clerk.io/incident/530137#8587063575e318c3a8ca0d5cf9289bdfdefa9df1d42ff7204d6e65cd580488a2 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530137 Tue, 01 Apr 2025 23:00:16 -0000 https://status.clerk.io/incident/530137#8587063575e318c3a8ca0d5cf9289bdfdefa9df1d42ff7204d6e65cd580488a2 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/530137 Tue, 01 Apr 2025 23:00:16 -0000 https://status.clerk.io/incident/530137#8587063575e318c3a8ca0d5cf9289bdfdefa9df1d42ff7204d6e65cd580488a2 Our services will be partially unavailable for around 10 minutes while we upgrade our database infrastructure. We apologize for the inconvenience. We do these upgrades to keep up the best possible experience for you and your customers. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:45:00 -0000 https://status.clerk.io/incident/474789#089f99b6308ad8b1793cc1843095d52b34a58d9f81aaf5aac6a65036c9fa8dc1 Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:45:00 -0000 https://status.clerk.io/incident/474789#089f99b6308ad8b1793cc1843095d52b34a58d9f81aaf5aac6a65036c9fa8dc1 Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:45:00 -0000 https://status.clerk.io/incident/474789#089f99b6308ad8b1793cc1843095d52b34a58d9f81aaf5aac6a65036c9fa8dc1 Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:45:00 -0000 https://status.clerk.io/incident/474789#089f99b6308ad8b1793cc1843095d52b34a58d9f81aaf5aac6a65036c9fa8dc1 Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:45:00 -0000 https://status.clerk.io/incident/474789#089f99b6308ad8b1793cc1843095d52b34a58d9f81aaf5aac6a65036c9fa8dc1 Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:45:00 -0000 https://status.clerk.io/incident/474789#089f99b6308ad8b1793cc1843095d52b34a58d9f81aaf5aac6a65036c9fa8dc1 Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:45:00 -0000 https://status.clerk.io/incident/474789#089f99b6308ad8b1793cc1843095d52b34a58d9f81aaf5aac6a65036c9fa8dc1 Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:45:00 -0000 https://status.clerk.io/incident/474789#089f99b6308ad8b1793cc1843095d52b34a58d9f81aaf5aac6a65036c9fa8dc1 Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:39:00 -0000 https://status.clerk.io/incident/474789#6bc2e410d35a6c71c2658c8538cefc0c41f0511a0be804b638cd5e73c62b64b1 All functionality is back. The issue was an internal issue in one of our database servers, it was isolated and resolved without any data lose. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:39:00 -0000 https://status.clerk.io/incident/474789#6bc2e410d35a6c71c2658c8538cefc0c41f0511a0be804b638cd5e73c62b64b1 All functionality is back. The issue was an internal issue in one of our database servers, it was isolated and resolved without any data lose. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:39:00 -0000 https://status.clerk.io/incident/474789#6bc2e410d35a6c71c2658c8538cefc0c41f0511a0be804b638cd5e73c62b64b1 All functionality is back. The issue was an internal issue in one of our database servers, it was isolated and resolved without any data lose. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:39:00 -0000 https://status.clerk.io/incident/474789#6bc2e410d35a6c71c2658c8538cefc0c41f0511a0be804b638cd5e73c62b64b1 All functionality is back. The issue was an internal issue in one of our database servers, it was isolated and resolved without any data lose. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:39:00 -0000 https://status.clerk.io/incident/474789#6bc2e410d35a6c71c2658c8538cefc0c41f0511a0be804b638cd5e73c62b64b1 All functionality is back. The issue was an internal issue in one of our database servers, it was isolated and resolved without any data lose. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:39:00 -0000 https://status.clerk.io/incident/474789#6bc2e410d35a6c71c2658c8538cefc0c41f0511a0be804b638cd5e73c62b64b1 All functionality is back. The issue was an internal issue in one of our database servers, it was isolated and resolved without any data lose. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:39:00 -0000 https://status.clerk.io/incident/474789#6bc2e410d35a6c71c2658c8538cefc0c41f0511a0be804b638cd5e73c62b64b1 All functionality is back. The issue was an internal issue in one of our database servers, it was isolated and resolved without any data lose. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:39:00 -0000 https://status.clerk.io/incident/474789#6bc2e410d35a6c71c2658c8538cefc0c41f0511a0be804b638cd5e73c62b64b1 All functionality is back. The issue was an internal issue in one of our database servers, it was isolated and resolved without any data lose. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:10:00 -0000 https://status.clerk.io/incident/474789#1514cef25731b429fa0b63c00f16edc904615fe9cd8648c0ab34f407c5814600 We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:10:00 -0000 https://status.clerk.io/incident/474789#1514cef25731b429fa0b63c00f16edc904615fe9cd8648c0ab34f407c5814600 We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:10:00 -0000 https://status.clerk.io/incident/474789#1514cef25731b429fa0b63c00f16edc904615fe9cd8648c0ab34f407c5814600 We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:10:00 -0000 https://status.clerk.io/incident/474789#1514cef25731b429fa0b63c00f16edc904615fe9cd8648c0ab34f407c5814600 We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:10:00 -0000 https://status.clerk.io/incident/474789#1514cef25731b429fa0b63c00f16edc904615fe9cd8648c0ab34f407c5814600 We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:10:00 -0000 https://status.clerk.io/incident/474789#1514cef25731b429fa0b63c00f16edc904615fe9cd8648c0ab34f407c5814600 We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:10:00 -0000 https://status.clerk.io/incident/474789#1514cef25731b429fa0b63c00f16edc904615fe9cd8648c0ab34f407c5814600 We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2. https://status.clerk.io/incident/474789 Mon, 09 Dec 2024 14:10:00 -0000 https://status.clerk.io/incident/474789#1514cef25731b429fa0b63c00f16edc904615fe9cd8648c0ab34f407c5814600 We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2.