Preliminary incident report: API availability

Date: 2026-09-02 T 07:05 CEST
Duration: Approximately 10 minutes

Impact: Some API requests and background operations were delayed or failed for a subset of stores. This also had a knock-on effect on the API availability for other stores as workers got tied up, until automatic scaling added new servers.

What happened

One of our database servers stopped processing queries after an internal MariaDB/InnoDB synchronization operation became stuck.

The database remained online at the network level but could not complete normal work. After its internal safety timeout was reached, MariaDB terminated the stalled process and recovered automatically during restart.

Service returned to normal once database recovery completed.

Cause

The incident was caused by an internal database metadata lock that failed to make progress. The likely trigger was an unusual interaction between a high-volume background table rebuild and database checkpoint processing.

This was not caused by increased customer traffic or an individual customer's actions. The same background processing normally completes without affecting availability.

The precise database thread responsible could not be identified because detailed crash diagnostics were not enabled at the time.

Data integrity

MariaDB completed its normal crash-recovery process successfully. Interrupted transactions were rolled back automatically, and we found no evidence of database corruption.

No customer action is required.

Improvements

We have increased the database log capacity to provide more processing headroom during large background operations.

We are also preparing the following changes:

• Remove unnecessary database metadata operations from bulk imports.
• Upgrade MariaDB to the latest maintenance release.
• Collect detailed diagnostics if a similar database failure occurs again.
• Reduce the time MariaDB waits before recovering from a confirmed internal stall.
• Improve monitoring and recovery automation for database operations that appear online but cannot process real queries.
• Extend failure handling for background processing as well as customer-facing API requests.

These changes are intended both to reduce the likelihood of recurrence and to restore service more quickly if another internal database stall occurs.

The service is fully recovered and we are investigating the database crash to find the exact cause and future fixes and mitigations.

Today between 07:05 CEST and 07:13 CEST the main API (search and recommendations) was partially unavailable because of a database crash.

We are currently investigating the crash and will be working to mitigate this in the future.

We are very sorry for any inconvenience this has caused for you and your customers.