On 2026-07-19, our production services experienced an extended outage following a planned database security maintenance announced by OVH. While the maintenance was initially expected to last no more than 30 minutes, an infrastructure failure at the OVH datacenter significantly prolonged the interruption.
After assessing the situation and the available recovery options, we activated our Business Continuity Plan (BCP) by restoring the production database from the latest available backup and upgrading it to PostgreSQL 18. All services were fully restored on 2026-07-20 at 11:15 UTC.
During the incident, Golive and Time Squad were unavailable for approximately 20 hours, preventing customers from accessing the applications and performing normal operations.
As part of the recovery process, we restored the database from the latest available backup. The most recent backup had been completed 25 minutes before the outage began, resulting in a potential data loss window of up to 25 minutes.
A review of the database audit logs showed that, due to the weekend period, only a limited number of write operations occurred during this timeframe. These updates were primarily related to deployment and monitoring activities, significantly reducing the effective customer impact.
The root cause of the incident was a hardware failure affecting a storage device in the OVH datacenter during a planned security maintenance operation on the managed PostgreSQL infrastructure operated by Aiven on behalf of OVH.
Initially, the incident was communicated as an extension of the planned maintenance, delaying visibility into the actual nature and severity of the problem. Several hours later, OVH published a dedicated incident confirming that the outage was caused by a hardware failure rather than the maintenance activity itself.
The incident occurred during a planned maintenance window announced by OVH on 2026-07-16. OVH informed customers that they would apply a security patch to our production database over the weekend. The expected service interruption was no longer than 30 minutes.
14:39 UTC
Our monitoring systems and availability probes detected an outage affecting our production services. As this matched the scheduled maintenance window, our support engineers considered the interruption expected.
16:00 UTC
After allowing an additional one-hour buffer beyond the planned maintenance window, our monitoring continued to report that the database was unavailable. Our support engineers checked the OVH status page, which still showed the maintenance as ongoing with no additional information. The OVH control panel also reported the database status as "Updating". At this stage, the extended downtime was still considered part of the maintenance.
17:15 UTC
The OVH status page remained unchanged. To gather additional information, our engineers consulted the public OVH Discord community, where other customers reported experiencing the same issue: databases remained in the "Updating" state while waiting for the security patch to complete.
21:20 UTC
With the announced maintenance window approaching its end, the database still unavailable, and business hours about to begin in the Australia/Oceania region, our support engineers opened a Priority 1 (P1) support ticket with OVH.
21:25 UTC
OVH acknowledged receipt of the incident and confirmed that the ticket had been assigned.
21:41 UTC
OVH confirmed an incident involving Aiven, the service responsible for managing the cloud database platform. They indicated that the issue was related to the planned maintenance operation and that the outage was taking significantly longer than the initially announced 30 minutes. They committed to providing further updates.
https://public-cloud.status-ovhcloud.com/incidents/f02sjn0qw27f
At this point, we updated the Golive and Time Squad status pages to report a service outage.
00:46 UTC
After more than ten hours of downtime without any meaningful update, our support engineers requested additional information from OVH to determine whether our Business Continuity Plan (BCP) should be activated. During this investigation, we also discovered that the OVH user interface did not allow us to create a database fork from an existing backup.
04:55 UTC
As no progress had been communicated publicly, our support engineers contacted OVH support again and also reached out to the on-call support team.
05:30 UTC
OVH support confirmed that the planned maintenance was still in progress but could not provide further details regarding the broader incident. We expressed our concerns about the lack of communication and requested additional information to support our operational decision-making. We also requested that OVH create a dedicated public incident rather than continuing to associate the outage solely with the maintenance operation.
07:20 UTC
Our engineers discovered that creating a database fork from a backup using the OVH REST API was possible, even though the option was unavailable in the web interface.
The most recent backup had been created at 14:15 UTC, while the outage started at 14:40 UTC, resulting in a potential data loss window of approximately 25 minutes.
After reviewing the database audit logs, our engineers confirmed that only a limited number of write operations had occurred during this period, primarily related to deployment and status monitoring activities due to the weekend. Based on this assessment, management decided to activate the Business Continuity Plan.
07:48 UTC
OVH published a dedicated incident, confirming that the outage was caused by a hardware failure rather than simply an extended maintenance window.
https://public-cloud.status-ovhcloud.com/incidents/bn54j3232tlv
08:00 UTC
Because our PostgreSQL version did not support database forking through the OVH web interface, we took advantage of the BCP activation to upgrade the database to PostgreSQL 18, a version that had already been successfully validated in our development and staging environments.
08:50 UTC
The restored database became available on PostgreSQL 18. Database integrity and sanity checks completed successfully, and Time Squad was restarted.
08:55 UTC
Time Squad was fully operational. Our status page was updated, and Golive services were restarted.
09:05 UTC
Despite several restart attempts, some Golive components failed to start correctly. Our engineers began investigating the root cause.
09:30 UTC
The investigation identified the issue as a backlog of asynchronous Atlassian webhook events (issue creation, updates, etc.) that had accumulated during the outage. Once connectivity was restored, Atlassian attempted to replay all missed events simultaneously, creating a significant load on our infrastructure before caches had been rebuilt.
09:55 UTC
A temporary mitigation was implemented by blocking incoming webhook traffic. This allowed Golive to complete its startup sequence and resume serving normal transactional traffic. The status page was updated accordingly.
10:13 UTC
A performance improvement that had already been validated in our development and staging environments and scheduled for an upcoming release was deployed to production. Webhook traffic was then re-enabled, and the combination of rebuilt caches and the performance improvement proved effective.
11:15 UTC
After more than one hour of stable event processing and normal system behavior, the incident was declared resolved. The status page was updated to indicate that all systems were fully operational.
During the incident, we initially could not activate our Business Continuity Plan because the OVH web interface did not allow us to create a database fork from a backup. This limitation was caused by the legacy PostgreSQL version running on our production environment.
As part of the recovery process, we upgraded our production database to PostgreSQL 18, the latest version supported by OVH and already validated in our non-production environments. This upgrade removes this limitation and enables faster recovery should a similar incident occur in the future.