Delays observed at JSM, Opsgenie and Compass alert search functionality in US Region
Postmortem
SUMMARY
Between September 10, 2025, at 13:56 UTC and September 11, 2025, at 14:40 UTC, Jira Service Management, Compass Cloud Operations, and Opsgenie customers in the U.S. region experienced degraded performance across web and mobile applications, as well as Alert REST APIs. Certain customers experienced difficulties with alert searches and page loading, particularly on alert-related pages. This incident was initiated by an EBS volume upgrade conducted within the cloud-based managed ElasticSearch clusters.
Our automated monitoring systems identified the incident within minutes, and it was resolved after 24 hours and 43 minutes by manually implementing vertical and horizontal scaling measures.
IMPACT
Between September 10, 2025, 13:56 UTC and September 11, 2025, 14:40 UTC, some customers in the U.S. region experienced degraded performance in Jira Service Management, Opsgenie, and Compass Cloud. During this time, web and mobile applications, as well as the Alert REST APIs, were impacted. Some customers may have seen issues with alert searches and slow loading of alert pages. During the incident, 22% of customers in the US region experienced failures in web API functionality, 8.6% encountered failures with the REST Alert API, and 1.3% experienced delays in notification delivery. Importantly, there was no loss of data during this event.
ROOT CAUSE
The degradation was caused by an EBS volume upgrade in the cloud-based managed ElasticSearch clusters, which necessitated a Blue/Green deployment strategy. One of the ElasticSearch nodes approached its shard size threshold, prompting the upgrade and subsequent deployment. This deployment resulted in elevated latency, increased 4xx HTTP response codes, and timeouts affecting both search and indexing operations. Recovery time exceeded expectations due to the prolonged blue/green deployment. After completion, the ElasticSearch cluster remained unhealthy and did not return to its normal state. Additionally, the switchover to the backup region failed because it was configured similarly in size and setup to the primary cluster.
REMEDIAL ACTIONS PLAN & NEXT STEPS
We understand that outages can affect your productivity. We are prioritizing the following improvement actions to help prevent similar incidents in the future:
- Increase ElasticSearch cluster capacity and optimize settings to handle peak search loads efficiently.
- Strengthen our high availability (HA) and failover architecture to ensure rapid and reliable recovery in the event of primary region failures.
- Optimize cluster upgrade and shard rebalancing strategies to prevent similar issues.
We apologize to customers whose services were impacted during this incident. We are taking steps to improve the platform’s performance and availability.
Thanks,
Atlassian Customer Support
Resolved
Monitoring
We will continue providing specific updates when available.
Update
Alerts continue to successfully notify users, in some cases the responder name may be missing. Our goal continues to restore full functionality to users from Jira Service Management, Opsgenie and Compass.
We will continue providing specific updates when available and will ensure we have further update within the hour.
Update
Alerts continue to successfully notify users, in some cases the responder name may be missing. Our goal continues to restore full functionality to users from Jira Service Management, Opsgenie and Compass.
We will continue providing specific updates when available and will ensure we have further update within the hour.
Update
Currently, the alerts continue to successfully notify users, except that the responder name may be missing.
We will continue providing specific updates when available and will ensure we have further update within the hour.
Update
Currently, the alerts continue to successfully notify users, except that the responder name may be missing.
We will continue providing specific updates when available and will ensure we have further update within the hour.
Update
Escalation teams also continue to remain engaged on this incident to ensure our focus is not diluted from a resolution as soon as possible.
Currently, the alerts continue to successfully notify users, except that the responder name may be missing.
We will continue providing specific updates when available and will ensure we have further update within the hour.
Update
Escalation teams are also engaged on this incident to ensure we can get to a resolution as soon as possible.
At this point, the alerts continue to successfully notify users, except that the responder name may be missing.
We will continue providing specific updates when available and will ensure we have further update within the hour.
Update
Escalation teams are also engaged on this incident to try and ensure we can get to a resolution as soon as possible.
Currently, the alerts continue to successfully notify users, but without the intended content within the alerts.
We will continue providing specific updates when available and will ensure we have further update within the hour.
Update
Escalation teams are fully engaged on this incident to try and ensure we can get to a resolution as soon as possible.
At this time these alerts are still successfully notifying users, but without the intended content within the alerts.
We will continue providing specific updates when available and will ensure we have further update within the hour.
Update
New alerts are continuing to notify users, however we are still prioritising infrastructure to restore the content inside these alerts for Web and Mobile UIs.
We will continue providing updates when available and will ensure we have further update within the hour.
Update
At this time we are prioritising infrastructure to try and restore new and active alerts to be populated with their alert content as soon as possible, while continuing to deliver existing messages as they are ready.
We will continue to provide updates as we progress, and will ensure we have an update posted within the hour.
Update
Please be aware we also have teams actively investigating these issues to ensure as fast a resolution as possible.
We want to reiterate that your alerts are still sending notifications correctly, and there is no data loss occurring. However, the underlying issue is affecting the propagation of alert details to notifications, and to the Web and Mobile UI.
Viewing schedules may also be impacted at this time while we continue to process the tasks required for recovery.
We will provide further updates as soon as possible.
Update
Our teams are still actively engaging with infrastructure teams to try and expedite historical processing of messages to fully restore services.
We will provide further updates within the hour.
Update
Our team is continuing to investigate with urgency potential infrastructure options to expedite this process.
We will provide further updates within the hour.
Update
There are a large number of missed alerts to be processed due to the prior service issue and our team is actively investigating methods to help expedite this processing.
Alerts should trigger in the meantime but the content of these alerts may not be visible at this time.
Our teams are continuing to investigate with urgency.
We will provide further updates within the hour.
Update
We are now continuing to see recovery of Jira Service Management operations and Opsgenie services.
Existing alerts are continuing to be processed at this time.
We will provide further update as soon as possible.