On Tuesday, September 5, 2022, at 10:20 PM UTC, Atlassian customers using Jira Service Management, Jira Software, and Jira Work Management products started getting errors while creating subtasks for issues with required fields. The error was limited to users who had required fields for creating subtasks. This incident was caused by a change pushed to production that started deploying on September 5, 2022, at 10:15 PM UTC. The incident was detected within one hour and 20 minutes and the problem was mitigated for 90 percent of affected customers in the next 6 hours, by 5:48 AM UTC on September 6. The first mitigation to unblock the majority of customers was to revert back to the old issue create experience. The problem was completely resolved for the remaining ten percent in seven hours, by deploying the actual fix. The total time to resolution for all customers was ten hours and 45 minutes.
The overall impact was from 10:20 PM UTC on September 5, 2022, to 9:05 AM UTC on September 6, 2022, and affected customers of Jira Service Management, Jira Software, Jira Work Management products. The incident caused service disruption to users when they were creating subtasks with required fields in Jira.
The root cause was a recent change in create issue code which was part of an ongoing improvement task. This change was missed by our automated continuous deployment suites and manual tests in our testing environment and got deployed to the production environment. This change affected the creation of subtasks with required fields in Jira. We acknowledge that the incident could not be mitigated earlier because we missed following our own process, to Feature Flag every single change we make, for part of the change.
We know that outages impact your productivity. While we have a number of testing and preventative processes in place, our automated continuous deployment suites and test scripts didn’t cover this use case.
We are prioritizing the following improvement actions to avoid repeating this type of incident:
We apologize to customers whose services were affected during this incident; we are taking immediate steps to improve the platform’s performance and availability.
Atlassian Customer Support