Resolved -
We have investigated and resolved a problem with Jira custom fields, impacting the Jira issue view user experience.
Because of this problem, all the Forge-powered custom fields visible in the issue view page were rendered every time a user interacted with the issue, for example, by typing in the comment field. This resulted in a slow user experience and excessive calls made to apps.
We were able to identify the source of the problem, a change controlled by a feature flag, and mitigate it by turning the flag off.
Timeline
- 2022-11-14 at 10:30 UTC - impact started as the rollout of the feature flag started, only 10 per cent of sites received the new change.
- 2022-11-15 at 02:32 UTC - the change has been deployed to 50 per cent of the sites via feature flag.
- 2022-11-16 at 12:30 UTC - the change has been deployed to all sites, and the feature flag rollout reached 100 per cent of sites.
- 2022-11-19 at 00:13 UTC - the feature flag was turned off, and the impact was mitigated.
The incident is now resolved. This is the final update.We apologize for any inconveniences this may have caused you, your team, and our mutual customers. We are committed to finding and preventing the root cause of the issue.
Nov 19, 00:28 UTC
Identified -
We are investigating a problem with Jira custom fields, impacting the Jira issue view user experience.
It appears all the custom fields visible in the issue view page are rendered while the user is interacting with the issue, for example, typing in the comment field. This results in a slow user experience and excessive calls made to apps.
We believe this is only impacting the custom fields powered by Forge and we have been able to establish that Connect apps and OAuth integrations are not impacted.
We were able to identify the root cause of the problem which was a change controlled by a feature flag, and we are currently working on mitigating the problem.
Timeline
2022-11-14 at 10:30 UTC - impact started as the rollout of the feature flag started, only 10 per cent of sites received the new change
2022-11-15 at 02:32 UTC - the change has been deployed to 50 per cent of the sites via feature flag
2022-11-16 at 12:30 UTC - the change has been deployed to all sites, and feature flag rollout reached 100 per cent of sites
The current ETA for resolving this incident is 4 hours. We are assessing if we can safely disable the feature flag and we will send another update once the assessment is done.
We will provide another update by 01:00 UTC, 2 hours from now.
Nov 19, 00:05 UTC
Update -
We have identified a problem with Jira custom fields where the user is interacting with the issue. We believe this is only impacting the custom fields powered by Forge, but still investigating if Connect is also impacted.
It appears all the custom fields visible in the issue view field are rendered while user is typing in the comment field. This results in a slow user experience and excessive calls made to the app.
The impact has started at least a day ago, as reported by a Marketplace partner, and we are working on determining the exact time it has started. We are still assessing if other integrations are impacted.
We have been able to reproduce the problem and we are investigating the root cause. There is no known workaround as of now.
We do not currently have an ETA for the resolution. We will provide another update by 23:30 UTC, 90 minutes from now.
Nov 18, 22:07 UTC
Investigating -
We have identified a problem with Jira custom fields where the user is interacting with the issue. We believe this is only impacting the custom fields powered by Forge, but investigating if the impact is beyond that. We are investigating the root cause and will provide another update by 21:30 UTC, an hour from now.
Nov 18, 20:34 UTC
