SUMMARY

On November 11, 2023, between 9:07 and 10:03 UTC, customers using Atlassian Analytics were unable to view charts on their dashboards. The event was triggered by a routine upgrade of a database cluster used by the service responsible for querying the data used in Atlassian Analytics charts. The incident was detected within 10 minutes by our monitoring system and mitigated by redeploying the impacted services which put affected Atlassian systems into a known good state. The total time to resolution was about 56 minutes.

IMPACT

The overall impact was between 9:07 UTC and 10:03 UTC on November 11, 2023. The incident caused a service disruption for all customers using Atlassian Analytics during the outage window. More specifically, the service responsible for querying data to populate charts experienced errors leading to failures when viewing and creating charts.

ROOT CAUSE

The issue was triggered by a routine upgrade of a database cluster which caused an unexpected loss of connectivity with the service responsible for issuing queries for Atlassian Analytics. Investigation during the incident identified that this was due to the database cluster changing the IP address after the upgrade. The service did not react to this change in an expected manner

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that outages impact your productivity. While we design our services to be highly available and resilient to most failures, this scenario was not handled as expected.

We are prioritizing the following actions to avoid repeating this type of incident:

• Implement more robust automatic database failure recovery.
• Improve existing and implement new monitoring processes in order to detect similar issues faster.
• Improve our internal documentation to more quickly resolve similar issues.

We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

Thanks,

Atlassian Customer Support