Charts are not loading on Atlassian Analytics
Incident Report for Atlassian Analytics
Postmortem

SUMMARY

On November 11, 2023, between 9:07 and 10:03 UTC, customers using Atlassian Analytics were unable to view charts on their dashboards. The event was triggered by a routine upgrade of a database cluster used by the service responsible for querying the data used in Atlassian Analytics charts. The incident was detected within 10 minutes by our monitoring system and mitigated by redeploying the impacted services which put affected Atlassian systems into a known good state. The total time to resolution was about 56 minutes.

IMPACT

The overall impact was between 9:07 UTC and 10:03 UTC on November 11, 2023. The incident caused a service disruption for all customers using Atlassian Analytics during the outage window. More specifically, the service responsible for querying data to populate charts experienced errors leading to failures when viewing and creating charts.

ROOT CAUSE

The issue was triggered by a routine upgrade of a database cluster which caused an unexpected loss of connectivity with the service responsible for issuing queries for Atlassian Analytics. Investigation during the incident identified that this was due to the database cluster changing the IP address after the upgrade. The service did not react to this change in an expected manner

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that outages impact your productivity. While we design our services to be highly available and resilient to most failures, this scenario was not handled as expected.

We are prioritizing the following actions to avoid repeating this type of incident:

  • Implement more robust automatic database failure recovery.
  • Improve existing and implement new monitoring processes in order to detect similar issues faster.
  • Improve our internal documentation to more quickly resolve similar issues.

We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

Thanks,

Atlassian Customer Support

Posted Nov 22, 2023 - 17:16 UTC

Resolved
Simple
Between 09:05 UTC to 10:00 UTC, we experienced charts failing to load for Atlassian Analytics. The issue has been resolved and the service is operating normally.

Detailed
Between 09:05 UTC to 10:00 UTC, all customers experienced charts failing to load for Atlassian Analytics. The root cause was a upstream update that caused database connectivity problems. There is no impact to the Atlassian Data Lake. We have deployed a fix to mitigate the issue and have verified that the services have recovered. The conditions that cause the bug have been addressed and we’re actively working on a permanent fix. The issue has been resolved and the service is operating normally.
Posted Nov 11, 2023 - 10:09 UTC
Investigating
We are investigating an issue with charts that is impacting Atlassian Analytics customers. We will provide more details within the next hour.
Posted Nov 11, 2023 - 10:02 UTC
This incident affected: Dashboards.