Performance degradation with Git and website
Incident Report for Atlassian Bitbucket
Postmortem

An article covering our response to this and other recent incidents has been published to Atlassian Community.

Posted May 04, 2021 - 14:32 UTC

Resolved
Our services have been operating smoothly for several hours now. Response times and error rates continue to stay at healthy levels.

Bitbucket Cloud engineering teams are working diligently on remedial actions to defend our systems from the impact of the performance degradation we experienced this morning.

We will be following up this incident with a public postmortem, here and on the Bitbucket Cloud blog.
Posted Apr 20, 2021 - 21:59 UTC
Monitoring
Based on our metrics, all services have returned to normal operation and our response times and error rates have reached healthy levels. We are continuing to monitor the situation and making action plans to prevent these issues from occurring again.
Posted Apr 20, 2021 - 16:17 UTC
Update
Our team is continuing to investigate performance issues. We have already added capacity to our website services but have not observed the performance improvements we had hoped for.

Thus far we have struggled to identify a root cause. Each component of our infrastructure appears to be operating normally, but throughput and overall response times are significantly elevated. We are also trying to understand why website and Git services are affected while API services continue to meet standard performance targets.

At the moment we are in the process of rolling out some configuration changes to offload pressure from our file storage to a caching layer while continuing to explore other mitigation options, such as identifying hot spots and disabling non-critical functionality temporarily to unblock customers who are currently prevented from using Bitbucket's core features.

We can observe response times trending downward and expect this issue to be resolved within the next hour, though we are doing our best to reach resolution sooner than that.
Posted Apr 20, 2021 - 14:55 UTC
Update
We are continuing to investigate reported performance issues and will provide a status update within the next hour.
Posted Apr 20, 2021 - 12:15 UTC
Investigating
We are investigating cases of degraded performance for some of our Atlassian Bitbucket Cloud customers. We will provide more details within the next hour.
Posted Apr 20, 2021 - 09:17 UTC
This incident affected: Website, SSH, Git via HTTPS, and Pipelines.