Pipelines slow to start due to Google Container Registry latency issues
Incident Report for Atlassian Bitbucket
Resolved
This incident has been resolved. We no longer see long image pull times from GCR.
Posted Apr 29, 2022 - 05:36 UTC
Update
We continue to see slow image pulls from GCR. Google estimate a mitigation to be in place by Thursday, 2022-04-28 18:15 US/Pacific. Please follow their status page for updates at https://status.cloud.google.com/. Image pulls from Artifact Registry are not affected.

As a mitigation, we recommend either setting a max-time on steps, to prevent them consuming excess build minutes if you do experience a slow image pull, or transitioning to Artifact Registry, which is now the recommended way of storing container images - https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr.
Posted Apr 28, 2022 - 23:29 UTC
Monitoring
Google have confirmed an issue with Google Container Registry that is causing high latency when pulling images. Please monitor their service status page at https://status.cloud.google.com/ for further updates.
Posted Apr 28, 2022 - 00:27 UTC
Identified
We've identified that images hosted in Google Container Registry (gcr.io) are slow to pull, whilst images hosted on Artifact Registry (pkg.dev) are not impacted. We’ve contact Google for further information.

For anyone using gcr.io/google.com/cloudsdktool/cloud-sdk, you can swap to using the same image hosted at Dockerhub (https://hub.docker.com/r/google/cloud-sdk/).
Posted Apr 27, 2022 - 03:56 UTC
Investigating
We are investigating reports that some pipelines are slow to start, and stuck pulling images for several minutes.
Posted Apr 26, 2022 - 22:47 UTC
This incident affected: Pipelines.