Pipelines slow to start due to Google Container Registry latency issues

Incident Report for Atlassian Bitbucket

Resolved

This incident has been resolved. We no longer see long image pull times from GCR.
Posted Apr 29, 2022 - 05:36 UTC

Update

We continue to see slow image pulls from GCR. Google estimate a mitigation to be in place by Thursday, 2022-04-28 18:15 US/Pacific. Please follow their status page for updates at https://status.cloud.google.com/. Image pulls from Artifact Registry are not affected.

As a mitigation, we recommend either setting a max-time on steps, to prevent them consuming excess build minutes if you do experience a slow image pull, or transitioning to Artifact Registry, which is now the recommended way of storing container images - https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr.
Posted Apr 28, 2022 - 23:29 UTC

Monitoring

Google have confirmed an issue with Google Container Registry that is causing high latency when pulling images. Please monitor their service status page at https://status.cloud.google.com/ for further updates.
Posted Apr 28, 2022 - 00:27 UTC

Identified

We've identified that images hosted in Google Container Registry (gcr.io) are slow to pull, whilst images hosted on Artifact Registry (pkg.dev) are not impacted. We’ve contact Google for further information.

For anyone using gcr.io/google.com/cloudsdktool/cloud-sdk, you can swap to using the same image hosted at Dockerhub (https://hub.docker.com/r/google/cloud-sdk/).
Posted Apr 27, 2022 - 03:56 UTC

Investigating

We are investigating reports that some pipelines are slow to start, and stuck pulling images for several minutes.
Posted Apr 26, 2022 - 22:47 UTC
This incident affected: Pipelines.