Weaviate Docker Image Pull Failure On GitLab Runner During CI

Q: 1. Verify runner’s Docker configuration

# On the runner host docker info --format '{{json .}}' | jq .RegistryConfig Expected output (when auth is configured):

Q: 2. Test manual pull from the runner

# Directly on the runner host (outside CI) docker pull semitechnologies/weaviate:1.22.0 If the same “unauthorized” error appears, the problem is not CI‑specific.

Q: 3. Check Docker Hub rate‑limit headers

# Use curl with a token‑less request curl -I https://registry-1.docker.io/v2/semitechnologies/weaviate/manifests/1.22.0 Sample response when limit is exceeded:

Q: 4. Inspect GitLab Runner config

# /etc/gitlab-runner/config.toml [[runners]] name = "docker-auto-scale" url = "https://gitlab.com/" token = "xxxxxxxx" executor = "docker" [runners.docker] privileged = true tls_verify = false image = "docker:latest" # Missing: auth_config = "/etc/docker/auth.json" Absence of auth_config indicates no credentials are passed to Docker.

Q: 5. Validate DNS from within the job container

# Add a debug step in .gitlab-ci.yml debug: image: busybox script: - nslookup registry-1.docker.io If nslookup fails, the network namespace cannot resolve Docker Hub.

Q: 1. Provide Docker Hub credentials to the runner

Store a Docker Hub service account in GitLab CI variables (DOCKERHUB_USER and DOCKERHUB_PASSWORD) and generate an auth.json file at runner startup.

In this article

Table of Contents

Problem

A GitLab CI job that sets up integration tests fails during the docker pull step for the semitechnologies/weaviate image. The runner aborts with a non‑zero exit code, preventing the pipeline from progressing.

Typical CI log excerpt:

Running with gitlab-runner 16.2.0 (c8b1c8c5)
  on docker-auto-scale 1234567890abcdef
Preparing the "docker" executor
Using Docker image sha256:7c3b... for build image
Pulling docker image semitechnologies/weaviate:1.22.0 ...
ERROR: failed to pull image "semitechnologies/weaviate:1.22.0": Get "https://registry-1.docker.io/v2/semitechnologies/weaviate/manifests/1.22.0": unauthorized: authentication required
ERROR: Job failed: exit status 1

Impact:

Integration tests never start, causing false negatives in CI.
Pipeline throughput drops as jobs are retried.
Developers lose visibility into actual code failures.

Root Cause

The failure originates from Docker Hub denying the image pull. The underlying reasons can be grouped into three categories:

Rate‑limit enforcement – Unauthenticated requests to Docker Hub are limited to 100 pulls per 6 hours per IP. The self‑hosted runner shares a public IP with other services, quickly exhausting the quota.
Missing authentication credentials – The runner is not configured to pass a Docker Hub username/password or a CI_JOB_TOKEN to the Docker daemon, so Docker treats the request as anonymous.
Runner network isolation – The Docker executor runs inside a custom network namespace that lacks proper DNS resolution or outbound connectivity, causing the request to fail before reaching Docker Hub.

In most production incidents the immediate error is “unauthorized: authentication required”, which is Docker Hub’s generic response for both rate‑limit and auth failures. Without additional context the CI job appears to “just fail”.

Debug

Follow these steps to isolate the exact trigger.

1. Verify runner’s Docker configuration

# On the runner host
docker info --format '{{json .}}' | jq .RegistryConfig

Expected output (when auth is configured):

{
  "IndexConfigs": {
    "docker.io": {
      "Name": "docker.io",
      "Mirrors": null,
      "Official": true,
      "Secure": true
    }
  },
  "InsecureRegistryCIDRs": [],
  "Mirrors": null
}

2. Test manual pull from the runner

# Directly on the runner host (outside CI)
docker pull semitechnologies/weaviate:1.22.0

If the same “unauthorized” error appears, the problem is not CI‑specific.

3. Check Docker Hub rate‑limit headers

# Use curl with a token‑less request
curl -I https://registry-1.docker.io/v2/semitechnologies/weaviate/manifests/1.22.0

Sample response when limit is exceeded:

HTTP/1.1 429 Too Many Requests
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 1622548800

4. Inspect GitLab Runner config

# /etc/gitlab-runner/config.toml
[[runners]]
  name = "docker-auto-scale"
  url = "https://gitlab.com/"
  token = "xxxxxxxx"
  executor = "docker"
  [runners.docker]
    privileged = true
    tls_verify = false
    image = "docker:latest"
    # Missing: auth_config = "/etc/docker/auth.json"

Absence of auth_config indicates no credentials are passed to Docker.

5. Validate DNS from within the job container

# Add a debug step in .gitlab-ci.yml
debug:
  image: busybox
  script:
    - nslookup registry-1.docker.io

If nslookup fails, the network namespace cannot resolve Docker Hub.

Solution

Apply the fixes that address the identified root cause. The most robust approach combines authentication with a pull‑through cache to mitigate rate limits.

1. Provide Docker Hub credentials to the runner

Store a Docker Hub service account in GitLab CI variables (DOCKERHUB_USER and DOCKERHUB_PASSWORD) and generate an auth.json file at runner startup.

Before (no auth):

# /etc/gitlab-runner/config.toml
[runners.docker]
  privileged = true

After (auth enabled):

# /etc/gitlab-runner/config.toml
[runners.docker]
  privileged = true
  auth_config = "/etc/docker/auth.json"

Create the auth file (executed by a systemd unit or a pre‑start script):

cat > /etc/docker/auth.json <<EOF
{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "$(echo -n "$DOCKERHUB_USER:$DOCKERHUB_PASSWORD" | base64)"
    }
  }
}
EOF
chmod 600 /etc/docker/auth.json

2. Enable a local pull‑through cache (optional but recommended)

Deploy a lightweight registry mirror (e.g., registry:2) on the same network and configure the runner to use it.

Docker daemon daemon.json before:

{
  "registry-mirrors": []
}

After adding the mirror:

{
  "registry-mirrors": ["http://registry-mirror.local:5000"]
}

Restart Docker on the runner host:

systemctl restart docker

3. Ensure privileged mode and network access

If the runner was missing privileged = true, Docker pull may be blocked by AppArmor or SELinux. Verify the flag is set (see config snippet above). Also confirm outbound port 443 is allowed through any host firewall.

4. Pin a specific Weaviate version and avoid “latest” tag

Using an immutable tag prevents accidental pulls of a newer image that may have different rate‑limit behavior.

# .gitlab-ci.yml
services:
  - name: semitechnologies/weaviate:1.22.0
    alias: weaviate

Verify

Run the pipeline after applying the changes and confirm the pull succeeds.

# Expected CI log fragment
Pulling docker image semitechnologies/weaviate:1.22.0 ...
Using Docker image sha256:9f2c... for weaviate
Pull complete

Additional verification steps:

Check docker images inside the job container for the Weaviate image.
Execute a simple health‑check against the service:

curl -sSf http://localhost:8080/v1/.well-known/ready
# Expected output: {"status":"ready"}

Inspect the runner’s Docker daemon logs for any residual authentication errors:

journalctl -u docker -f | grep "weaviate"

Prevent

Cache pulls: Keep a private registry mirror to absorb Docker Hub rate‑limit spikes.
Credential rotation: Store Docker Hub credentials as protected CI variables and rotate them quarterly.
Monitor rate limits: Enable a Prometheus metric scrape of Docker Hub response headers (e.g., RateLimit-Remaining) and alert when below 20.
Pin images: Always reference a specific semantic version or digest to avoid accidental “latest” pulls.
Runner health checks: Periodically run a lightweight job that pulls a known small image (e.g., alpine:3.18) and fails the pipeline if the pull takes >30 seconds.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the pull succeed locally but fail in CI?
Because the local machine authenticates with Docker Hub (via docker login) while the CI runner runs without credentials, hitting the anonymous rate‑limit.
Can I use the GitLab CI_JOB_TOKEN to authenticate to Docker Hub?
No. CI_JOB_TOKEN only works for the GitLab Container Registry. For Docker Hub you must supply a Docker Hub username/password or a personal access token.
Is the “unauthorized” error always a rate‑limit issue?
Not always. It can also indicate missing or malformed auth JSON. Checking the HTTP RateLimit-Remaining header distinguishes the two.
Do I need privileged = true for image pulls?
Privileged mode is required for Docker‑in‑Docker scenarios where the job itself runs Docker commands. Without it, the Docker client cannot access the host daemon, leading to pull failures.
How do I verify which registry mirror was used for a pull?
Enable Docker daemon debug logging ("debug": true in daemon.json) and look for lines containing “pulling from” in journalctl -u docker. The URL printed indicates the mirror.