Troubleshooting Registry Issues

Failed to push or pull image

Failing to authenticate with a container registry can be hard to debug. To be able to troubleshoot effectively, one needs to understand how registry authentication works in Tekton. For the full details, see the Tekton Authentication docs. In short:

  1. Tekton uses a ServiceAccount (build-pipeline-$COMPONENT_NAME for build and release-service-account for release by default in Konflux) to run your Pipeline.

  2. This ServiceAccount has a list of imagePullSecrets and secrets.

    1. Tekton uses the imagePullSecrets when pulling the images that execute the Tasks in your pipeline. These images are typically hard-coded in the Task definition and publicly accessible. As a Konflux user, you’re more interested in the secrets.

    2. Tekton injects the secrets into the execution environment so that the tools executed inside the Tasks (e.g. buildah, skopeo, oras, cosign) can authenticate to registries.

Tekton takes all the dockercfg and dockerconfigjson secrets linked in secrets (as well as specially annotated basic-auth secrets), merges them into a single file and injects the file into the Task Pod at ~/.docker/config.json. The format of the file is described in the containers-auth.json manpage.

Note that the merged file can contain multiple credentials for a single registry, distinguished by path. For example:

{
  "auths": {
    "quay.io/my-org": {
      "auth": "..."
    },
    "quay.io": {
      "auth": "..."
    }
  }
}

The most specific path takes precedence. When accessing images in quay.io/my-org/, the tool will prefer the quay.io/my-org credential over the generic quay.io credential (assuming the tool implements the containers-auth.json spec correctly).

More tips and tricks for debugging below.

Check if a secret has access to a registry / to a specific image

Prerequisites: jq, kubectl or oc, access to the Secrets in the namespace.

secret_name=secret-for-my-registry
should_have_access_to=my.registry.io/my-org/my-image

kubectl get secret "$secret_name" -o json |
    jq '.data[] | @base64d | fromjson | {auths: (.auths // .)}' |
    tee /tmp/auth.json

# Check if a tool can use the authfile to access a registry / an image. E.g. skopeo:
skopeo login --authfile /tmp/auth.json "$should_have_access_to"

Note: works for dockercfg and dockerconfigjson secrets, not basic-auth.

Check if the secret is linked to a service account

In order to connect the new secret to pipeline run, you need to link it from service account used by Tekton to run the pipeline run. Check if your secret appears in the secrets section:

kubectl get sa $SERVICE_ACCOUNT_NAME -o json

If not, link it to the $SERVICE_ACCOUNT_NAME service account with:

secret_name=secret-for-my-registry

# using kubectl
kubectl patch serviceaccount $SERVICE_ACCOUNT_NAME -p "{\"secrets\": [{\"name\": \"$secret_name\"}]}"

# using oc
oc secrets link $SERVICE_ACCOUNT_NAME "$secret_name"

Get the merged registry auth file

Prerequisites: jq, kubectl or oc, access to the Secrets in the namespace.

This script roughly approximates Tekton’s merging of dockercfg and dockerconfigjson secrets. It does not handle basic-auth secrets.

The output is a containers-auth.json file (you can e.g. save it as /tmp/auth.json and use it the same way as in the example above). Each entry in the file has an extra _from_secret attribute showing which Secret provides the entry. This may be useful to determine which Secret is introducing problematic content into the merged file.

linked_secrets=$(kubectl get sa $SERVICE_ACCOUNT_NAME -o json | jq '.secrets | map(.name)')

kubectl get secrets -o json |
    jq --argjson linked_secrets "$linked_secrets" '
        .items | map(
            . as $secret |
            select($linked_secrets | index($secret.metadata.name)) |
            .data | .[".dockercfg"], .[".dockerconfigjson"] | select(. != null) |
            @base64d | fromjson |
            .auths // . |
            to_entries[] |
            {registry: .key, config: .value, secret: $secret.metadata.name}
        ) |
        reduce .[] as $x ({}; .[$x.registry] = $x.config + {_from_secret: $x.secret}) |
        {auths: .}
    '