Troubleshooting

No space left on device

Tasks may fail with an error message mentioning No space left on device as the underlying error. This likely means your build pipeline wrote more data than expected into a shared volume.

Typically seen in the clone-repository or prefetch-dependencies task in a build pipeline.

For the clone task, the error message may look similar to:

[clone] {"level":"error","ts":1721904304.0047252,"caller":"git/git.go:53","msg":"Error running git [checkout -f FETCH_HEAD]: exit status 128\nerror: unable to write file ...: No space left on device\n"

The device that’s running out of space is most likely the workspace declared in your PipelineRun YAML files. The solution is to request more disk space. In the .spec.workspaces section in all the relevant PipelineRun files, increase the storage request.

spec:
  # ...
  workspaces:
    # ...
    - name: workspace
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 1Gi  # increase accordingly

Pipeline Run Times Out

Tasks may fail with an error message mentioning PipelineRun <pipelineName> failed to finish within "1h0m0s".

If you see this error message, it means that the pipeline run has exceeded the default one hour time limit set for PipelineRuns. You can increase the timeout if necessary, see Configuring timeouts.

Failed to push or pull image

Failing to authenticate with a container registry can be hard to debug. To be able to troubleshoot effectively, one needs to understand how registry authentication works in Tekton. For the full details, see the Tekton Authentication docs. In short:

  1. Tekton uses a ServiceAccount (named appstudio-pipeline by default in Konflux) to run your Pipeline.

  2. This ServiceAccount has a list of imagePullSecrets and secrets.

    1. Tekton uses the imagePullSecrets when pulling the images that execute the Tasks in your pipeline. These images are typically hardcoded in the Task definition and publicly accessible. As a Konflux user, you’re more interested in the secrets.

    2. Tekton injects the secrets into the execution environment so that the tools executed inside the Tasks (e.g. buildah, skopeo, oras, cosign) can authenticate to registries.

Tekton takes all the dockercfg and dockerconfigjson secrets linked in secrets (as well as specially annotated basic-auth secrets), merges them into a single file and injects the file into the Task Pod at ~/.docker/config.json. The format of the file is described in the containers-auth.json manpage.

Note that the merged file can contain multiple credentials for a single registry, distinguished by path. For example:

{
  "auths": {
    "quay.io/my-org": {
      "auth": "..."
    },
    "quay.io": {
      "auth": "..."
    }
  }
}

The most specific path takes precedence. When accessing images in quay.io/my-org/, the tool will prefer the quay.io/my-org credential over the generic quay.io credential (assuming the tool implements the containers-auth.json spec correctly).

More tips and tricks for debugging below.

Check if a secret has access to a registry / to a specific image

Prerequisites: jq, kubectl or oc, access to the Secrets in the namespace.

secret_name=secret-for-my-registry
should_have_access_to=my.registry.io/my-org/my-image

kubectl get secret "$secret_name" -o json |
    jq '.data[] | @base64d | fromjson | {auths: (.auths // .)}' |
    tee /tmp/auth.json

# Check if a tool can use the authfile to access a registry / an image. E.g. skopeo:
skopeo login --authfile /tmp/auth.json "$should_have_access_to"

Note: works for dockercfg and dockerconfigjson secrets, not basic-auth.

Check if the secret is linked to the appstudio-pipeline service account

In order to connect the new secret to pipeline run, you need to link it from service account used by Tekton to run the pipeline run. Check if your secret appears in the secrets section:

kubectl get sa appstudio-pipeline -o json

If not, link it to the appstudio-pipeline service account with:

secret_name=secret-for-my-registry

# using kubectl
kubectl patch serviceaccount appstudio-pipeline -p "{\"secrets\": [{\"name\": \"$secret_name\"}]}"

# using oc
oc secrets link appstudio-pipeline "$secret_name"

Get the merged registry auth file

Prerequisites: jq, kubectl or oc, access to the Secrets in the namespace.

This script roughly approximates Tekton’s merging of dockercfg and dockerconfigjson secrets. It does not handle basic-auth secrets.

The output is a containers-auth.json file (you can e.g. save it as /tmp/auth.json and use it the same way as in the example above). Each entry in the file has an extra _from_secret attribute showing which Secret provides the entry. This may be useful to determine which Secret is introducing problematic content into the merged file.

linked_secrets=$(kubectl get sa appstudio-pipeline -o json | jq '.secrets | map(.name)')

kubectl get secrets -o json |
    jq --argjson linked_secrets "$linked_secrets" '
        .items | map(
            . as $secret |
            select($linked_secrets | index($secret.metadata.name)) |
            .data | .[".dockercfg"], .[".dockerconfigjson"] | select(. != null) |
            @base64d | fromjson |
            .auths // . |
            to_entries[] |
            {registry: .key, config: .value, secret: $secret.metadata.name}
        ) |
        reduce .[] as $x ({}; .[$x.registry] = $x.config + {_from_secret: $x.secret}) |
        {auths: .}
    '