Troubleshooting Builds

No space left on device

Tasks may fail with an error message mentioning No space left on device as the underlying error. This likely means your build pipeline wrote more data than expected into a shared volume.

Typically seen in the clone-repository or prefetch-dependencies task in a build pipeline.

For the clone task, the error message may look similar to:

[clone] {"level":"error","ts":1721904304.0047252,"caller":"git/git.go:53","msg":"Error running git [checkout -f FETCH_HEAD]: exit status 128\nerror: unable to write file ...: No space left on device\n"

The device that’s running out of space is most likely the workspace declared in your PipelineRun YAML files. The solution is to request more disk space. In the .spec.workspaces section in all the relevant PipelineRun files, increase the storage request.

spec:
  # ...
  workspaces:
    # ...
    - name: workspace
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 1Gi  # increase accordingly

Pipeline Run Times Out

Tasks may fail with an error message mentioning PipelineRun <pipelineName> failed to finish within "1h0m0s".

If you see this error message, it means that the pipeline run has exceeded the default one hour time limit set for PipelineRuns. You can increase the timeout if necessary, see Configuring timeouts.

Manually Update Task Bundles

Usually, Konflux users rely on renovate to update the various Task bundle references in the build Pipelines. However, it is also possible to update these references manually if needed. For example, consider a build Pipeline that includes the following Task:

- name: init
  params:
    - name: image-url
      value: $(params.output-image)
    - name: rebuild
      value: $(params.rebuild)
    - name: skip-checks
      value: $(params.skip-checks)
  taskRef:
    params:
      - name: name
        value: init
      - name: bundle
        value: quay.io/konflux-ci/tekton-catalog/task-init:0.2@sha256:284e3029cce3ae5ee0b05866100e300046359f53ae4c77fe6b34c05aa7a72cee
      - name: kind
        value: task
    resolver: bundles

You can find the newest digest for the Task bundle with skopeo and jq. You must first remove the digest from the existing reference. For example:

skopeo inspect --no-tags docker://quay.io/konflux-ci/tekton-catalog/task-init:0.2 | jq -r '.Digest'

The output will contain a new digest, e.g. sha256:4c6712db9419461b8c8a39523c012cb0dc061fb58563bb9170b3777d74f54659. Update the Task bundle reference in your build Pipeline to use the new digest.

The script below provides a working example of how to achieve this for all the Task bundle references in a given build Pipeline file.

cat <<'EOF' > update-tekton-task-bundles.sh
#!/bin/bash

# Use this script to update the Tekton Task Bundle references used in a Pipeline or a PipelineRun.
# update-tekton-task-bundles.sh .tekton/*.yaml

set -euo pipefail

FILES=$@

# Find existing image references
OLD_REFS="$(\
    yq '... | select(has("resolver")) | .params // [] | .[] | select(.name == "bundle") | .value'  $FILES | \
    grep -v -- '---' | \
    sort -u \
)"

# Find updates for image references
for old_ref in ${OLD_REFS}; do
    repo_tag="${old_ref%@*}"
    new_digest="$(skopeo inspect --no-tags docker://${repo_tag} | yq '.Digest')"
    new_ref="${repo_tag}@${new_digest}"
    [[ $new_ref == $old_ref ]] && continue
    echo "New digest found! $new_ref"
    for file in $FILES; do
        sed -i -e "s!${old_ref}!${new_ref}!g" $file
    done
done
EOF

chmod +x update-tekton-task-bundles.sh

./update-tekton-task-bundles.sh .tekton/*.yaml