70. Support for zstd:chunked image compression

Date: 2026-05-13

# Status

Proposed

# Context

Konflux compresses all container image layers with gzip. zstd:chunked is a newer format built on the Zstandard algorithm that offers smaller compressed sizes, faster decompression, and support for partial-layer downloads. There are reports of up to 90% speed increase in large image pulls (e.g. CUDA) with zstd:chunked.

The problem is backward compatibility. Older Docker versions (pre-23.0) try to read zstd layers as gzip and fail with archive/tar: invalid tar header. Other tools like buildah (1.36+), podman (5.1+), skopeo (1.15+), and CRI-O already support zstd. Docker is the main blocker.

As described in build-definitions#1264, pushing both gzip and zstd:chunked variants into the same OCI image index works as long as gzip entries are listed first. While the OCI image index spec leaves platform selection implementation-defined, older Docker clients predictably select the first matching platform entry, so they will pick gzip. Podman reads the io.github.containers.compression.zstd annotation and prefers the zstd variant automatically.

This was originally proposed in build-definitions#1264. It was also demanded in build-definitions#3188 for large CUDA images, and release-service-catalog#634 already noted the need for zstd layer handling in Pyxis.

# Decision

Add a dual-compression setting to all build pipelines. In dual mode, the build task pushes both gzip and zstd:chunked variants and bundles them into a per-arch OCI image index. The task exposes the index digest as IMAGE_DIGEST, and explicitly lists the individual gzip and zstd child manifest digests in the IMAGES result.

This is an experimental approach with a known constraint on Tekton result sizes (the 4KB limit). TEP-0164 (tektoncd/community#1248) proposes external storage for Tekton results which will resolve this limitation. Until then, dual-compression is opt-in. It also risks breaking release pipeline tasks that assume one manifest per architecture, so those need deduplication fixes before the default can change.

While the implementation details below focus on the buildah task, this convention applies to all image-producing tasks and pipelines in the Konflux ecosystem (e.g. oci-copy, modelcar, and any future build tasks). Authors of those tasks should follow the same dual-compression pattern when adding zstd support.

# Implementation details

The buildah task will get 2 new params:

• COMPRESSION_FORMAT — gzip (default), zstd-chunked, or dual
• FORCE_COMPRESSION — when true, recompresses all layers, including the base image layers

In dual mode, the push step creates a per-arch index containing both compression variants:

1. Build the image locally
2. Push the gzip variant to the registry
3. Push the zstd:chunked variant to the registry
4. Create a per-arch index with buildah manifest create + buildah manifest add, adding the gzip manifest first
5. Push the per-arch index to the unique tag and the real image tag

The individual manifests do not need their own temporary tags; they are pushed by digest and referenced from the per-arch index directly.

Both variant pushes must use --format=oci. Mixing OCI and Docker manifest formats within a single index causes “manifest invalid” error on Quay. If COMPRESSION_FORMAT=dual and the image format is not OCI, the task must detect this incompatible combination and error out early with a clear message.

IMAGE_DIGEST points to the per-arch index, which contains both variants. Because Tekton Chains does not currently support recursive signing of child manifests within an index (tektoncd/chains#1070), the build task explicitly exports the digests for both the gzip and zstd child manifests in its IMAGES result. This allows Chains to generate provenance for each manifest directly from the trusted build task, closing potential trust gaps. Downstream tasks, SBOM generation, and signing consume IMAGE_DIGEST as the primary artifact. The io.github.containers.compression.zstd annotation is set on the zstd manifest descriptor within the index, which is how Podman and CRI-O identify and prefer the zstd variant.

In gzip or zstd-chunked mode, the build task pushes a single manifest as it does today. IMAGE_DIGEST holds the primary digest (gzip in gzip mode, zstd in zstd-chunked mode).

The build-image-index task requires no new parameters. Each entry in the existing IMAGES array is now a per-arch index (in dual mode) instead of a single manifest. The task’s existing --all flag on buildah manifest add extracts child manifests from per-arch indexes and flattens them into the final multi-arch index. The --all preserves both the gzip-first ordering and the io.github.containers.compression.zstd annotation on the zstd child manifest descriptors. ADR 0069 proposes a platform sort in build-image-index for reproducibility. That sort must either include compression type in the sort key or use a stable sort to preserve the gzip-first invariant.

SBOM generation requires an update. Each image in the per-arch index (the index itself, the gzip child, and the zstd child) needs its own SBOM attached. Although both variants share the same package list, the SBOM’s main component field includes the image digest, so each SBOM must reference the digest of the image it is attached to. This means Mobster needs to run separately for each variant with the correct digest, rather than once with the index digest.

The per-arch index also needs an SBOM. For single-arch builds, the per-arch index is the final output. For multi-arch builds, the per-arch index persists in the registry by its unique tag even after build-image-index flattens it into the final index. The per-arch index SBOM should follow the same structure as the final multi-arch index SBOM — one package for the per-arch index itself (with the index digest as the main component), one package for the gzip child, and one package for the zstd child, each with their own digest.

The final multi-arch index SBOM (created by build-image-index) also needs an update. It currently lists one entry per architecture. With dual-compression it will list 2N entries for N architectures, and each entry should distinguish between gzip and zstd so consumers can tell them apart.

The mapping data for which child is gzip and which is zstd must be supplied by the build pipeline, since Mobster cannot gather this information on its own.

Six release tasks in release-service-catalog call get-image-architectures and loop over all the manifests. With dual-compression they will see duplicate entries per architecture. Each task needs a jq dedup filter that groups by (architecture, os) and keeps the non-zstd entry. The zstd manifests remain in the released image index and are available for pull; the filter only prevents duplicate metadata in Pyxis, advisory content, and related systems.

• create-pyxis-image — would create duplicate Pyxis entries and doesn’t have zstd decompression logic
• populate-release-notes — would list the same architecture twice in advisory content
• filter-already-released-advisory-images — would treat zstd variants as “unreleased” since they have new digests
• extract-oot-kmods — would miscount architectures and fail extracting kmods from zstd layers
• push-rpm-data-to-pyxis — uses cosign download to fetch SBOMs from child manifest digests, which works since each variant has its own SBOM with the correct digest as the main component
• rpms-signature-scan — crashes extracting zstd layers with archive/tar: invalid tar header because the RPM verifier tool does not understand zstd compression

Signing tasks are not affected; they sign all manifests in the index, which is correct for both variants.

# Rollout

The default value of COMPRESSION_FORMAT stays as gzip until the release task fixes are merged and tested. Only then will the default change to dual. Conforma also needs follow-up work before the default can flip: Chains generates a single provenance attestation with all three images (index, gzip child, zstd child) as subjects, and attaches that same attestation to all three. Conforma’s subject_build_task_matches rule checks each subject against IMAGE_URL@IMAGE_DIGEST from the build task. Since IMAGE_DIGEST is the per-arch index, only the index subject matches. The gzip and zstd child subjects do not match IMAGE_DIGEST, triggering violations on all three images being validated. Once dual-compression adoption is stable and older Docker versions are no longer a concern, the default can move to zstd-chunked only, dropping the gzip variant entirely. There may be additional considerations to take into account when deciding to switch the default to dual or zstd-chunked. For example, if major installations are still requiring the Docker format instead of OCI for images, then changing this default might need to wait until those installations are ready to move.

# Alternatives Considered

# Add compression at index push time

Buildah’s --add-compression flag automatically creates zstd variants at index push time. This is simpler, but the zstd manifests are created during buildah manifest push in the build-image-index task. They are not results of any TaskRun, so Chains cannot generate provenance for them. At release time, Conforma would find manifests without provenance and block the release.

# Push two individual manifests with separate results

Instead of creating a per-arch index, the buildah task could push two individual manifests and expose both digests as separate results (IMAGE_DIGEST for gzip, ZSTD_IMAGE_DIGEST for zstd). This requires a new ZSTD_IMAGES parameter on build-image-index and raises several concerns: orphaned manifests subject to garbage collection if not referenced by a tag or index, and breakage in gzip-only mode when the zstd result is empty. The per-arch index avoids all of these by bundling both variants under a single digest.

# Separate intermediary task

A new task could accept the gzip manifest digest, pull the image, and re-push it with zstd compression. This works for provenance, but adds complexity, a new task definition, new pipeline wiring, and an extra cycle. Pushing twice from the existing buildah task is simpler and keeps everything in one place.

# Consequences

Pros:

• Faster pulls for modern clients. Users with Podman and CRI-O will automatically benefit from zstd:chunked without changing their pipeline configuration. Docker 23.0+ clients gain tolerance for the zstd layer, but will continue to receive the gzip variant via first-match selection.
• Full backward compatibility. Old Docker clients continue to work because gzip is listed first in the image index.
• Provenance and signing. Because the build task explicitly exports the child manifests in IMAGES, Tekton Chains generates a single provenance attestation with all images (index, gzip child, zstd child) as subjects, avoiding trust gaps where manifests are only discovered by downstream tasks.

Cons:

• The final image index contains 2N manifests instead of N (one per architecture per compression format). Exposing all of these in the IMAGES result increases the Tekton result size, pushing closer to the 4KB limit for large matrix builds (e.g., builds exceeding 4-5 architectures may hit this limit).
• Registry storage per image roughly doubles. Each layer is stored in both gzip and zstd formats. Users who don’t want this can keep COMPRESSION_FORMAT=gzip.
• Builds take longer in dual mode. The zstd push recompresses all layers, adding time proportional to image size.
• Release pipeline tasks must be updated to handle duplicate manifests before the default can safely change to dual.