Managed pipeline automatic retries

A Managed Pipeline can fail for reasons that have nothing to do with the content being released, such as a Task running out of memory. Re-running the Pipeline with more memory or more time is often enough for the Release to succeed. Konflux does this automatically. When a Managed Pipeline fails for these reasons, it retries the Pipeline multiplying memory or increasing timeouts for the retry attempt.

Automatic retries are managed by Konflux and are not user defined. From your ReleasePlanAdmission (RPA) you can check whether retries are active for your Managed Pipeline and adjust how many retries are allowed.

Check if your pipeline is configured for retries

Check your RPA’s status to see whether retries are active for your Managed Pipeline:

$ kubectl get releaseplanadmission <rpa-name> -n <managed-tenant-namespace> -o yaml

Look for the retryInfo field under status:

status:
  retryInfo:
    enabled: true (1)
    maxRetries: 3 (2)
    disablingTags: (3)
      - '.*\{\{\s*incrementer\s*\}\}.*'
    reason: retries enabled by policy (4)
1 Whether automatic retries are active for this RPA’s Managed Pipeline.
2 The number of retry attempts that will be made after the initial Pipeline fails.
3 The tag patterns that disable retries when they match a tag in the Release, ReleasePlan or ReleasePlanAdmission data. See Tags that disable retries.
4 A short explanation of the result, for example pipeline not configured for retries, retries disabled by RPA pipeline override, or disabled by ReleasePlan or ReleasePlanAdmission tag(s): <tags> when a tag is blocking retries.

If retries aren’t configured for the Managed Pipeline, retryInfo will show:

status:
  retryInfo:
    enabled: false
    reason: pipeline not configured for retries

Check the attempts on the Release

You can see the status of each attempt on the Release CR itself, under status.managedPipelineAttempts. Every attempt records which PipelineRun ran. If it failed it also includes the failure reason and the last Task and Step that were running.

When an attempt fails for a retriable reason, the next attempt records the mitigation that was applied to it:

status:
  managedPipelineAttempts:
    - failureReason: (1)
      lastTask: (2)
      lastStep: (3)
      pipelineRun: rhtap-releng-tenant/managed-qd8pz
      status: Failed
    - mitigation: (4)
        memoryRequest: 400Mi
        memoryLimit: 400Mi
      pipelineRun: rhtap-releng-tenant/managed-x7kf2
      status: Succeeded
1 The failure reason for this attempt, such as OOMKill, PipelineRunTimeout, or TaskRunTimeout. A retriable reason triggers a retry.
2 The last Task that was running when the attempt failed.
3 The last Step that was running when the attempt failed.
4 The mitigation applied to the retry, such as an multipily memory or an increased timeout.

If a retry was skipped because a tag on the Release matched a disabling tag, the attempt records retryDisabledReason instead of retrying (see Tags that disable retries):

status:
  managedPipelineAttempts:
    - failureReason: (1)
      lastTask: verify-access-to-resources
      lastStep: verify-access-to-resources
      pipelineRun: rhtap-releng-tenant/managed-qd8pz
      retryDisabledReason: 'disabled by Release tag(s): v1.0.0-{{ incrementer }}' (2)
      status: Failed
1 The failure reason was retriable, so the attempt would normally be retried.
2 The tag on the Release CR that matched a disabling tag and stopped the retry.

Understand which failures trigger a retry

Not every failure is retried. Retrying a genuine error is unlikely to make a Release succeed. Konflux only retries a Managed Pipeline for failures that can be fixed with more resources or time. The following failure reasons trigger a retry:

  • OOMKill: A Task or Step was killed for exceeding its memory limit.

  • PipelineRunTimeout: The overall PipelineRun didn’t finish within its timeout.

  • TaskRunTimeout: An individual TaskRun didn’t finish within its timeout.

An Error failure is never retried automatically.

Override the retry count

If your Pipeline is configured for retries, you can override how many attempts are allowed by setting maxRetries on the pipeline field of your RPA:

apiVersion: appstudio.redhat.com/v1alpha1
kind: ReleasePlanAdmission
metadata:
  name: <rpa-name>
  namespace: <managed-tenant-namespace>
spec:
  applications:
    - demo-app
  data: <key>
  origin: <dev-tenant-namespace>
  pipeline:
    pipelineRef:
      resolver: git
      params:
        - name: url
          value: "<url-to-repo>.git"
        - name: revision
          value: <revision>
        - name: pathInRepo
          value: "<path-to-your-pipeline>"
    serviceAccountName: release-service-account
    maxRetries: 5 (1)
  policy: <policy>
1 Overrides the default retry count for this RPA. Set it higher to allow more attempts or set it to 0 to disable automatic retries.
maxRetries only has an effect on a Pipeline that Konflux already retries.

Retries can adjust resources and timeouts

Retrying a failed attempt with the same resources or timeouts usually produces the same failure. To give the retry a chance of succeeding, Konflux applies mitigations to the failing Pipeline or Task on each attempt:

  • After an OOMKill, the Task or Step’s memory request and limit are increased by a configured multiplier, up to a maximum.

  • After a PipelineRunTimeout or TaskRunTimeout, the relevant timeout is increased by a configured increment, up to a maximum.

Mitigations are managed by Konflux and are not user defined.

Understand what the mitigation multiplies

A mitigation multiplies or increments whatever resources or timeout the failed attempt ran with, not the Catalog’s default. If you’ve overridden a Task or Step’s computeResources using taskRunSpecs (See Adjusting timeouts and resources), that override is the starting point for the next retry’s mitigation. The same applies to timeouts you’ve set on the Pipeline.

Mitigations accumulate. Each retry builds on the previous attempt’s not the original, so resources or timeouts keep growing until the Release succeeds or the configured maximum is reached.

Tags that disable retries

A Pipeline configured for retries can still skip retries when specific tags appear in your release data. This is useful for tags that should never be retried, such as an unresolved template variable like {{ incrementer }} that would push a different image tag on every attempt.

The tags that disable retries are configured by Konflux and matched against the tags in your data. Each configured tag can be either a static tag or a regular expression:

  • A static tag, such as {{ incrementer }}, matches that exact tag.

  • A regular expression matches any tag that fits the pattern. For example, .\{\{\s*incrementer\s\}\}.* matches both {{ incrementer }} and v1.0.0-{{ incrementer }}.

Konflux checks for disabling tags in two places:

  • Tags in your ReleasePlan or ReleasePlanAdmission data disable retries for the Managed Pipeline entirely. When this happens the RPA’s status.retryInfo.enabled is false and status.retryInfo.reason reads disabled by ReleasePlan or ReleasePlanAdmission tag(s): <tags>.

  • Tags in an individual Release data are checked when its Managed Pipeline fails for a retriable reason and would otherwise be retried. A matching tag skips the retry for that Release only, and the attempt records retryDisabledReason: disabled by Release tag(s): <tags>.

When more than one tag matches, all matching tags are listed, separated by commas.