Prefetching package manager dependencies for hermetic builds

In Konflux, you can run a hermetic build by restricting network access to the build, but without network a build can’t fetch component dependencies from a repository and might fail. To avoid that, Konflux can prefetch dependencies for your hermetic builds using Cachi2.

For every build, Cachi2 generates a software bill of materials (SBOM) where all dependencies are properly declared and pinned to specific versions. Also, Cachi2 ensures that arbitrary code is never executed during the prefetch, meaning, for example, that the build doesn’t pull any undeclared dependencies. Such measures result in very accurate SBOMs and improve the build reproducibility. For more information about SBOMs, see Inspecting SBOMs.

Table 1. Available package managers

Package manager

Programming language

gomod

Go

pip

Python

npm

JavaScript

yarn

JavaScript

bundler

Ruby

rpm*

N/A

generic

N/A
*The rpm-lockfile-prototype and the rpm package manager for cachi2 are not fully supported. You can use them to prefetch rpms for your hermetic builds, but the file format and technology may change in the future. If you’re interested in the future of this topic, join the discussion at rpm-software-management/dnf5#833.

Procedure

To prefetch dependencies for a component build, complete the following steps:

  1. Go to the .tekton directory and find the .yaml files related to the pull request and push processes.

  2. Configure the hermetic pipeline by adding the following parameters in both .yaml files:

    pipelineSpec:
  params:
    ...
    - name: hermetic
      type: string
      description: Execute the build with network isolation
      default: "true"
    spec:
  params:
    ...
    - name: prefetch-input
      value: '{"type": "<package_manager>", "path": ".", "some_other_key": "some_other_value"}' (1)
    1 The prefetch-input parameter specifies the path to the directory of the project. In this example, the . indicates the repository root. If you have multiple directories, you can provide the path to those directories in the JSON array format: [{"type": "<package_manager>", "path": "."}, {"type": "<package_manager>", "path": "subpath/to/the/other/directory"}]. Additionally, you can add other keys and values to the JSON object supported by specific package managers.

  3. Create a pull request by committing your changes to the repository of the component.

  4. Review and merge the pull request.

Verification

  • From the Konflux Applications view, go to Activity > Pipeline runs.

    • Go to the pipeline run with Build in the Type column and confirm that the prefetch-dependencies stage displays a green checkmark. This indicates that the build process successfully fetched all dependencies.

  • From the Konflux Applications view, go to Activity > Latest commits.

Troubleshooting

If your build fails, be sure to look at your logs: In Konflux, from the Applications view, select the application build you want to troubleshoot, then from the resulting Overview page, select the Activity tab. From there, under Activity By, select Pipeline runs. From the Name column, select the build whose logs you want to check, then from the resulting Pipeline run details view, do one of the following:

  • Select the Logs tab.

  • Alternatively, you can click build-container. When the right panel opens, select the Logs tab to see a partial view of the log for that build.

Enabling prefetch builds for gomod

Prerequisites

  • You have a go.mod file in your repository that lists all the dependencies.

Prefetching Go modules from private repos

To enable downloading modules from private repos, Go supports authentication via .netrc. You can create a netrc Secret for your pipeline as described below. For more Go-specific .netrc details, see the Go docs.

Enabling prefetch builds for pip

Cachi2 supports pip by parsing of pip requirements files, including but not limited to, requirements.txt files placed in the root of your repository. By generically parsing pip requirements files, Cachi2 downloads the specified dependencies.

The requirements file can have a different name because you can use multiple files to provide the dependencies. These requirements files function as lockfiles, encompassing all transitive dependencies. You must actively pin each transitive dependency listed in the requirements file to a specific version.
Prerequisites

  • You have an environment that closely matches the environment in the container build, meaning it has the same operating system and the same python $major.$minor version.

  • You have installed the pip-tools package.

To prefetch dependencies for a component build, complete the following steps:

  1. Download the pip_find_builddeps.py script directly from GitHub.

    This script has no runtime dependency other than pip.

  2. Add the script that you downloaded in a directory that is already included in your $PATH. For example, you can use the ~/bin directory in your home folder. Ensure that it exists or create it if needed. To add it to the $PATH permanently, you can modify the shell configuration file (for example, .bashrc, .bash_profile, or .zshrc) and restart the terminal after appending the following line:

    export PATH="$HOME/bin:$PATH"

  3. Open the terminal and go to the directory where you placed the pip_find_builddeps.py script and run the following command to make it executable:

    chmod +x pip_find_builddeps.py

  4. Go to your component’s source code.

  5. Review the root of your repository for a metadata file, for example, pyproject.toml, setup.py, or setup.cfg. If there is no metadata file, create one, because Cachi2 looks for the name and version of your project in the metadata files.

    [metadata]
name = "my_package"
version = "0.1.0"
    Instead of a pyproject.toml file, you can also create a setup.py or setup.cfg file. For information about the metadata of these files, see Project metadata.

  6. Generate a fully resolved requirements.txt file that contains all the transitive dependencies and pins them to a specific version and hash by using the following command:

    $ pip-compile pyproject.toml --generate-hashes

    • To successfully run the previous command, your environment must be as close as possible to the environment in the container build. That is, the environment should have the same operating system and the same Python $major.$minor version.

    • The previous command assumes that you have defined project dependencies in pyproject.toml. However, if you have defined the project dependencies in either the setup.py, requirements.txt, or requirements.in files, make sure you update the command accordingly.

  7. Add the requirements.txt file to the root of your component source code.

  8. In the root of your repository create a requirements-build.in file.

  9. Copy the build system requirements from the pyproject.toml file to the requirements-build.in file.

  1. Run the pip_find_builddeps.py script and pip-compile the outputs by using the following command:

    $ pip_find_builddeps.py requirements.txt \
--append \
--only-write-on-update \
-o requirements-build.in

  2. Use the pip-compile command to convert the requirements-build.in file in to the requirements-build.txt file by using the following command:

    $ pip-compile requirements-build.in --allow-unsafe --generate-hashes

  3. Add the requirement-build.txt file to your project. It does not require any changes to your build process.

    pip automatically installs the build dependencies when needed for explicit installation. The purpose of the requirement-build.txt file is to enable Cachi2 to fetch the build dependencies and provide them to pip for offline installation in a network-isolated environment.

    • By default, Cachi2 processes requirements.txt and requirements-build.txt at a specified path.

    • When adding these parameters, you can safely ignore the default values for the pipelineSpec.params in the .yaml files.

    1. Optional: For requirements files without default names and path, add the following hermetic pipeline prefetch-input parameters in both the .yaml files:

    spec:
    params:
        -   ...
        -   name: prefetch-input
            value: '{"type": "pip", "path": ".", "requirements_files": ["requirements.txt", "requirements-extras.txt", "tests/requirements.txt"]}' (1)
    1 The prefetch-input parameter specifies the path to the directory that has the lockfile and the package metadata files. In the previous example, the . indicates that the package manager lockfile is located in the root of the repository. Additionally, if you have multiple directories, you can provide the path to those directories in the JSON array format. For example, [{"path": ".", "type": "pip", , "requirements_files": ["requirements.txt", "requirements-extras.txt", "tests/requirements.txt"]}, {"path": "subpath/to/the/other/directory", "type": "pip", "requirements_files": ["requirements.txt", "requirements-extras.txt", "tests/requirements.txt"]}].
To troubleshoot any issues you might experience when you enable prefetch builds for pip with source dependencies, see cachi2 documentation

Prefetching pip dependencies from custom index servers

Cachi2 supports the --index-url option. You can add this option to your requirements.txt file(s), instructing Cachi2 to download packages from the specified index server. For example:

--index-url=https://example.pypi.org/simple/
requests==2.32.2 \
    --hash=sha256:dd951ff5ecf3e3b3aa26b40703ba77495dab41da839ae72ef3c8e5d8e2433289 \
    --hash=sha256:fc06670dd0ed212426dfeb94fc1b983d917c4f9847c863f313c9dfaaffb7c23c

# ...other packages
Do not include credentials in the index URL. If needed, provide authentication through a .netrc file (as described below). For more pip-specific details on netrc files, review the pip documentation for netrc support.

Enabling prefetch builds for npm

Cachi2 supports npm by fetching any dependencies you declare in your package.json and package-lock.json project files. The npm CLI manages the package-lock.json file automatically, and Cachi2 fetches any dependencies and enables your build to install them without network access.

Prerequisites

  • You have an up-to-date package-lock.json file, newer than version 1, in your source repository. To make sure that you have the latest package-lock.json file, or to create a lockfile, run the npm-install command. You can also look at the lockfileVersion attribute in your package-lock.json file to make sure its value is a number greater than 1.

Enabling prefetch builds for yarn

Supported versions: 1.x and 3.x. Cachi2 automatically detects the version of yarn and fetches any dependencies you declare in your package.json and yarn.lock project files.

Prerequisites

  • You have an up-to-date yarn.lock file in your source repository. To ensure you have the latest yarn.lock file or to create it, run the yarn install command. If yarn.lock is not up-to-date, Cachi2 will not fetch the dependencies.

Enabling prefetch builds for bundler

Prerequisites

You have a Gemfile and a Gemfile.lock file in your repository that lists all the dependencies. The Gemfile.lock is generated from the Gemfile and contains all transitive dependencies pinned to specific versions.

Enabling prefetch builds for rpm

Cachi2 has a dev-preview package manager capable of fetching rpm dependencies. This requires the use of a pair of rpms.in.yaml and rpms.lock.yaml files to be committed to your repository. You write a rpms.in.yaml file and the rpm-lockfile-prototype CLI tool resolves that to produce a rpms.lock.yaml file. Cachi2 fetches those specific rpms and enables your build to install them without network access.

Prerequisites

To prefetch dependencies for a component build, complete the following steps:

  1. Create a rpms.in.yaml file in your git repository, with the following contents:

    packages: [nethack] (1)
contentOrigin:
  repofiles: ["./fedora.repo"] (2)
arches: [x86_64, aarch64] (3)
    1 The packages list is the list of packages you want to install in your Container. You don’t have to declare transitive dependencies here. The rpm-lockfile-prototype tool will resolve them for you.
    2 This should be a reference to a repo file, like those found in /etc/yum.repos.d/. This tells the tooling where to find your rpm and its dependencies.
    3 The arches array allows you to specify which architectures the dependencies should be downloaded for. If you’re building a multi-arch container this array is mandatory, otherwise the build task will fail.

  2. Copy any necessary yum/dnf repo files into your git repository. If you are using a fedora rawhide base image, that looks like:

    $ BASE_IMAGE=quay.io/fedora/fedora:rawhide
$ podman run -it $BASE_IMAGE cat /etc/yum.repos.d/fedora.repo > fedora.repo
    For every repository defined in your set of repo files, make sure to add the corresponding sources repo (or make sure to enable them, if they’re already present). Otherwise, the lockfile generator will not include any SRPMs in your lockfile, cachi2 won’t download any SRPMs and the source container for your build will be incomplete.

  3. Run the following command to resolve your rpms.in.yaml file and produce a rpms.lock.yaml file.

    $ BASE_IMAGE=quay.io/fedora/fedora:rawhide
$ rpm-lockfile-prototype --image $BASE_IMAGE rpms.in.yaml (1)
    1 The produced rpms.lock.yaml file will include only your requested dependency plus its transitive dependencies, minus any rpms that are already installed in the provided base image.

  4. Additionally, pass an extra parameter to the prefetch-dependencies task in the .spec.pipelineSpec.tasks section to indicate that "dev package managers" should be enabled.

        tasks:
        -   ...
        -   name: prefetch-dependencies
            params:
                - ...
                - name: dev-package-managers (1)
                  value: "true"
    1 You won’t find dev-package-managers as a param on the prefetch-dependencies task. You have to add it, and set it to true. This is because Cachi2 hasn’t declared stable support for rpm lockfile processing yet. It’s new technology and the conversation about which way forward in the dnf community is still ongoing.
Konflux also supports prefetching RPM content which requires a Red Hat subscription. For more information see Using Red Hat activation keys to access subscription content.

Enabling prefetch builds for generic fetcher

If you need to prefetch arbitrary files for your build, Cachi2 supports generic fetcher for that purpose. It uses a custom lockfile named artifacts.lock.yaml to achieve this. This file needs to be either commited in the source repository, or explicitly specified as an absolute path. The latter is useful in case you for some reason need the lockfile to be dynamic and committing it to the repository would be problematic. For more information on supported types of artifacts, see Cachi2 documentation.

To prefetch dependencies for a component build, complete the following steps:

  1. Create a artifacts.lock.yaml file in your git repository, with a list of files to prefetch, their checksums, and optionally their filenames. See Cachi2 documentation for complete overview of the lockfile format.

    ---
metadata: (1)
  version: "1.0"
artifacts:
  - download_url: "https://github.com/jeremylong/DependencyCheck/releases/download/v11.1.0/dependency-check-11.1.0-release.zip"
    checksum: "sha256:c5b5b9e592682b700e17c28f489fe50644ef54370edeb2c53d18b70824de1e22" (2)
    filename: "dependency-check.zip" (3)
    1 metadata section is required and needs to specify lockfile version
    2 checksum is expected to be specified as algorith:hash
    3 If no filename is specified, it will be derived from the URL.

Creating the netrc secret

The prefetch-dependencies task supports .netrc files for authentication. For example, it is useful for:

To create a netrc Secret for use in your pipelines:

  1. Create the .netrc file locally. See the .netrc docs for more details on the file format.

  2. Create a key/value secret (see creating secrets for your builds). Set .netrc as the key. Upload the content of your .netrc file as the value.

  3. In your .tekton/ PipelineRun files, add the newly created secret to the .spec.workspaces section:

    spec:
  # ...
  workspaces:
    # ...
    - name: netrc
      secret:
        secretName: my-netrc  # the name you assigned to the secret in step 2

  4. In the .spec.pipelineSpec.tasks section, find the entry with name: prefetch-dependencies. Add the netrc workspace to the list of workspaces (if not present):

        tasks:
      # ...
      - name: prefetch-dependencies
        # ...
        workspaces:
          # ...
          - name: netrc
            workspace: netrc