Building upstream projects with git submodules

This document outlines a basic guide for synchronizing changes between an upstream and downstream repository using Git submodules.

By leveraging Git submodules, it is possible to maintain a consistent and up-to-date relationship between related repositories (upstream and downstream) while having more control of how the project is built, tested and released.

Additionally, Konflux can automate key aspects of this process, ensuring that changes from the upstream repository are efficiently and reliably propagated to the downstream repository.

Basic definitions

  • Upstream repository: a community project which you do not have push access to (or which you do not want to push CI changes into).

  • Downstream repository: a self-owned copy of the upstream repository whose build, test, release lifecycle you control completely.

Benefits of this guide

  1. Separation of Concerns: The downstream repository can stay focused on its specific development goals while the submodule handles its own code. This keeps your downstream repository clean and modular.

  2. Automatic Dependency Syncing with Renovate: By automating the dependency sync process, whether its syncing the git submodule, the Containerfile, etc. your downstream repository can always use the latest code from the upstream repository(s), ensuring that your project benefits from new features, bug fixes, and security updates.

  3. Version Control for upstream repositories: You can lock the submodule to a specific commit or version, giving you control over when and how updates are integrated. This is especially useful if an upstream update introduces breaking changes.

  4. Konflux CI: Automated building and testing of your software artifacts help ensure that changes in the upstream repository don’t break your downstream project.

Potential drawbacks

  1. Complexity: Managing submodules and automating their updates adds complexity to your workflow, such as:

    • Debugging issues related to submodule updates

    • Your automation configuration can be more challenging.

    • Updates in the submodules can be opaque which can result in drift between your build process and that of the original repository. There are some ways to manage this.

Procedure

By the end of this guide, you will have created a downstream repository which does the following:

  • holds the git submodule reference to the upstream repository.

  • uses the contents of the git submodule in a Containerfile for further edits.

  • is onboarded to Konflux for builds.

Create a downstream repository

Create a downstream repository in the source control system of your choice, GitHub or Gitlab.

Forgejo and Gitea are not yet fully supported.

Your downstream repository is a standalone repository. It should not be a fork of the upstream repository.

Create the git submodule for the upstream repository to track it

After creating the downstream repository, add a git submodule to it that tracks the upstream repository:

git submodule add -b <upstream-branch> <upstream-git-url>
git commit -m 'Adding git submodule tracking <upstream-branch> branch of <upstream-git-url>'

Check and configure the git submodule config:

# .gitmodules
[submodule "<project-name>"]
  path = <path-in-downstream-repository>
  url = https://github.com/<namespace>/<project>.git
  branch = <upstream-repository-branch>
By doing this, you have created a git submodule reference that tracks a branch, but it is currently pinned to a particular upstream commit. When new upstream commits land, your repository will not instantaneously update to consume them. Instead, an automated process in Konflux called mintmaker will propose pull requests that keep your submodule up to date.
See the Git documentation for more details on configuring git submodules.

Create a Containerfile

In the Containerfile you can specify the git submodule directories and add any necessary edits that differ from the upstream repository. Here is a basic example:

FROM <base-image>

# Install necessary dependencies
RUN dnf -y install <dependencies>

# Set the working directory
ENV <SUBMODULEPKG> /path/to/submodule
WORKDIR ${<SUBMODULEPKG>}

# Commands to update the submodule for the current image build
RUN sed -i 's/<old-text>/<new-text>/g' file/in/submodule/directory
# Commands to build binary from submodule sources
RUN go build file/in/submodule/directory

WORKDIR /app

LABEL name="<name" \
      summary="<summary>" \
      description="<description>"

ENTRYPOINT ["<entrypoint-executable>"]

You should be able to prove to yourself that the downstream Containerfile builds locally with podman build ..

You can see examples of Containerfiles that follow this pattern in Konflux’s own yq-container and oras-container repositories.

Onboard the component onto Konflux

After creating the downstream repository you can onboard the component. See Creating applications and components for more details.

Alternative: Use an args file

Depending on the nature of your upstream, it may be possible to build from a downstream repo without creating a downstream Containerfile. If the upstream repository’s Containerfile supports ARG values, you can create only a "build arguments" file in your downstream repository, and configure your Konflux build pipeline to use the upstream Containerfile with downstream args at build time. This has the advantage of not having to maintain a copy of the Containerfile in the downstream repo.

See examples in passing buildah arguments.

Using private repositories as a git submodule:

When using private repositories you must give Konflux access to them.

Private Gitlab submodules are currently untested.