Building upstream projects with git submodules

This document outlines some basic guidelines for synchronizing changes between an upstream and downstream repository using Git submodules.

By leveraging Git submodules, it is possible to maintain a consistent and up-to-date relationship between related repositories (upstream and downstream) while having more control of how the project is built, tested and released.

Additionally, Konflux can automate key aspects of this process, ensuring that changes from the upstream repository are efficiently and reliably propagated to the downstream repository.

Implementing this guideline

Basic definitions

  • Upstream repository: a community project which you do not have push access to (or which you do not want to push CI changes into).

  • Downstream repository: a self-owned copy of the upstream repository whose build, test, release lifecycle you want to control completely.

Create a downstream repository

The downstream repository can be used to do the following:

  • hold the git submodule reference to the upstream repository.

  • use the contents of the git submodule in a Containerfile for further edits.

  • onboarded onto Konflux.

Configuring components

Make sure to properly set the context and Containerfile path as required for the artifact builds. You can keep the context value at the the repository root where the Containerfile and git submodules are located.

See Creating applications and components for more details.

Configuring the Containerfile

In the Containerfile you can specify the git submodule directories and add any necessary edits that differ from the upstream repository. Here is a basic example:

FROM <base-image>

# Install necessary dependencies
RUN dnf -y install <dependencies>

# Set the working directory
ENV <SUBMODULEPKG> /path/to/submodule
WORKDIR ${<SUBMODULEPKG>}

# Commands to update the submodule for the current image build
RUN sed -i 's/<old-text>/<new-text>/g' file/in/submodule/directory
# Commands to build binary from submodule sources
RUN go build file/in/submodule/directory

WORKDIR /app

LABEL name="<name" \
      summary="<summary>" \
      description="<description>"

ENTRYPOINT ["<entrypoint-executable>"]

Create the git submodule for the upstream repository to track it

After creating the downstream repository, add a git submodule for the upstream repository:

git submodule add <upstream-url>

Check and configure the git submodule config:

# .gitmodules
[submodule "<project-name>"]
  path = <path-in-downstream-repository>
  url = https://github.com/<namespace>/<project>.git
  # Please note: by default a branch is not specified
  # in the git submodule configuration file.
  # Please consider setting one to prevent using the wrong branch.
  # Here is an example of a branch being set:
  branch = <designated-repository-branch>

See the Git documentation for more details on configuring git submodules.

Onboard the component onto Konflux

After creating the downstream repository you can onboard the component. See Creating applications and components for more details.

Using private repositories as a git submodule:

When using private repositories you must give Konflux access to them.

Private Gitlab submodules are currently untested.

Benefits of this guideline

  1. Separation of Concerns: The downstream repository can stay focused on its specific development goals while the submodule handles its own code. This keeps your downstream repository clean and modular.

  2. Automatic Dependency Syncing with Renovate: By automating the dependency sync process, whether its syncing the git submodule, the Containerfile, etc. your downstream repository can always use the latest code from the upstream repository(s), ensuring that your project benefits from new features, bug fixes, and security updates.

  3. Version Control for upstream repositories: You can lock the submodule to a specific commit or version, giving you control over when and how updates are integrated. This is especially useful if an upstream update introduces breaking changes.

  4. Konflux CI: Automated building and testing of your software artifacts help ensure that changes in the upstream repository don’t break your downstream project.

Potential drawbacks

  1. Complexity: Managing submodules and automating their updates adds complexity to your workflow, such as:

    • Debugging issues related to submodule updates

    • Your automation configuration can be more challenging.

    • Updates in the submodules can be opaque which can result in drift between your build process and that of the original repository.