The internet is full of articles explaining why you shouldn’t use Git submodules. But what alternatives are there? One option is Git Subtree, which I’d like to take a closer look at in this article.
Git Subtree connects an external repository with the main repository, as if it were a simple folder. The history of the external repository is preserved in the process. Developers can make changes in the external repository and transfer them to the main repository with ease. Likewise, they can pull changes from the main repository into the external repository.
Why Using Git Submodules Can Be a Bad Idea
Before we take a closer look at Git Subtree, here are once again the reasons why using Git submodules is unpopular with some developers:
Cloning complexity: Git does not clone submodules by default, which leads to increased complexity. You must use either
git submodule update --init --recursiveorgit clone --recursive <repository-url>to clone submodules.Manual synchronization required: Git submodules are not automatically synchronized. This creates additional effort, as you have to run
git submodule updateto bring the submodules up to date.Additional steps when making changes: If files have been changed in both the main module and the submodule, this requires additional steps such as committing/pushing or reverting changes in both modules. This can easily lead to confusion and errors.
Difficulties with merge conflicts: Resolving merge conflicts can be particularly challenging when using Git submodules, especially when changes have been made in both the main repository and the submodule repository.
Complicated comparison of file revisions: Moving files or directories between the main project and the submodule repository can make comparing file revisions considerably more difficult.
In summary, the entire team needs a fair amount of knowledge to handle submodules. This creates friction during development that distracts from the actual development work.
What Are the Pros and Cons of Using Git Subtree
Now let’s take a closer look at Git Subtree. What exactly do you gain or lose by using this approach?
Advantages:
- Immediate access to subproject code: After cloning the main project, the subproject code is directly available.
- No additional learning for repository users: Users don’t need to deal with new concepts, as they can ignore the use of Subtree for managing dependencies.
- No additional metadata files: Unlike submodules, Subtree does not add new metadata files like .gitmodule.
- Flexibility with module changes: The module content can be modified without a separate copy of the dependency in the repository.
Disadvantages:
- Need to learn a new merge strategy: Learning the Subtree merge strategy is required, but only for those who want to work with the Subtree’s remote.
- Complexity when contributing subproject code upstream: Bringing changes back into the subprojects is somewhat more complex.
- Responsibility for separating main and subproject code: It is your responsibility to keep the code of main and subprojects cleanly separated in commits.
Practice Makes Perfect
The use of Git Subtree is best explained through examples. To do this, I’d like to walk through some typical use cases that one encounters when working with Git Subtree.
Preparation: Create Two Local Test Repos
First, we need a repository that will be integrated as a Subtree in a project. For testing purposes, I’ll create a local Git repository with three commits below.
mkdir repo1
cd repo1
git init
touch file1.txt
git add --all
git commit --message "Initial commit"
echo "Change 1" >> file1.txt
git add --all
git commit --message "First change"
echo "Change 2" >> file1.txt
cat file1.txt
git add --all
git commit --message "Second change"
cd ..
Next, I need a Git repository into which the other repository will be integrated as a Subtree.
mkdir subtree_test
cd subtree_test
git init
touch README.md
git add --all
git commit --message "Initial commit"
cd ..
Now that we have created both repositories, we can proceed and apply the Subtree concept in practice.
Use Case 1: Adding a Subtree to a Project
First, I add the repository “repo1” as a Subtree to my repository “subtree_test.”
cd subtree_test
git subtree add --prefix my_repo1 ../repo1 master
The command line option “–prefix” specifies the directory in which the Subtree should be created. In our case, this is “my_repo1.” This is followed by the upstream reference, in the example our local repository. In the general case, this is usually a link to a Git server.
After executing the command, the contents of the repository “repo1” are created in the directory “./my_repo1.”
The git commit graph (git log --graph --oneline) of the repository “subtree_test” then looks like this:
* 0745df0 (HEAD -> master) Add 'my_repo1/' from commit '4499ee8ec47f747f2beb30512db9202d6a76f650'
|\
| * 4499ee8 Second change
| * 99564a0 First change
| * 33c5799 Initial commit
* 5beb502 Initial commit
Note the following: There are now two root commits – one for the repository “subtree_test” and another for “repo1.” Additionally, adding a Subtree integrates the entire history of the added repository into the main repository, enabling a clean and traceable connection between both repositories. The metadata, such as the origin of the added Subtree and the commits involved, is stored in the commit message to ensure a better overview and easier tracking of changes.
Use Case 2: Adding a Subtree but Squashing the History
Let’s reset the master back to the “Initial commit.”
git reset --hard HEAD~1
And now use the command line option --squash:
git subtree add --prefix my_repo1 ../repo1 master --squash
The Git commit graph (git log --graph --oneline) of the repository “subtree_test” now looks like this:
* cdea121 (HEAD -> master) Merge commit '7d5bc4a1dcafca9be8bcd161fa0f038655001695' as 'my_repo1'
|\
| * 7d5bc4a Squashed 'my_repo1/' content from commit 4499ee8
* 5beb502 Initial commit
“repo1” is still part of the “subtree_test” repo. However, the history of “repo1” has now been consolidated beforehand. In Git jargon, this is called a “squash.” This is useful, among other things, when you don’t need the history of “repo1” in your main project.
Use Case 3: Pulling a Change from “repo1” into “subtree_test”
Now let’s switch to “repo1” and add another change:
cd ..
cd repo1
echo "Change 3" >> file1.txt
git add --all
git commit --message "Third change"
cd ..
Now let’s try to pull this change into our main repo “subtree_test”:
cd subtree_test
git subtree pull --prefix my_repo1 ../repo1 master --squash
Here a weakness of Git Subtree becomes apparent: No link to “repo1” is stored in the Git repository. If we want to work with a remote, we must always specify the link to that remote again.
CAUTION: There is a bug that prevents a “subtree pull” without --squash if --squash was used previously. This bug has, in my opinion, been fixed in Git 2.40.x. However, this is usually irrelevant since people generally work with --squash anyway.
The Git commit graph (git log --graph --oneline) of the repository “subtree_test” now looks like this.
* 19d5c22 (HEAD -> master) Merge commit '702d51b28ba5daa1b1d99431a071ea5d4df81b6b'
|\
| * 702d51b Squashed 'my_repo1/' changes from 4499ee8..50d8d46
* | cdea121 Merge commit '7d5bc4a1dcafca9be8bcd161fa0f038655001695' as 'my_repo1'
|\|
| * 7d5bc4a Squashed 'my_repo1/' content from commit 4499ee8
* 5beb502 Initial commit
As you can see, the changes were again consolidated in their own “squash” commit. They were then integrated into the master branch in a “merge” commit.
Use Case 4: Pushing a Change from “subtree_test” into “repo1”
Since Git does not allow pushing to a checked-out branch by default, our “repo1” must first be set to a different branch. This is done as follows:
cd ..
cd repo1
git checkout -b test_branch
cd ..
This step is unnecessary when working with real “remotes” instead of local repositories. E.g., with GitHub.
Now you can push your changes into “repo1” as follows:
cd subtree_test
cd my_repo1
echo "Change 4" >> file1.txt
git add --all
git commit --message "Change in subtree_test"
cd..
git subtree push --prefix my_repo1 ../repo1 master
The command now performs a “split.” This separates the changes specific to the Subtree from the changes in the main repository. The changes from the Subtree are then pushed into the “repo1” repository. This process ensures that only the relevant changes in the Subtree are transferred to the external repository and the history of both projects remains clean.
Now if you look at the commit graph (git log --graph --oneline master) of “repo1,” you get the following result:
8313d6a (master) Change in subtree_test
* 50d8d46 (HEAD -> test_branch) Third change
* 4499ee8 Second change
* 99564a0 First change
* 33c5799 Initial commit
The change made in “subtree_test” was isolated and then transferred to “repo1.”
Conclusion
Git Subtree is a powerful alternative to Git submodules that allows developers to integrate external repositories directly into the main repository and thus seamlessly exchange changes between both repositories. Compared to Git submodules, Git Subtree offers several advantages such as easier handling and no additional metadata. However, there are also disadvantages, such as the need to learn a new merge strategy and the responsibility not to mix the code of main and subprojects in commits. The use cases presented in this article serve as practical guides for working with Git Subtree and show how to use the concept efficiently in real development projects.