Handy Scripts For Doing Things
License: MIT License
For easy installation, use rcm, which you can install with the following command:
brew tap thoughtbot/formulae
brew install rcm
You can then symlink the utilities in this repo to your ~/bin directory by executing the following command:
$ RCRC="$(pwd)/.rcrc" rcup -v
Remove the symlinks using:
$ RCRC="$(pwd)/.rcrc" rcdn -v
Happy CLI'ing.
Case: The file history of a submodule in Gitlab is no longer the history of the current file, but all the commits of the submodule.
I did a little research on this situation and found that it was caused by the git log --follow file1 used in GitLab's file history, with the --follow option listing the files that were moved/rename. If just git log file, then everything is fine.（https://gitlab.com/gitlab-org/gitlab/-/issues/215094）
Any ideas, guys?
Thanks！
This script is great, but I'm having problems dealing with my multi-branch repo.
Let's say I have master and dev, each of which has different commits, so I need to run the script for each of them. But this causes them both to have a huge commit: "Merge submodule contents for xxx"", which results in a very large .git file after the merge request, is there a way to slove this?
Thanks!
The documentation for the .gitmodules file states that a url property may be a path relative to the superproject's URI. E.g. if foo has a submodule bar, the url for bar maybe ../bar if bar's origin is relative to foo's. The git-submodule-rewrite script doesn't handle this case. Specifically the url extracted on line 187 will be relative, and the clone on line 84 will fail.
I had a Git tag in my submodule which is not present anymore after merging. This is no problem for me personally because it was just one tag I have now manually readded. But in general, perhaps the tool should warn users about this?
If you run git-submodule-rewrite on a submodule that has its own submodules they will be lost after the consolidation, rather than being merged into the parent repository's .gitmodules.
(Yes, we did have far too many submodules!)
After integrating a submodule into the main repo, we noticed that there were some key pieces of our submodule that were missing and eventually determined that it was because they were now subject to the top level .gitignore rules instead of their own.
This should be fairly simple to reproduce, have a parent repository that has build/ in its .gitignore and then have the submodule have a wanted directory with build/ in the filepath and then run the script. All the files in the submodule will be on your local, but anything under the build/ directory won't be pushed.
The script should have a section that combines the .gitignore files so that the former submodule won't now be subject to the old parent's rules.
I am studying using this script, but it just took two hours to process one branch, and it's configured to operate on a temporary path that it deletes afterwards, so as soon as I'll want to merge another branch, I think I'm in for another two hour processing per branch
While googling around, I found something quite promising under the name of git filter-repo (https://github.com/newren/git-filter-repo/) whose use is even advised in the documentation of git filter-branch itself
So the question is, how hard would it be to change the script to use filter-repo instead of filter-branch and would there be any reason not to ? While glancing at it, it seems like it could solve a few problems at once: first, I would not have to reprocess the commits for each branch (granted I hook a non-temporary working path in the arguments too instead of the temp dir), and could process the whole repository at once, and second, the perf is promised to be much, much better
And then the corollary question would be, if I change the script to use filter-repo, would you be interested in a PR ?
As the submodule commits are rewritten for the prefix path (causing the hash to change), it would be great to be able to rebase them instead of merge, so the commits are linear/sequential and not strewn throughout the git history (which isn't helpful anyway, as the old submodule pointer might not point to it right after the commit is made)
➤ git-submodule-rewrite --verbose dev-scripts myco && echo "finished" git:dev
This script will convert your "dev-scripts" git submodule into
a simple subdirectory in the parent repository while retaining all
contents, file history and its own submodules.
The script will:
* delete the dev-scripts submodule configuration from .gitmodules and
.git/config and commit it.
* rewrite the entire history of the dev-scripts submodule so that all
paths are prefixed by dev-scripts.
This ensures that git log will correctly follow the original file
history.
* merge the submodule's tags into its parent repository and commit
each tag merge individually.
(only those tags are considered which are reachable from
the tip of dev-scripts/myco)
* merge the submodule into its parent repository and commit it.
* reinstate any of the submodule's own submodules as part of the parent
repository
NOTE: This script might completely garble your repository, so PLEASE apply
this only to a fresh clone of the repository where it does not matter if
the repo is destroyed. It would be wise to keep a backup clone of your
repository, so that you can reconstitute it if need be. You have been
warned. Use at your own risk.
Do you want to proceed? (y/n) y
+ git config -f .gitmodules --remove-section submodule.dev-scripts
+ git config -f .git/config --get submodule.dev-scripts.url
https://gitlab.biw-services.com/east-php/q360-fed/dev-scripts.git
+ git config -f .git/config --remove-section submodule.dev-scripts
+ rm -rf dev-scripts
+ git add -A .
+ git commit -m 'Remove submodule dev-scripts'
[dev 6746478b4] Remove submodule dev-scripts
2 files changed, 12 deletions(-)
delete mode 160000 dev-scripts
+ rm -rf .git/modules/dev-scripts
++ mktemp -d -t submodule-rewrite-XXXXXX
+ local tmpdir=/var/folders/dt/215qvyd54xb2fchcpqj8x4fc0000gn/T/submodule-rewrite-XXXXXX.0uXEZxvu
+ git clone -b myco https://gitlab.biw-services.com/east-php/q360-fed/dev-scripts.git /var/folders/dt/215qvyd54xb2fchcpqj8x4fc0000gn/T/submodule-rewrite-XXXXXX.0uXEZxvu
Cloning into '/var/folders/dt/215qvyd54xb2fchcpqj8x4fc0000gn/T/submodule-rewrite-XXXXXX.0uXEZxvu'...
remote: Enumerating objects: 348, done.
remote: Counting objects: 100% (348/348), done.
remote: Compressing objects: 100% (154/154), done.
remote: Total 348 (delta 215), reused 321 (delta 194)
Receiving objects: 100% (348/348), 111.64 KiB | 367.00 KiB/s, done.
Resolving deltas: 100% (215/215), done.
+ git fetch --tags
+ pushd /var/folders/dt/215qvyd54xb2fchcpqj8x4fc0000gn/T/submodule-rewrite-XXXXXX.0uXEZxvu
/var/folders/dt/215qvyd54xb2fchcpqj8x4fc0000gn/T/submodule-rewrite-XXXXXX.0uXEZxvu ~/repos/mhp
++ printf '\t'
+ local 'tab= '
+ local 'filter=git ls-files -s | sed "s: : dev-scripts/:" | GIT_INDEX_FILE=${GIT_INDEX_FILE}.new git update-index --index-info && mv ${GIT_INDEX_FILE}.new ${GIT_INDEX_FILE} || true'
+ git filter-branch --index-filter 'git ls-files -s | sed "s: : dev-scripts/:" | GIT_INDEX_FILE=${GIT_INDEX_FILE}.new git update-index --index-info && mv ${GIT_INDEX_FILE}.new ${GIT_INDEX_FILE} || true' HEAD
WARNING: git-filter-branch has a glut of gotchas generating mangled history
rewrites. Hit Ctrl-C before proceeding to abort, then use an
alternative filtering tool such as 'git filter-repo'
(https://github.com/newren/git-filter-repo/) instead. See the
filter-branch manual page for more details; to squelch this warning,
set FILTER_BRANCH_SQUELCH_WARNING=1.
Proceeding with filter-branch...
Rewrite e3e9213b7144b3f6ecbe18e0bfe69feb81a4b733 (95/105) (5 seconds passed, remaining 0 predicted)
Ref 'refs/heads/myco' was rewritten
+ popd
~/repos/mhp
+ git remote add dev-scripts /var/folders/dt/215qvyd54xb2fchcpqj8x4fc0000gn/T/submodule-rewrite-XXXXXX.0uXEZxvu
+ git_version_lte 2.8.4
+++ echo 2.8.4
+++ tr . '\n'
+++ head -n 4
++ printf %03d%03d%03d%03d 2 8 4
+ OP_VERSION=002008004000
++ git version
+ GIT_VERSION='git version 2.24.1'
+++ echo 2.24.1
+++ sed -E 's/([0-9.]*).*/\1/'
+++ tr . '\n'
+++ head -n 4
++ printf %03d%03d%03d%03d 2 24 1
+ GIT_VERSION=002024001000
+ echo -e '002024001000\n002008004000'
+ sort
+ head -n1
002008004000
+ '[' 002024001000 -le 002008004000 ']'
+ ALLOW_UNRELATED_HISTORIES=--allow-unrelated-histories
++ git tag --list --merged dev-scripts/myco
fatal: malformed object name dev-scripts/myco
+ git merge -s ours --no-commit --allow-unrelated-histories dev-scripts/myco
merge: dev-scripts/myco - not something we can merge
Also had this happen with a similar submodule that was in the root. Before I ran this I commented out this git fetch --tags "${sub} line as it caused issues during previous attempts
You have no idea how much your script about un-submodule has been a blessing for me.
I just landed on a project that have been struggling with a lot of problems. Having too many repository (90+) is one of them, and thanks to your script, I managed to easily do something that would have been a struggle for me.
This is obviously not an issue, but it is important to thank the community for its helpful contribution.
Thank you Jeremy !
If I just git-clone + git-submodule-rewrite without git submodule update --init, then the script fails with "fatal: No such section!".
on L134, the retrieved tags can be out of order depending on the versioning scheme. I'd recommend adding for submodule_tag in $(git tag --list --merged "${sub}/${branch}" | sort --version-sort --field-separator=.) to assure the tags are applied in the correct order.
I have a repo, A, containing a submodule B.
Repo B is on branch B-branch and has a submodule C.
Submodule C in B is at a specific commit, xxxxxxxxxxxxxxx, that happens not to be the head of the master branch for C.
All the repos (A, submodule B and sub-submodule C) are clean, pushed, etc.
I go into submodule B within A and do
$ git submodule status
xxxxxxxxxxxxxxx C
I then go into the root of repo A and do git-submodule-rewrite B B-branch. It is apparently successful, and I get
Submodule merge complete. Push changes after review.
However, when I do git submodule status, I get
yyyyyyyyyyyyyy B/C
and not
xxxxxxxxxxxxxxx B/C
The merge has resulted in the C sub-sub-module moving to the tip of its master branch (which happens to be commit yyyyyyyyyyyyyy), rather than staying where it should be at commit xxxxxxxxxxxxxxx.
Hello Jeremy,
This script looks really promising but is there any guidelines for using it?
I am trying to use this script for merging our submodules into the main repo. it is taking 40 mins to to rewrite the history but when I am trying to push the changes it has only 2 changes in it.
I have cloned the main repo and executed the script in following manner
git-submodule-rewrite libraries/app develop
name of submodule is libraries/app. can you please point me out to the guidelines or what is wrong I am doing?
Hi Jeremy,
I am very grateful to you for having git-submodule-rewrite script. It saved a lot of my time when getting rid of submodules in my project.
I'm just curious, how can I integrate a submodule into the parent monorepository with all branches it has?
Thank you in advance!
If the remote of the main repo is not called "origin", then the script fails with
dirname: missing operand
Try 'dirname --help' for more information.
/path/to/git-submodule-rewrite: line 78: $2: unbound variable
Error: Could not determine the remote origin for this superproject:
Script git-submodule-rewrite always failed for me.
It prints can not merge unrelated histories,
I have to add --allow-unrelated-histories to git merge --allow-unrelated-histories -s ours --no-commit ${ALLOW_UNRELATED_HISTORIES} "${sub}/master" to fix this issue.
Hope it would be fixed in main repo also.
git-submodule-rewrite: line 214: [: missing `]'
Fix: s/"]/" ]/
Hi - I've been experimenting with your git-submodule-rewrite script.
I had the assumption that, since submodules are pinned at a specific revision, when it merged the submodule into the main repository it would be done at the specific revision and not the master branch (by default).
Attempting to specify a revision in place of the branch parameter doesn't appear to work.
I'm happy to dig deeper into the script and see if I can make it work, but your choice of master as default instead of the pinned revision makes me suspect you encountered some problem with this approach beforehand that makes it difficult.
git-submodule-rewrite failed to merge a simple submodule:
../jeremysears/scripts/bin/git-submodule-rewrite -v hw-verify
This script will convert your "hw-verify" git submodule into
a simple subdirectory in the parent repository while retaining all
contents, file history and its own submodules.
The script will:
* delete the hw-verify submodule configuration from .gitmodules and
.git/config and commit it.
* rewrite the entire history of the hw-verify submodule so that all
paths are prefixed by hw-verify.
This ensures that git log will correctly follow the original file
history.
* merge the submodule into its parent repository and commit it.
* reinstate any of the submodule's own submodules as part of the parent
repository
NOTE: This script might completely garble your repository, so PLEASE apply
this only to a fresh clone of the repository where it does not matter if
the repo is destroyed. It would be wise to keep a backup clone of your
repository, so that you can reconstitute it if need be. You have been
warned. Use at your own risk.
Do you want to proceed? (y/n) y
+ git config -f .gitmodules --remove-section submodule.hw-verify
+ git config -f .git/config --get submodule.hw-verify.url
https://github.com/somecompany/hw-verify.git
+ git config -f .git/config --remove-section submodule.hw-verify
+ rm -rf hw-verify
+ git add -A .
+ git commit -m 'Remove submodule hw-verify'
[master 307e0e9] Remove submodule hw-verify
2 files changed, 4 deletions(-)
delete mode 160000 hw-verify
+ rm -rf .git/modules/hw-verify
++ mktemp -d -t submodule-rewrite-XXXXXX
+ local tmpdir=/tmp/submodule-rewrite-vD5x6o
+ git clone -b master https://github.com/somecompany/hw-verify.git /tmp/submodule-rewrite-vD5x6o
Cloning into '/tmp/submodule-rewrite-vD5x6o'...
remote: Counting objects: 123, done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 123 (delta 69), reused 123 (delta 69), pack-reused 0
Receiving objects: 100% (123/123), 38.62 KiB | 0 bytes/s, done.
Resolving deltas: 100% (69/69), done.
Checking connectivity... done.
+ pushd /tmp/submodule-rewrite-vD5x6o
/tmp/submodule-rewrite-vD5x6o ~/projects/m5
++ printf '\t'
+ local 'tab= '
+ local 'filter=git ls-files -s | sed "s: : hw-verify/:" | GIT_INDEX_FILE=${GIT_INDEX_FILE}.new git update-index --index-info && mv ${GIT_INDEX_FILE}.new ${GIT_INDEX_FILE} || true'
+ git filter-branch --index-filter 'git ls-files -s | sed "s: : hw-verify/:" | GIT_INDEX_FILE=${GIT_INDEX_FILE}.new git update-index --index-info && mv ${GIT_INDEX_FILE}.new ${GIT_INDEX_FILE} || true' HEAD
Rewrite 902d922429ca56430397428f4d539f27499f363f (16/16)
Ref 'refs/heads/master' was rewritten
+ popd
~/projects/m5
+ git remote add hw-verify /tmp/submodule-rewrite-vD5x6o
+ git fetch hw-verify
warning: no common commits
remote: Counting objects: 139, done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 139 (delta 69), reused 107 (delta 69)
Receiving objects: 100% (139/139), 39.32 KiB | 0 bytes/s, done.
Resolving deltas: 100% (69/69), done.
From /tmp/submodule-rewrite-vD5x6o
* [new branch] master -> hw-verify/master
+ git_version_lte 2.8.4
+++ echo 2.8.4
+++ tr . '\n'
+++ head -n 4
++ printf %03d%03d%03d%03d 2 8 4
+ OP_VERSION=002008004000
++ git version
+ GIT_VERSION='git version 2.1.4'
+++ echo ' 2.1.4'
+++ tr . '\n'
+++ head -n 4
++ printf %03d%03d%03d%03d 2 1 4
+ GIT_VERSION=002001004000
+ echo -e '002001004000\n002008004000'
+ sort
+ head -n1
002001004000
+ '[' 002001004000 -le 002008004000 ']'
+ ALLOW_UNRELATED_HISTORIES=
+ git merge -s ours --no-commit hw-verify/master
Automatic merge went well; stopped before committing as requested
+ rm -rf tmpdir
+ git clone -b master https://github.com/somecompany/hw-verify.git hw-verify
Cloning into 'hw-verify'...
remote: Counting objects: 123, done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 123 (delta 69), reused 123 (delta 69), pack-reused 0
Receiving objects: 100% (123/123), 38.62 KiB | 0 bytes/s, done.
Resolving deltas: 100% (69/69), done.
Checking connectivity... done.
+ add_submod_cmds=
+ '[' -f hw-verify/.gitmodules ']'
++ git config -f hw-verify/.gitmodules --name-only --get-regexp path
++ sed -re 's/^submodule\.(.+)\.path$/\1/g'
error: unknown option `name-only'
usage: git config [options]
Config file location
--global use global config file
--system use system config file
--local use repository config file
-f, --file <file> use given config file
--blob <blob-id> read config from given blob object
Action
--get get value: name [value-regex]
--get-all get all values: key [value-regex]
--get-regexp get values for regexp: name-regex [value-regex]
--get-urlmatch get value specific for the URL: section[.var] URL
--replace-all replace all matching variables: name value [value_regex]
--add add a new variable: name value
--unset remove a variable: name [value-regex]
--unset-all remove all matches: name [value-regex]
--rename-section rename section: old-name new-name
--remove-section remove a section: name
-l, --list list all
-e, --edit open an editor
--get-color <slot> find the color configured: [default]
--get-colorbool <slot>
find the color setting: [stdout-is-tty]
Type
--bool value is "true" or "false"
--int value is decimal number
--bool-or-int value is --bool or --int
--path value is a path (file or directory name)
Other
-z, --null terminate values with NUL byte
--includes respect include directives on lookup
+ sub_names=
about my system:
# uname -a
Linux aspire-v15 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u2 (2017-06-26) x86_64 GNU/Linux
# cat /etc/debian_version
8.9
# git --version
git version 2.1.4
# apt-get upgrade git
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... git is already the newest version.
Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
option name-only seems to appear in git 2.7.5.
The script works great, but in the output I noticed the following error:
/c/Users/path/to/git-submodule-rewrite.sh: line 59: printf: windows: invalid number
002008004000
Hello, just used your script to fix up a massive project with many nested submodules.
I ran into an interesting bug, which hopefully will be easy to replicate. If you need more information let me know.
I had a tree of several super and sub repos that I progressively rewrote upwards using the script. I noticed that after bringing in two submodules in my middle-level repo (which was disconnected from the highest-level super repo at the time of running the script), the script would fail subtly on a subsequent run and not keep the history of the middle-level repo when rewriting it into the highest-level repo. Instead there would be a clone of the "submodule"'s old repo that was untracked in the super repo.
After some debugging, I found that my middle-level repo had a .gitmodules file that was empty. Deleting this empty file would restore normal functionality to the script. It appears that the script does not delete an empty .gitmodules file after the last submodule is removed. When this empty file exists in the repo (and this repo is itself a submodule of another repo), the script does not properly apply the final step(s) that attach the newly-rewritten submodule repository to the super repository.
I hope that my description was clear. I don't have time at the moment to visually display the issue, but if you need more details I will be happy to add them. I'm currently scrambling to use the results of my efforts at my workplace :)
I think the repro steps would be as such:
Our normal git repo's url is like: https://gitlab.our-co.com/usonly-projects/client-A
in my .gitmodules file I had relative urls which are literally like: ../common-code - Somehow, these relative urls workaround an issue with authentication for cloning submodules in gitlab CI.
Anyway, I had to change these to the full urls for this script to work, which was very easy, I expect almost anyone else running into the issue will spot it and immediately know what to fix.
onward to a true monorepo. No more forking and and trying to sync submodules anymore.
I wanted to setup a simple example with sub-modules to check/demonstrate a bug, but the script doesn't appear to work with relative paths for submodules:
# Initialise two empty repositories
git init a
git init b
# Create and commit something in b
cd ../b
touch c
git add c
git commit -m "Added empty c" c
# Create b as a submodule within a and commit
cd ../a
git submodule add ../b b
git commit -m "Added b submodule" .gitmodules
# Try to rewrite b ...
git-submodule-rewrite b
Get...
dirname: missing operand
Try 'dirname --help' for more information.
../git-submodule-rewrite: line 73: $2: unbound variable
Error: Could not determine the remote origin for this superproject:
Usage: ../git-submodule-rewrite <submodule-name> [<submodule-branch>]
Merge a single branch of <submodule-name> into a repo, retaining file history.
If provided then <submodule-branch> will be merged, otherwise master.
options:
-h, --help Print this message
-v, --verbose Display verbose output
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.