Getting a crash when running about bfg-repo-cleaner HOT 28 CLOSED

Hi @akac - I'm assuming you had to delete some lines from that stacktrace, as a stackoverflow would usually involve more repeated lines, many more times?

Is this repo public - can I see a copy of it?

from bfg-repo-cleaner.

Its a private repo. And yes, it repeats that a few thousand times :)

So I tried it like this:

java -jar ~/Downloads/bfg-1.11.0.jar --delete-folders .git .

and now its just stuck.

Using repo : /Projects/GitHubTemp/iOS-Development.git

Found 2670 objects to protect
Found 3 tag-pointing refs : refs/tags/3.30.17, refs/tags/3.30.18, refs/tags/3.30.35
Found 299 commit-pointing refs : HEAD, refs/heads/1.65, refs/heads/2.01, ...

Protected commits

These are your protected commits, and so their contents will NOT be altered:

• commit 7cdc6873 (protected by 'HEAD') - contains 21 dirty files :
  • three20/.git/HEAD (23 B)
  • three20/.git/config (281 B)
  • ...

WARNING: The dirty content above may be removed from other commits, but as
the protected commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

/Projects/GitHubTemp/iOS-Development.git.bfg-report/2013-11-29T11-17/protected-dirt/

If you really want this content gone, make a manual commit that removes it,
and then run the BFG on a fresh copy of your repo.

Cleaning

Found 9099 commits
Cleaning commits: 94% (8554/9099)

from bfg-repo-cleaner.

I realize a public repo would be more helpful, but not sure how to help otherwise.

The issue is that we're converting from a Kiln GIT repo that was at one point Mercurial to a github repo and of course we've got some history in the past with .git folders in there.

So we are trying to clean that up. Running on OS X Mavericks.

from bfg-repo-cleaner.

While the BFG is in the stuck state, could you run jstack and attach the full output returned?

It's going to be very hard to diagnose this problem without getting my hands on a copy of the repo. A possible cause of the problem is commit message updating. If you're not able to send me the full repo, would you be able to email me the full output of git log?

from bfg-repo-cleaner.

I can email you the git log and jstack - as github won't let me paste it here. What email address can I do that with?

from bfg-repo-cleaner.

You can email [email protected] - please send the git log in these two formats:

git log --oneline --abbrev-commit --all --graph

git log --all

from bfg-repo-cleaner.

Thanks for providing those diagnostics @akac - the jstack is particularily puzzling as the main thread trace appears to have lost the initial frames (should include eg com.madgag.git.bfg.cleaner.RepoRewriter). Could you send a full copy of the stacktrace thrown on the console with your initial command?

I'll think I'll be able to send you a custom diagnostic build at some point this weekend.

from bfg-repo-cleaner.

HI @akac - looks like the problem is probably with the id-substitution that occurs against commit messages (changing the old commit-ids into new). The commit messages in your repo have a slightly higher than usual incidence of commit ids (because some merges are specified by commit-id rather than branch) - ie:

Merge commit '9e5a840a88b0aa54f88a1133160fb394e25c6428' into After-321

I've shared a custom BFG build with you on Google Drive, it disables object-id-substitution and will probably allow the BFG to run on your repo. It'll also produce an original-commit-ids.txt file under the .bfg-report directory which will give me a bit more information about the order the BFG approached the id-substitution if you send it to me.

from bfg-repo-cleaner.

Got it. Trying it now.

from bfg-repo-cleaner.

Here is the problem I'm trying to solve:
Counting objects: 99050, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (24114/24114), done.
Writing objects: 100% (99050/99050), 269.50 MiB | 2.72 MiB/s, done.
Total 99050 (delta 75227), reused 88584 (delta 73859)
remote: error: object f2371b54e17a355e23e06649d17b19e7e96130ab:contains '.git'
remote: fatal: Error in object
error: unpack failed: index-pack abnormal exit

So I've got a .git folder in my source. So I'm running:

java -jar ~/Downloads/bfg-assembly-1.11.1-SNAPSHOT.jar --delete-folders .git iOS-Development.git/

But after doing that I'm still getting the same error in pushing to git. Did the logs help? Am I not calling the command line properly?

from bfg-repo-cleaner.

Ah, I see - its protecting the content:
hese are your protected commits, and so their contents will NOT be altered:

• commit 56445f8b (protected by 'HEAD') - contains 21 dirty files :
  • three20/.git/HEAD (23 B)
  • three20/.git/config (281 B)
  • ...

WARNING: The dirty content above may be removed from other commits, but as
the protected commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

/Projects/GitHubTemp/iOS-Development.git/..bfg-report/2013-12-01T15-09/protected-dirt/

If you really want this content gone, make a manual commit that removes it,
and then run the BFG on a fresh copy of your repo.

But I don't want anything protected. The problem is that I think that HEAD actually comes from when we converted from SVN, and I've never been able to get rid of it. Its been the bane of my source control existence for years - which is why I wanted it gone.

from bfg-repo-cleaner.

OK I used --no-blob-protection - that worked. Now lets try the push - arg. Still the same:
remote: error: object f2371b54e17a355e23e06649d17b19e7e96130ab:contains '.git'

I wonder if there is a way to simply have git drop all commits before a certain point…

BTW - I'm working on a bare mirror, so its not a working copy. Just the raw git repo.

from bfg-repo-cleaner.

If I do

git show f2371b54e17a355e23e06649d17b19e7e96130ab
tree f2371b54e17a355e23e06649d17b19e7e96130ab

.git
.gitignore
AppledocSettings.plist
Core/
DTFoundation.podspec
DTFoundation.xcodeproj/
Demo/
Documentation/
LICENSE
Test/
readme.markdown

I see that its still there. Am I using the command correctly?

from bfg-repo-cleaner.

OK I used --no-blob-protection - that worked. Now lets try the push - arg. Still the same:
remote: error: object f2371b5:contains '.git'

Just checking, did you run these commands after The BFG ran? :

$ git reflog expire --expire=now --all
$ git gc --prune=now --aggressive

(these commands are specified in http://rtyley.github.io/bfg-repo-cleaner/#usage , but you might have missed them)

from bfg-repo-cleaner.

Sigh… :) Trying that now. I must have missed that when I went straight to the examples.

from bfg-repo-cleaner.

OK - so yeah I obviously missed that but now I've got a different object says its now failing on:

remote: error: object d1ca5b477096c281877fcd531610e471780063a5

though I suppose that's just different objects being rewritten.

git show d1ca5b477096c281877fcd531610e471780063a5
tree d1ca5b477096c281877fcd531610e471780063a5

.git
.gitignore
AppledocSettings.plist
Core/
DTFoundation.podspec
DTFoundation.xcodeproj/
Demo/
Documentation/
LICENSE
Test/
readme.markdown

So running bfg on it again:

java -jar ~/Downloads/bfg-assembly-1.11.1-SNAPSHOT.jar --no-blob-protection --delete-folders .git .

Using repo : /Projects/GitHubTemp/iOS-Development.git/.

Found 0 objects to protect
Found 3 tag-pointing refs : refs/tags/3.30.17, refs/tags/3.30.18, refs/tags/3.30.35
Found 299 commit-pointing refs : HEAD, refs/heads/1.65, refs/heads/2.01, ...

Protected commits
-----------------

You're not protecting any commits, which means the BFG will modify the contents of even *current* commits.

This isn't recommended - ideally, if your current commits are dirty, you should fix up your working copy and commit that, check that your build still works, and only then run the BFG to clean up your history.

Cleaning
--------

Found 9099 commits
Cleaning commits:       100% (9099/9099)
Cleaning commits completed in 2,723 ms.

BFG aborting: No refs to update - no dirty commits found??

But obviously there is a .git folder.

from bfg-repo-cleaner.

Could you try enclosing the .git in quotes, to see if that makes a difference? ie --delete-folders ".git"

Also, on your updated repo, could you run this log command, and send me the output:

git log --all  --format=format:"%H %T %d : %s"

It'll give me an idea of which commit this corrupted tree is sitting in.

from bfg-repo-cleaner.

So, looking at the output of git log --all --format=format:"%H %T %d : %s" that you sent me, the tree hash d1ca5b477096c281877fcd531610e471780063a5 doesn't occur anywhere, though I would expect it to if that's a top-level folder in any commit.

Does the directory listing from git show d1ca5b47 shows a top level folder within your project or actually a sub-folder?

from bfg-repo-cleaner.

its a subfolder. Essentially its a project that we'd pull down via git to that spot (because we were using SVN we couldn't use submodules).

from bfg-repo-cleaner.

Ok - could you send me the (rather larger) output of this command so I can see what commit it's in?

git log --all --format=format:"%H" | xargs -I cid sh -c 'echo commit cid; git ls-tree -r -d cid'

from bfg-repo-cleaner.

OK so instead of sending that to you - I checked the 300MB file myself. That text file does not show .git in there. I even looked for commit d1ca5b477096c281877fcd531610e471780063a5 and could not find anything with git

But yet if I type git show I get:

git show d1ca5b477096c281877fcd531610e471780063a5
tree d1ca5b477096c281877fcd531610e471780063a5

.git
.gitignore
AppledocSettings.plist
Core/
DTFoundation.podspec
DTFoundation.xcodeproj/
Demo/
Documentation/
LICENSE
Test/
readme.markdown

So I'm not sure what that means. If you still want the logs - I can dropbox that to you, just let me know.

from bfg-repo-cleaner.

I might try from scratch getting the clean rep, running BFG on it, and then expire/prune and see if that works.

from bfg-repo-cleaner.

Yep- you might want to try doing:

git push --all - pushes just branches...

git push --tags

...separately, to see which fails. It might also be worth deleting the GitHub project and recreating it, just in case there's some weird state hanging around in it from earlier.

And definitely don't forget the:

$ git reflog expire --expire=now --all
$ git gc --prune=now --aggressive

...before you push.

from bfg-repo-cleaner.

1. Pulled down the latest repo from Kiln using git clone --mirror
2. Copied it for a backup
3. java -jar ~/Downloads/bfg-assembly-1.11.1-SNAPSHOT.jar --no-blob-protection --delete-folders ".git"
4. git reflog expire --expire=now --all
5. git gc --prune=now --aggressive
6. git push --all

Sigh:

Writing objects: 100% (99073/99073), 251.40 MiB | 2.51 MiB/s, done.
Total 99073 (delta 78084), reused 99068 (delta 78082)
remote: error: object c074e07cf851ecfffa0fafec3d72dfaf37c55649:contains '.git'
remote: fatal: Error in object
error: unpack failed: index-pack abnormal exit

I wish git wasn't so stubborn. I'm really kind of at a loss what to do next.

from bfg-repo-cleaner.

OK so this time I tried something different. I told it to also delete FILES, not just dirs. Now it worked.

from bfg-repo-cleaner.

Oh, so there was a file called .git too... well done on working that out...!

from bfg-repo-cleaner.

@rtyley I'm having this symptom (you called the issue: "id-substitution that occurs against commit messages") too. Reading this issue, it looks like there was a private build that worked around the problem...is there a way I could get a look at that version if you still have it around, please?

from bfg-repo-cleaner.

@rtyley It seems that I'm having the same problem as @mdengler. Is there meanwhile a solution?

from bfg-repo-cleaner.