Comments (21)
After I running
git filter-branch -f --prune-empty -- --all
on my repogit sizer
was happy.
@larsxschneider: presumably your repository had replace references or grafts, and the reason that git filter-branch
fixed the problem is that it "baked them into" the actual repository structure. I doubt that the --prune-empty
option was actually necessary.
from git-sizer.
Does GIT_GRAFT_FILE=NUL also work?
Yes, it does work, too!
from git-sizer.
Are you using the newest version (v1.1) of git-sizer
?
Is the repository publicly available so that I can try to reproduce the problem?
from git-sizer.
I've experienced the same with v1.1, this happens on rails/rails for me.
from git-sizer.
Yes, I was using v1.1, sorry but repo I tried is not publicly available.
from git-sizer.
Hmm, I can't reproduce this bug. I tried cloning rails/rails from github then ran git-sizer -v
against it (version 1.1), but no error. I tried using Git 2.3.16, 2.14.2, and master. @jeremywrowe, which Git version do you have in your PATH?
from git-sizer.
I experienced the same error too with git-sizer v1.1 and git 2.16.2 on Linux.
from git-sizer.
git-sizer v1.1 and git 2.16.2 on os x
from git-sizer.
I think empty commits might be problematic in certain cases. After I running git filter-branch -f --prune-empty -- --all
on my repo git sizer
was happy.
from git-sizer.
I tried in the simplest possible way to create an empty commit git commit --allow-empty [...]
then run git-sizer
on the repo. It didn't fail. Also, it seems that an empty commit would be an unlikely trigger for a git-sizer
failure.
My guess is that git filter-branch -f --prune-empty -- --all
rearranged a lot of things (e.g., including the commit topology) in such a way that the bug wasn't triggered.
If I could get my hands on a repo that triggers this bug reliably, it would be a big help. Until then, I'd just be guessing.
from git-sizer.
I forgot to ask an important question: does git fsck --no-dangling
run cleanly on the affected repositories?
Meanwhile, it occurred to me that I have about 80 Million Git repositories that I can run git-sizer
against (output anonymized, of course) to look for problems. Maybe I can find one there that triggers the bug.
from git-sizer.
git fsck --no-dangling
did not fix the problem π’
I can't share the repo but I am happy to run a git-sizer version with relevant debug output for you.
from git-sizer.
git fsck --no-dangling
did not fix the problem
@larsxschneider: Just to be sure we understand each other...
git fsck
doesn't actually fix anything in a Git repository. If git fsck --no-dangling
reports errors in a repository, then the repository is corrupt in one way or another and it would not be surprising for git-sizer
to fail on that repo. If git fsck --no-dangling
doesn't report any problems, then the repository is presumably not corrupt, so git-sizer
should hopefully succeed.
I assume that you meant that "git fsck --no-dangling
doesn't report any problems, so repository corruption doesn't seem to be the explanation for why git-sizer
fails." But if you meant something else, please clarify.
I've run git-sizer
on tens of thousands of repositories and haven't yet found an error that wasn't explained by some kind of mild repository corruption. But I'll keep trying.
If I come up with any theories for why the failure occurs, then I might take you up on your offer to run a debugging version of git-sizer
against your repository.
from git-sizer.
Oops. I am sorry, of course git fsck
doesn't fix anything.
It doesn't report any errors on my repo either:
$ git fsck --no-dangling
Checking object directories: 100% (256/256), done.
Checking objects: 100% (2265686/2265686), done.
Checking connectivity: 1932408, done.
$ ~/bin/git-sizer
Processing blobs: 700857
Processing trees: 998950
panic: commit is not available
goroutine 1 [running]:
github.com/github/git-sizer/sizes.(*Graph).GetCommitSize(0xc4200c6000, 0x765aca5c538afed6, 0xf79c1b412029dceb, 0x5f4071f7, 0xc30000000f)
/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/sizes/graph.go:630 +0xc6
github.com/github/git-sizer/sizes.(*Graph).RegisterCommit(0xc4200c6000, 0x1efdd41421eb6764, 0x9c655f3f1847d6b, 0xc4cef55c9a, 0xc43b2ec880)
/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/sizes/graph.go:656 +0x1e1
github.com/github/git-sizer/sizes.ScanRepositoryUsingGraph(0xc4200c2000, 0x5687e8, 0x2, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/sizes/graph.go:281 +0x11e8
main.mainImplementation(0x0, 0x0)
/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/git-sizer.go:150 +0x8ef
main.main()
/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/git-sizer.go:46 +0x26
from git-sizer.
I'm hit by that bug/message to. In my case .git/info/grafts
is the culprit. When I remove that file it works, when I move it back, it breaks.
See git h repository-layout
for a short description of that file.
from git-sizer.
I'm hit by that bug/message to. In my case
.git/info/grafts
is the culprit.
Oh, that makes perfect sense. info/grafts
or replace references could break invariants that git-sizer
depends on. @larsxschneider and others, do your repos all have grafts and/or replace references?
I think there are env variables that can be used to disable both of those. I'll check the details when I'm at a computer again.
BTW, I ran git-sizer
against over 100k repositories and none failed except for reasons of minor corruption. But none of those repos had grafts, and they all had replace refs turned off.
from git-sizer.
@mhagger Well, neither grafts nor replace references are cloned/pushed/fetched by default (grafts cannot, replacements can but not by default). So both things actually live only in local repositories where real users work.
from git-sizer.
I looked into how to disable these things.
- Replace references can be disabled by running Git with
git --no-replace-objects [...]
or by setting the (undocumented) environment variableGIT_NO_REPLACE_OBJECTS=1
. - Grafts can be disabled by setting the (undocumented) environment variable
GIT_GRAFT_FILE=<filename>
, where<filename>
is an empty or non-existent file. - Shallow clones (detectable from the existence of
$GIT_DIR/shallow
, I believe) would also be a problem. (The location of theshallow
file can be influenced by the environment variableGIT_SHALLOW_FILE
, but I don't think that's useful.) We probably need to disallow runninggit-sizer
in a shallow repository at all.
We should build these things into git-sizer
. Meanwhile, those of you who experienced this bug should be able to get around it by running
GIT_GRAFT_FILE=/dev/null git --no-replace-objects sizer [...]
Please confirm whether that fixes your problems.
from git-sizer.
@mhagger : π―yes. That is exactly what happened. @asheiduk thanks for bringing up the grafts. I totally forgot about them in my repo.
from git-sizer.
GIT_GRAFT_FILE=/dev/null git --no-replace-objects sizer
@mhagger FWIW this works on Windows (I tested specifically the /dev/null
part). It also works in CMD (of course you have to set the environment variable separately, then).
from git-sizer.
@dscho: Thanks for checking. Does GIT_GRAFT_FILE=NUL
also work? That's what #31 currently implements, and the code is IMO more straightforward that way (just rely on os.DevNull
rather than having to worry that some other OS uses something different here).
from git-sizer.
Related Issues (20)
- Question: How is logical file size calculated (Total size of files) HOT 1
- Obtain a list of biggest files in the repository. HOT 1
- Question: is there anyway I can get the count of files in particular language for example cuda statistics HOT 1
- Can it work on a bare repo? HOT 1
- MASTER Test HOT 1
- Get a top 10 of biggest checkouts in repository
- Approximating size shown at https://github.com/settings/repositories HOT 4
- Please update to Go 1.16 and also release for macOS ARM64 HOT 4
- Add suggestion on how to resolve spotted issues HOT 1
- "git-sizer --version" should show a version number. HOT 5
- `git-sizer --help` should exit with 0 HOT 1
- Add new convenience options like `--no-branches`, `--stash`, etc. HOT 1
- Release 1.4.0 Windows builds flagged as malicious by multiple vendors HOT 3
- error: couldn't open Git repository: git rev-parse failed: Unknown option: -C HOT 4
- Detect & report tmp_pack files HOT 1
- Idea: have `git-sizer` read some defaults from the gitconfig
- Impove NewObjectIter method HOT 2
- [Doc] Record Golang dependency in readme.
- git-sizer error: "running 'git config': exit status 129" HOT 7
- go get: installing executables with 'go get' in module mode is deprecated HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from git-sizer.