Remove runtime dependency on `git` package about nix HOT 11 OPEN

I've used a very teeny slice of libgit2 indirectly through python and rust bindings and my experience was that it's a mix when it comes to performance.

In particular, I found diffing was quite a bit slower (especially on big repos) via libgit2 than by shelling out to git. (Some other stuff was quite a bit faster, though.)

I think it just merits benchmarking each implementation against a good mix of repo sizes. Maybe an easy-to-run ~benchmarking suite pays off over time by warding off problems and encouraging people to experiment with optimizations.

Edit: there's also the rust gitoxide project. It's an alternative to the rust binding, but not a drop-in replacement. It also has a nascent gix cli, but IIRC it is likewise not a drop-in replacement. If there are any obvious performance hotspots, it might be worth exploring whether an equivalent implementation can use gitoxide or gix to see how they perform. The maintainer's pretty responsive with respect to tracking/prioritizing missing features to help onboard projects.

from nix.

Here's an example of the pain it's caused with cargo: rust-lang/cargo#11567 (comment)

It's plausible that nix's use case of pulling objects out of the store for a vfs would be faster on libgit2 and yet also that clone would be unacceptably slow without instead shelling out to git. I can believe that various use cases end up with very different perf results.

from nix.

Regarding libgit2 performance, romkatv has made optimizations that may be interesting to upstream.
libgit2/libgit2#5044 attempted to upstream a number of them. Perhaps we could cherry-pick ones that are relevant to us and turn them into smaller PRs that are easier for upstream to merge?
Their motivating use case is a highly optimized git status: https://github.com/romkatv/gitstatus/blob/master/README.md#why-fast

from nix.

You should be aware that libgit2 has severe performance problems, which would be rather unfortunate given that git is already not the fastest thing in the world. It seems much more sensible to deliberately put a fallback git into the closure of Nix, so it will consistently be available, and be of the better implementation.

from nix.

You should be aware that libgit2 has severe performance problems

We've seen some performance regressions that we expect to be offset by the ability to operate on subtrees of repositories, and not adding everything to the store. I haven't seen anything that I'd have to call severe.
@lf- Could you elaborate on that? Do you happen to know why this might be?

Note also that use libgit2 to sidestep not entirely reproducible behaviors in the git checkout and archive logic, such as smudge filters and export-ignore. This is solved simply and reliably by accessing the tree objects directly.

deliberately put a fallback git [command] into the closure

There's no doubt that we keep it as long as we need it. Performance can be a valid reason to keep it.

from nix.

Their motivating use case is a highly optimized git status

Interesting. Hadn't spotted that before. Amusingly, my motivating case is more or less the same (I implemented lilgit as a more minimal alternative to gitstatus to see if i could claw back more time) :)

from nix.

Related to:

• #3533

from nix.

I filed #10567 which goes in the opposite direction of this issue. Great input by @fricklerhandwerk makes me think that removing git as a runtime dependency is the way to go. However, on an IMO more basic level, there should be an interface for plugging fetchers. See also my #10567 (comment). Commenting this here for reach regarding "pluggable fetchers".

from nix.

When libgit2 supports remote helpers, #10567 and removing the git command dependency won't be in opposition.

I've added "Do not regress in functionality" to the issue description, because I'm certainly not advocating for breaking people's stuff for some minor technical reason.

from nix.

When libgit2 supports remote helpers, #10567 and removing the git command dependency won't be in opposition.

Right. But do you have any indication that libgit2 would consider adding this, or even better someone working on it? I searched their repo but couldn't find anything. Don't get me wrong, I think it'd be very cool if it would support remote helpers, I am just genuinely asking.

from nix.

I haven't talked to them about these features; I've only opened this issue to track what we'd need, because it came up in review and discussions. I don't think this issue is strategically important for Nix, but any efforts will be appreciated.

from nix.