Faster cloning about isomorphic-git HOT 8 OPEN

Oooooookay. Well, we've set the bar. Let's raise it. :) Time writing files: 248 minutes.

> isogit clone --url=https://gitlab.com/gitlab-org/gitlab-ce.git
git.clone({fs, dir: '.', "url":"https://gitlab.com/gitlab-org/gitlab-ce.git"})
path = remote.origin.url
value = https://gitlab.com/gitlab-org/gitlab-ce.git
Total time unpacking objects: 15735618.409464035
Total time applying deltas: 11397.829457527987
Total time reading files: 774297.640982891
Total time writing files: 14918749.676409185
✓ (4h30m) [2017-12-19 00:15:07]

On the bright side, Time Applying Deltas is only 11 seconds. Time Reading Files is 12 minutes. So the bottleneck is simply NTFS doesn't handle lots of small file writes to .git/objects/ well. Not unpacking the packfile should easily bring some big performance gains. This has all been on the desktop... the relative gain may be even bigger in the browser environment which is currently using IndexedDB for file writes.

from isomorphic-git.

Since the original acceptance criteria I wrote was "Acceptance criteria: Be able to clone https://gitlab.com/gitlab-org/gitlab-ce.git" and technically it can do that I'm going to break this into more specific tasks.

from isomorphic-git.

I got my implementation of git index-pack down from 72 minutes to 15. Cloning still takes 35 minutes but that's because my "git checkout" code is redundant. With a proper one-pass clone, I think I can realistically get it down to 18 minutes.

from isomorphic-git.

With #305 indexPack is down from 22 minutes to 10 minutes. (New hard drive, so don't compare with the old numbers.)

from isomorphic-git.

On the bright side, I can report that isomorphic-git in Node is stable. It's been unpacking "gitlab" for over two hours. Also, its memory usage is dropping at about the same rate that my disk space is shrinking. All this basically confirms I was correct when I labeled #8 a bug, not an enhancement request.

from isomorphic-git.

Awesome! A good baseline to work off of. What are the basic machine / os specs that you ran this on?

from isomorphic-git.

Updated baseline: 84 minutes (2.9x speedup)

> isogit clone --url=https://gitlab.com/gitlab-org/gitlab-ce
>> git.clone({fs, dir: '.', "url":"https://gitlab.com/gitlab-org/gitlab-ce"})
✓ (1h24m) [2017-12-24 18:54:38] Will@WILL-PC /e/git/wmhilton-contrib/gitlab-2 (master)

This is using new code that doesn't unpack the packfile, simply downloads, indexes, and checks out the files.

Git manages to run git index-pack on the gitlab-ce packfile (318 MB) in less than a minute, whereas my implementation takes 72 minutes. I'm working on optimizing that now. Git is doing it in multiple passes while I was trying to do it in one pass, and that may make a big difference.

from isomorphic-git.

Status update:

This still needs to be done. Basically the idea is add a 'checkoutRef' option to GitPackIndex.fromPack that will cause the desired files to saved the working directory during its indexing pass through the packfile.

from isomorphic-git.