Slow performance on checking out assets about uegitplugin HOT 9 CLOSED

Could you run this in a console in your repo directory:

GIT_TRACE=1 git lfs lock filename

and share the output? It'll help to see what's happening in your environment to see what's taking up the time

from uegitplugin.

Here's running the command from a Git bash. I had to redact names to anonymize the result but the output is otherwise the same:

`$ GIT_TRACE=1 git lfs lock TestAsset.uasset
15:27:00.968108 exec-cmd.c:237 trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
15:27:00.970092 git.c:734 trace: exec: git-lfs lock TestAsset.uasset
15:27:00.970092 run-command.c:667 trace: run_command: git-lfs lock TestAsset.uasset
15:27:01.231499 trace git-lfs: exec: git 'version'
15:27:01.480010 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--show-toplevel'
15:27:01.736454 trace git-lfs: exec: uname
15:27:01.985960 trace git-lfs: exec: cygpath '-w' 'C:/GitProjects/UnrealTest'
15:27:02.210976 trace git-lfs: exec: cygpath '-w' 'C:\GitProjects\UnrealTest'
15:27:02.421929 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
15:27:02.604963 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
15:27:02.814782 trace git-lfs: exec: cygpath '-w' 'C:/GitProjects/UnrealTest/.git'
15:27:03.013687 trace git-lfs: exec: cygpath '-w' 'C:/GitProjects/UnrealTest'
15:27:03.205520 trace git-lfs: exec: git 'config' '-l'
15:27:03.419656 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
15:27:03.620077 trace git-lfs: exec: git 'config' '-l' '--blob' ':.lfsconfig'
15:27:03.835465 trace git-lfs: exec: git 'config' '-l' '--blob' 'HEAD:.lfsconfig'
15:27:04.068595 trace git-lfs: run_command: ssh -- [email protected] git-lfs-authenticate TestGames/UnrealTest.git upload
15:27:04.080999 trace git-lfs: exec: ssh '--' '[email protected]' 'git-lfs-authenticate TestGames/UnrealTest.git upload'
15:27:07.223953 trace git-lfs: HTTP: POST https://github.com/_lfs/TestGames/UnrealTest/locks
15:27:09.367613 trace git-lfs: HTTP: 201
15:27:09.367996 trace git-lfs: HTTP: {"lock":{"id":"345","path":"Content/Blueprints/Actors/TestAsset.uasset","locked_at":"2022-05-16T19:27:09.38374612Z"}}

15:27:09.370090 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--show-toplevel'
15:27:09.630527 trace git-lfs: exec: cygpath '-w' 'C:/GitProjects/UnrealTest'
15:27:09.866219 trace git-lfs: exec: cygpath '-w' 'C:\GitProjects\UnrealTest'
Locked Content/Blueprints/Actors/TestAsset.uasset
15:27:10.073222 trace git-lfs: filepathfilter: rewrite ".git" as "/.git/" (strict: false)
15:27:10.073222 trace git-lfs: filepathfilter: rewrite "/.git" as "/.git" (strict: false)
15:27:10.073222 trace git-lfs: filepathfilter: accepting "tmp"`

Note that the same amount of files runs much quicker in our own tool written in .NET. Using multithreading we've been able to lock 30+ files in less than 5 seconds, without locking the UI.

from uegitplugin.

We use HTTPS instead of SSH for auth, which should speed things up by a few seconds. But, not sure why it would take up to a minute. Usually takes 10 seconds for us at most. Will look to see if we have any other configuration changes in Unreal or git.

The multi-threaded processing is done for multiple files at once, and is in the git-lfs binary in the root of the repo.

from uegitplugin.

Hi, could you try this in DefaultEngine.ini:

[SystemSettingsEditor]
r.Editor.SkipSourceControlCheckForEditablePackages=1

from uegitplugin.

Hi,

We're already using that parameter and it doesn't seem to affect the performance.

Regarding the multi-threaded processing, do you mean to say the threading is done inside git-lfs ?
Or is it being done inside Unreal ?

from uegitplugin.

Inside the git-lfs binary, it multithreads the locking/unlocking.

from uegitplugin.

Do you have any reference for your .NET tooling? Would love to see the differences in how they're processing it, so I can get a better understanding of why one works vs. the other. I understand though if it's proprietary.

from uegitplugin.

Pushed new Git LFS binaries (latest 3.3.0 release): https://github.com/ProjectBorealis/UEGitPlugin/tree/dev

Let me know if this resolves the issue.

from uegitplugin.

I've also included a lot of performance improvements in the latest patches, now released!

from uegitplugin.