random 401 status in lfs.log causing client-side smudge errors about lfs-test-server HOT 7 OPEN

Hey @KC-SFC, thanks for opening this issue up, and sorry that you're having trouble!

Except that it randomly will throw a 401 status when asked for file. The git repo has about 100 binary files in the tree, and there's no telling when you'll get the 401. If you repeatedly pull, the client-side cache will eventually populate and the "error goes away" for the user.

This seems like a big clue to me. I'm wondering if there's either:

1. An issue with timing de-sync between the client and server (since lfs-test-server does not implement the new relative expires_in property), or...
2. An issue with a perculiar credential helper set-up on the client end not responding with correct or up-to-date credentials.

Do you have anything interesting going on that might pertain to number (2)?

from lfs-test-server.

Except that it randomly will throw a 401 status when asked for file. The git repo has about 100 binary files in the tree, and there's no telling when you'll get the 401. If you repeatedly pull, the client-side cache will eventually populate and the "error goes away" for the user.

What happens when you get a 401? Are you prompted to enter a user and pass? Does it go away for a little bit? If so, try upgrading to LFS v2.3.4. It caches those command prompts in memory for a single git lfs command, even if you have no credential helper configured.

Unfortunately, this is a test server that's only used as a reference implementation, and has never been run in a long term production scenario by any of the authors. I'd highly recommend upgrading to a self hosted Git installation with LFS support baked in. If not, maybe something like Artifactory would work better for you.

from lfs-test-server.

Hi Team - and I have to say a big THANK YOU for the hard work in the implementation of this test server code. I knew what I was doing when I used it - it does say in BIG letters "not for production". We don't use paid-for git hosting so yours was the only game in town for LFS, and it does work well; I hope you're proud of it; you should be. We've not lost a single file, we can get everything and it solves our what-do-you-do-with-binaries problem (I'm not keen on "use SVN" as the answer to that question)

The username an password for LFS is baked-in to URL in .lfsconfig i.e. 'https://someuser:somepassword@ourserver:9999", and AFAIK no credentials helper set up. We use ssh for our normal git clones, so don't need one to help authenticate for that; the joy of linux .ssh/config eh?

The 401 failure just results in our scripts failing on the clone. Run it again and the failing object comes down fine, with a 401 on another file in the updated list.

I've built the latest git for the clients and with sort of the latest LFS, so if that changes anything. The only place for a 401 is in the server.go code, L567 - just after the Header set which mentions the basic realm of git-lfs-server. Bearing in mind I've got no idea what this is doing (!) but is there an issue with us running 9 instances of git-lfs-server with the same realm name?

Or is it some problem with the meta store not returning the right data quickly enough?

from lfs-test-server.

@KC-SFC Thanks, and you'll have to bear with me since it's been a little while since the last time I've looked through this code. The *MetaStore authenticates here:

https://github.com/git-lfs/lfs-test-server/blob/master/meta_store.go#L442

Which returns false if the password is the empty string,"". Can you double check that the user is sending the right password, and that is has length greater than zero?

from lfs-test-server.

That's a great question. The username and password length should be straight from the .lsconfig - which is committed into the repo. It should be the same on every call to the API.

I wonder if the IT networking team are messing with our traffic.

TY for the help. I have some digging to do.

from lfs-test-server.

If you want to confirm whether the Git LFS client is doing the right thing, run this:

$ LFS_DEBUG_HTTP=1 GIT_CURL_VERBOSE=1 git ...

This will add the full http request and responses to the command output. LFS_DEBUG_HTTP is a custom flag that enables LFS to dump the Authorization header too. If you're running a command like git pull, then you'll see Git's HTTP requests too. A command like git lfs pull should be enough to check just LFS.

from lfs-test-server.

Stumbled here looking to solve https://github.com/git-lfs/git-lfs/issues/3416, which may be related. I assumed it was a git-lfs issue, but it may be an lfs-test-server issue.

We are also playing with using lfs-test-server in a production-ish environment (with daily backups to deal with failures). We use github.com, but it isn't a viable solution for our large files both because of the 2GB size limit and the massive time hit we would take uploading and downloading giant files over the internet. We don't want to do a self-hosted git, but maybe we have to consider it.

from lfs-test-server.