git-annex diff about datalad HOT 8 CLOSED

In that thread, I suggested using git diff --ext-diff with a external diff program that could look at the contents of the annexed symlinks.

That still seems to me to be basically doable.

joey@darkstar:~/tmp/rr2>GIT_EXTERNAL_DIFF=echo git diff HEAD^..HEAD
baz /home/joey/tmp/X1qzaf_baz 3a1d59ec93df0f689f2fe219ac37c804660beb4a 120000 /home/joey/tmp/9PBJaf_baz d98fd2c46cc683ff157be03a5b42fbd303c58d93 120000

I see that the external diff helper is passed two temp files. When annexed files are being diffed, the temp files, while not actually being symlinks (arguably a git bug!) do contain the link target that got committed to git for that file. (Even if they were symlinks, they're located in $TMP, so they won't point at the annexed content.)

So, what could be done is, adjust those temp files to be proper symlinks to the right path into .git/annex, and then run $whatever to diff those 2 files. Optionally git-annex getting any missing content of either of the two versions of the file.

This could be scripted w/o modifying git-annex, or git-annex could provide a git-annex diff, that overrides GIT_EXTERNAL_DIFF="git-annex fixdiff", which would handle the above fixup and then chain off to the real external diff program. (Either the original setting of GIT_EXTERNAL_DIFF, or the diff driver specified by the diff gitattribute.)

from datalad.

I'm not sure about git-annex getting missing contents while diffing.. It may make diffing an unexpectedly expensive or failure-prone process. Any thoughts on that?

from datalad.

On Mon, 10 Nov 2014, Joey Hess wrote:

This could be scripted w/o modifying git-annex, or git-annex could provide
a git-annex diff, that overrides GIT_EXTERNAL_DIFF="git-annex fixdiff",
which would handle the above fixup and then chain off to the real external
diff program. (Either the original setting of GIT_EXTERNAL_DIFF, or the
diff driver specified by the diff gitattribute.)

Thank for looking into it!

Since I expect it might come handy for other users as well, may be
indeed 'git-annex diff' decorator which would override GIT_EXTERNAL_DIFF
to later invoke original GIT_EXTERNAL_DIFF on those two target symlinks
be useful for general annex audience.

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik

from datalad.

On Mon, 10 Nov 2014, Joey Hess wrote:

I'm not sure about git-annex getting missing contents while diffing.. It
may make diffing an unexpectedly expensive or failure-prone process. Any
thoughts on that?

fail by default and add a (configurable) option
(annex.retrieve-to-diff) to retrieve?

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik

from datalad.

About Diff, do you know the Coopy toolbox ? http://share.find.coop and all the things around it ? like this one http://datacommons.coop & http://paulfitz.github.io/daff or http://data.okfn.org & http://dataprotocols.org

from datalad.

Looking into this some more, git log and perhaps other things can call git diff and for this to work with an external diff driver, having the diff only be started by a git annex diff command is not the right approach.

So, my current plan is to make git-annex be able to function an an external diff driver shim. To make git use it, set GIT_EXTERNAL_DIFF=git-annex diffdriver.

Then, git-annex will need to read the .gitattributes files to find out the real diff driver that is configured to be used for files, and pass the diff requests on to them, after fixing up annexed file links.

Some workflows might involve just setting GIT_EXTERNAL_DIFF="foo --opts" to configure the external diff driver to use. To support this, we can use GIT_EXTERNAL_DIFF="git-annex diffdriver -- foo --opts"

from datalad.

After implementing that, I discovered that GIT_EXTERNAL_DIFF does not override .gitattributes. Instead, if .gitattributes is set, git diff uses it in preference to looking at the environment.

Horrors! I've emailed the git list to see if this is a bug.

If it's intentional, I think a different approach yet is called for. Actually, I think it will be simpler anyway, so I'll just do it:

Make "git annex diffdriver -- cmd params --" be what the user configures as the external diff driver command. Which will in turn run "cmd params" with the fixed up files.

This avoids needing to set GIT_EXTERNAL_DIFF, so will actually simplify the code quite a lot, as git-annex won't need to look at .gitattributes itself.

Wish I had thought of this approach faster!

from datalad.

Current state of my "affairs" in a demo:

❯ cat ~/bin/git-annex-diff-wrapper
#!/usr/bin/env bash                                                             
set -eu
# echo "Called with $# args: $@ under $PWD" >> calls
LANG=C colordiff --fakeexitcode --color=auto -u  "$2" "$5"
❯ git config alias.annexdiff
!GIT_EXTERNAL_DIFF='git-annex diffdriver -- git-annex-diff-wrapper --' git diff
❯ git annexdiff HEAD^^..HEAD^
--- .git/annex/objects/30/zX/SHA256E-s2--4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865.dat/SHA256E-s2--4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865.dat    2023-06-07 12:15:03.348666122 -0400
+++ .git/annex/objects/03/xx/SHA256E-s4--a6e2b7a040683432de03a18fd8a1939a2fdf82585b364bfc874bdd4095c4cae1.dat/SHA256E-s4--a6e2b7a040683432de03a18fd8a1939a2fdf82585b364bfc874bdd4095c4cae1.dat    2023-06-07 12:15:03.556667579 -0400
@@ -1 +1,2 @@
 1
+2
❯ GIT_EXTERNAL_DIFF='git-annex diffdriver -- git-annex-diff-wrapper --' git log -p --ext-diff | head -n 20
commit 4bf38f60fb5a21fc67222bfe1de1d777384049b8
Author: Yaroslav Halchenko <[email protected]>
Date:   Wed Jun 7 12:15:03 2023 -0400

    change 3

--- .git/annex/objects/03/xx/SHA256E-s4--a6e2b7a040683432de03a18fd8a1939a2fdf82585b364bfc874bdd4095c4cae1.dat/SHA256E-s4--a6e2b7a040683432de03a18fd8a1939a2fdf82585b364bfc874bdd4095c4cae1.dat	2023-06-07 12:15:03.556667579 -0400
+++ .git/annex/objects/50/6x/SHA256E-s6--14c5e74c4b96ccef41cd94db73a9ec3348038ac094feca4fd897cecffa07cdae.dat/SHA256E-s6--14c5e74c4b96ccef41cd94db73a9ec3348038ac094feca4fd897cecffa07cdae.dat	2023-06-07 12:15:03.756668982 -0400
@@ -1,2 +1,3 @@
 1
 2
+3

from datalad.