cc -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c dulwich/_objects.c -o build/temp.macosx-10.9-intel-2.7/dulwich/_objects.o

clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]

clang: note: this will be a hard error (cannot be downgraded to a warning) in the future

error: command 'cc' failed with exit status 1

See https://developer.apple.com/library/mac/releasenotes/DeveloperTools/RN-Xcode/xc5_release_notes/xc5_release_notes.html#//apple_ref/doc/uid/TP40001051-CH2-SW12

Hi,

in Index object, I think it would be good to have a method to get all file of one folder.

Example :

>>> [i for i in index]
['index.html', 'folder1/a.html', 'folder1/b.html', 'folder2/c.html']
>>> index['/']
['index.html', 'folder1', 'folder2']
>>> index['folder1']
['a.html', 'b.html']

What are your comment about it ?

Best regards,
Stephane

is there currently support for prompting garbage collection? I'm not seeing anything poking through the docs and the source. If not, could you consider this a feature request?

thanks so much for a great library!

Append one function to get datas that I can read with "git status"

See patch: http://bitbucket.org/ashcrow/dulwich/changeset/2083399ddcdc/

C:\work\dul>python c:\Python26\Scripts\dulwich clone git://github.com/jelmer/dulwich.git dulwich
Counting objects: 3841, done.
Compressing objects: 100% (2823/2823), done.
Total 3841 (delta 1383), reused 3392 (delta 960)
Traceback (most recent call last):
File "c:\Python26\Scripts\dulwich", line 5, in
pkg_resources.run_script('dulwich==0.5.1', 'dulwich')
File "C:\Python26\lib\site-packages\setuptools-0.6c9-py2.6.egg\pkg_resources.py", line 448, in run
script
File "C:\Python26\lib\site-packages\setuptools-0.6c9-py2.6.egg\pkg_resources.py", line 1166, in ru
n_script
File "c:\python26\lib\site-packages\dulwich-0.5.1-py2.6-win32.egg\EGG-INFO\scripts\dulwich", line
186, in
commandscmd
File "c:\python26\lib\site-packages\dulwich-0.5.1-py2.6-win32.egg\EGG-INFO\scripts\dulwich", line
154, in cmd_clone
r["HEAD"] = remote_refs["HEAD"]
File "C:\Python26\lib\site-packages\dulwich-0.5.1-py2.6-win32.egg\dulwich\repo.py", line 853, in _
setitem
self.refs[name] = value
File "C:\Python26\lib\site-packages\dulwich-0.5.1-py2.6-win32.egg\dulwich\repo.py", line 498, in _
setitem_
self.set_if_equals(name, None, ref)
File "C:\Python26\lib\site-packages\dulwich-0.5.1-py2.6-win32.egg\dulwich\repo.py", line 442, in s
et_if_equals
realname, _ = self._follow(name)
File "C:\Python26\lib\site-packages\dulwich-0.5.1-py2.6-win32.egg\dulwich\repo.py", line 241, in _
follow
contents = self.read_ref(refname)
File "C:\Python26\lib\site-packages\dulwich-0.5.1-py2.6-win32.egg\dulwich\repo.py", line 216, in r
ead_ref
contents = self.read_loose_ref(refname)
File "C:\Python26\lib\site-packages\dulwich-0.5.1-py2.6-win32.egg\dulwich\repo.py", line 392, in r
ead_loose_ref
f = GitFile(filename, 'rb')
File "C:\Python26\lib\site-packages\dulwich-0.5.1-py2.6-win32.egg\dulwich\file.py", line 70, in Gi
tFile
return file(filename, mode, bufsize)
IOError: [Errno 22] invalid mode ('rb') or filename: 'dulwich.git\refs\heads\master\r'

C:\work\dul>

Python 2.6.5
XP SP3

I know this isn't a minor change and you probably don't want to make it for compatibility reasons. But I really think the data type used for strings should be unicode. I.e. dulwich should decode commit messages, author names, ... to unicode.

I'd be happy to take a stab at this. You don't get around it anyway when you port it to Python 3.

I am new to the code base and have been trying to get my head around it (though pretty easy to follow along).

With the repo I am working with I am seeing this:

>>> from dulwich.repo import Repo
>>> repo = Repo("<my bare repo path>")
>>> repo.head()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/brian/Development/dulwich/dulwich/repo.py", line 308, in head
    return self.refs.follow('HEAD')
  File "/Users/brian/Development/dulwich/dulwich/repo.py", line 82, in follow
    return follow_ref(self, name)
  File "/Users/brian/Development/dulwich/dulwich/repo.py", line 65, in follow_ref
    return follow_ref(container, ref)
  File "/Users/brian/Development/dulwich/dulwich/repo.py", line 60, in follow_ref
    contents = container[name]
  File "/Users/brian/Development/dulwich/dulwich/repo.py", line 163, in __getitem__
    raise KeyError(name)
KeyError: 'refs/heads/master'

Digging in a bit deeper I see that refs/heads/master is a packed ref which is why it cannot find it in refs/heads. Is there any reason why Repo.head uses self.refs.follow over self.ref? Hence the title. ref looks in packed refs, but doesn't know what HEAD is equal to to pull out the right ref.

I am sort of lost in the implementation to find the correct place to solve this.

Hi,

this is the offending piece of code:

def ack(self, sha):
    """..."""
    if sha in self.heads:
        self.heads.remove(sha)
    if sha in self.parents:
        for p in self.parents[sha]:
            self.ack(p)

Most of the time, python can push ahead with this and complete if self.parents is short. However, I had a pretty sizeable repository, and this was taking forever to complete.

Please see my branch for a test that demonstrates this and a fix.

repo['refs/heads/1234567-123456-12345678912345']

results in a TypeError with a traceback somewhat like this:

File "dulwich/repo.py", line 1113, in __getitem__
    return self.object_store[name]
  File "dulwich/object_store.py", line 115, in __getitem__
    type_num, uncomp = self.get_raw(sha)
  File "dulwich/object_store.py", line 362, in get_raw
    sha = hex_to_sha(name)
  File "dulwich/objects.py", line 92, in hex_to_sha
    return binascii.unhexlify(hex)
TypeError: Non-hexadecimal digit found

In the case when HEAD is not symbolically referencing another branch, we need to be able to set it directly to a commit. Currently doesn't seem to have a easy way to do so.

2a8548b, latest from git.

I got a local copy and then, from the top directory in a windows command prompt, python setup.py install.
I tried with Python 2.7 and 3.2.3. Both ended with
running build_ext
building 'dulwich._objects' extension
error: Unable to find vcvarsall.bat

It was unclear to me from your web pages and internal documentation if this was how I was supposed to install dulwich, or what versions of python it is compatible with.

My python setup may be imperfect since I installed as a local user on my locked down Windows 7 system (I'm also unsure whether I need admin rights to install dulwich). In particular, my PATH did not include the directory for python until I edited it to get the 3.2 version.

When performing a git ls-remote on a repo served be dulwich i got a stacktrace from dulwich. Indeed determine_wants in server.py can return empty dict if client doesn't send wanted refs (the case for a ls-remote or delete branch). Here is the stacktrace :

2013-09-01 18:19:17,416 INFO: Listening for TCP connections on localhost:9418
2013-09-01 18:19:23,288 INFO: Handling request from ('127.0.0.1', 48555)
2013-09-01 18:19:23,288 INFO: Handling git-upload-pack request, args=['/home/fabien/enovance/git-swift/smart/repo2', 'host=localhost']
2013-09-01 18:19:23,292 ERROR: Exception happened during processing of request from ('127.0.0.1', 48555)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/home/fabien/git/enovance/dulwich/dulwich/server.py", line 750, in _make_handler
return TCPGitRequestHandler(self.handlers, _args, *_kwargs)
File "/home/fabien/git/enovance/dulwich/dulwich/server.py", line 730, in init
SocketServer.StreamRequestHandler.init(self, _args, *_kwargs)
File "/usr/lib/python2.7/SocketServer.py", line 649, in init
self.handle()
File "/home/fabien/git/enovance/dulwich/dulwich/server.py", line 741, in handle
h.handle()
File "/home/fabien/git/enovance/dulwich/dulwich/server.py", line 272, in handle
get_tagged=self.get_tagged)
File "/home/fabien/git/enovance/dulwich/dulwich/repo.py", line 903, in fetch_objects
haves = self.object_store.find_common_revisions(graph_walker)
File "/home/fabien/git/enovance/dulwich/dulwich/object_store.py", line 186, in find_common_revisions
sha = graphwalker.next()
File "/home/fabien/git/enovance/dulwich/dulwich/server.py", line 420, in next
return self._impl.next()
AttributeError: 'NoneType' object has no attribute 'next'

I assume None shoud be return instead of [] in that case : https://github.com/jelmer/dulwich/blob/master/dulwich/server.py#L384
Or fetch_object must handle the case by "if not want:" https://github.com/jelmer/dulwich/blob/master/dulwich/repo.py#L899

Cheers,

Fbo

I can't build dulwich anymore on Windows 7 x64. I get this error using Windows SDK 7.0 (MSVC9).

"_pack.c(21) fatal error C1083: Cannot open include file: 'stdint.h': No such file or directory"

I tried using this version
http://msinttypes.googlecode.com/svn/trunk/stdint.h.
by copying "stdint.h" into dulwich folder, and I also tried compiling with sdk 7.1 (MSVC10) which includes stdint.h. Dulwich compiles without errors for both cases, and most of the tests pass, except for test_pack.py, which errors in TestThinPack, and almost all of the unittests in tests/compat fail.

The failures are all either the file is in use by another process (ThinPack), access denied (dulwich.tests.compat.test_client.DulwichMockSSHClientTest), or timed out (dulwich.tests.compat.test_client.DulwichTCPClientTest).

I think it's built fine, but I'm not 100% Are the tests supposed to fail, or are there some tests that I can run to see if _pack.pyd as built is functioning correctly? Thanks!

Pushing to a local git repo using hg-git, I got the following output:

(actually some of the "garbage" characters didn't pass through the coding system correctly, so you can download the raw output here)

Specific steps to reproduce:

1. Install mercurial and hg-git
2. hg clone https://[email protected]/dabrahams/pip/
3. mkdir pip-git; cd pip-git; git init; cd ..
4. cd pip
5. hg push ../pip-git

pushing to /Users/dave/src/pip-git2
importing Hg objects into Git
creating and sending data
** unknown exception encountered, details follow
** report bug details to http://mercurial.selenic.com/bts/
** or [email protected]
** Mercurial Distributed SCM (version 1.5.1)
** Extensions loaded: bookmarks, graphlog, hggit
Traceback (most recent call last):
  File "/opt/local/bin/hg", line 27, in 
    mercurial.dispatch.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 16, in run
    sys.exit(dispatch(sys.argv[1:]))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 30, in dispatch
    return _runcatch(u, args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 47, in _runcatch
    return _dispatch(ui, args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 467, in _dispatch
    return runcommand(lui, repo, cmd, fullargs, ui, options, d)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 337, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 518, in _runcommand
    return checkargs()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 472, in checkargs
    return cmdfunc()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 466, in 
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
    return func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/commands.py", line 2445, in push
    r = repo.push(other, opts.get('force'), revs=revs)
  File "/Users/dave/src/hg-git/hggit/hgrepo.py", line 19, in push
    git.push(remote.path, revs, force)
  File "/Users/dave/src/hg-git/hggit/git_handler.py", line 147, in push
    changed_refs = self.upload_pack(remote, revs, force)
  File "/Users/dave/src/hg-git/hggit/git_handler.py", line 545, in upload_pack
    changed_refs = client.send_pack(path, changed, genpack)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/dulwich-0.5.1-py2.6-macosx-10.6-x86_64.egg/dulwich/client.py", line 269, in send_pack
    return client.send_pack(path, changed_refs, generate_pack_contents)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/dulwich-0.5.1-py2.6-macosx-10.6-x86_64.egg/dulwich/client.py", line 131, in send_pack
    raise ChecksumMismatch(sha, client_sha)
dulwich.errors.ChecksumMismatch: Checksum mismatch: Expected (�B‘Ú§eNçhÚØ�˙h’�@"–, got �error: refusing to update checked out branch: refs/heads/master

When receiving a 302 redirect to a POST response, dulwich re-sends the request as a GET (without data) instead. This causes a 405 to be returned by some smart HTTP servers (e.g. gerrit).

This issue also happens to be in git 1.7.1, but appears to be fixed in git 1.8.1.2.

An example of a repository affected by this bug is: http://git.opendaylight.org/gerrit/p/affinity.git

I'm not sure how to get a trace of the HTTP traffic from dulwich without actually resorting to using something like mocker or tcpdump but here's the trace from git 1.7 with GIT_CURL_VERBOSE=1:

Initialized empty Git repository in /tmp/affinity/.git/
* Couldn't find host git.opendaylight.org in the .netrc file; using defaults
* About to connect() to git.opendaylight.org port 80 (#0)
*   Trying 140.211.169.21... * Connected to git.opendaylight.org (140.211.169.21) port 80 (#0)
> GET /gerrit/p/affinity.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.1
Host: git.opendaylight.org
Accept: */*
Pragma: no-cache

< HTTP/1.1 301 Moved Permanently
< Date: Fri, 04 Oct 2013 06:28:23 GMT
< Server: Apache/2.2.15 (Red Hat)
< Location: https://git.opendaylight.org/gerrit/p/affinity.git/info/refs?service=git-upload-pack
< Content-Length: 292
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<
* Closing connection #0
* Issue another request to this URL: 'https://git.opendaylight.org/gerrit/p/affinity.git/info/refs?service=git-upload-pack'
* Couldn't find host git.opendaylight.org in the .netrc file; using defaults
* About to connect() to git.opendaylight.org port 443 (#0)
*   Trying 140.211.169.21... * Connected to git.opendaylight.org (140.211.169.21) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: [email protected],CN=*.opendaylight.org,O=The Linux Foundation,L=San Francisco,ST=California,C=US,OID.2.5.4.13=Jd4wx0gq5e9VdOEO
*       start date: Mar 05 05:03:06 2013 GMT
*       expire date: Mar 06 01:11:53 2015 GMT
*       common name: *.opendaylight.org
*       issuer: CN=StartCom Class 2 Primary Intermediate Server CA,OU=Secure Digital Certificate Signing,O=StartCom Ltd.,C=IL
> GET /gerrit/p/affinity.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.1
Host: git.opendaylight.org
Accept: */*
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Fri, 04 Oct 2013 06:28:24 GMT
< Expires: Tue, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Type: application/x-git-upload-pack-advertisement
< Content-Length: 7054
< Connection: close
<
* Expire cleared
* Closing connection #0
* Couldn't find host git.opendaylight.org in the .netrc file; using defaults
* About to connect() to git.opendaylight.org port 80 (#0)
*   Trying 140.211.169.21... * connected
* Connected to git.opendaylight.org (140.211.169.21) port 80 (#0)
> POST /gerrit/p/affinity.git/git-upload-pack HTTP/1.1
User-Agent: git/1.7.1
Host: git.opendaylight.org
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 116

< HTTP/1.1 301 Moved Permanently
< Date: Fri, 04 Oct 2013 06:28:25 GMT
< Server: Apache/2.2.15 (Red Hat)
< Location: https://git.opendaylight.org/gerrit/p/affinity.git/git-upload-pack
< Content-Length: 274
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<
* Closing connection #0
* Issue another request to this URL: 'https://git.opendaylight.org/gerrit/p/affinity.git/git-upload-pack'
* Violate RFC 2616/10.3.2 and switch from POST to GET
* Couldn't find host git.opendaylight.org in the .netrc file; using defaults
* About to connect() to git.opendaylight.org port 443 (#0)
*   Trying 140.211.169.21... * connected
* Connected to git.opendaylight.org (140.211.169.21) port 443 (#0)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: [email protected],CN=*.opendaylight.org,O=The Linux Foundation,L=San Francisco,ST=California,C=US,OID.2.5.4.13=Jd4wx0gq5e9VdOEO
*       start date: Mar 05 05:03:06 2013 GMT
*       expire date: Mar 06 01:11:53 2015 GMT
*       common name: *.opendaylight.org
*       issuer: CN=StartCom Class 2 Primary Intermediate Server CA,OU=Secure Digital Certificate Signing,O=StartCom Ltd.,C=IL
> GET /gerrit/p/affinity.git/git-upload-pack HTTP/1.1
User-Agent: git/1.7.1
Host: git.opendaylight.org
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result

* The requested URL returned error: 405
* Closing connection #0
error: RPC failed; result=22, HTTP code = 405

And here's the output from git 1.8:

Cloning into '/tmp/affinity'...
* Couldn't find host git.opendaylight.org in the .netrc file; using defaults
* About to connect() to git.opendaylight.org port 80 (#0)
*   Trying 140.211.169.21...
* Connected to git.opendaylight.org (140.211.169.21) port 80 (#0)
> GET /gerrit/p/affinity.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.2
Host: git.opendaylight.org
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 301 Moved Permanently
< Date: Fri, 04 Oct 2013 06:30:35 GMT
< Server: Apache/2.2.15 (Red Hat)
< Location: https://git.opendaylight.org/gerrit/p/affinity.git/info/refs?service=git-upload-pack
< Content-Length: 292
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
* Closing connection 0
* Issue another request to this URL: 'https://git.opendaylight.org/gerrit/p/affinity.git/info/refs?service=git-upload-pack'
* Couldn't find host git.opendaylight.org in the .netrc file; using defaults
* About to connect() to git.opendaylight.org port 443 (#1)
*   Trying 140.211.169.21...
* Connected to git.opendaylight.org (140.211.169.21) port 443 (#1)
* found 160 certificates in /etc/ssl/certs/ca-certificates.crt
*    server certificate verification OK
*    common name: *.opendaylight.org (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: 2.5.4.13=#13104a6434777830677135653956644f454f,C=US,ST=California,L=San Francisco,O=The Linux Foundation,CN=*.opendaylight.org,[email protected]
*    start date: Tue, 05 Mar 2013 05:03:06 GMT

*    expire date: Fri, 06 Mar 2015 01:11:53 GMT

*    issuer: C=IL,O=StartCom Ltd.,OU=Secure Digital Certificate Signing,CN=StartCom Class 2 Primary Intermediate Server CA
*    compression: NULL
*    cipher: AES-256-CBC
*    MAC: SHA1
> GET /gerrit/p/affinity.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.2
Host: git.opendaylight.org
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Fri, 04 Oct 2013 06:30:36 GMT
< Expires: Tue, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Type: application/x-git-upload-pack-advertisement
< Content-Encoding: gzip
< Content-Length: 3217
< Connection: close
< 
* Closing connection 1
* Couldn't find host git.opendaylight.org in the .netrc file; using defaults
* About to connect() to git.opendaylight.org port 80 (#2)
*   Trying 140.211.169.21...
* Connected to git.opendaylight.org (140.211.169.21) port 80 (#2)
> POST /gerrit/p/affinity.git/git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.2
Host: git.opendaylight.org
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 186

* upload completely sent off: 186 out of 186 bytes
< HTTP/1.1 301 Moved Permanently
< Date: Fri, 04 Oct 2013 06:30:37 GMT
< Server: Apache/2.2.15 (Red Hat)
< Location: https://git.opendaylight.org/gerrit/p/affinity.git/git-upload-pack
< Content-Length: 274
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
* Closing connection 2
* Issue another request to this URL: 'https://git.opendaylight.org/gerrit/p/affinity.git/git-upload-pack'
* Couldn't find host git.opendaylight.org in the .netrc file; using defaults
* About to connect() to git.opendaylight.org port 443 (#3)
*   Trying 140.211.169.21...
* Connected to git.opendaylight.org (140.211.169.21) port 443 (#3)
* found 160 certificates in /etc/ssl/certs/ca-certificates.crt
* SSL re-using session ID
*    server certificate verification OK
*    common name: *.opendaylight.org (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: 2.5.4.13=#13104a6434777830677135653956644f454f,C=US,ST=California,L=San Francisco,O=The Linux Foundation,CN=*.opendaylight.org,[email protected]
*    start date: Tue, 05 Mar 2013 05:03:06 GMT

*    expire date: Fri, 06 Mar 2015 01:11:53 GMT

*    issuer: C=IL,O=StartCom Ltd.,OU=Secure Digital Certificate Signing,CN=StartCom Class 2 Primary Intermediate Server CA
*    compression: NULL
*    cipher: AES-256-CBC
*    MAC: SHA1
> POST /gerrit/p/affinity.git/git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.2
Host: git.opendaylight.org
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 186

* upload completely sent off: 186 out of 186 bytes
< HTTP/1.1 200 OK
< Date: Fri, 04 Oct 2013 06:30:38 GMT
< Expires: Tue, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Type: application/x-git-upload-pack-result
< Connection: close
< Transfer-Encoding: chunked
< 
* Closing connection 3

Commit objects should have an hash method that returns self.id so they can be put into sets() safely.

Here's my proposed version:

[tox]
downloadcache = {toxworkdir}/cache/
envlist = py24, py25, py26, py27, pypy

[testenv]
deps =
    unittest2

commands = make check
recreate = True
whitelist_externals = make

[testenv:py24]
setenv =
    PIP_INSECURE=1


[testenv:py25]
setenv =
    PIP_INSECURE=1

I'm not making a PR because i have odd issues with tests hanging on python <2.7.

I never got passed:
Doctest: remote.txt

That can go from as little as all the files in the pushed revisions (not just the changed files) to the whole branch back to its root (in case a branch is made off an old revision).
This script should be equivalent to git rev-list --objects (except that it doesn't resolve refs or short hashes), but it gives much more, for example:
for a single revision :
$ git rev-list --objects 088be0574f98b5eca85f94e38f7647ef465876a9 ^a5d43f6cc88834d881e5ed763cb08d14b4afd7d0 |wc -l
5
$ python ../dul-rev-list 088be0574f98b5eca85f94e38f7647ef465876a9 ^a5d43f6cc88834d881e5ed763cb08d14b4afd7d0 |wc -l
1762
for a branch off an old revision
$ python ../dul-rev-list 051a598458657f2a2fd28ab791d053952274db82 ^a5d43f6cc88834d881e5ed763cb08d14b4afd7d0 |wc -l
48723
$ git rev-list --objects 051a598458657f2a2fd28ab791d053952274db82 ^a5d43f6cc88834d881e5ed763cb08d14b4afd7d0 |wc -l
424

Here is the script : http://gist.github.com/316958

I have an odd case from one of RhodeCode users, attached a stacktrace from call that causes a RuntimeError exception in dulwich code. I could not reproduce this, and i don't have any access to repo that is causing this.

>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:155 in __call__
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:131 in __call__
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:107 in __call__
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:312 in dispatch
>>  return controller(environ, start_response)
Module rhodecode.lib.base:287 in __call__
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:211 in __call__
>>  response = self._dispatch_call()
Module pylons.controllers.core:162 in _dispatch_call
>>  response = self._inspect_call(func)
Module pylons.controllers.core:105 in _inspect_call
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:57 in _perform_call
>>  return func(**args)
Module rhodecode.controllers.summary:141 in index
>>  run_task(get_commits_stats, c.dbrepo.repo_name, ts_min_y, ts_max_y)
Module rhodecode.lib.celerylib:79 in run_task
>>  return ResultWrapper(task(*args, **kwargs))
Module celery.task.base:241 in __call__
>>  return self.run(*args, **kwargs)
Module celery.app:141 in run
>>  return fun(*args, **kwargs)
Module rhodecode.lib.celerylib.tasks:2 in get_commits_stats
Module rhodecode.lib.celerylib:122 in __wrapper
>>  ret = func(*fargs, **fkwargs)
Module rhodecode.lib.celerylib.tasks:171 in get_commits_stats
>>  datadict["added"] += len(cs.added)
Module rhodecode.lib.vcs.utils.lazy:27 in __get__
>>  result = obj.__dict__[self.__name__] = self._func(obj)
Module rhodecode.lib.vcs.backends.git.changeset:522 in added
>>  self._get_paths_for_status('added')], self)
Module rhodecode.lib.vcs.backends.git.changeset:507 in _get_paths_for_status
>>  a, m, d = self._changes_cache
Module rhodecode.lib.vcs.utils.lazy:27 in __get__
>>  result = obj.__dict__[self.__name__] = self._func(obj)
Module rhodecode.lib.vcs.backends.git.changeset:492 in _changes_cache
>>  for (oldpath, newpath), (_, _), (_, _) in changes:
Module dulwich.object_store:142 in tree_changes
>>  want_unchanged=want_unchanged):
Module dulwich.diff_tree:183 in tree_changes
>>  for entry1, entry2 in entries:
Module dulwich.diff_tree:148 in walk_trees
>>  tree1 = is_tree1 and store[entry1.sha] or None
Module dulwich.object_store:112 in __getitem__
>>  type_num, uncomp = self.get_raw(sha)
Module dulwich.object_store:333 in get_raw
>>  return pack.get_raw(sha)
Module dulwich.pack:1844 in get_raw
>>  type_num, chunks = self.data.resolve_object(offset, obj_type, obj)
Module dulwich.pack:1055 in resolve_object
>>  type, base_chunks = self.resolve_object(base_offset, type, base_obj)
Module dulwich.pack:1055 in resolve_object
>>  type, base_chunks = self.resolve_object(base_offset, type, base_obj)
...
Module dulwich.pack:1055 in resolve_object
>>  type, base_chunks = self.resolve_object(base_offset, type, base_obj)
Module dulwich.pack:1055 in resolve_object
>>  type, base_chunks = self.resolve_object(base_offset, type, base_obj)
Module dulwich.pack:1055 in resolve_object
>>  type, base_chunks = self.resolve_object(base_offset, type, base_obj)
Module dulwich.pack:1055 in resolve_object
>>  type, base_chunks = self.resolve_object(base_offset, type, base_obj)
Module dulwich.pack:1055 in resolve_object
>>  type, base_chunks = self.resolve_object(base_offset, type, base_obj)
Module dulwich.pack:1062 in resolve_object
>>  self._offset_cache[offset] = type, chunks
Module dulwich.lru_cache:198 in __setitem__
>>  self.add(key, value, cleanup=None)
Module dulwich.lru_cache:330 in add
>>  node = _LRUNode(key, value, cleanup=cleanup)
RuntimeError: maximum recursion depth exceeded

But i think we can ask question to this user undere here: https://bitbucket.org/marcinkuzminski/rhodecode/issue/642/

The fact that dulwich is currently GPL licensed removes a lot of usefulness of the project as a storage format for other things. Are there any chances that the license could change to something like BSD?

Check out the diff stat here.

Use read_pkt_line() instead of assuming the server response is of length 20.

Also, since we're now using read_pkt_line(), catch HangupException. This may happen when talking to old versions of git-daemon (before v1.6.6.1-26-g38a81b4).

Hi,

This description follows the previous bug report on launchpad: https://bugs.launchpad.net/dulwich/+bug/953916

I've been able to reproduce the same error on a public git repository, so it can now easily be reproduced.
I can reproduce this problem with at least two versions of dulwich: the current pypi version and the latest github master version at 2a8548b (Dec 04, 2012).

It happens during a git pull operation of a repository served by http.
Here are the steps to reproduce:

$ cd /tmp 
$ git clone https://github.com/asaladin/testdul_repo.git ptools
$ git clone https://github.com/asaladin/testdul_git2.git git2

$ cat test_dulwich/test_dulwich.py
##################
from dulwich.repo import Repo
from dulwich.web import HTTPGitApplication
from dulwich import server as dulserver

repo_path = "/tmp/ptools"

_d = {'/repo' : Repo(repo_path)}
backend = dulserver.DictBackend(_d)
gitserve = HTTPGitApplication(backend)

def app(environ, start_response):
   return gitserve(environ, start_response)
##################

#run on another shell with: $ gunicorn test_dulwich:app
$ cd git2
$ git pull http://localhost:8000/repo 
fatal: git fetch_pack: expected ACK/NAK, got 'dul-daemon says what'

cheers,

when pip installing on Red Hat Enterprise Linux 6, which has py2.6, i get this error::

SyntaxError: ('invalid syntax', ('/usr/lib64/python2.6/site-packages/dulwich/client.py', 1015, 11, '        for key in SSHGitClient.SSH_KWARGS_KEYS\n'))

This is because dict comprehension syntax isn't supported in 2.6. A compatible syntax is::

# Incompatible with py2.6
#ssh_kwargs = {
#    key: kwargs.pop(key, None)
#    for key in SSHGitClient.SSH_KWARGS_KEYS
#}
# dict comprehension for py26 compatability
ssh_kwargs = dict(
    [(key, kwargs.pop(key, None)) for key in SSHGitClient.SSH_KWARGS_KEYS]
)

index.py, line 258, has this :

def __setitem__(self, name, x):
    assert isinstance(name, str)

Unfortunatly, this kills ducktyping. You can't use a subclass of unicode for example.

Something like :

def __setitem__(self, name, x):
    assert isinstance(name, basetring)

Would be better. It allows the use of libs such as path.py (https://github.com/jaraco/path.py) without getting an AssertionError. With Python 3.4 comming with pathlib (http://docs.python.org/3.4/library/pathlib.html), this will become more and more important.

Even better would be to get rid of the test completly, as it prevents use of transparent string buffers and such, that does not necessary inherit from basestring.

fetch documentation says determine_wants is optional, however, if it's not set, it doesn't set one for fetch_objects (which requires it):

http://github.com/jelmer/dulwich/blob/4a321d8d3dba90c7c4481a306d797d3bf2306d4b/dulwich/repo.py#L253

Sorry, I don't know if this is a dulwich or hg-git problem:

pulling from /Users/dave/work/pipsync/ian-bare.git
importing Hg objects into Git
Counting objects: 2104, done.
Compressing objects:   0% (1/1495)   
Compressing objects:   1% (15/1495)   
Compressing objects:   2% (30/1495)   
Compressing objects:   3% (45/1495)   
Compressing objects:   4% (60/1495)   
Compressing objects:   5% (75/1495)   
Compressing objects:   6% (90/1495)   
Compressing objects:   7% (105/1495)   
Compressing objects:   8% (120/1495)   
Compressing objects:   9% (135/1495)   
Compressing objects:  10% (150/1495)   
Compressing objects:  11% (165/1495)   
Compressing objects:  12% (180/1495)   
Compressing objects:  13% (195/1495)   
Compressing objects:  14% (210/1495)   
Compressing objects:  15% (225/1495)   
Compressing objects:  16% (240/1495)   
Compressing objects:  17% (255/1495)   
Compressing objects:  18% (270/1495)   
Compressing objects:  19% (285/1495)   
Compressing objects:  20% (299/1495)   
Compressing objects:  21% (314/1495)   
Compressing objects:  22% (329/1495)   
Compressing objects:  23% (344/1495)   
Compressing objects:  24% (359/1495)   
Compressing objects:  25% (374/1495)   
Compressing objects:  26% (389/1495)   
Compressing objects:  27% (404/1495)   
Compressing objects:  28% (419/1495)   
Compressing objects:  29% (434/1495)   
Compressing objects:  30% (449/1495)   
Compressing objects:  31% (464/1495)   
Compressing objects:  32% (479/1495)   
Compressing objects:  33% (494/1495)   
Compressing objects:  34% (509/1495)   
Compressing objects:  35% (524/1495)   
Compressing objects:  36% (539/1495)   
Compressing objects:  37% (554/1495)   
Compressing objects:  38% (569/1495)   
Compressing objects:  39% (584/1495)   
Compressing objects:  40% (598/1495)   
Compressing objects:  41% (613/1495)   
Compressing objects:  42% (628/1495)   
Compressing objects:  43% (643/1495)   
Compressing objects:  44% (658/1495)   
Compressing objects:  45% (673/1495)   
Compressing objects:  46% (688/1495)   
Compressing objects:  47% (703/1495)   
Compressing objects:  48% (718/1495)   
Compressing objects:  49% (733/1495)   
Compressing objects:  50% (748/1495)   
Compressing objects:  51% (763/1495)   
Compressing objects:  52% (778/1495)   
Compressing objects:  53% (793/1495)   
Compressing objects:  54% (808/1495)   
Compressing objects:  55% (823/1495)   
Compressing objects:  56% (838/1495)   
Compressing objects:  57% (853/1495)   
Compressing objects:  58% (868/1495)   
Compressing objects:  59% (883/1495)   
Compressing objects:  60% (897/1495)   
Compressing objects:  61% (912/1495)   
Compressing objects:  62% (927/1495)   
Compressing objects:  63% (942/1495)   
Compressing objects:  64% (957/1495)   
Compressing objects:  65% (972/1495)   
Compressing objects:  66% (987/1495)   
Compressing objects:  67% (1002/1495)   
Compressing objects:  68% (1017/1495)   
Compressing objects:  69% (1032/1495)   
Compressing objects:  70% (1047/1495)   
Compressing objects:  71% (1062/1495)   
Compressing objects:  72% (1077/1495)   
Compressing objects:  73% (1092/1495)   
Compressing objects:  74% (1107/1495)   
Compressing objects:  75% (1122/1495)   
Compressing objects:  76% (1137/1495)   
Compressing objects:  77% (1152/1495)   
Compressing objects:  78% (1167/1495)   
Compressing objects:  79% (1182/1495)   
Compressing objects:  80% (1196/1495)   
Compressing objects:  81% (1211/1495)   
Compressing objects:  82% (1226/1495)   
Compressing objects:  83% (1241/1495)   
Compressing objects:  84% (1256/1495)   
Compressing objects:  85% (1271/1495)   
Compressing objects:  86% (1286/1495)   
Compressing objects:  87% (1301/1495)   
Compressing objects:  88% (1316/1495)   
Compressing objects:  89% (1331/1495)   
Compressing objects:  90% (1346/1495)   
Compressing objects:  91% (1361/1495)   
Compressing objects:  92% (1376/1495)   
Compressing objects:  93% (1391/1495)   
Compressing objects:  94% (1406/1495)   
Compressing objects:  95% (1421/1495)   
Compressing objects:  96% (1436/1495)   
Compressing objects:  97% (1451/1495)   
Compressing objects:  98% (1466/1495)   
Compressing objects:  99% (1481/1495)   
Compressing objects: 100% (1495/1495)   
Compressing objects: 100% (1495/1495), done.
Total 2042 (delta 861), reused 1702 (delta 541)
** unknown exception encountered, details follow
** report bug details to http://mercurial.selenic.com/bts/
** or [email protected]
** Mercurial Distributed SCM (version 1.5.1)
** Extensions loaded: bookmarks, graphlog, hggit, rebase, color, highlight, transplant
Traceback (most recent call last):
  File "/opt/local/bin/hg", line 27, in 
    mercurial.dispatch.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 16, in run
    sys.exit(dispatch(sys.argv[1:]))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 30, in dispatch
    return _runcatch(u, args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 47, in _runcatch
    return _dispatch(ui, args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 467, in _dispatch
    return runcommand(lui, repo, cmd, fullargs, ui, options, d)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 337, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 518, in _runcommand
    return checkargs()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 472, in checkargs
    return cmdfunc()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py", line 466, in 
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
    return func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/extensions.py", line 116, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
    return func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/hgext/rebase.py", line 514, in pullrebase
    orig(ui, repo, *args, **opts)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
    return func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/commands.py", line 2406, in pull
    modheads = repo.pull(other, heads=revs, force=opts.get('force'))
  File "/Users/dave/src/hg-git/hggit/hgrepo.py", line 12, in pull
    git.fetch(remote.path, heads)
  File "/Users/dave/src/hg-git/hggit/git_handler.py", line 95, in fetch
    refs = self.fetch_pack(remote, heads)
  File "/Users/dave/src/hg-git/hggit/git_handler.py", line 646, in fetch_pack
    commit()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/dulwich-0.5.1-py2.6-macosx-10.6-x86_64.egg/dulwich/object_store.py", line 457, in commit
    self.move_in_pack(path)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/dulwich-0.5.1-py2.6-macosx-10.6-x86_64.egg/dulwich/object_store.py", line 422, in move_in_pack
    entries = p.sorted_entries()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/dulwich-0.5.1-py2.6-macosx-10.6-x86_64.egg/dulwich/pack.py", line 666, in sorted_entries
    ret = list(self.iterentries(resolve_ext_ref, progress=progress))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/dulwich-0.5.1-py2.6-macosx-10.6-x86_64.egg/dulwich/pack.py", line 655, in iterentries
    raise KeyError([sha_to_hex(h) for h in postponed.keys()])
KeyError: ['c144c2259bf875b2cb015bc3126f0e3decb34d7c', 'c086d101ff9a0e31f7136116f8b85d20fa2c4a01', '60102de957439851f4fe46a3ac09cfdcd6d31af3', '4a99dd1aebfa96cbdc39381649700cfab9c56e32', 'f55b3f7be0d8fa00c7a025572031a0c675b1f125', '7ebbf6bd598970bc32309e20484ce3330928ec76', '6f6d49dbba343fc2d5f58433483660eef162fab3', 'b04a8639ce0ae246c457a33d59674f59cb5fb6f1', 'a52f43d26a753894f597497d83c9fc48e52e9955', 'f274f8958ed1bad2f078c68230e03c9ef411a665', 'a72cee9339bd86a65c2efa75aa16a45f313d964d', '17a15620065e6b7d7b75064e499050078151b7ef', '5aef9a71778f510b734e7ef9d778e551145720b4', 'd9ab07598abd0fb7c17b24a7525130259ca6d6ea', '58dbfc88524569f18cd7f30f7f0a7174061f1db0', '97020d6551aee7c8d2a8ba0c4c0b32f24fa3214c', '263bb581e9b0b1f459c601eabf3435c6dfc286a2', '292020c7a432be70c18fa356fdfca0f16bd3e97d', '04c5aff13b6fb1b5522418ad4d736deefa3dab42', '8d8b832dfbee982beeb2125c097011724416f1ac', 'dabea0e5cd6b62ccb568fc3658371b37b7733df7', '1911f04c389db9436ef95066eee2d74d22bde3eb', '9fefa4339c009257ab1e45b7da84050278682afb', 'f52bb1ac2d96c53a941b45f3def97289561607c2', 'b04c913a8d6c172eab2a07a9df35d9639b161341', '6d0fd5e998846d7488e7aa919a83c972aeedabef', 'ff39e10db43297a78df7c774188d11e5a4375473', '9fe66aa52ad044109af9409fb0caacc7a8eb62de', 'f89a02220f9bdb239917aae2bdda7b1260f4535d', 'b3af6c2b64de3ff5223168273c6c635f5b28290d', '1cfe703417d9bc026341c9fa555d9fe0f1dcf087', '7f0fa7db8e32e9290f685d7be70e9887d9f78581', 'c3f2dba389c00fa63d509239330f3718a382ebed', 'd3607c8c469d7d24d3ea4ea28a6d903560bd0e0d', 'b41363058aafda0155fecd5b44247c3ea89f99f7', '6ca91a475b621db9aecdf3a49bccaf26529bbcd0', 'c08709dadd3901379166d2aca113d6207cc60868', '260bd4600edb5c6dc583820611d3c151c93012eb', '02a20b046a44ae2b66fc7b5e2a9bae9e5d420e8a', 'a4f115221e08a049b15d3408152220b8fa9999ad', '6fd89958c8291723301fcf7d6c992589fb01d20f', 'a60f57ce3de0bbd393cfdd8bee654b8bda279fcf', '5be54085cb7c611969c236cec1f432f8214ad2c8', 'bae317d79ad0f8939f587dd91b0f9d03e4d580f8', '2a31c4ad4886a210aa872390cfa6276c2d71f239', '9790e8aa1124a6b94997e3b13f9911109ea47f5a', '55ebf02d06e30617b64e7ce16ca29ce5aa477436', '2f6b13164745116171b80a9cbbf5f1bd1c310ccc', '864ff01556786b41956ccb150382496c14f5e97e']

Hi,

Using the master branch of Dulwich I got an error and I cannot figure out what is the
problem for now.
I have the dul daemon running with the FileSystemBackend. I clone a repo from github (that case openstack/swift) and I push the master branch on an alternate remote (localhost dulwich). Then I push an other branch (origin/stable/grizzly) from the clone to the local Dulwich and here it fails like as follow :
...
File "/usr/lib/python2.7/SocketServer.py", line 638, in init
self.handle()
File "/home/fabien/dulwich_upstream/dulwich/server.py", line 756, in handle
h.handle()
File "/home/fabien/dulwich_upstream/dulwich/server.py", line 726, in handle
status = self._apply_pack(client_refs)
File "/home/fabien/dulwich_upstream/dulwich/server.py", line 634, in _apply_pack
p = self.repo.object_store.add_thin_pack(self.proto.read, recv)
File "/home/fabien/dulwich_upstream/dulwich/object_store.py", line 607, in add_thin_pack
return self._complete_thin_pack(f, path, copier, indexer)
File "/home/fabien/dulwich_upstream/dulwich/object_store.py", line 539, in _complete_thin_pack
entries = list(indexer)
File "/home/fabien/dulwich_upstream/dulwich/pack.py", line 1250, in _walk_all_chains
for result in self._walk_ref_chains():
File "/home/fabien/dulwich_upstream/dulwich/pack.py", line 1272, in _walk_ref_chains
self._pending_ref.pop(base_sha)
KeyError: "U\x0e\xfb<=\x02\x86\xd1~dC1DN?'\x13Rn\xe3"

I've try the same procedure with another base repo (openstack/cinder) and it fails the same way.

Have you any idea which situation raises this problem ?

Not sure if this is by design or not, but you can't use MemoryObjectStore with http server.

dulwich.web.GunzipFilter is calls gzip.GzipFile with environ['wsgi.input'] as input file object. GzipFile expects it to have a .seek(...) method but it doesn't necessarily have one (not specified in PEP 3333)

I've seen this approach used before to automate a push to remote to update a site.

http://toroid.org/ams/git-website-howto

I'd like to use dulwich in some of our deploy fabric scripts to make this process a little saner and not rely on a repo embedded with the production code with fixed paths.

I know the checkout method that was patched in is intended only for freshly cloned repos to checkout the master branch. Is it possible to call checkout with a path that will move through the objects in a branch and output the entire structure to a target directory? If not, is this something I could volunteer some work on?

See patch: http://bitbucket.org/ashcrow/dulwich/changeset/afa4e2413529/

Sometimes files could not be closed on windows systems after a fetch. I found one possible place where this could happen but I think there are more places where file descriptors are not closed correctly.

In client.py line 204 I added this try finally block:

        try:
            result = self.fetch_pack(path, determine_wants,
                    target.get_graph_walker(), f.write, progress)
        finally:
            commit()

I will test this code in a production env and will report if this fixes this issue.

Check out the diffstat here.

A wrapper is provided to emulate test skipping. For people without nose, simply return. (I thought about raising an Exception subclass, but that would make the test fail.)

It would also ease efforts to upgrade to python 2.7, which provides `unittest.skip()``.

The open_repository method of DictBackend checks if given path is valid and raises an NotGitRepository exception when path is invalid, only problem is that a

   raise NotGitRepository("No git repository was found at %(path)s", path=path)

Code raises an TypeError because of path=path is a keyword expression later passed to Exception class and that is raising a TypeError

I'll try to fix it today and add a test and send pull request for this.

Hi,

I am using dulwich with hg-git on windows and have run into a problem committing:

(2) C:\s\p\Powershell2 -» hg push git://github.com/cbilson/Powershell2.git
pushing to git://github.com/cbilson/Powershell2.git
importing Hg objects into Git
at: 0/3
Path:
Path: Automating_Your_.NET_Application
Path: Automating_Your_.NET_Application\TipCalculator
** unknown exception encountered, details follow
** report bug details to http://mercurial.selenic.com/bts/
** or [email protected]
** Mercurial Distributed SCM (version 1.4.1)
** Extensions loaded: bookmarks, extdiff, fetch, graphlog, mq, purge, record, rebase, activity, hgshelve, histedit, svn, hggit
Traceback (most recent call last):
  File "C:\Python26\Scripts\hg", line 5, in 
    pkg_resources.run_script('mercurial==1.4.1', 'hg')
  File "C:\Python26\lib\site-packages\pkg_resources.py", line 489, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "C:\Python26\lib\site-packages\pkg_resources.py", line 1207, in run_script
    execfile(script_filename, namespace, namespace)
  File "c:\python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\EGG-INFO\scripts\hg", line 27, in 
    mercurial.dispatch.run()
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\dispatch.py", line 16, in run
    sys.exit(dispatch(sys.argv[1:]))
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\dispatch.py", line 30, in dispatch
    return _runcatch(u, args)
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\dispatch.py", line 46, in _runcatch
    return _dispatch(ui, args)
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\dispatch.py", line 449, in _dispatch
    return runcommand(lui, repo, cmd, fullargs, ui, options, d)
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\dispatch.py", line 319, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\dispatch.py", line 500, in _runcommand
    return checkargs()
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\dispatch.py", line 454, in checkargs
    return cmdfunc()
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\dispatch.py", line 448, in 
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\util.py", line 386, in check
    return func(*args, **kwargs)
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\extensions.py", line 116, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\util.py", line 386, in check
    return func(*args, **kwargs)
  File "C:\src\ThirdParty\hgsubversion\hgsubversion\wrappers.py", line 419, in generic
    return orig(ui, repo, *args, **opts)
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\util.py", line 386, in check
    return func(*args, **kwargs)
  File "C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg\mercurial\commands.py", line 2356, in push
    r = repo.push(other, opts.get('force'), revs=revs)
  File "c:\src\ThirdParty\hg-git\hggit\hgrepo.py", line 19, in push
    git.push(remote.path, revs, force)
  File "c:\src\ThirdParty\hg-git\hggit\git_handler.py", line 145, in push
    self.export_commits()
  File "c:\src\ThirdParty\hg-git\hggit\git_handler.py", line 113, in export_commits
    self.export_git_objects()
  File "c:\src\ThirdParty\hg-git\hggit\git_handler.py", line 189, in export_git_objects
    self.export_hg_commit(rev)
  File "c:\src\ThirdParty\hg-git\hggit\git_handler.py", line 238, in export_hg_commit
    tree_sha = commit_tree(self.git.object_store, self.iterblobs(ctx))
  File "C:\Python26\lib\site-packages\dulwich-0.4.1-py2.6-win32.egg\dulwich\index.py", line 301, in commit_tree
    return build_tree("")
  File "C:\Python26\lib\site-packages\dulwich-0.4.1-py2.6-win32.egg\dulwich\index.py", line 295, in build_tree
    sha = build_tree(os.path.join(path, basename))
  File "C:\Python26\lib\site-packages\dulwich-0.4.1-py2.6-win32.egg\dulwich\index.py", line 295, in build_tree
    sha = build_tree(os.path.join(path, basename))
  File "C:\Python26\lib\site-packages\dulwich-0.4.1-py2.6-win32.egg\dulwich\index.py", line 292, in build_tree
    for basename, entry in trees[path].iteritems():
KeyError: 'Automating_Your_.NET_Application\\TipCalculator'

I don't really know python, but was able to get the debugger to run hg-git and let me stop when the error occurred. I noticed that if I replaced the "" with "/" in the path in "trees[path]" it worked:

diff -r 03f98a25f119 dulwich/index.py
--- a/dulwich/index.py  Mon Dec 07 14:51:49 2009 +0100
+++ b/dulwich/index.py  Sun Jan 10 17:14:04 2010 -0800
@@ -288,7 +288,9 @@

     def build_tree(path):
         tree = Tree()
-        for basename, entry in trees[path].iteritems():
+
+        treePath = path.replace("\\", "/")
+        for basename, entry in trees[treePath].iteritems():
             if type(entry) == dict:
                 mode = stat.S_IFDIR
                 sha = build_tree(os.path.join(path, basename))

It worked, but I suspect there will be other problems like this elsewhere? I don't mind trying to find and fix them myself, but would just like to talk with someone about this first. Any thoughts? Thanks!

BaseObjectStore._collect_ancestors() iterates across Tags and Commits, but Tags do not have the "parents" attribute:

    while queue:
        e = queue.pop(0)
        if e in common:
            bases.add(e)
        elif e not in commits:
            commits.add(e)
            cmt = self[e]
            queue.extend(cmt.parents)

The following failure occurs:

thayne@lava:~/dev/omtr/git/dw-all-tharbaug.hg$ hg push -f -B issue-DMARE-224
pushing to git+ssh://[email protected]:tharbaug/dw-all.git
["git-receive-pack 'tharbaug/dw-all.git'"]
X11 forwarding request failed on channel 0
searching for changes
** Unknown exception encountered with possibly-broken third-party extension git
** which supports versions 2.3.1 of Mercurial.
** Please disable git and try your action again.
** If that fixes the bug please report it to https://bitbucket.org/durin42/hg-git/issues
** Python 2.7.5+ (default, Sep 17 2013, 15:31:50) [GCC 4.8.1]
** Mercurial Distributed SCM (version 2.7.2)
** Extensions loaded: convert, extdiff, fetch, git, gpg, graphlog, hgk, mq, purge, rebase, record, transplant
Traceback (most recent call last):
  File "/usr/bin/hg", line 38, in <module>
    mercurial.dispatch.run()
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 28, in run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 69, in dispatch
    ret = _runcatch(req)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 97, in _runcatch
    return _dispatch(req)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 778, in _dispatch
    cmdpats, cmdoptions)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 549, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 869, in _runcommand
    return checkargs()
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 840, in checkargs
    return cmdfunc()
  File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 775, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 507, in check
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mercurial/extensions.py", line 143, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 507, in check
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/hgext/mq.py", line 3533, in mqcommand
    return orig(ui, repo, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 507, in check
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mercurial/commands.py", line 4694, in push
    newbranch=opts.get('new_branch'))
  File "/usr/lib/python2.7/dist-packages/hgext/git/hgrepo.py", line 22, in push
    return git.push(remote.path, revs, force)
  File "/usr/lib/python2.7/dist-packages/hgext/git/git_handler.py", line 276, in push
    old_refs, new_refs = self.upload_pack(remote, revs, force)
  File "/usr/lib/python2.7/dist-packages/hgext/git/git_handler.py", line 917, in upload_pack
    new_refs = client.send_pack(path, changed, genpack)
  File "/usr/lib/python2.7/dist-packages/dulwich/client.py", line 501, in send_pack
    objects = generate_pack_contents(have, want)
  File "/usr/lib/python2.7/dist-packages/hgext/git/git_handler.py", line 900, in genpack
    for mo in self.git.object_store.find_missing_objects(have, want):
  File "/usr/lib/python2.7/dist-packages/dulwich/object_store.py", line 176, in find_missing_objects
    finder = MissingObjectFinder(self, haves, wants, progress, get_tagged)
  File "/usr/lib/python2.7/dist-packages/dulwich/object_store.py", line 989, in __init__
    all_ancestors = object_store._collect_ancestors(have_commits)[0]
  File "/usr/lib/python2.7/dist-packages/dulwich/object_store.py", line 248, in _collect_ancestors
    queue.extend(cmt.parents)
AttributeError: 'Tag' object has no attribute 'parents'

I have temporarily skipped objects other than commits in the iteration:

        while queue:
            e = queue.pop(0)
            if e in common:
                bases.add(e)
            elif e not in commits:
                commits.add(e)
                cmt = self[e]
                if 'commit' != cmt.type_name:
                    continue
                queue.extend(cmt.parents)

I am unsure if this is an appropriate fix.

When running dul-receive-pack on a repository I get this output:

005a23b87ffabd30448e05e1e28cafa660e4580fbb3a HEADreport-status delete-refs side-band-64k
004023b87ffabd30448e05e1e28cafa660e4580fbb3a refs/heads/branch2
003f5130a31543f90c68636a676e203ba2a975b713e7 refs/heads/master
0000

Whereas, git-receive-pack gives:
007923b87ffabd30448e05e1e28cafa660e4580fbb3a refs/heads/branch2 report-status delete-refs side-band-64k quiet ofs-delta
003f5130a31543f90c68636a676e203ba2a975b713e7 refs/heads/master
0000

I'm not sure if the extra line describing HEAD is an issue. However, the missing space between HEAD and report-status should cause the send-pack process to parse the capabilities incorrectly.

I have a repo served by Dulwich with FileSystemBackend (server.py). With my cgit client i try to remove a remote branch like as follow :
git push origin :mybranch5

But Dulwich hangs as it will wait for pack data that the git client will never send. I have confirmed that by sniffing packets. So I think if there is only one command sent by git client we must skip the add_thin_pack method https://github.com/jelmer/dulwich/blob/master/dulwich/server.py#L623

I've tried to fix that in the pull request I'll send after.

But I'm not sure about the status returned to the client as the protocol wants an unpack status in the status. [['ref', 'ok']] is not sufficient, so I add ['unpack', 'ok'] too.
https://github.com/git/git/blob/master/Documentation/technical/pack-protocol.txt#L504

Cheers,

Fbo

I've made some patches some time ago, they are in my fork branch pack. All but the last patch are ready for me.
Basically, I've fixed and enabled the commented out code of writing deltas, the result is that it's very slow, I've tried to speed things up by various means : reuse deltas in existing packs, reusing the SequenceMatcher, etc. The last patch contains a C extension for diffing (actually finding blocks that match), I'm not sure this is the best idea, it reduced the time for a writing a pack (push to an empty repository) from 2m20 to 1m30. I still find this excessive, so I tried other things: doing a line-based (as opposed to character-diff) diff is way faster, only 10 seconds, but the pack is somewhat larger (216k vs 161k). "word" based diff (splitting on spaces) is somewhere in the middle (30 seconds, 166k). I find that this, or some "adaptive" method (like splitting on the most common char, may be the best idea. (the line-based and word based are both using pure python)

Can anyone explain the background of this FIXME:

https://github.com/jelmer/dulwich/blob/master/dulwich/pack.py#L1510

Since I need thin pack, I would like to fix it. Before I do something stupid, can someone give me some background?

Check out the diff stat here.

The first patch creates the pack subdirectory, if necessary.

The second patch changes test_object_store to use tempfile.mkdtemp() instead of the directory "foo" in the pwd.

It'd be nice to have dulwich hosted on PyPI, like the majority of packages. Downloading dulwich usually accounts for 90% of my installation times (due to fast PyPI mirrors/caches), since samba.org delivers a solid 3 kb/s or so to my machine.

Is there a good reason dulwich is not on PyPI? Maybe one that can be alleviated?

I can't compile this on OS X 10.6. It seems to think I'm running 10.3, and then complains about not finding an SDK.

python setup.py build
running build
running build_py
creating build
creating build/lib.macosx-10.3-fat-2.6
creating build/lib.macosx-10.3-fat-2.6/dulwich
...
building 'dulwich._objects' extension
creating build/temp.macosx-10.3-fat-2.6
creating build/temp.macosx-10.3-fat-2.6/dulwich
Compiling with an SDK that doesn't seem to exist: /Developer/SDKs/MacOSX10.4u.sdk
Please check your Xcode installation

But Xcode is there, even installed in its default location etc. The same thing happens BTW when trying to install it via easy_install. I don't know Python and its setup tools well enough to diagnose this further, but I'd be happy to provide any more information you need to figure this out.

% hg clone git://anongit.freedesktop.org/git/cairo
destination directory: cairo
importing Hg objects into Git
Counting objects: 46840, done.
Compressing objects: 100% (15632/15632), done.
Total 46840 (delta 34929), reused 41900 (delta 31166)
importing Git objects into Hg
** unknown exception encountered, details follow
** report bug details to http://mercurial.selenic.com/bts/
** or [email protected]
** Mercurial Distributed SCM (version c564dbb2a6d5)
** Extensions loaded: bookmarks, color, convert, graphlog, highlight, hgk, mq, parentrevspec, patchbomb, purge, rebase, record, transplant, hgsubversion, histedit, hggit
Traceback (most recent call last):
  File "/opt/hg/bin/hg", line 27, in 
    mercurial.dispatch.run()
  File "/opt/hg/lib/python/mercurial/dispatch.py", line 16, in run
    sys.exit(dispatch(sys.argv[1:]))
  File "/opt/hg/lib/python/mercurial/dispatch.py", line 27, in dispatch
    return _runcatch(u, args)
  File "/opt/hg/lib/python/mercurial/dispatch.py", line 43, in _runcatch
    return _dispatch(ui, args)
  File "/opt/hg/lib/python/mercurial/dispatch.py", line 457, in _dispatch
    return runcommand(lui, repo, cmd, fullargs, ui, options, d)
  File "/opt/hg/lib/python/mercurial/dispatch.py", line 317, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/opt/hg/lib/python/mercurial/dispatch.py", line 509, in _runcommand
    return checkargs()
  File "/opt/hg/lib/python/mercurial/dispatch.py", line 462, in checkargs
    return cmdfunc()
  File "/opt/hg/lib/python/mercurial/dispatch.py", line 456, in 
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/opt/hg/lib/python/mercurial/util.py", line 400, in check
    return func(*args, **kwargs)
  File "/opt/hg/lib/python/mercurial/extensions.py", line 106, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/opt/hg/lib/python/mercurial/util.py", line 400, in check
    return func(*args, **kwargs)
  File "/opt/hg/lib/python/hgsubversion/wrappers.py", line 374, in clone
    update=not opts.get('noupdate'))
  File "/opt/hg/lib/python/mercurial/hg.py", line 292, in clone
    dest_repo.clone(src_repo, heads=revs, stream=stream)
  File "/opt/hg/lib/python/mercurial/localrepo.py", line 2151, in clone
    return self.pull(remote, heads)
  File "/Users/augie/hg-git/hggit/hgrepo.py", line 12, in pull
    git.fetch(remote.path, heads)
  File "/Users/augie/hg-git/hggit/git_handler.py", line 103, in fetch
    self.import_git_objects(remote_name, refs)
  File "/Users/augie/hg-git/hggit/git_handler.py", line 383, in import_git_objects
    (obj_type, obj_sha) = obj.get_object()
  File "/opt/hg/lib/python/dulwich-0.4.0-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 346, in get_object
    self._ensure_parsed()
  File "/opt/hg/lib/python/dulwich-0.4.0-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 140, in _ensure_parsed
    self._parse_text()
  File "/opt/hg/lib/python/dulwich-0.4.0-py2.6-macosx-10.6-universal.egg/dulwich/objects.py", line 331, in _parse_text
    sep = value.index("> ")
ValueError: substring not found

The SubprocessGitClient._connect method is passing args and kwargs to GitClient, where it should be passing self._args and self._kwargs.

Here's a patch:

--- a/dulwich/client.py
+++ b/dulwich/client.py
@@ -248,12 +248,14 @@
         self.proc = subprocess.Popen(argv, bufsize=0,
                                 stdin=subprocess.PIPE,
                                 stdout=subprocess.PIPE)
+        def can_read_fn():
+            return _fileno_can_read(self.proc.stdout.fileno())
         def read_fn(size):
             return self.proc.stdout.read(size)
         def write_fn(data):
             self.proc.stdin.write(data)
             self.proc.stdin.flush()
-        return GitClient(lambda: _fileno_can_read(self.proc.stdout.fileno()), read_fn, write_fn, *args, **kwargs)
+        return GitClient(can_read_fn, read_fn, write_fn, *self._args, **self._kwargs)
 
     def send_pack(self, path, changed_refs, generate_pack_contents):
         client = self._connect("git-receive-pack", path)

repo[some_unicode_commit_sha] throws a KeyError when it should raise a TypeError (because unicode is now allowed).

    def __getitem__(self, name):
        ### Skipped because isinstance check does not hold
        if len(name) in (20, 40) and isinstance(name, str):
            try:
                return self.object_store[name]
            except (KeyError, ValueError):
                pass
        ### Fails because sha is no ref
        try:
            return self.object_store[self.refs[name]]
        except RefFormatError:
            raise KeyError(name)

I used dulwich last week. I failed to find API for branch merge and rebase. Dulwich has not arrived this issuses?

some_repo[some_key_of_length_20]

results in a TypeError if the key is not a valid SHA string.

Dulwich currently checks for

len(x) in (20, 40)

This should be replaced by something like

len(x) in (20, 40) and x.islower() and x.isalpha()

https://github.com/jelmer/dulwich/blob/master/dulwich/porcelain.py#L206

After a test run it seems files under .git can be staged when "paths = None". Also, filtering of .gitignore doesn't seem to be happening either.