smartfile / python-librsync Goto Github PK

in the function patch, the read_cb callback reads the same file position repeated.
this leads to poor performance on big files

how to repeat:
to be able to see the problem, add a line to the read_cb function
see:
https://github.com/smartfile/python-librsync/blob/master/librsync/__init__.py#L221

add the follwing line before the f.seek(pos) line
print "pos:",pos," length:",length

the code should now look:

...
def read_cb(opaque, pos, length, buff):
        print "pos:",pos," length:",length
        f.seek(pos)
        block = f.read(length)
....

now store this testcase as file and execute it:

import librsync, os, shutil
#create the datfiles only once
file_1='1mb_a.dat'
file_2='1mb_b.dat'
file_new='1mb_c.dat'

if (not os.path.exists(file_1)):
    rnd = open('/dev/random','rb')
    dst = open(file_1, 'wb')
    dst.seek(0)
    dst.write(rnd.read(1000000));
    dst.close()
    rnd.close()

if (not os.path.exists(file_2)):
    src = open(file_1,'rb')
    cnt = src.read()
    src.close()

    #make a change in the content of file_1
    cnt_a=bytearray(cnt)
    if (cnt_a[10] != 'A'):
        cnt_a[10] = 'A'
    else:
        cnt_a[10] = 'B'

    #and store the changed content as file_2
    dst = open(file_2, 'wb')
    dst.seek(0)
    dst.write(str(cnt_a))
    dst.close()

#now create the signature, delta
dst = open(file_1, 'rb')
src = open(file_2, 'rb')

synced = open(file_new, 'wb')
signature = librsync.signature(dst)
delta = librsync.delta(src, signature)

# Step 3: synchronize the files.
librsync.patch(dst, delta, synced)

the output is:
python redundant-read.py
pos: 0 length: 2048
pos: 0 length: 65536
pos: 0 length: 131072
pos: 0 length: 196608
pos: 0 length: 262144
pos: 0 length: 327680
pos: 0 length: 393216
pos: 0 length: 458752
pos: 0 length: 524288
pos: 0 length: 589824
pos: 0 length: 655360
pos: 0 length: 720896
pos: 0 length: 786432
pos: 0 length: 851968
pos: 0 length: 917504
pos: 0 length: 983040

here, the read_cb callback reads the file from position 0 again and again

Just pip installed it and quickly found out one serious bug

Python 3.4.3 (default, Sep 14 2015, 17:11:46) 
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> for i in range(50, 70):
...     s = b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'*i
...     d = b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'*i + b'2'
...     src = io.BytesIO(s)
...     dst = io.BytesIO(d)
...     signature = librsync.signature(dst)
...     delta = librsync.delta(src, signature)
...     synced = librsync.patch(dst, delta)
...     _= src.seek(0)
...     src.read() == synced.read()
... 
True
True
True
True
True
True
True
True
True
False
False
False
False
False
False
False
False
False
False
False

tried with current master and all seems ok, so maybe it will be worth it to publish a new release on pypi :)

Trying to work with files bigger than 4MB results in misleading errors.

Changing the size in the tests.py to (1024**2 * 5) will result in:

File "python-librsync/librsync/__init__.py", line 126, in _execute
  block = f.read(RS_JOB_BLOCKSIZE)
File "/usr/lib64/python2.7/tempfile.py", line 581, in read
  return self._file.read(*args)

IOError: File not open for reading

Is there any chance for adding librsync 2.0.0 compatibility?
Or is this project dead?

I received the following error when attempting to use sync:

Traceback (most recent call last):
File "test.py", line 11, in
main()
File "test.py", line 8, in main
sync.download('/home/travis/smartfile-sync/test.txt', '/test.txt')
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/smartfile/sync.py", line 117, in download
self.sync(RemoteFile(remote, self.api), LocalFile(local))
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/smartfile/sync.py", line 103, in sync
return dst.patch(src.delta(dst.signature(block_size=self.block_size)))
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/smartfile/sync.py", line 47, in patch
r = librsync.patch(reference, delta, output)
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/librsync/init.py", line 117, in wrapper
return f(_args, *_kwargs)
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/librsync/init.py", line 229, in patch
_execute(job, d, o)
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/librsync/init.py", line 130, in _execute
block = f.read(RS_JOB_BLOCKSIZE)
AttributeError: 'unicode' object has no attribute 'read'

If two files are the same content, applying the following procedure will fail ... diff mytest.1.txt and mytest3.txt will differ. Is this the intended behavior, or am I missing something.

#!/usr/bin/python

import librsync

with open("mytest1.txt", "wb") as f:
    f.write("hello world\n")

with open("mytest2.txt", "wb") as f:
    f.write("hello world\n")

with file("mytest1.txt", "rb") as src, file("mytest2.txt", "rb") as dest, \
    file("mytest3.txt", "wb") as patched:

    signature = librsync.signature(dest)
    delta = librsync.delta(src, signature)
    librsync.patch(dest, delta, patched)

I have two files:

OldFile.txt with content:

Text.

NewFile.txt with content:

New text.
Text.

Then I run the example provided in readme:

import librsync

dst = file('OldFile.txt', 'rb')
src = file('NewFile.txt', 'rb')
synced = file('Synced.txt', 'wb')

signature = librsync.signature(dst)
delta = librsync.delta(src, signature)
librsync.patch(dst, delta, synced)

And get the following error:

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 314, in 'calling callback function'
TypeError: read_cb() takes exactly 3 arguments (4 given)
python: ERROR: (rs_job_complete) patch job failed: unexplained problem
Traceback (most recent call last):
  File "RsyncTest.py", line 12, in <module>
    librsync.patch(dst, delta, synced)
  File "/usr/local/lib/python2.7/dist-packages/librsync/__init__.py", line 113, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/librsync/__init__.py", line 228, in patch
    _execute(job, d, o)
  File "/usr/local/lib/python2.7/dist-packages/librsync/__init__.py", line 141, in _execute
    raise LibrsyncError(r)
librsync.LibrsyncError: unexplained problem

I tried syncing many different files and all went well, but this one case just won't work.

Can I recursively rsync a directory using python-librsync? Will it accept a list of files names/directories to rsync (i.e. using rsync's --include-from=FILE or --files-from=FILE options).

Thanks! :)

python: ERROR: (rs_sig_begin) invalid sig_magic 0x1b5ea5a0
Segmentation fault (core dumped)

works with librsync 0.9.7

Traceback (most recent call last):
File "C:\Users\Devin Lucaschu\AppData\Local\Programs\Python\Python311\Lib\site-packages\librsync_init_.py", line 20, in
librsync = ctypes.cdll.librsync
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Devin Lucaschu\AppData\Local\Programs\Python\Python311\Lib\ctypes_init.py", line 446, in getattr
dll = self.dlltype(name)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Devin Lucaschu\AppData\Local\Programs\Python\Python311\Lib\ctypes_init.py", line 376, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'librsync' (or one of its dependencies). Try using the full path with constructor syntax.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "a:\User\Devin Lucaschu\Download\tex.py", line 2, in
import librsync
File "C:\Users\Devin Lucaschu\AppData\Local\Programs\Python\Python311\Lib\site-packages\librsync_init_.py", line 22, in
raise ImportError('Could not load librsync, make sure it is installed')
ImportError: Could not load librsync, make sure it is installed