smartfile / python-librsync Goto Github PK
View Code? Open in Web Editor NEWCtypes wrapper for librsync.
License: MIT License
Ctypes wrapper for librsync.
License: MIT License
Trying to work with files bigger than 4MB results in misleading errors.
Changing the size in the tests.py to (1024**2 * 5) will result in:
File "python-librsync/librsync/__init__.py", line 126, in _execute
block = f.read(RS_JOB_BLOCKSIZE)
File "/usr/lib64/python2.7/tempfile.py", line 581, in read
return self._file.read(*args)
IOError: File not open for reading
in the function patch, the read_cb callback reads the same file position repeated.
this leads to poor performance on big files
how to repeat:
to be able to see the problem, add a line to the read_cb function
see:
https://github.com/smartfile/python-librsync/blob/master/librsync/__init__.py#L221
add the follwing line before the f.seek(pos) line
print "pos:",pos," length:",length
the code should now look:
...
def read_cb(opaque, pos, length, buff):
print "pos:",pos," length:",length
f.seek(pos)
block = f.read(length)
....
now store this testcase as file and execute it:
import librsync, os, shutil
#create the datfiles only once
file_1='1mb_a.dat'
file_2='1mb_b.dat'
file_new='1mb_c.dat'
if (not os.path.exists(file_1)):
rnd = open('/dev/random','rb')
dst = open(file_1, 'wb')
dst.seek(0)
dst.write(rnd.read(1000000));
dst.close()
rnd.close()
if (not os.path.exists(file_2)):
src = open(file_1,'rb')
cnt = src.read()
src.close()
#make a change in the content of file_1
cnt_a=bytearray(cnt)
if (cnt_a[10] != 'A'):
cnt_a[10] = 'A'
else:
cnt_a[10] = 'B'
#and store the changed content as file_2
dst = open(file_2, 'wb')
dst.seek(0)
dst.write(str(cnt_a))
dst.close()
#now create the signature, delta
dst = open(file_1, 'rb')
src = open(file_2, 'rb')
synced = open(file_new, 'wb')
signature = librsync.signature(dst)
delta = librsync.delta(src, signature)
# Step 3: synchronize the files.
librsync.patch(dst, delta, synced)
the output is:
python redundant-read.py
pos: 0 length: 2048
pos: 0 length: 65536
pos: 0 length: 131072
pos: 0 length: 196608
pos: 0 length: 262144
pos: 0 length: 327680
pos: 0 length: 393216
pos: 0 length: 458752
pos: 0 length: 524288
pos: 0 length: 589824
pos: 0 length: 655360
pos: 0 length: 720896
pos: 0 length: 786432
pos: 0 length: 851968
pos: 0 length: 917504
pos: 0 length: 983040
here, the read_cb callback reads the file from position 0 again and again
python: ERROR: (rs_sig_begin) invalid sig_magic 0x1b5ea5a0
Segmentation fault (core dumped)
works with librsync 0.9.7
Traceback (most recent call last):
File "C:\Users\Devin Lucaschu\AppData\Local\Programs\Python\Python311\Lib\site-packages\librsync_init_.py", line 20, in
librsync = ctypes.cdll.librsync
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Devin Lucaschu\AppData\Local\Programs\Python\Python311\Lib\ctypes_init.py", line 446, in getattr
dll = self.dlltype(name)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Devin Lucaschu\AppData\Local\Programs\Python\Python311\Lib\ctypes_init.py", line 376, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'librsync' (or one of its dependencies). Try using the full path with constructor syntax.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "a:\User\Devin Lucaschu\Download\tex.py", line 2, in
import librsync
File "C:\Users\Devin Lucaschu\AppData\Local\Programs\Python\Python311\Lib\site-packages\librsync_init_.py", line 22, in
raise ImportError('Could not load librsync, make sure it is installed')
ImportError: Could not load librsync, make sure it is installed
Is there any chance for adding librsync 2.0.0 compatibility?
Or is this project dead?
If two files are the same content, applying the following procedure will fail ... diff mytest.1.txt and mytest3.txt
will differ. Is this the intended behavior, or am I missing something.
#!/usr/bin/python
import librsync
with open("mytest1.txt", "wb") as f:
f.write("hello world\n")
with open("mytest2.txt", "wb") as f:
f.write("hello world\n")
with file("mytest1.txt", "rb") as src, file("mytest2.txt", "rb") as dest, \
file("mytest3.txt", "wb") as patched:
signature = librsync.signature(dest)
delta = librsync.delta(src, signature)
librsync.patch(dest, delta, patched)
Can I recursively rsync a directory using python-librsync? Will it accept a list of files names/directories to rsync (i.e. using rsync's --include-from=FILE or --files-from=FILE options).
Thanks! :)
Just pip installed it and quickly found out one serious bug
Python 3.4.3 (default, Sep 14 2015, 17:11:46)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> for i in range(50, 70):
... s = b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'*i
... d = b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'*i + b'2'
... src = io.BytesIO(s)
... dst = io.BytesIO(d)
... signature = librsync.signature(dst)
... delta = librsync.delta(src, signature)
... synced = librsync.patch(dst, delta)
... _= src.seek(0)
... src.read() == synced.read()
...
True
True
True
True
True
True
True
True
True
False
False
False
False
False
False
False
False
False
False
False
tried with current master and all seems ok, so maybe it will be worth it to publish a new release on pypi :)
I have two files:
OldFile.txt with content:
Text.
NewFile.txt with content:
New text.
Text.
Then I run the example provided in readme:
import librsync
dst = file('OldFile.txt', 'rb')
src = file('NewFile.txt', 'rb')
synced = file('Synced.txt', 'wb')
signature = librsync.signature(dst)
delta = librsync.delta(src, signature)
librsync.patch(dst, delta, synced)
And get the following error:
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
TypeError: read_cb() takes exactly 3 arguments (4 given)
python: ERROR: (rs_job_complete) patch job failed: unexplained problem
Traceback (most recent call last):
File "RsyncTest.py", line 12, in <module>
librsync.patch(dst, delta, synced)
File "/usr/local/lib/python2.7/dist-packages/librsync/__init__.py", line 113, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/librsync/__init__.py", line 228, in patch
_execute(job, d, o)
File "/usr/local/lib/python2.7/dist-packages/librsync/__init__.py", line 141, in _execute
raise LibrsyncError(r)
librsync.LibrsyncError: unexplained problem
I tried syncing many different files and all went well, but this one case just won't work.
I received the following error when attempting to use sync:
Traceback (most recent call last):
File "test.py", line 11, in
main()
File "test.py", line 8, in main
sync.download('/home/travis/smartfile-sync/test.txt', '/test.txt')
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/smartfile/sync.py", line 117, in download
self.sync(RemoteFile(remote, self.api), LocalFile(local))
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/smartfile/sync.py", line 103, in sync
return dst.patch(src.delta(dst.signature(block_size=self.block_size)))
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/smartfile/sync.py", line 47, in patch
r = librsync.patch(reference, delta, output)
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/librsync/init.py", line 117, in wrapper
return f(_args, *_kwargs)
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/librsync/init.py", line 229, in patch
_execute(job, d, o)
File "/home/travis/virtualenv/local/lib/python2.7/site-packages/librsync/init.py", line 130, in _execute
block = f.read(RS_JOB_BLOCKSIZE)
AttributeError: 'unicode' object has no attribute 'read'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.