jonathonreinhart / syrep Goto Github PK
View Code? Open in Web Editor NEWhttp://0pointer.de/lennart/projects/syrep/
License: GNU General Public License v2.0
http://0pointer.de/lennart/projects/syrep/
License: GNU General Public License v2.0
syrep 0.9 Copyright 2003-2006 Lennart Poettering <mzflerc (at) 0pointer (dot) de> * [1]License * [2]News * [3]Overview * [4]Current Status * [5]Documentation * [6]Requirements * [7]Installation * [8]Acknowledgements * [9]Download License This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. News Fri May 19 2006: [10]Version 0.9 released; Changes include: ported to Win32/Cygwin (patches from Bohdan Futerko); compatibility with Berkeley DB 4.4 Sat Apr 22 2006: [11]Version 0.8 released; Changes include: when resuming a canceled merge operation, try harder not to loose any files; when merging and a file already exists, and the user chooses not to replace it, don't treat this as an error; when updating an existing snapshot locally which originates from a different host, update the origin field Mon Jul 1 2005: [12]Version 0.7 released; Changes include: fix trash cleanup; fix long standing file copying bug; don't save device info data in the MD cache any longer, use the new option --check-dev to reenable this feature; update to Berkeley DB 4.3 Wed Sep 22 2004: [13]Version 0.6 released; Changes include: fix an ugly bug which made snapshots where --forget was used unusable; add debian/ directory for easily making Debian packages Mon Jul 19 2004: [14]Version 0.5 released; Changes include: optionally show sizes of file on --diff, implement new command --forget, check for extended attribute user.syrep on --update on file systems that support it. Mon Mar 22 2004: [15]Version 0.4 released; Changes include: fix annonoying SIGBUS failure when working on files >= 100 MB, update to Berkeley DB 4.2, use madvise() to improve file copying throughput on newer kernels, minor other fixes Sun Nov 30 2003: [16]Version 0.3 released; Changes include: new options --sort, --check-md, --always-copy; implemented direct bi-directory merges, documentation updates, build system updates, assorted fixes. Tue Sep 9 2003: [17]Version 0.2 released; Fixes include: documentation update, --diff output improved, --merge output fixed. Mon Sep 8 2003: [18]Version 0.1 released. Overview syrep is a generic file repository synchronization tool. It may be used to synchronize large file hierarchies bidirectionally by exchanging patch files. Syrep is truely peer-to-peer, no central servers are involved. Synchronizations between more than two repositories are supported. The patch files may be transferred via offline media, e.g. removable hard disks or compact discs. Files are tracked by their message digests, currently MD5. The following file operations are tracked in the snapshot files: creation, deletion, modification, creation of new hard or symbolic links, renaming. (The latter is nothing more than a new hard link and removal of the old file). syrep doesn't distuinguish between soft and hard links. In fact even copies of files are treated as the same. Currently, syrep doesn't synchronize file attributes like access modes or modification times. syrep was written to facilitate the synchronization of two large digital music repositories without direct network connection. Patch files of several gigabytes are common in this situation. syrep is able to cope with 64 bit file sizes. (LFS) syrep is optimized for speed. It may make use of a message digest cache to accelerate the calculation of digests of a whole directory hierarchy. How does syrep compare with rsync, cvs, Subversion, arch/tla, BitKeeper, xdelta2, diff/patch? syrep is kind of a bidirectional rsync, but stores and makes use of a file hierarchy history. Synchronization with syrep is based on patch files, and doesn't require a direct connection between the synchronizing peers. syrep has many things in common with version control systems like CVS or SVN: it stores a history and has operations similar to update and commit. However: the history doesn't contain file contents and is not line based, it stores the MD5 digest and some meta data only. There is no central server, instead all peers have the same role. There is no distinction between repositories and checkouts. In fact checkout and repository are identical. syrep has even more things in common with arch/tla and BitKeeper. All three are patch based and are "peer-to-peer". However, there are certain differences: syrep doesn't differentiate between repositories and checkouts. syrep doesn't keep a file contents history of any kind. syrep resembles diff/patch or xdelta2 in some way. While the latter work on file contents, syrep works on file hierarchies. In contrast two most of the software mentioned above, syrep is capable of synchronizing repositories of several 100GB of size, with only a very small overhead. (i.e. 4 MB of control data for half a year history for 100 GB of user data) Current Status Version 0.9 is more or less stable and fulfills its purpose. Documentation Have a look on the man page [19]syrep(1). (A XSLT capable browser is required) Method of operation Syrep's operation relies on "snapshots" of a file repository. A snapshot contains information about all files existent in the hierarchy combined with a limited history log of file operations. Snapshots may be compared, files missing or deleted on one of both sides may be detected this way. Based on this knowledge patch files containing all missing files may be created and merged. To keep the file operation log in a sensible state it is crucial to update the snapshot frequently, probably by adding a new cron job. Example usage Fred and Karl want to synchronize their digital music libraries by exchanging an USB hard disk with patch files. As first step, both initialize their repositories for usage with syrep: fred$ syrep -zp --update ~/mp3/ ... karl$ syrep -zp --update ~/mp3/ Depending on the size of the repositories this takes a lot of time, since a message digest is calculated for every file. Since exact tracking of all file operations on the repository is crucial for effetive synchronization, they both use the time passing to add a new entry to their crontab: 1 3 * * * syrep -z --update ~/mp3/ When the snapshot creation finished, they send the newly created patch files ~/mp3/.syrep/curent.syrep to each other. As these snapshots are only about 400K of size for a 80GB repository they do that via email: fred$ mutt -a ~/mp3/.syrep/current.syrep -s "The current snapshot of fred" karl ... karl$ mutt -a ~/mp3/.syrep/current.syrep -s "The current snapshot of karl" fred When the mails arrive they both detach the snapshot and create a patch on their USB harddisk containing all local files not existing on the other siede: fred$ mount /mnt/usb fred$ syrep -p -o /mnt/usb/patch-for-karl --makepatch ~/mp3/ ~/karls-current.sy rep fred$ umount /mnt/usb ... karl$ mount /mnt/usb karl$ syrep -p -o /mnt/usb/patch-for-fred --makepatch ~/mp3/ ~/freds-current.sy rep karl$ umount /mnt/usb As next step they exchange their harddisks. Back at home they merge the newly acquired patch into their own repository: fred$ mount /mnt/usb fred$ syrep -pT --merge /mnt/usb/patch-for-fred ~/mp3/ fred$ umount /mnt/usb ... karl$ mount /mnt/usb karl$ syrep -pT --merge /mnt/usb/patch-for-karl ~/mp3/ karl$ umount /mnt/usb At this moment both have the same file hierarchy. To update the local snapshot log with the newly merged files they both should run --update now. This update run should be much quicker since the message digests of all unchanged files are read from a message cache created and update each time --update runs: fred$ syrep -zp --update ~/mp3/ ... karl$ syrep -zp --update ~/mp3/ Some time later Fred got plenty of new music files, while Karl didn't change anything on his repository. Thus, Fred is able to use the old snapshot he recieved from Karl to generate a new patch for him. He does it exactly the same way he did the last time, see above. And now, several iterations of the story described above follow. That's the end of the story. OK, not quite. Sometimes a conflict happens, e.g. at the same time both created a file foo.mp3 with different contents. When this happens the local copy is always copied into the patch and the user may decide during merge which file version he wants to have locally. Because of that merging is an interactive task and cannot be automated completely. There is no need that the synchronization operations happen in such a "symmetric" way as described above. Requirements syrep requires installed development versions of [20]zlib and [21]Berkeley DB 4.3. If you want build syrep with support for extended attributes (currently supported on Linux only) you have to install libattr and a kernel that supports it. syrep was developed and tested on Debian GNU/Linux "testing" from September 2003, it should work on most other Linux distributions and may be POSIX implementations since it uses GNU autoconf for source code configuration. Some support for for big endian architectures is included, however, it is incomplete. You're welcome to send me patches. If the syrep build system detects Oliver Kurth's [22]xmltoman the man page is rebuilt. Otherwise the pre-compiled versions shipped with syrep are used. Installation As this package is made with the GNU autotools you should run ./configure inside the distribution directory for configuring the source tree. After that you should run make for compilation and make install (as root) for installation of syrep. Acknowledgements This software includes an implementation of the MD5 algorithm by L. Peter Deutsch. Thanks to him for this. Bohdan Futerko for porting syrep to Win32/Cygwin. Download The newest release is always available from [23]http://0pointer.de/lennart/projects/syrep/ The current release is [24]0.9. [25]Win32/x86 binaries courtesy of Bohdan Futerko Get syrep's development sources from the [26]Subversion [27]repository ([28]viewcvs): svn checkout svn://svn.0pointer.net/syrep/trunk syrep You may find an up-to-date Debian package (i386) of syrep in my [29]local Debian package repository. If you want to be notified whenever I release a new version of this software use the subscription feature of [30]Freshmeat. _________________________________________________________________ Lennart Poettering <mzflerc (at) 0pointer (dot) de>, May 2006 $Id: README.html.in 113 2006-05-19 10:09:00Z lennart $ References 1. README#license 2. README#news 3. README#overview 4. README#status 5. README#documentation 6. README#requirements 7. README#installation 8. README#acks 9. README#download 10. http://0pointer.de/lennart/projects/syrep/syrep-0.9.tar.gz 11. http://0pointer.de/lennart/projects/syrep/syrep-0.8.tar.gz 12. http://0pointer.de/lennart/projects/syrep/syrep-0.7.tar.gz 13. http://0pointer.de/lennart/projects/syrep/syrep-0.6.tar.gz 14. http://0pointer.de/lennart/projects/syrep/syrep-0.5.tar.gz 15. http://0pointer.de/lennart/projects/syrep/syrep-0.4.tar.gz 16. http://0pointer.de/lennart/projects/syrep/syrep-0.3.tar.gz 17. http://0pointer.de/lennart/projects/syrep/syrep-0.2.tar.gz 18. http://0pointer.de/lennart/projects/syrep/syrep-0.1.tar.gz 19. http://0pointer.de/lennart/projects/syrep/syrep.1.xml 20. http://www.gzip.org/zlib/ 21. http://www.sleepycat.com/ 22. http://masqmail.cx/xml2man/ 23. http://0pointer.de/lennart/projects/syrep/ 24. http://0pointer.de/lennart/projects/syrep/syrep-0.9.tar.gz 25. http://0pointer.de/lennart/projects/syrep/syrep-0.9.zip 26. http://subversion.tigris.org/ 27. svn://svn.0pointer.net/syrep 28. http://0pointer.net/cgi-bin/viewcvs.cgi/?root=syrep 29. http://0pointer.de/debian/ 30. http://freshmeat.net/projects/syrep/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.