rsync

rsync

Original author(s)	Andrew Tridgell, Paul Mackerras
Developer(s)	Wayne Davison
Initial release	June 19, 1996 (1996-06-19)[1]
Stable release	3.0.9 [edit]  (September 23, 2011; 17 months ago (2011-09-23))[2] [±]
Development status	active
Written in	C
Platform	Unix-like, Windows
Type	Data transfer, Differential backup
License	GNU GPLv3
Website	rsync.samba.org

rsync is a utility software and network protocol for Unix-like systems (with ports to Windows) that synchronizes files and directories from one location to another while minimizing data transfer by using delta encoding when appropriate. Quoting the official website: "rsync is a file transfer program for Unix systems. rsync uses the 'rsync algorithm' which provides a very fast method for bringing remote files into sync."^[3] An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction.^[why?][4] rsync can copy or display directory contents and copy files, optionally using compression and recursion.

In daemon mode, rsync listens on the default TCP port of 873, serving files in the native rsync protocol or via a remote shell such as RSH or SSH.^{[citation needed][5][not in citation given]} In the latter case, the rsync client executable must be installed on the remote machine as well as on the local machine.

Released under the GNU General Public License version 3, rsync is free software. It is widely used.^[6][7][8][9]

History

Andrew Tridgell and Paul Mackerras wrote the original rsync. Tridgell discusses the design, implementation and performance of rsync in chapters 3 through 5 of his Australian National University Ph.D. thesis.^[10]

rsync was first announced on 19 June 1996.^[1] Rsync 3.0 was released on 1 March 2008.^[11]

Uses

rsync was originally written as a replacement for rcp and scp. As such, it has a similar syntax to its parent programs.^[12] Like its predecessors, it still requires a source and a destination to be specified, either of which may be remote. Because of the flexibility, speed and scriptability of rsync, it has become a standard Linux utility and is included in all popular Linux distributions. It has also been ported to Windows (via Cygwin, Grsync or SFU^[13]) and Mac OS.

Possible uses:

rsync [OPTION] … SRC [SRC] … [USER@]HOST:DESTrsync [OPTION] … [USER@]HOST:SRC [DEST]

^{[clarification needed]}

One of the earliest applications of rsync was to implement mirroring or backup for multiple Unix clients to a central Unix server using rsync/ssh and standard Unix accounts.^{[citation needed]}

With a scheduling utility such as cron, one can schedule automated encrypted rsync-based mirroring between multiple hosts and a central server.

Examples

A command line to mirror FreeBSD might look like:

% rsync -avz --delete ftp4.de.FreeBSD.org::FreeBSD/ /pub/FreeBSD/[14]

The Apache HTTP Server supports only rsync for updating mirrors.

rsync -avz --delete --safe-links rsync.apache.org::apache-dist /path/to/mirror[15]

The preferred (and simplest) way to mirror the PuTTY website to the current directory is to use rsync.

rsync -auH rsync://rsync.chiark.greenend.org.uk/ ftp/users/sgtatham/putty-website-mirr or/ .[16]

A way to mimic the capabilities of Time Machine (Mac OS) - see also tym.

#date=`date "+%Y-%m-%dT%H:%M:%S"`date=`date "+%FT%T"`rsync -aP --link-dest=$HOME/Backups/current /path/to/important_files $HOME/Backups/back-$dateln -nfs $HOME/Backups/back-$date $HOME/Backups/current[17]

Algorithm

Determining which files to send

By default rsync uses a quick method of determining which files differ between the sending and receiving systems, by checking the modification time and size of each file. This method uses very little CPU time, but it can occasionally miss files that have changed.

rsync can be forced to use a more comprehensive check by adding the --checksum flag, which causes rsync to do a full checksum on every file. This ensures that rsync does not miss any changed files, but it is much slower and uses a lot of CPU time.

Determining which parts of a file have changed

The rsync utility uses an algorithm invented by the Australian computer programmer Andrew Tridgell for efficiently transmitting a structure (such as a file) across a communications link when the receiving computer already has a similar, but not identical, version of the same structure.

The recipient splits its copy of the file into fixed-size non-overlapping chunks and computes two checksums for each chunk: the MD5 hash, and a weaker 'rolling checksum'. (Prior to version 30 of the protocol, released with rsync version 3.0.0, it used MD4 hashes rather than MD5.^[18]) It sends these checksums to the sender.

The sender computes the rolling checksum for every chunk of size in its own version of the file, even overlapping chunks. This can be calculated efficiently because of a special property of the rolling checksum: if the rolling checksum of bytes through is , the rolling checksum of bytes through can be computed from , byte , and byte without having to examine the intervening bytes. Thus, if one had already calculated the rolling checksum of bytes 1–25, one could calculate the rolling checksum of bytes 2–26 solely from the previous checksum (R), byte 1 (n), and byte 26 (n+S).

The rolling checksum used in rsync is based on Mark Adler's adler-32 checksum, which is used in zlib, and is itself based on Fletcher's checksum.

The sender then compares its rolling checksums with the set sent by the recipient to determine if any matches exist. If they do, it verifies the match by computing the hash for the matching block and by comparing it with the hash for that block sent by the recipient.

The sender then sends the recipient those parts of its file that did not match the recipient's blocks, along with information on where to merge these blocks into the recipient's version. This makes the copies identical. However, there is a small probability that differences between chunks in the sender and recipient are not detected, and thus remains uncorrected. This requires a simultaneous hash collision in MD5 and the rolling checksum. It is possible to generate MD5 collisions, and the rolling checksum is not cryptographically strong, but the chance for this to occur by accident is nevertheless extremely remote. With 128 bits from MD5 plus 32 bits from the rolling checksum, and assuming maximum entropy in these bits, the probability of a hash collision with this combined checksum is 2^−(128+32) = 2⁻¹⁶⁰. The actual probability is a few times higher, since good checksums approach maximum output entropy but very rarely achieve it.

If the sender's and recipient's versions of the file have many sections in common, the utility needs to transfer relatively little data to synchronize the files. Note that if usual data compression algorithms are used, files that are similar when uncompressed may be very different when compressed, and thus the entire file will need to be transferred – local changes in uncompressed files yield global changes in compressed files. This is particularly an issue with mirroring of archive files, such as disk images and compressed tarballs, where often individual files change. Some compression programs, such as gzip, provide a special "rsyncable" mode which allows these files to be efficiently rsynced, by ensuring that local changes in the uncompressed file yields only local changes in the compressed file.

While the rsync algorithm forms the heart of the rsync application that essentially optimizes transfers between two computers over TCP/IP, the rsync application supports other key features that aid significantly in data transfers or backup. They include compression and decompression of data block by block using zlib at sending and receiving ends, and support for protocols such as ssh that enables encrypted transmission of compressed and efficient differential data using rsync algorithm. Instead of ssh, stunnel can also be used to create an encrypted tunnel to secure the data transmitted.

Rsync is capable of limiting the bandwidth consumed during a transfer.

Variations

A utility called rdiff uses the rsync algorithm to generate delta files with the difference from file A to file B (like the utility diff, but in a different delta format). The delta file can then be applied to file A, turning it into file B (similar to the patch utility).

Unlike diff, the process of creating a delta file has two steps: first a signature file is created from file A, and then this (relatively small) signature and file B are used to create the delta file. Also unlike diff, rdiff works well with binary files.

• Using rdiff, a utility called rdiff-backup has been created, capable of maintaining a backup mirror of a file or directory either locally or remotely over the network, on another server. rdiff-backup stores incremental rdiff deltas with the backup, with which it is possible to recreate any backup point.^[19]

• Duplicity is a variation on rdiff-backup that allows for backups without cooperation from the storage server, as with simple storage services like Amazon S3. It works by generating the hashes for each block in advance, encrypting them, and storing them on the server, then retrieving them when doing an incremental backup. The rest of the data is also stored encrypted for security purposes.

• rsyncrypto is a utility to encrypt files in an rsync-friendly fashion. The rsyncrypto algorithm ensures that two almost identical files, when encrypted with rsyncrypto and the same key, will produce almost identical encrypted files. This allows for the low-overhead data transfer achieved by rsync while providing encryption for secure transfer and storage of sensitive data in a remote location.^[20]

An alternative to manually scripting rsync is the Free Software (FLOSS) GUI program BackupPC, which performs automatic scheduled backups to rsync servers.

As of Mac OS X 10.5 and later, there is a special -E or --extended-attributes switch which allows retaining much of the HFS file metadata when syncing between two machines supporting this feature. This is achieved by transmitting the proprietary Resource Fork along with the Data Fork.^[21]

Practical applications

Rsync is a very efficient replacement of copy or move operation. For example, Say you have an external hard drive that serves as a backup of your music and you keep updating your local copy by reorganizing folders and adding files. Manually updating your backup to reflect the same is virtually impossible without using atleast 4-5 tools together, even then backup can be an error prone and time consuming task. However, rsync can be used to scan all of the files in your music library, as well as the subdirectories, and to add only the ones that are not present on the external hard drive and replay your reorganization to move files as well. All this is done smartly in just one tool using some flags.

Solutions using Rsync

See also

Free software portal

• Remote Differential Compression

References

External links

• Official website
• rsync algorithm