Rsync
From Luke Jackson
Contents |
Summary
rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated.
The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.
Some of the additional features of rsync are:
- support for copying links, devices, owners, groups, and permissions
- exclude and exclude-from options similar to GNU tar
- a CVS exclude mode for ignoring the same files that CVS would ignore
- can use any transparent remote shell, including ssh or rsh
- does not require root privileges
- pipelining of file transfers to minimize latency costs
- support for anonymous or authenticated rsync servers (ideal for mirroring)
Usage
The basic syntax for rsync is simple enough -- just run rsync [options] source destination to copy the file or files provided as the source argument to the destination.
So, for example, if you want to copy some files under your home directory to a USB storage device, you might use rsync -a /home/user/dir/ /media/disk/dir/. By the way, "/home/user/dir/" and "/home/usr/dir" are not the same thing to rsync. Without the final slash, rsync will copy the directory in its entirety. With the trailing slash, it will copy the contents of the directory but won't recreate the directory. If you're trying to replicate a directory structure with rsync, you should omit the trailing slash -- for instance, if you're mirroring /var/www on another machine or something like that.
In this example, I included the archive option (-a), which actually combines several rsync options. It combines the recursive and copy symlinks options, preserves group and owner, and generally makes rsync suitable for making archive copies. Note that it doesn't preserve hardlinks; if you want to preserve them, you will need to add the hardlinks option (-H).
Another option you'll probably want to use most of the time is verbose (-v), which tells rsync to report lots of information about what it's doing. You can double and triple up on this option -- so using -v will give you some information, using -vv will give more, and using -vvv will tell you everything that rsync is doing.
rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
rsync [OPTION]... [USER@]HOST:SRC DEST
rsync [OPTION]... SRC [SRC]... DEST
rsync [OPTION]... [USER@]HOST::SRC [DEST]
rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
Options
Here is a short summary of the options available in rsync. Please refer to the detailed description below for a complete description.
-v, --verbose increase verbosity
-q, --quiet decrease verbosity
-c, --checksum always checksum
-a, --archive archive mode, equivalent to -rlptgoD
-r, --recursive recurse into directories
-R, --relative use relative path names
--no-relative turn off --relative
--no-implied-dirs don’t send implied dirs with -R
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir make backups into this directory
--suffix=SUFFIX backup suffix (default ~ w/o --backup-dir)
-u, --update update only (don’t overwrite newer files)
--inplace update the destination files inplace
-K, --keep-dirlinks treat symlinked dir on receiver as dir
-l, --links copy symlinks as symlinks
-L, --copy-links copy the referent of all symlinks
--copy-unsafe-links copy the referent of "unsafe" symlinks
--safe-links ignore "unsafe" symlinks
-H, --hard-links preserve hard links
-p, --perms preserve permissions
-o, --owner preserve owner (root only)
-g, --group preserve group
-D, --devices preserve devices (root only)
-t, --times preserve times
-S, --sparse handle sparse files efficiently
-n, --dry-run show what would have been transferred
-W, --whole-file copy whole files, no incremental checks
--no-whole-file turn off --whole-file
-x, --one-file-system don’t cross filesystem boundaries
-B, --block-size=SIZE force a fixed checksum block-size
-e, --rsh=COMMAND specify the remote shell
--rsync-path=PATH specify path to rsync on the remote machine
--existing only update files that already exist
--ignore-existing ignore files that already exist on receiver
--delete delete files that don’t exist on sender
--delete-excluded also delete excluded files on receiver
--delete-after receiver deletes after transfer, not before
--ignore-errors delete even if there are I/O errors
--max-delete=NUM don’t delete more than NUM files
--partial keep partially transferred files
--partial-dir=DIR put a partially transferred file into DIR
--force force deletion of dirs even if not empty
--numeric-ids don’t map uid/gid values by user/group name
--timeout=TIME set I/O timeout in seconds
-I, --ignore-times turn off mod time & file size quick check
--size-only ignore mod time for quick check (use size)
--modify-window=NUM compare mod times with reduced accuracy
-T --temp-dir=DIR create temporary files in directory DIR
--compare-dest=DIR also compare received files relative to DIR
--link-dest=DIR create hardlinks to DIR for unchanged files
-P equivalent to --partial --progress
-z, --compress compress file data
-C, --cvs-exclude auto ignore files in the same way CVS does
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE exclude patterns listed in FILE
--include=PATTERN don’t exclude files matching PATTERN
--include-from=FILE don’t exclude patterns listed in FILE
--files-from=FILE read FILE for list of source-file names
-0 --from0 all file lists are delimited by nulls
--version print version number
--daemon run as an rsync daemon
--no-detach do not detach from the parent
--address=ADDRESS bind to the specified address
--config=FILE specify alternate rsyncd.conf file
--port=PORT specify alternate rsyncd port number
--blocking-io use blocking I/O for the remote shell
--no-blocking-io turn off --blocking-io
--stats give some file transfer stats
--progress show progress during transfer
--log-format=FORMAT log file transfers using specified format
--password-file=FILE get password from FILE
--bwlimit=KBPS limit I/O bandwidth, KBytes per second
--write-batch=FILE write a batch to FILE
--read-batch=FILE read a batch from FILE
--checksum-seed=NUM set block/file checksum seed
-4 --ipv4 prefer IPv4
-6 --ipv6 prefer IPv6
-h, --help show this help screen
Hidden Files
rsync will move hidden files (files whose names begin with a .) by default. If you want to exclude (ignore) hidden files, you can use this option:
--exclude=".*/"
You can also use the --exclude option to prevent copying things like Vim's swap files (.swp) and automatic backups (.bak) created by some programs.
Examples
Echo all files from source location to target.
- Recursively sync files.
- Preserve Permissions.
- Preserve Owner.
- Preserve Group.
- Copy Whole File don't check for file updates.
- Delete all files that no longer exist on source.
- Ignore Existing files on target.
- Exclude specified folders.
- Exclude hidden files
rsync -vrpogW --ignore-existing --delete --exclude ".*" --exclude "System Volume Information" --exclude "RECYCLER" /SOURCE_MP3/ /TARGET_MP3
Making local copies
Suppose you have an external USB or FireWire drive, and you want to copy data from your home directory to your external drive. A good way to do this would be to keep all your important data under a single top-level directory and then copy it to a backup directory on the external drive using a command like:
rsync -avh /home/usr/dir/ /media/disk/backup/
If you want to make sure that local files you've deleted since the last time you ran rsync are deleted from the external system as well, you'll want to add the --deleted option, like so:
rsync -avh --delete /home/user/dir/ /media/disk/backup
Be very careful with the delete option; with it, you can whack a bunch of files without meaning to. In fact, while you're getting used to rsync, it's probably a good idea to use the --dry-run option with your commands to run through the transfer first, without actually copying or synching files. If you do start an rsync transfer and realize that you've botched the command in some way that might result in the destruction of data, press Ctrl-c immediately to terminate the transfer. Some files may be gone, but you may be able to save the rest. Making remote copies
What if you want to copy files offsite to a remote host? No problem -- all you need to do is add the host and user information. So, for instance, if you want to copy the same directory to a remote host, you'd use:
rsync -avhe ssh --delete /home/user/dir/ user@remote.host.com:dir/
If you want to know how fast the transfer is going, and how much remains to be copied, add the --progress option:
rsync --progress -avhe ssh --delete /home/user/dir/ user@remote.host.com:dir/
If you don't want to be prompted for a password each time rsync makes a connection -- and you don't -- make sure that you have rsync set up to log in using an SSH key rather than a password. To do this, create an SSH key on the local machine using ssh-keygen -t dsa, and press Enter when prompted for a passphrase. After the key is created, use ssh-copy-id -i .ssh/id_dsa.pub user@remote.host.com to copy the public key to the remote host.
What if you need to bring back some of the files you copied using rsync? Use the following syntax:
rsync -avze ssh remote.host.com:/home/user/dir/ /local/path/
The z option compresses data during the transfer. If the file you are copying exists on the local host, then rsync will just leave it alone -- the same as if you were copying files to a remote host. Wrapping it up with a script
Once you've figured out what directory or directories you want to sync up, and you've gotten the commands you need to sync everything, it's easy to wrap it all up with a simple script. Here's a short sample:
rsync --progress -avze ssh --delete /home/user/bin/ user@remote.host.com:bin/ rsync --progress -avze ssh --delete /home/user/local/data/ user@remote.host.com:local/data/ rsync --progress -avze ssh --delete /home/user/.tomboy/ user@remote.host.com:/.tomboy/
Use the --progress option if you're going to be running rsync interactively. If not, there's no need for it.
If you look at the rsync man page, you can easily be confused. However, after a little practice with rsync, you'll find that it's not hard to set up rsync jobs that will help you prepare for the day that your disk drive craps out and you need access to your data right away
Batch Download
rsync --bwlimit=1024 --progress -vrae 'ssh' user@server.net:/local/path/20110112_01_x2564_order_documents_f3_V.tar.gz .
rsync --bwlimit=1024 --existing --progress -vrae 'ssh' user@server.net:/local/path/20110112_01_x2564_order_documents_f3_V.tar.gz .
Local File Copy with Status and Speed
rsync --bwlimit=1024 --partial --progress -vra ~/Downloads/QuickP\ 1.0.97.rar .
rsync -vra io.com/ io.com.svn/io_wos_base/
SKUs Sync
rsync --bwlimit=1024 --partial --progress -vvrae 'ssh -p 2222' na_io85_dump_20150505.sql.gz root@SERVER:/www/io85.com/dd/htdocs/skus/
Escaping Spaces with (~) Tilda Support and Brace Expansion
Below is an example of how to work with spaces in directory structure:
scp user@server.ext:'/home/samba/Service\ Deliver/System\ Admin/file\ name.sxw' .
scp user@host:"'filename with spaces'" destination
rsync --bwlimit=2048 --partial --progress -vvrae 'ssh' user@server:~/"'files/Name - With A lot Of spaces'" .
rsync --bwlimit=2048 --partial --progress -vvrae 'ssh' user@server:~/"'files/Name - With A lot Of spaces'"\{5..6\}"'.More name with spaces'"* .
rsync --bwlimit=2048 --partial --progress -vvrae 'ssh' user@server:~/"'files/File.S03E'"\{0..1\}{0..9\}* .