Categories
Planning

How to exclude files from rsync

How to exclude files from rsync

Rsync (Remote Sync) is a most commonly used command for copying and synchronizing files and directories remotely as well as locally in Linux/Unix systems. With the help of rsync command you can copy and synchronize your data remotely and locally across directories, across disks and networks, perform data backups and mirroring between two Linux machines.

Rsync is one of the most useful utilities for a server administrator, but it syncs everything by default, which can be annoying if your application creates a lot of temporary files. Here’s how to exclude files when using rsync.

Important features of rsync command

  • Speed : First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
  • Security : rsync allows encryption of data using ssh protocol during transfer.
  • Low Bandwidth : rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols like FTP.
  • Privileges : No special privileges are required to install and execute rsync.

The basic syntax of rsync is very straight forward, and operates in a way that is similar to ssh, scp, and cp.

  • -v : verbose.
  • -r : copies data recursively (but don’t preserve timestamps and permission while transferring data).
  • -a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps.
  • -z : compress file data.
  • -h : human-readable, output numbers in a human-readable format.
  • -d : transfer directories without recursing -e : specify the ssh as remote shell.

First, create a sample directory structure as shown below (with some empty files) that can be used for testing purpose.

The above commands will create a source directory (under home directory) with the following structure,To see the structure of source directory type below command.

1) Exclude a specific directory

Frist we remove destination directory if it exist by typing below command.

If we don’t want to sync the dir3 and its subdirectories from the source to the destination folder, we use the rsync –exclude option as shown below.

For remote sync, we can type command below.

2) Exclude a specific file

If we don’t want to sync the file3.txt from the source to the destination folder, we type the command below.

For remote sync, we can type command below.

For example in real time where you wish to rsync /var to a backup directory but you don’t wish to include cache and tmp folder that usually don’t hold important content between restarts you can use the following command:

3) Exclude a specific file type

To exclude a specific with specific extension, we use the appropriate pattern. For example, to exclude all the files that contains .txt as extension, we type the command below.

For remote sync, we can type command below.

4) Exclude multiple directories that matches a pattern

We will exclude any directory (or subdirectories) under source/ that starts with “dir”

For remote sync, we can type command below.

5) Exclude multiple files and directories at the same time

First, create a text file with a list of all the files and directories you don’t want to backup. This is the list of files and directories you want to exclude from the rsync.

Next, execute the rsync using –exclude-from option with the exclude-list.txt as shown below.

For remote sync, we can type command below.

6) Exclude path is always relative

The exclude option seems to have a full path (i.e /dir3/file4.txt). But, from rsync point of view, exclude path is always relative, and it will be treated as dir3/file4.txt. In the example below, rsync will look for dir3 under source directory (and not under / root directory).

For remote sync, we can type command below.

It is similar to command below.

7) Exclude files from the command line

can also exclude files directly from the command line, this is useful when you have a smaller number of files to exclude. For example if we wish to sync /source to a destination directory but you don’t wish to include file3.txt and file4.txt files, we can use the following command.

For remote sync, we can type command below.

8) Exclude multiple files or folders

When we want to exclude type of files *.txt and specific folder dir3 , we can type command below.

For remote sync, we can type command below.

9) Exclude Multiple Directories with Specific Pattern

We can also provide multiple directories with specific pattern for exclude. We will use wildcard * to complete dir directory name.

For remote sync, we can type command below.

10) Exclude Certain Size Files

We encounter during the backup process some large size files took really long time to copy. we have to copy some movies which are over 1GB in size. So, We think that it would be better if we could simply exclude the big files or any unimportant files in order to complete the backup as soon as possible.

For example, we want to exclude files bigger than 3 MB in size. All we have to do is use –max-size=SIZE option with rsync command. This option will not transfer any file larger than the specified size.

For remote sync, we can type command below.

We can also use –min-size=SIZE to exclude any file smaller than the specified size.

for example, to exclude files which are smaller than 7 MB, run the following command.

The suffixes of sizes are as follows:

  • “K” (or “KiB”) is a kilobyte (1024)
  • “M” (or “MiB”) is a megabyte (1024*1024)
  • “G” (or “GiB”) is a gigabyte (1024*1024*1024)

If you want the multiplier to be 1000 instead of 1024, use “KB”, “MB”, or “GB” (Note: lower-case is also accepted for all values).

Hope you enjoyed this tutorial! Please leave your comments

@lowellheddings
January 16, 2014, 12:34pm EDT

How to exclude files from rsync

Rsync is one of the most useful utilities for a server administrator, but it syncs everything by default, which can be annoying if your application creates a lot of temporary files. Here’s how to exclude files when using rsync.

Excluding from a List in a File

This is the ideal method for excluding files and folders, since you can always edit the list and tweak things if necessary. Here is the syntax:

The tricky thing with rsync is that you need to use a relative path when you are trying to exclude things, because when it tries to match the exclusions it won’t use the first part of the path for the match… it’s weird.

Say, for example, you are trying to backup /data/web/ and send it to another server, so you use a command like rsync -a /data/web/ [email protected]:/backups/data/web/ to make it happen… but you’d really like to skip syncing the /data/web/cache/ folder. When rsync goes to check your exclusion list for each item that it syncs, it won’t check /data/web/cache/ since your original rsync command is based in the /data/web/ folder. It’ll just check “cache/” against the list. So you’ll need to put “cache” into the list, not the full path. Example:

Now to exclude /data/web/cache and /data/web/temp from rsync using this command, we’d remove the /data/web/ portion of the path and the /data/exclusions file would simply contain this:

You’ll note that I added the * into the path, to make sure that it matches anything that starts with “cache” at the beginning. You can use this star pattern for more useful reasons if you wanted — say you want to exclude all .txt files from being synced. You’d start the pattern with the star to make sure that always matches, and add this:

That would ensure that those types of files are skipped during a sync. It’s pretty simple beyond that.

Excluding a Single Item

This technique is much less useful, but you can use it on the fly if you need to. If you are setting up a script to use rsync, which you usually are, you should take the extra minute to exclude from a file list instead to make future maintenance easier. The syntax is very similar:

The same relative path should apply here as above.

Posted Feb 20, 2019

How to exclude files from rsync

Rsync is a fast and versatile command line utility that synchronizes files and folders between two locations over a remote shell.

With Rsync you can mirror data, create incremental backups and copy files between systems. When copying data, you may want to exclude one or more files or directories based on their name or location.

In this tutorial, we will show you how to exclude files and directories with rsync.

Before You Begin #

You should have a basic knowledge of how rsync works .

In the examples below, we will use rsync with the -a , option. This tells rsync to syncs directories recursively, transfer special and block devices and preserve symbolic links, modification times, group, ownership, and permissions.

When excluding files or directories you need to use their relative paths to the source directory.

There are two options to specify the files and directories you want to exclude:

  • From a command line, using the –exclude option.
  • From a file, using the –exclude-from option.

Exclude a Specific File #

To exclude a specific file, pass the relative path to the file to the –exclude option.

In the following example the file src_directory/file.txt will not be transferred:

Exclude a Specific Directory #

Excluding a specific directory is same as excluding a file, just pass the relative path to the directory to the –exclude option as shown below:

If you want to exclude the directory content but not the directory itself use dir1/* instead of dir1 :

Exclude Multiple Files or Directories #

To exclude multiple files or directories simply specify multiple –exclude options:

If you prefer to use a single –exclude option you can list the files and directories you want to exclude in curly braces <> separated by a comma as shown below:

If the number of the files and/or directories you want to exclude is large, instead of using multiple –exclude options you can specify the files and directories you want to exclude in a file and pass the file to the –exclude-from option.

The command below does exactly the same as the one above:

Exclude Multiple Files or Directories Based on a Pattern #

With rsync you can also exclude files and directories based on a pattern that matches the file or directory name.

For example, to exclude all .jpg files you would run:

It is little trickier to exclude all other files and directories except those that match a certain pattern. Let’s say you want to exclude all other files and directories except the files ending with .jpg .

One option is to use the following command:

When using multiple include/exclude option, the first matching rule applies.

  • –include=’*.jpg’ – First we are including all .jpg files.
  • –include=’*/’ – Then we are including all directories inside the in src_directory directory. Without this rsync will only copy *.jpg files in the top level directory.
  • -m – Removes the empty directories.

Another option would be to pipe the output of the find command to rsync:

  • -printf %Pn – will remove the src_directory/ from the file path.
  • –files-from=- – means include only the files from the standard input (files passed from the find command).

Conclusion #

In this tutorial, you learned how to exclude files and directories when transferring data with Rsync. There’s lots more to learn about Rsync at Rsync User’s Manual page.

How to exclude files from rsync

  • About Us
  • Privacy Policy
  • Terms & Conditions
  • Disclaimer
  • Contact

Last Updated: January 4th, 2020 by How to exclude files from rsync Hitesh J in Guides , Linux

How to exclude files from rsync

Rsync is a very powerful tool to backup, copy and transfer files and directories between two locations and servers, much like the SCP Command.

In some cases, you need to ensure Rsync Excludes some files and directories from a transfer or backup.

By default, rsync syncs everything.

So you can use the rsync –exclude option to exclude your required directories and files from sync.

In this tutorial, we will show you how to exclude files and directories with rsync through different examples.

Create a Directory Structure

For testing purposes, you will need to create a directory structure in your system.

You can create it with the following command:

mkdir -p source/d1/d2
mkdir -p source/d3/d4
mkdir -p source/d5
touch source/f1.txt
touch source/f2.txt
touch source/f3.txt
touch source/d1/d2/f4.txt
touch source/d5/f5.txt
touch source/d3/f6.txt

You can see your directory structure with the following command:

You should see the following screen:

How to exclude files from rsync

Exclude a Specific File and Directory

You can exclude a specific file and directory using –exclude option with rsync.

For example, to exclude d3 directory from the source directory to destination directory run the following command:

rsync -avrz –exclude=d3 source/ destination

You should see the following output:

sending incremental file list
created directory destination
./
f1.txt
f2.txt
f3.txt
d1/
d1/d2/
d1/d2/f4.txt
d5/
d5/f5.txt
sent 460 bytes received 168 bytes 1,256.00 bytes/sec
total size is 0 speedup is 0.00

How to exclude files from rsync

To exclude the file f1.txt and f4.txt from the source directory, run the following command:

rsync -avz –exclude=f1.txt –exclude=d1/d2/f4.txt source/ destination

You should see the following output:

sending incremental file list
created directory destination
./
f2.txt
f3.txt
d1/
d1/d2/
d3/
d3/f6.txt
d3/d4/
d5/
d5/f5.txt
sent 465 bytes received 157 bytes 1,244.00 bytes/sec
total size is 0 speedup is 0.00

How to exclude files from rsync

Exclude a Specific file Type

You can also exclude a file with a specific extension.

For example, to exclude all the files with .txt extension run the following command:

rsync -avz –exclude=*.txt source/ destination

You should see the following output:

sending incremental file list
created directory destination
./
d1/
d1/d2/
d3/
d3/d4/
d5/
sent 201 bytes received 73 bytes 548.00 bytes/sec
total size is 0 speedup is 0.00

How to exclude files from rsync

Exclude multiple directories that match a pattern

You can also exclude any directories that match with a specific pattern.

For example, exclude all the directory under source that starts with the letter “d”:

rsync -avrz –exclude=d* source/ destination

You should see the following output:

sending incremental file list
created directory destination
./
f1.txt
f2.txt
f3.txt
sent 234 bytes received 110 bytes 688.00 bytes/sec
total size is 0 speedup is 0.00

How to exclude files from rsync

Exclude multiple files and directories

You can also exclude multiple files and directories at the same time.

You can achieve it using the option –exclude-from with Rsync.

First, create a file that contains the path of all files and directories that you want to exclude:

Add the following lines:

Save and close the file when you are finished.

Then, run the following command using –exclude-from option with the file-list.txt as shown below:

rsync -avrz –exclude-from=file-list.txt source/ destination

You should see the following output:

sending incremental file list
created directory destination
./
f2.txt
f3.txt
d1/
d1/d2/
d3/
d3/f6.txt
d3/d4/
d5/
d5/f5.txt
sent 465 bytes received 157 bytes 1,244.00 bytes/sec
total size is 0 speedup is 0.00

How to exclude files from rsync

Exclude Specific Size File

In some cases, you need to exclude files based on their size.

For example, exclude files bigger than 1 MB size using the option –max-size with Rsync:

rsync -avrz –max-size=1m source/ destination

You should see the following output:

sending incremental file list
created directory destination
./
f1.txt
f2.txt
f3.txt
d1/
d1/d2/
d1/d2/f4.txt
d3/
d3/f6.txt
d3/d4/
d5/
d5/f5.txt
sent 581 bytes received 195 bytes 1,552.00 bytes/sec
total size is 0 speedup is 0.00

How to exclude files from rsync

To exclude files smaller than 20 KB using the option –min-size with Rsync:

rsync -avrz –min-size=20k source/ destination

You should see the following output:

sending incremental file list
created directory destination
./
d1/
d1/d2/
d3/
d3/d4/
d5/
sent 357 bytes received 73 bytes 860.00 bytes/sec
total size is 0 speedup is 0.00

How to exclude files from rsync

Conclusion

In the above tutorial, we’ve learned what the Rsync Exclude files and directories from copying in Linux.

These options are very useful in the day-to-day backup process in order to save time and save disk space.

Rsync can recursively traverse the source directory and copy every file in source to destination. But there might be some junk (temp files, trash, and web-browser caches) that you rather not copy.

Excludes prevent rsync from backing up junk. The rsync –exclude-from option is similar to the one in GNU tar. Here is the option definition from the rsync man page:

The”exclude-from” file

The exclude file is a list of directory and file names to be excluded from the rsync destination e.g.

+ /source1/.fileA
– /source1/.*
– /source1/junk/

The format of the exclude file is one pattern per line. The pattern can be a literal string, wildcard, or character range.

In UNIX, hidden files start with a dot (.fileA is a hidden file), and * is a wildcard (.* means all hidden files).

A leading “+” means include the pattern. A leading “-” means exclude the pattern.

A path with a leading-slash must start with the source directory name (not the entire path).

Trailing slash is a directory (not a file). No trailing slash can be a directory or a file.

Lines in an exclude file are read verbatim. One frequent error is leaving some extra whitespace after a file name.

The “exclude-from” and “recursive” options

This tutorial uses the –recursive (-r) option (which is implied by -a). The default behavior of recursion is to traverse every branch of each source directory from the top down. Here is an example usage:

The following is a description of how rsync excludes or includes names, where “name” is the name of a file or folder.

If a directory name is included, the branch is traversed.

Setup source1 directory for examples 2 & 3

The tutorial examples use a small directory. Create the demo2 directory in your home directory:

user> mkdir demo2

Change to the rsync directory:

Create the source1 directory:

/demo2$ mkdir source1 source1/junk source1/junk/keep

Now populate the source1 directory with files:

/demo2$ touch source1/.fileA source1/.fileB source1/file source1/junk/file source1/junk/keep/fileX source1/junk/keep/fileY

Your source directory should look like this:

/demo2$ ls source1 -AFR
source1:
file .fileA .fileB junk/

source1/junk:
file keep/

source1/junk/keep:
fileX fileY

In the following examples, all commands are from the rsync directory.

Example 2 “exclude-from” file

This example shows why the order of patterns matters. Save the following text as rsync/exclude:

+ /source1/.fileA
– /source1/.*
– /source1/junk/

Assign permissions to the exclude file:

/demo2$ chmod 755 exclude

Make sure that destination is removed:

/demo2$ rm -r dest

Run rsync with the –exclude-from option:

/demo2$ rsync -ain –exclude-from=exclude source1 dest
cd+++++++++ source1/
>f+++++++++ source1/.fileA
>f+++++++++ source1/file

rsync read the exclude list top down. As expected, .fileA was backed up, and all other hidden files (.fileB) were skipped.

Now move the “- /source1/.*” line to the top of the exclude file and save:

– /source1/.*
+ /source1/.fileA
– /source1/junk/

Run rsync again:

/demo2$ rsync -ain –exclude-from=exclude source1 dest
cd+++++++++ source1/
>f+++++++++ source1/file

This time .fileA did not get backed up. Here is what happened: rsync traversed the source1 directory. When it visited .fileA, rsync read the exclude file top down, and acted on the first matching pattern. The first matching pattern was “- /source1/.*”, so .fileA was excluded. Order of exclude files matters because rsync reads the exclude file top down.

Example 3 “exclude-from” file

In this example we exclude an entire junk directory, except for one file.

Replace the contents of the exclude file with following lines and save:

+ /source1/junk/keep/
+ /source1/junk/keep/fileX
– /source1/junk/*
– /source1/junk/keep/*

The exclude file’s strategy is to enter all the directories leading to fileX, and skip all the unwanted junk files and directories not leading to fileX. To see how this works, trace the exclude file against the “rsync –recursive –exclude-from=FILE” algorithm near the top of this page.

/demo2$ rsync -ain –exclude-from=exclude source1 dest
cd+++++++++ source1/
>f+++++++++ source1/.fileA
>f+++++++++ source1/.fileB
>f+++++++++ source1/file
cd+++++++++ source1/junk/
cd+++++++++ source1/junk/keep/
>f+++++++++ source1/junk/keep/fileX

As planned, fileX is the only file in the junk directory that was backed up.

A common error is to forget specific include/exclude rules for all the parent directories that need to be visited. For example, this exclude file will not backup fileX:

/demo2$ sync -ain –exclude-from=exclude source1 dest
cd+++++++++ source1/
>f+++++++++ source1/.fileA
>f+++++++++ source1/.fileB
>f+++++++++ source1/file

Here is what happened: rsync traversed the source1 directory. When it visited junk/, rsync read the exclude file top down, and acted on the first matching pattern. The first matching pattern was “- /source1/junk/”, so entire junk directory was excluded.

exclude-file suggestions

Consider the following items for your –exclude-from file:

#configuration files (Ubuntu 9.04)
+ /user/.config/
+ /user/.gnome2/

#desktop (Ubuntu 9.04)
– /user/Desktop/
– /user/examples.desktop

#firefox bookmarks, where “xxxxxxxx” represents a random string of 8 characters.
#http://support.mozilla.com/en-US/kb/Backing+up+your+information#Locate_your_profile_folder
+ /user/.mozilla/firefox/
+ /user/.mozilla/firefox/xxxxxxxx.default/
+ /user/.mozilla/firefox/xxxxxxxx.default/bookmarkbackups/
– /user/.mozilla/*
– /user/.mozilla/firefox/*
– /user/.mozilla/firefox/xxxxxxxx.default/*

#temporary files
– *.tmp
– *.temp

I am trying to backup my home folder on my NAS drive. I am giving this:

where rsync-exclude.txt has this content:

and it is in the same folder I execute rsync (home folder).

However the rsync tries to copy this folder, too.

What am I doing wrong?

How to exclude files from rsync

2 Answers 2

You are providing absolute paths in your exclude list.

With rsync, all exclude (or include!) paths beginning with / are are anchored to the “root of transfer“.

The root of transfer in this case is /home/chris . If you did:

rsync -Paz –exclude-from ‘rsync-exclude.txt’ / [email protected]:

. then your exclusions should work (but you’d be copying everything else on that filesystem!).

But since you’re just trying to sync your home directory, and there is no subdirectory of /home/chris named “home/chris/Downloads” , rsync finds nothing that matches.

So try removing the /home/chris parts from your rsync-exclude.txt file.

Actually, you should just need a single line in the file:

Note that if you don’t specify the leading / , and you happen to have other directories named “Downloads” , those would also be excluded. I’m assuming you only want to exclude your “top-level” (relative to the source directory, aka the “root of transfer”) Downloads directory, so you’ll want the leading / .

THE EASIEST WAY (to exclude only a few paths)

If you only need to exclude one directory, just do this (avoiding a separate file):

You can also chain together –exclude tags, like so:

Note that since there’s no slash, that one will exclude .hiddenFile from any every directory it copies!

But if you have more than a few exclusions, you’re better off with –exclude-from and a file.

Note

I see that you got it right, but those new to rsync should note the slash at the end of /home/chris/

To quote the rsync man page, “You can think of a trailing / on a source as meaning ‘copy the contents of this directory’ as opposed to ‘copy the directory by name’.”

So if you left off that trailing slash, you would end up with a directory called chris within the target directory, containing everything from /home/chris (except the original Downloads directory, of course!).

I’m having problems getting my rsync syntax right and I’m wondering if my scenario can actually be handled with rsync . First, I’ve confirmed that rsync is working just fine between my local host and my remote host. Doing a straight sync on a directory is successful.

Here’s what my filesystem looks like:

What I want to do is run rsync only on files that begin with “file_11_” in the subdirectories and I want to be able to run just one rsync job to sync all of these files in the subdirectories.

Here’s the command that I’m trying:

rsync -nrv –include=”**/file_11*.jpg” –exclude=”*” /Storage/uploads/ /website/uploads/

This results in 0 files being marked for transfer in my dry run. I’ve tried various other combinations of –include and –exclude statements, but either continued to get no results or got everything as if no include or exclude options were set.

Anyone have any idea how to do this?

4 Answers 4

The problem is that –exclude=”*” says to exclude (for example) the 1260000000/ directory, so rsync never examines the contents of that directory, so never notices that the directory contains files that would have been matched by your –include .

I think the closest thing to what you want is this:

(which will include all directories, and all files matching file_11*.jpg , but no other files), or maybe this:

(same concept, but much pickier about the directories it will include).

rsync include exclude pattern examples:

And final note is that simply dont rely on asterisks that are used in the beginning for evaluating paths; like “**dir” (its ok to use them for single folders or files but not paths) and note that more than two asterisks dont work for file names.

How to exclude files from rsync

Add -m to the recommended answer above to prune empty directories.

Here’s my “teach a person to fish” answer:

Rsync’s syntax is definitely non-intuitive, but it is worth understanding.

  1. First, use -vvv to see the debug info for rsync.

The key concept here is that rsync applies the include/exclude patterns for each directory recursively. As soon as the first include/exclude is matched, the processing stops.

The first directory it evaluates is /Storage/uploads . Storage/uploads has 1280000000/, 1260000000/, 1270000000/ dirs/files. None of them match file_11*.jpg to include. All of them match * to exclude. So they are excluded, and rsync ends.

  1. The solution is to include all dirs ( */ ) first. Then the first dir component will be 1260000000/, 1270000000/, 1280000000/ since they match */ . The next dir component will be 1260000000/ . In 1260000000/ , file_11_00.jpg matches –include=”file_11*.jpg” , so it is included. And so forth.

Last Validated on November 18, 2020 Originally Published on September 10, 2013

Introduction

Rsync, which stands for “remote sync”, is a remote and local file synchronization tool. It uses an algorithm that minimizes the amount of data copied by only moving the portions of files that have changed.

In this guide, we will cover the basic usage of this powerful utility.

What Is Rsync?

Rsync is a very flexible network-enabled syncing tool. Due to its ubiquity on Linux and Unix-like systems and its popularity as a tool for system scripts, it is included on most Linux distributions by default.

Basic Syntax

The basic syntax of rsync is very straightforward, and operates in a way that is similar to ssh, scp, and cp.

We will create two test directories and some test files with the following commands:

We now have a directory called dir1 with 100 empty files in it.

We also have an empty directory called dir2 .

To sync the contents of dir1 to dir2 on the same system, type:

The -r option means recursive, which is necessary for directory syncing.

We could also use the -a flag instead:

The -a option is a combination flag. It stands for “archive” and syncs recursively and preserves symbolic links, special and device files, modification times, group, owner, and permissions. It is more commonly used than -r and is usually what you want to use.

An Important Note

You may have noticed that there is a trailing slash ( / ) at the end of the first argument in the above commands:

This is necessary to mean “the contents of dir1 ”. The alternative, without the trailing slash, would place dir1 , including the directory, within dir2 . This would create a hierarchy that looks like:

Always double-check your arguments before executing an rsync command. Rsync provides a method for doing this by passing the -n or –dry-run options. The -v flag (for verbose) is also necessary to get the appropriate output:

Compare this output to the output we get when we remove the trailing slash:

You can see here that the directory itself is transferred.

How To Use Rsync to Sync with a Remote System

Syncing to a remote system is trivial if you have SSH access to the remote machine and rsync installed on both sides. Once you have SSH access verified between the two machines, you can sync the dir1 folder from earlier to a remote computer by using this syntax (note that we want to transfer the actual directory in this case, so we omit the trailing slash):

This is called a “push” operation because it pushes a directory from the local system to a remote system. The opposite operation is “pull”. It is used to sync a remote directory to the local system. If the dir1 were on the remote system instead of our local system, the syntax would be:

Like cp and similar tools, the source is always the first argument, and the destination is always the second.

Useful Options for Rsync

Rsync provides many options for altering the default behavior of the utility. We have already discussed some of the more necessary flags.

If you are transferring files that have not already been compressed, like text files, you can reduce the network transfer by adding compression with the -z option:

The -P flag is very helpful. It combines the flags –progress and –partial . The first of these gives you a progress bar for the transfers and the second allows you to resume interrupted transfers:

If we run the command again, we will get a shorter output, because no changes have been made. This illustrates rsync’s ability to use modification times to determine if changes have been made.

We can update the modification time on some of the files and see that rsync intelligently re-copies only the changed files:

In order to keep two directories truly in sync, it is necessary to delete files from the destination directory if they are removed from the source. By default, rsync does not delete anything from the destination directory.

We can change this behavior with the –delete option. Before using this option, use the –dry-run option and do testing to prevent data loss:

If you wish to exclude certain files or directories located inside a directory you are syncing, you can do so by specifying them in a comma-separated list following the –exclude= option:

If we have specified a pattern to exclude, we can override that exclusion for files that match a different pattern by using the –include= option.

Finally, rsync’s –backup option can be used to store backups of important files. It is used in conjunction with the –backup-dir option, which specifies the directory where the backup files should be stored.

Conclusion

Rsync can simplify file transfers over networked connections and add robustness to local directory syncing. The flexibility of rsync makes it a good option for many different file-level operations.

A mastery of rsync allows you to design complex backup operations and obtain fine-grained control over what is transferred and how.

I’ve about 50 or so files in various sub-directories that I’d like to push to a remote server. I figured rsync would be able to do this for me using the –include-from option. Without the –exclude=”*” option, all the files in the directory are being synced, with the option, no files are.

I’m running it as dry initially and 0.0.0.0 is obviously replaced by the IP of the remote server. The contents of rsync_include.txt is a new line separated list of relative paths to the files I want to upload.

Is there a better way of doing this that is escaping me on a Monday morning?

8 Answers 8

Edit: Josip Rodin’s answer below is better. Please use that one!

You might have an easier time, if you’re looking for a specific list of files, putting them directly on the command line instead:

This is assuming, however, that your list isn’t so long that the command line length will be a problem and that the rsync_include.txt file contains just real paths (i.e. no comments, and no regexps).

How to exclude files from rsync

There is a flag –files-from that does exactly what you want. From man rsync :

Using this option allows you to specify the exact list of files to transfer (as read from the specified FILE or – for standard input). It also tweaks the default behavior of rsync to make transferring just the specified files and directories easier:

The –relative (-R) option is implied, which preserves the path information that is specified for each item in the file (use –no-relative or –no-R if you want to turn that off).

The –dirs (-d) option is implied, which will create directories specified in the list on the destination rather than noisily skipping them (use –no-dirs or –no-d if you want to turn that off).

The –archive (-a) option’s behavior does not imply –recursive (-r), so specify it explicitly, if you want it.

These side-effects change the default state of rsync, so the position of the –files-from option on the command-line has no bearing on how other options are parsed (e.g. -a works the same before or after –files-from, as does –no-R and all other options).

The filenames that are read from the FILE are all relative to the source dir — any leading slashes are removed and no “..” references are allowed to go higher than the source dir. For example, take this command:

If /tmp/foo contains the string “bin” (or even “/bin”), the /usr/bin directory will be created as /backup/bin on the remote host. If it contains “bin/” (note the trailing slash), the immediate contents of the directory would also be sent (without needing to be explicitly mentioned in the file — this began in version 2.6.4). In both cases, if the -r option was enabled, that dir’s entire hierarchy would also be transferred (keep in mind that -r needs to be specified explicitly with –files-from, since it is not implied by -a). Also note that the effect of the (enabled by default) –relative option is to duplicate only the path info that is read from the file — it does not force the duplication of the source-spec path (/usr in this case).

In addition, the –files-from file can be read from the remote host instead of the local host if you specify a “host:” in front of the file (the host must match one end of the transfer). As a short-cut, you can specify just a prefix of “:” to mean “use the remote end of the transfer”. For example:

This would copy all the files specified in the /path/file-list file that was located on the remote “src” host.

If the –iconv and –protect-args options are specified and the –files-from filenames are being sent from one host to another, the filenames will be translated from the sending host’s charset to the receiving host’s charset.

NOTE: sorting the list of files in the –files-from input helps rsync to be more efficient, as it will avoid re-visiting the path elements that are shared between adjacent entries. If the input is not sorted, some path elements (implied directories) may end up being scanned multiple times, and rsync will eventually unduplicate them after they get turned into file-list elements.