Categories
Interior

How to use rsync to backup your data on linux

Table of Contents

Introduction

  • I have to sync up the two directory many times on Unix boxes.The simple steps would be to delete the directory from destination and take a backup on source destination and copy it to the destination directory.
  • It was good for small directories but This process will consume lot of time as directory size grow up.
  • I searched around and find that The fastest way to do this process would be to use RSYNC utility
  • It uses ‘rsync algorithm’ which provides a very fast method for syncing the directories or file systems. An important feature of rsync is that the mirroring takes place with only one transmission in each direction and which is not available in other similar programs.
  • Rsync’s default port is 873 and its an opensource software. Rsync is available for Unix,Linux and windows operating systems. You can freely download rsync source code from rsync.samba web portal.

Key features of RSYNC command

  • Support for copying links, devices, owners, groups and permissions
  • Exclude and exclude-from options similar to GNU tar
  • A CVS exclude mode for ignoring the same files that CVS would ignore
  • Does not require root privileges
  • Pipelining of file transfers to minimize latency costs
  • Support for anonymous or authenticated rsync servers (ideal for mirroring)

I would be giving the basics on RSYNC and how to use it effectively to sync the directories/code

How to use RSYNC – sync the data between two directory

-a option is called the archive option,It means following things
Recursive mode
Preserves symbolic links
Preserves permissions
Preserves timestamp
Preserves owner and group
-v Verbose option

This command once run will ask for the password of oracle user on Destination machine.Once password is given it will sync the change files in the directory to the destination server

If u need to enable this sync through a automated job, then this script should be working without prompting for password.

We can password less ssh setup between the machine to achieve it

Automated script would be like

The same command of RSYNC can be used in the same server also

Some more points to remember

(1)If the directories contain multiple owner,then I would suggest doing it using root user

(2) We can delete the files at destination site which are deleted on source site by adding –delete in RSYNC

(3) it can also be achieved using RSYNC daemon which is configured through /etc/rsyncd.conf on Solaris.But it is not necessary to use that.

Benefits of RSYNC command

(1) First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
(2) We can use ssh protocol for rsync which allows encryption of data during transfer.So it is quite secure
(3) Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.

Practical RSYNC Command

  1. Synchronizing the directories on the local server

2. Synchronizing the directories on the remote server

3. Synchronizing the directory structure on the remote server

4. How to Include and Exclude Pattern during File Transfer

The above command exclude any file starting with U

5. How to delete files at the destination

I hope you like this article on RSYNC command and add value to the technical work you are doing in your organization

Please do provide feedback for this article on RSYNC command

How to use rsync to backup your data on linux

Create a backup using rsync If you need to back up your system, there is no better way than to back up using rsync.

Rsync (Remote Synchronization) is a popular and powerful tool used to copy and synchronize files and directories between remote or local Linux/Unix systems. With rsync, we can easily copy/sync data between local and remote directories, on different drives and on different networks.

In this tutorial, I’ll show you how to back up with rsync using the Linux terminal. If you prefer graphical programs, you can also use rsync with a graphical interface.

Create a backup with rsync

For this tutorial, I will be using Arch Linux in a virtual machine created with VirtualBox. To simulate an external hard drive, I’ll plug in a USB stick where the backup will be stored and then restored. I recommend that you do the same to check your backup. This will give you confidence that your backup is working because an unverified backup is not a backup.

In this case, we’ll use this whole command:

sudo rsync -aAXv –delete –dry-run –exclude=/dev/* –exclude=/proc/* –exclude=/sys/* –exclude=/tmp/* –exclude=/run/* –exclude=/mnt/* –exclude=/media/* –exclude=”swapfile” –exclude=”lost+found” –exclude=”.cache” –exclude=”Downloads” –exclude=”.VirtualBoxVMs”–exclude=”.ecryptfs” / /run/media/alu/ALU/

To make a backup using rsync, we usually use the command line. I know that not everyone is sure about the command line tools, but you will find that the process is not that complicated and you can also back up your system using the command line.

We now turn to explain what this command means:
sudo – execute the command as superuser. Mandatory use.
rsync is the program itself to be used.
-a – archive mode.
-A – save the access control list.
-X – keep extended attributes.
Basically, these three options mean preserving all the attributes of your files. Owner attributes or rights will not be changed during the backup process.
-v – show the progress of the backup.
–delete – This option allows you to make an incremental backup. This means that if this is not your first backup, it will only keep the difference between your source and destination. Thus, it will only back up new files and changed files, and it also deletes all files in the backup that have been deleted from your system. Be careful with this option.
–dry-run – This option simulates a backup. Useful for checking if it is being executed.
–exclude – Excludes folders and files from the backup. I entered exclude as a separate parameter for each directory. You can also use it this way –exclude= . But before running rsync, make sure you change the working directory to root (cd /), otherwise the shared exclusion option may not work.

Excluded folders are directly dependent on each of us, but folders /dev/, /proc/, /proc/ /sys/ /tmp/ /run/ /mnt/ и /media are not important for backups because rsync will not copy their contents. /mnt/ it is vital to exclude them if we plug in a USB stick.
/ – What we want to keep.
/run/media/alu/ALU is what you need for backups. I recommend encrypting the destination to keep your data safe.
Press Enter, the command will be executed in simulation mode (due to the –dry-run option). This way we test it to make sure everything is in order. If you are sure everything is running the way you want, you remove –dry-run from the command and run it again.
Note. It is recommended that the backup disk have a Linux compatible filesystem like ext4.

Restore the backup using rsync

To restore the backup we created, we’re going to boot from the live ISO. Since we are working with Arch Linux, the ISO must be from Arch Linux. Next, we need to mount our USB stick.
After logging in from the live image, we have to create two folders, one for the system on the hard drive, and the other where the created backup will be mounted:

Next, we need to check the names of our devices:

Then we have to mount the filesystem and the backup on the USB flash drive:

mount /dev/sda1 /mnt/system mount /dev/sdb1 /mnt/usb

Finally, let’s start restoring our backup. To do this, run:

rsync -aAXv –delete –exclude=”lost+found” /mnt/usb/ /mnt/system/

By executing this command, we restore a backup of our system.

How to use rsync to backup your data on linux

Conclusion

Rsync is a powerful tool that can be used from the command line, with several options to adapt to any need. Of course, you can use GUI backup programs. There are also some really good non-open source graphical backup software available.
To keep your backup safe, I also recommend encrypting your backup drive.
Have you already made a backup? Let me know your experience with rsync.
Share this article on your social networks.

How to use rsync to backup your data on linux

Create a backup using rsync If you need to back up your system, there is no better way than to back up using rsync.

Rsync (Remote Synchronization) is a popular and powerful tool used to copy and synchronize files and directories between remote or local Linux/Unix systems. With rsync, we can easily copy/sync data between local and remote directories, on different drives and on different networks.

In this tutorial, I’ll show you how to back up with rsync using the Linux terminal. If you prefer graphical programs, you can also use rsync with a graphical interface.

Create a backup with rsync

For this tutorial, I will be using Arch Linux in a virtual machine created with VirtualBox. To simulate an external hard drive, I’ll plug in a USB stick where the backup will be stored and then restored. I recommend that you do the same to check your backup. This will give you confidence that your backup is working because an unverified backup is not a backup.

In this case, we’ll use this whole command:

sudo rsync -aAXv –delete –dry-run –exclude=/dev/* –exclude=/proc/* –exclude=/sys/* –exclude=/tmp/* –exclude=/run/* –exclude=/mnt/* –exclude=/media/* –exclude=”swapfile” –exclude=”lost+found” –exclude=”.cache” –exclude=”Downloads” –exclude=”.VirtualBoxVMs”–exclude=”.ecryptfs” / /run/media/alu/ALU/

To make a backup using rsync, we usually use the command line. I know that not everyone is sure about the command line tools, but you will find that the process is not that complicated and you can also back up your system using the command line.

We now turn to explain what this command means:
sudo – execute the command as superuser. Mandatory use.
rsync is the program itself to be used.
-a – archive mode.
-A – save the access control list.
-X – keep extended attributes.
Basically, these three options mean preserving all the attributes of your files. Owner attributes or rights will not be changed during the backup process.
-v – show the progress of the backup.
–delete – This option allows you to make an incremental backup. This means that if this is not your first backup, it will only keep the difference between your source and destination. Thus, it will only back up new files and changed files, and it also deletes all files in the backup that have been deleted from your system. Be careful with this option.
–dry-run – This option simulates a backup. Useful for checking if it is being executed.
–exclude – Excludes folders and files from the backup. I entered exclude as a separate parameter for each directory. You can also use it this way –exclude= . But before running rsync, make sure you change the working directory to root (cd /), otherwise the shared exclusion option may not work.

Excluded folders are directly dependent on each of us, but folders /dev/, /proc/, /proc/ /sys/ /tmp/ /run/ /mnt/ и /media are not important for backups because rsync will not copy their contents. /mnt/ it is vital to exclude them if we plug in a USB stick.
/ – What we want to keep.
/run/media/alu/ALU is what you need for backups. I recommend encrypting the destination to keep your data safe.
Press Enter, the command will be executed in simulation mode (due to the –dry-run option). This way we test it to make sure everything is in order. If you are sure everything is running the way you want, you remove –dry-run from the command and run it again.
Note. It is recommended that the backup disk have a Linux compatible filesystem like ext4.

Restore the backup using rsync

To restore the backup we created, we’re going to boot from the live ISO. Since we are working with Arch Linux, the ISO must be from Arch Linux. Next, we need to mount our USB stick.
After logging in from the live image, we have to create two folders, one for the system on the hard drive, and the other where the created backup will be mounted:

Next, we need to check the names of our devices:

Then we have to mount the filesystem and the backup on the USB flash drive:

mount /dev/sda1 /mnt/system mount /dev/sdb1 /mnt/usb

Finally, let’s start restoring our backup. To do this, run:

rsync -aAXv –delete –exclude=”lost+found” /mnt/usb/ /mnt/system/

By executing this command, we restore a backup of our system.

How to use rsync to backup your data on linux

Conclusion

Rsync is a powerful tool that can be used from the command line, with several options to adapt to any need. Of course, you can use GUI backup programs. There are also some really good non-open source graphical backup software available.
To keep your backup safe, I also recommend encrypting your backup drive.
Have you already made a backup? Let me know your experience with rsync.
Share this article on your social networks.

Home » Web Servers » How to Use rsync to Back up Data

Backing up data is an essential part of both individual and enterprise infrastructures. Machines with the Linux operating system can use rsync and ssh to facilitate the process.

Rsync is a command-line utility that lets you transfer files to local and remote locations. Rsync is convenient to use since it comes by default with most Linux distributions. You can customize the tool by using many of the available options.

In this use case, we will use SSH in combination with rsync to secure the file transfer.

Follow this tutorial to learn how to use rsync to backup data. The guide will list a few examples to illustrate how the process works.

How to use rsync to backup your data on linux

  • Sudo or root privileges or a user with access to backup and destination directories
  • SSH access to a server via command line/terminal window
  • Rsync installed on the local and destination machine

Basic Rsync Syntax for Local and External Transfers

The syntax for using the rsync tool is different for local and remote transfers.

For local backups, the syntax follows this basic pattern:

To transfer files to an external location, we will use a pattern that is a bit different:

In both cases, the source and destination are a directory or a file path.

Backing up Data with Rsync

For a better understanding of what rsync does, we will use the verbose switch -v . Additionally, since we will be backing up data in directories, we will use the archive mode -a for recursive syncing. There are many rsync options and examples, so use any of them you need for your use case.

Rsync Dry Run – Precautions

The rsync utility allows you to manipulate your data in different ways. So, be careful when backing up your files. If you use a wrong option or a wrong destination, you may end up mixing your data. Even worse, you may unintentionally overwrite or delete files.

For that reason, use the –dry-run option to confirm the tool does what you want to do. Accidental data loss can occur, but this option helps prevent it.

For simple transfers, you may not need to use –dry-run , but when a larger set of data is in question, we strongly advise that you do.

Use the basic syntax format and add –dry-run :

Use Rsync to Back up Data Locally

We will start by performing a backup of a directory on the same Linux machine. The path can be any location – another partition, hard drive, external storage, etc.

Use the full path for both the source and destination to avoid errors.

For example, to back up a Dir1 from Documents to /media/hdd2/rscync_backup, use the rsync command in this form:

How to use rsync to backup your data on linux

The output shows the list of transferred files and directories and other transfer details.

Note: To create a new directory at the destination and back up your files there, add a trailing slash ( / ) at the end of the destination path. If you add the trailing slash to the source, then the source directory will not be created at the destination. Rsync only transfers its content in that case.

Use Rsync to Back up Data over Network

For secure data backup over the network, rsync uses SSH for transfers. Your server needs to be set to allow SSH connection.

Once you manage to connect to the remote machine over SSH, you can start backing up your data to a location on that machine.

For example, to back up Dir1 to backup on another machine over the network, enter:

How to use rsync to backup your data on linux

The output lists the directories and files rsync transferred to another machine.

You can check if the files really are on the remote server:

If you are connecting for the first time, you will need to enter your password and confirm when you get a prompt. There is no need to enter a username for remote transfers if you want to connect as the current user.

Note: You can evade entering a password every time you want to back up data with rsync over SSH. Set up SSH key-based authentication, and you will be able to use passwordless login to the remote machine.

The example we used here assumes that SSH uses the default port. If you need to specify a different port for the SSH connection, use the -e flag and enter SSH options.

To specify port 4455, for example, run the above command in this format:

When needed, you can delete the source files after you transfer them to another location.

Compress Data when Backing up with Rsync

To save some space, you can compress your data before transferring it to another location. You can use rsync’s built-in option to compress data, or you can use a different tool to do that before running rsync.

To compress data during transfer, use the -z switch with your rsync command.

Another option is to use the zip command to zip your files or directory and then run rsync . In our case, we will zip Dir1 into Dir.zip:

Then transfer that file to another location:

Now, you have a zipped copy of your directory on a remote server. You can also do this for local transfers if you want to have a backup on another drive or partition.

This tutorial showed you how to back up data using rsync both locally and over a network. Take caution when using this tool and make sure you do a dry run if you are unsure about the rsync options you want to use.

You can check our other guides for more rsync examples or you can learn how to exclude files and directories with rsync.

Use these ‘rsync’ examples to learn how to copy from the command line

How to use rsync to backup your data on linux

  • Tweet
  • Share
  • Email

What to Know

  • Copy based on file type: rsync /home/jon/Desktop/data/*.jpg /home/jon/Desktop/backupdata/
  • Copy based on file size: rsync –max-size=2k /home/jon/Desktop/data/ /home/jon/Desktop/backupdata/
  • Copy entire folders: rsync –recursive /home/jon/Desktop/data /home/jon/Desktop/data2

This article explains how to use the rsync file-transfer program for Linux to copy directories and files, and even exclude files in a systematic way. As such, it backs up files intended for archiving while avoiding everything else.

Command Syntax

Using the rsync command properly requires that you follow the correct syntax:

Some commonly used option switches include:

  • -v, –verbose: Increase verbosity (provides more details about what the command is doing).
    • –info=FLAGS: Provides detailed informational messages.
    • –debug=FLAGS: Provides detailed debug messages.
    • –msgs2stderr: Special output handling for debugging.
  • -q, –quiet: Suppresses non-error messages.
    • –no-motd: Suppresses daemon-mode message of the day.
  • -c, –checksum: Skips files based on checksum, not mod-time and size.
  • -r, –recursive: Browse into sub-directories for additional files.
  • -b, –backup: Make backups.
    • –backup-dir=DIR: Make backups into a matching directory hierarchy.
    • –suffix=SUFFIX: Adds suffix text to the end of backed up files.
  • -d, –dirs: Transfer only directories without browsing inside of them.

Command Examples

Use rsync with some of those options to fine-tune your backup strategy.

Selective Copying Based on File Type

In this example, all of the JPG files from the /data/ folder copy to the /backupdata/ folder on the user Jon’s Desktop folder.

Copying Files Based on Size

This rsync example is a bit more complicated since it’s set up to not copy files if they’re larger than 2,048 KB. It only copies files smaller than the specified size.

Use k, m, or g to indicate kilobytes, megabytes, and gigabytes in the 1,024 multiplier, or kb, mb, or gb to use 1,000.

The same can be done for –min-size, too. In this example, rsync only copies files that are 30 MB or larger.

Use the –progress option to watch the process work up to 100 percent—handy when you’re copying very large files.

Copy Entire Folders

The –recursive option provides an easy way to copy an entire folder to a different location, like to the /data2/ folder in the example above.

This command copies the entire folder and all of its contents to the new location.

Exclude Certain Files

Copy a whole folder but exclude files of a certain file extension, such as DEB files, in this example above. The whole /data/ folder is copied to /backupdata/ as in the previous example, but all DEB files are excluded from the copy.

How to use rsync to backup your data on linux

The rsync command in Linux is mainly used to synchronize the files on your machine locally or with a remote machine. When your work is being carried out from multiple locations or on more than just a single machine, synchronizing files will be a extremely useful.

Thankfully, we have the rsync command in Linux to help us out by automating our tasks. This is a versatile and fast Linux file copying utility. It makes use of remote shells such as ssh and rsh. Alternatively, it can even sync files over TCP with help of the rsync daemon.

Table of Contents

Breaking down the rsync command

The rsync command in Linux makes use of a special ‘delta-transfer algorithm’ which copies only the files which don’t match between the source and the destination. Now, the best way to learn any command best through understanding it’s syntax. Here is how the syntax for the rsync command in Linux looks like.

The rsync command provides a long list of options to customize the data we receive while analyzing our network. Here is a list of some of the most popular rsync options.

Option Function
-4 / -6 This is used to specify the IPv4 (for -4) or IPv6 (for -6) resolution of IP address for bridging a connection to the remote machine.
-a This makes the rsync utility to work in archive mode. Here, all the directories are synced in a recursive manner. All block and special devices are transferred and all meta-data like symbolic links, ownerships and permissions are preserved.
-z This option is used for forcing the rsync utility to compress data before the transfer. It is helpful in case the sync is taking place over a weak network connection.
-P Display a progress bar when a data transfer is underway. Further, it retains files that have been transferred only partially. This is an important function in the event of transferring large files over an unstable or slow connection.
-q This makes the rsync utility to work in archive mode. Here, all the messages, which aren’t an error message, are not displayed on the output screen.
–delete This option tells the rsync command to delete all irrelevant files at the destination location while syncing it with the source. This is useful to make the destination ‘mirror’ the source.
–help This is used to display the help page for the rsync command and exit.

rsync command options

These were only the most commonly used options available with the rsync command in Linux. Make sure to check the other options using the man command to utilize the full functionality of the rsync command.

Practical Use of the rsync command in Linux

Now that we understand how the rsync command in Linux works, it is time to learn how to use it. Here we explore some common useful applications of the rsync command. The commands remain the same local and remote backups and for this demonstration, I’ll perform local backups. You simply need to change the destination address for remote backups (demonstrated in subsection 5 here)

1. Syncing directories using the rsync command

The most basic use of the rsync command in Linux is to sync files and directories. In this example, we will sync the contents of one directory to another. Here is the command for syncing two directories.

If the destination directory does not exist, it will be created by rsync command. As you can see in the following screenshot, the source and the destination are synced

2. Setting up incremental backups

Backups are an essential safety precaution against data loss. When your latest copy gets deleted or corrupted and there is no way for recovery, backups allow you some compensation. With them, you only need to re-do stuff since your last backup. And rsync can help with this.

Suppose you wish to create daily backups of your work. This can be done with the following command.

As we can see in the screenshot above, all the files which aren’t in the source were deleted from the destination.

We use cronjob to ensure the command automatically runs every day. You can learn more about automating tasks with cronjobs here.

It is generally advised to create at least two types of backups – daily/weekly or weekly/monthly – depending on your requirements. The second backup is a fail-safe in case you wish something was deleted in the daily/weekly sync and you need it back. The weekly/monthly backup should still have such a file.

3. Deleting Source Files After Sync

Mirroring is useful when you are working on a project with multiple systems. In such a case, it is important that when you delete something on one system, it should be removed from all the systems to keep them in sync. This can be done easily with the rsync command. When you wish to delete files from the destination which are removed from the source, we use the following command.

4. Setting speed limits for sync

When we are transferring files using rsync, it can hog up our system resources (when used locally) and the network bandwidth (when used to sync with a remote host). To prevent this, we can limit the rate at which the data transfer takes place. Here is how we do so.

Using the –bwlimit function in this command, we cap off the data transfer rate for this sync at 2000 KBPS (KiloBytes, not kilobits). This ensures that we can utilize system resources for other uses, even while a sync is in progress.

5. Using rsync for remote directories

The main purpose of the utility is for remote syncing files between servers or systems. But the command remains the same essentially. Since the command uses ssh and rsh protocols to communicate over the network, the destination directory can be specified in the same format as we use while connecting to a ssh server.

If we interachange the source and the destination in the above command, we can also sync files from the remote server to a directory on our local machine.

Wrapping up

The rsync command is a powerful utility to fulfill the copying and syncing requirements of a Linux user. It has a wide variety of applications and optimizes the process of data transfer and backup with its special delta transfer algorithm. We hope this tutorial was able to help you understand the rsync command in Linux. If you have any feedback, queries, or corrections, feel free to leave them in the comments below.

If you need to backup your system, there is no better way than making a backup with rsync.

Rsync (Remote Sync) is a popular and powerful tool used to copy and synchronize files and directories between remote or local Linux/Unix systems. With the help of rsync, we can easily copy/synchronize data between local and remote directories, across different drives and networks.

In this tutorial, I will show you how to make a backup with rsync using the Linux terminal. If you prefer graphical programs, you can use rsync with graphical interface too.

Video Tutorial

Making the backup with rsync

To make this tutorial, I will use Arch Linux in a virtual machine built with VirtualBox. To simulate an external hard drive, I will connect a USB flash drive where the backup will be stored and then restored. I recommend you do the same to test your backup. This will give you the confidence to know that your backup works because an untested backup is not a backup.

For this case we will use this entire command:

To make a backup with rsync, we usually use the command line. I know that not everyone is confident with the command line tools, but you will realize that the process is not that complicated and you can also back up your system using the command line.

We now proceed to explain what this command means:

sudo – to execute the command as a superuser. Mandatory use.

rsync – is the program itself to use.

-a – archive mode.

-A – preserve Access Control List.

-X – preserve extended attributes.

Basically, these three options mean to preserve all the attributes of your files. Owner attributes or permissions will not be modified during the backup process.

-v – It will show the progress of the backup.

–delete – this option allows you to make an incremental backup. That means, if it is not your first backup, it will backup only the difference between your source and the destination. So, it will backup only new files and modified files and it will also delete all the files in the backup which were deleted on your system. Be careful with this option.

–dry-run – This option simulates the backup. Useful to test its execution.

–exclude – Excludes folders and files from backup. I typed exclude as a separate option for every directory. You can also use it this way –exclude= . But make sure you change your working directory to root ( cd / ) before you run rsync, otherwise the joint exclude option may not work.

The excluded folders depend directly on each of us, however, the /dev/, /proc/, /proc/ /sys/ /tmp/ /run/ /mnt/ and /media folders are not important to backup because rsyn will not copy their content. /mnt/ it is vital to exclude them if we connect a USB memory.

/ – What we want to back up.

/run/media/alu/ALU – This is where you what to backup. I recommend to enrypt the destination, to make your data safe.

We press enter, the command will execute in a simulation mode (because of the –dry-run option) This way we test it to make sure everything is okay. When you’re sure that everything is performed as you want, you remove –dry-run from the command and run it again.

Note: It is recommended that the backup drive has a Linux compatible file system as ext4.

Restore the backup with rsync

To restore the backup we have made, we are going to boot from a live ISO. Since we are working with Arch Linux, then the iso image must be from Arch Linux. Next, we must mount our USB flash drive.

Once logged in from the live image, we must create two folders, one for the system on the hard disk and the other where the backup created will be mounted:

Next, we need to check the names of our devices:

Then, we must mount the file system and the backup on the USB flash device:

Finally, we proceed to perform the restoration of our backup. For this we run:

By running this command, we restore the backup of our system.

Conclusion

Rsync is a powerful tool that can be used from the command line, with multiple options that adapt to any need. You can of course use backup programs with graphical interface. There are also some really good non-open sources graphical backup programs.

Have you already done your backup? Let me know about your experience with rsync.

Please share this article through your social networks.

Basic rsync commands are usually enough to manage your Linux backups, but a few extra options add speed and power to large backup sets.

How to use rsync to backup your data on linux

Subscribe now

Get the highlights in your inbox every week.

It seems clear that backups are always a hot topic in the Linux world. Back in 2017, David Both offered Opensource.com readers tips on “Using rsync to back up your Linux system,” and earlier this year, he published a poll asking us, “What’s your primary backup strategy for the /home directory in Linux?” In another poll this year, Don Watkins asked, “Which open source backup solution do you use?”

My response is rsync. I really like rsync! There are plenty of large and complex tools on the market that may be necessary for managing tape drives or storage library devices, but a simple open source command line tool may be all you need.

Basic rsync

I managed the binary repository system for a global organization that had roughly 35,000 developers with multiple terabytes of files. I regularly moved or archived hundreds of gigabytes of data at a time. Rsync was used. This experience gave me confidence in this simple tool. (So, yes, I use it at home to back up my Linux systems.)

The basic rsync command is simple.

Indeed, the rsync commands taught in any tutorial will work fine for most general situations. However, suppose we need to back up a very large amount of data. Something like a directory with 2,000 sub-directories, each holding anywhere from 50GB to 700GB of data. Running rsync on this directory could take a tremendous amount of time, particularly if you’re using the checksum option, which I prefer.

Performance is likely to suffer if we try to sync large amounts of data or sync across slow network connections. Let me show you some methods I use to ensure good performance and reliability.

Advanced rsync

  • Linux commands cheat sheet
  • Advanced Linux commands cheat sheet
  • Free online course: RHEL Technical Overview
  • Linux networking cheat sheet
  • SELinux cheat sheet
  • Linux common commands cheat sheet
  • What are Linux containers?
  • Our latest Linux articles

Here’s an example based on this scenario. Let’s say we have a directory called /storage that we want to back up to an external USB device mounted at /media/WDPassport.

If we want to back up /storage to a USB external drive, we could use this command:

The c option tells rsync to use file checksums instead of timestamps to determine changed files, and this usually takes longer. In order to break down the /storage directory, I sync by subdirectory, using the find command. Here’s an example:

This looks OK, but if there are any files in the /storage directory, they will not be copied. So, how can we sync the files in /storage? There is also a small nuance where certain options will cause rsync to sync the . directory, which is the root of the source directory; this means it will sync the subdirectories twice, and we don’t want that.

Long story short, the solution I settled on is a “double-incremental” script. This allows me to break down a directory, for example, breaking /home into the individual users’ home directories or in cases when you have multiple large directories, such as music or family photos.

Here is an example of my script:

The first rsync command copies the files and directories that it finds in the source directory. However, it leaves the directories empty so we can iterate through them using the find command. This is done by passing the d argument, which tells rsync not to recurse the directory.

The find command then passes each directory to rsync individually. Rsync then copies the directories’ contents. This is done by passing the r argument, which tells rsync to recurse the directory.

This keeps the increment file that rsync uses to a manageable size.

Most rsync tutorials use the a (or archive) argument for convenience. This is actually a compound argument.

The other arguments that I pass would have been included in the a; those are l, p, t, g, and o.

The –delete option tells rsync to remove any files on the destination that no longer exist on the source. This way, the result is an exact duplication. You can also add an exclude for the .Trash directories or perhaps the .DS_Store files created by MacOS.

Be careful

One final recommendation: rsync can be a destructive command. Luckily, its thoughtful creators provided the ability to do “dry runs.” If we include the n option, rsync will display the expected output without writing any data.

This script is scalable to very large storage sizes and large latency or slow link situations. I’m sure there is still room for improvement, as there always is. If you have suggestions, please share them in the comments.

How to use rsync to backup your data on linux

This article we bring shell scripts to backup your files and directories from you local Linux machine to a remote Linux server using rsync command. This would be an interactive way to perform backup, where you need to provide remote backup server hostname/ip address and folder location. We keep a separate file where you need to provide files and directories that need backup. We have added two scripts where first script ask password after each file had been copied (if you have enabled ssh authentication keys , then password will be not be asked) and in second script password will be prompted only once.

We are going to backup bckup.txt, dataconfig.txt, docs and oracledb.

[[email protected] tmp]# ls -l
total 12
-rw-r–r–. 1 root root 0 May 15 10:43 bckrsync.sh
-rw-r–r–. 1 root root 0 May 15 10:44 bckup.txt
-rw-r–r–. 1 root root 0 May 15 10:46 dataconfig.txt
drwxr-xr-x. 2 root root 4096 May 15 10:45 docs
drwxr-xr-x. 2 root root 4096 May 15 10:44 oracledb

This file contains backup files / dir details

[[email protected] tmp]# cat /tmp/bckup.txt
/tmp/oracledb
/tmp/dataconfig.txt
/tmp/docs
[[email protected] tmp]#

Script 1 :

Running the script with output

Script 2 :

Screenshot running the second script with output

How to use rsync to backup your data on linux

Hope these scripts help you to take backup !!

Configure an automatic backup in Linux now and you will thank yourself in the future.

Creating automatic backup will not only save the time you would spend creating the backups manually, but it may save you days, weeks, months or even years of work when something goes wrong and you lose some important data. Everyone knows about this but still a few users schedule an automatic backup in Linux. Even I have postponed configuring scheduled backups in my new KDE Neon install for a few months already. Luckily, nothing wrong happened with my data during this time.

I have just configured my system for automatic daily, weekly, and monthly backups with rsync and I would like to share how I did that. Keep in mind that rsync is a command line tool if you prefer graphical backup programs have a look at LuckyBackup or CloudBerry Backup.

How to schedule rsync

Rsync is installed on all popular Linux systems and you can easily configure it to run on schedule in the background. The most popular and simplest way to do that is to use the cron job scheduler. Cron is installed and configured on all Debian-based systems like Ubuntu, Linux Mint, KDE Neon, etc.

You simply need to add the schedule of the rsync commands you want to run to the crontab file:

I am not going to explain the rsync command. I have done it already in this post. Here, I will only explain the schedule format. You can see the schedule format from the comments in the crontab file. The columns abbreviation means the following:

  • m – minute
  • h – hour
  • dom – day_of_month
  • m – month
  • dow – day_of_week
  • command – the command to execute

Hence, my schedule command means that rsync will run at 12:00 daily, at 15:00 on Friday weekly, and at 16:00 on the first day of every month. To configure the time of your automatic backup in Linux, change these time points as you need.

From the above command, you can also see that daily and weekly backups will be overwritten, while all monthly backups will be kept with the date in their names. For example, my backup created on April 1, 2019, has the following name monthly_20190401 . If you have a limited amount of space, you can set monthly backups to be overwritten too. Just removed _$(date +%Y%m%d) .

I think this is the most reasonable way to automatically backup Linux. You can also compress monthly and weekly backups, but in my experience, compression is not practical. First, it takes a lot of time to run. Second, looking for specific files in a compressed backup is slow.

NOTE that your computer must be on during the scheduled time to execute the rsync commands. Cron will not run missed jobs. To run missed jobs, look at anacron.

Where to backup

You should never store your backup on the same hard drive which you backup from. If it breaks, you will lose all data. I recommend using a separate hard drive either installed in your computer or connected externally through the USB.

I personally have a WD 4 TB hard drive installed on my system that is used for automatic backup, and I also copy monthly backups manually to an external WD 20 TB RAID hard drive. If my computer gets stolen, I will still have a backup of all data on my external hard drive.

You can purchase a hard drive using my Amazon page, where I share the hard drives I use. They all are Linux compatible.

CONCLUSION

Automatic backup in Linux takes only three lines of simple code in your crontab file, but it can be a lifesaver one day. So, do not wait any more, go and configure your automatic backup in Linux.