The Problem:
I had been backing data from my stand-alone server (Mac Pro G5 with Ubuntu Server 13.04 software) to an external USB drive on an iMac, then to a hard drive on a HP laptop with Debian OS, and finally to SkyDrive (a Microsoft cloud storage option). I used a “Norton Commander – like” file manager, MU Commander, to perform backups by copying the directories to each backup location. MU Commander allowed me to overwrite existing files/data if they were older on the storage device than those being copied from the server. That was the extent of the sophistication of my backup process.
Though the process I used was simple, there were problems:
- Changes in the file structure from the sending location (server) would not always be copied to the receiving location (both local drives), for example: when I deleted a file on the source, it would still be on the backup drives after the backup process was completed. In short, the source files and backup files did not mirror each other.
- It was time consuming backing up to the local drives, and it took forever to back up to the SkyDrive.
- I found the SkyDrive software buggy to the point that I had to do some time consuming work-arounds to get my data on the cloud. (As a sidebar after trying several cloud-based storage services I have come to this conclusion: If you are backing up a few gigabytes to them, they are probably useful. On the other hand, if are backing up 50 gigs or more, to me, cloud storage is a colossal waste of time.)
Clearly, I needed a new plan.
The Plan:
I decided my new backup strategy would eliminate cloud storage. I will continue to backup data to both my laptop and my external USB drive. In addition, after backing up data on my external USB hard drive, the drive itself will be removed from my office and stored in my car. This way, should something catastrophic happen to my home/office, I will still have access to all my computer generated data, photos, music, movies, etc. To keep the data on the USB drive secure, I will encrypt the drive. Finally, I will setup a scheduled backup process.
Now that I had the process figured out, I needed to find software that will make this process more efficient, and make data on the backup storage hard drives mirror my the data on my server. After doing some looking around, I could not find software that would do exactly what I wanted. Then a family member recommended rsync, and it was exactly what I wanted.
Here is how it is described from the official website (rsync.samba.org):
“Rsync is a (command line) file transfer program for Unix systems. Rsync uses the “rsync algorithm” which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand. Some features of rsync include: it can update whole directory trees and filesystems, optionally preserves symbolic links, hard links, file ownership, permissions, devices and times, requires no special privileges to install, internal pipelining reduces latency for multiple files, can use rsh, ssh or direct sockets as the transport, (and) supports anonymous rsync which is ideal for mirroring.”
Rsync backup software is open source, and is free backup software! Finding the software was simple. Rsync backup software is natively installed in many operating systems. It was already installed on two of my machines, an iMac with Mavericks OSX, and my Server with Ubuntu Server 13.04. I wasn’t so lucky on my Debian/HP laptop, – I needed to download and install the software. Here’s how to get it on Debian/Ubuntu/Mint Linux distributions: At the command line type, “sudo aptitude update”, then, after adding your password when requested, “sudo aptitude upgrade”, ditto again on the password, and install any new software. Now that the repositories for the Debian are up to date, type “sudo aptitude install rsync”.
Configuring A RSYNC Backup
Rsync has many uses, and can backup files to different folders, folders to different directories, directories to different drives, data to thumbnail drives, external USB drives, and more. Add this flexibility to a large number of different flags and options, and you have a highly flexible backup software that can tackle almost any backup duty. For most of us, the fact that it runs from the “command line” makes it look a little daunting. With regard to some of rsync’s ability to function as a solution for very complex backup issues, the software probably is too daunting for most of us, but for doing “everyday” backups, rsync is pretty easy. Here are two examples of how I use rsync backup:
Example 1. I want to back up my data files (which coincidentally, are my network Samba shares) from my server to an external USB hard drive attached to my iMac. In addition, I want to do this securely and without any potential network authorization issues. As a result, I will be using a secure shell, or SSH (For information about using SSH, click here.) Here is the syntax, entered from the prompt in the Terminal program:
sudo rsync -ah –progress –delete -e ssh larry@prometheus:/mnt/Main /Volumes/250GB/Backup
Here is what is happening when you have completed typing the syntax, press the Enter key:
sudo This is a program that will let me run as a superuser or root. This will allow the user (me) administrative privileges. Though it may not be necessary in all cases, without it, you may not be able to use ssh, or copy, change directories, delete files, etc. with rsync. At this point on the iMac, or Linux, you will asked for you local (the machine you are using) computer password.
rsync The backup program is initialized and is ready for options.
-ah Here, two options are joined together (just a little space saving). The “a”, this is a quick way of saying you want recursion, and want to preserve nearly everything. The “h” is setting the data up to be “human readable”. You can also add “n” here, to perform a test run of the backup, without the software copying or changing anything.
— progress We are telling rsync to print on the monitor the progress of the transfer. If you are backing up a large number of files, and/or a lot of gigabytes, this is useful to get a “feel” for how things are going. Also, it makes your computer look like its running some supposedly high tech program like those used on computer props on the NCIS TV show.
–delete I am always nervous when the word “delete” enters some syntax. In this case, “delete” does some important syncing. “Delete” tells rsync to delete any files on the receiving side that are not on the sending side. That being said, this could still spell disaster if there are files on the receiving side that you want to continue to archive. Did this make you a little nervous? This will make you feel better, add the “n” (perform a test run) option as mentioned above, and you will be able to see what will be added and/or deleted in your backup. In my experience, it is best to test “-n” by doing some sample backups of a few files.
-e By default, rsync wants to use rsh as a remote shell, – by adding the “-e” to the syntax, you are telling rsync you want to use another shell: ssh.
larry@prometheus:mnt/Main At this point, we are telling the rsync backup software where to find the data (directories, folders, files, etc.) we want to back up. By using ssh, we are going to login as “larry”, on a computer named “prometheus” (usually, an the IP address of the target computer can be substituted for the computer name). Then after the “:”, we are telling rsync to go to the sending computer’s “mnt” directory, and backup everything in the “Main” folder. Note: once ssh has access to the sending computer, the user’s password for that computer is requested. Be sure you are a “user” on the sending computer, and have a password on file with that computer or you will not gain access.
/Volumes/250GB/Backup Finally, rsync needs to know where you want to place the copied files. In this example, they will go to the “Volumes” directory. Here, on an iMac, you will find mounted drives for the network, a thumbnail drive, a external USB drive or external firewire drive. The “250GB”, refers to the Volume Name of the external USB drive; so we are now telling rsync software to send the copied data to the drive named 250GB, and place the data in a folder on that external USB drive named “Backup”.
Example 2. Here, I want to backup the same files, but to one of the internal hard drives in my Debian Linux based laptop computer. Here is the syntax:
rsync -ah –progress –delete -e ssh larry@prometheus:/mnt/Main /mnt/Backup
These are the differences from Example 1:
- I did not use sudo. Why? Debian has built-in Superuser terminal, so when I use this terminal I am already a “superuser”, and have administrative priviledges.
- Rsync software in placing the backed up information in the “mnt” directory on the laptop’s internal hard drive, in a file called “Backup”.
The Result:
The data placed on the receiving drives, including file structure and permissions, is identical to the data from the sending location (in this case, my server). Subsequent backups will be incremental, – only backing up changes to the sending machine’s data, permissions, and/or file structure. After completing a backup, the data that was backed up will always mirror the source computer’s data.
Conclusion:
Hard to beat. Rsync backup software is a FREE, feature rich, well sorted out program, that does exactly what you want it to do. Yes, it does not have a pretty GUI, but it does work the way you want it to, time after time, after time……… Finally, as an aside, cloud storage is a great backup solution for a small amount of data, but not for power-users.
Some Additional Information On Rsync:
Wikepedia – rsync http://en.wikipedia.org/wiki/Rsync
“How To Use rsync For Transferring Files Under Linux or UNIX” http://www.cyberciti.biz/tips/linux-use-rsync-transfer-mirror-files-directories.html
“How to Setup rsync Backups on Linux The Easy Way” http://www.howtogeek.com/66348/how-to-setup-rsync-backups-on-linux-the-easy-way/
“Using rsync to Transfer and Synchronize Local and Remote Systems” http://www.liquidweb.com/kb/using-rsync-to-transfer-and-synchronize-local-and-remote-systems/
Hey there, thanks a lot for this guide, I am just setting up rsync-use for myself and your description and weblinks are quite helpful!
Glad you found the article helpful. Here is the “correct” syntax is I use for rsync (WordPress for some inexplicable reason would not add the needed two hyphens in front of “progress” and “delete*):
sudo rsync -ah –progress –delete -e ssh larry@prometheus:/mnt/Main /Volumes/250GB/Backup
I still use rsync, backing my server up to two other computers, and a detachable hard drive that I store in my car when not in use (having lost all my data a number of years ago, I am now paranoid about it. The drive I keep in my car, is for protection in case my desktop and laptop computers are stolen, or destroyed in some natural disaster. I might be paranoid, but if a disaster occurrs, I have a greater chance of not losing my data).
Note: After posting this, I find that WordPress will not provide me a way to display the proper syntax for rsync, soooooooooooo…..there are two hyphen’s in front of “progress” and two hyphen’s in front of “delete”. The rest of the syntax is correct. Sorry for the confusion.
Larry
Hi,
Just reading this and wondered if the rsync command can be used to copy the entire contents of one drive to another including hidden files?
Chris,
In short, it’s doable. Using some other software to clone the drive might be a better choice. Which way to go probably depends on why you want to copy the entire drive. Rsync for example, won’t capture (at least, not that I am aware of…) the Master Boot Record of a drive. In any event,rsync is a great backup tool.
Larry
Hi Larry,
I gave it a try to copy the contents of my Documents folder to another Networked drive but to copy just 45Gb was taking forever. I’ll stick to Carbon Copy Cloner as that it a million times faster using a disk image process even though it makes calls to rsync to get the job done.
Thanks for responding.
Chris