My Perfect Data Backup Solution

This might already exist, and I might just be uninformed. When I look for a backup tool that will do what I need, usually I get a scoff in the form of “rsync, dummy”. My computing equipment typically consists of a laptop that I use everywhere, and sometimes even at home, a headless computer at home that I use mostly for music and movies (with a projector), and several external hard drives with varying capacities.

These are my requirements for an awesome backup solution:

  • When I connect to my home wireless network I want it to automatically start syncing and backing up data to the headless computer.
  • I don’t want it to saturate the network or be too taxing on disk I/O, so that regular computing tasks could be resumed unhindered.
  • I want it to be resumable, so if I leave the house while it was syncing some huge file, it will just continue where it left off when I get back home.
  • A visual status indication of whether a sync is taking place. A way to pause it.
  • I want it to have 3 different kinds of backup modes for 3 different types of data:
  • Home folder
    Snapshot based backups, that allows me to easily roll back and view previous revisions of my home directory.
    Media
    Accumulative backup. See what new music, photos or videos I have on my laptop, and copy them to the backup storage. This mode never deletes media that existed in previous syncs. I should be able to download or create new media, and have it stored on the high capacity storage at home, and only carry with me the media that I am currently consuming (I really don’t want to clutter my laptop disk with The Godfather trilogy, but I might want to take it with me on a flight).
    Virtual machines
    I usually have a small collection of virtual machines on my laptop. They are by far the largest single type of data I have on my machine. Since they support virtual disk snapshots natively, I don’t really care for revisioning like the home folder, virtual machine disks could clobber older versions of themselves on the backup storage.

There are countless backup solutions (read: rsync wrappers) out there. Do any of them answer those needs in a user-friendly way?

My Perfect Data Backup Solution

13 thoughts on “My Perfect Data Backup Solution

  1. Leon says:

    Try Back In Time. It’s rsync based and does the snapshot approach you want for your /home in a very nice and visual way.

  2. Zippy says:

    Most of what you want, you’d get from using a good filesystem like ZFS, which has lean, mean snapshotting built in.

  3. Alex says:

    I am currently working on a system for synchronizing directories between multiple machines. It does not keep track of history much (current design keeps track of moves, copies, and deletions until all participating machines are up to date), and it tries to limit the amount of data transmitted over the network when only a small part of the file changes (should work great for VM images). My main use cases for this system will be synchronization of music and video between my various machines. As I design it, I am trying to make it as user-friendly and idiot-proof as is feasible.

    As for your needs, I think my system will be able to handle media and virtual machines, but not your home directory.

    Personally, the only important things in my home directory are in the Documents, secrets, and src directories. The contents of Documents and secrets are stored in git repositories that are pushed to and pulled from my server. The same is done with stuff in src on directory-by-directory basis. Most directories are stuff I downloaded, so they doesn’t need to be backed up, while others are personal projects that do need to be backed up. I also have a git repository for the home directory itself, but only a few files are in that (.zshrc, .emacs, etc). The server runs gitosis, which makes it fairly easy to accomplish all this.

    Feel free to contact me by email if you need more information.

  4. I’ve split laptop backup into two parts: synchronization and server backups.

    I currently use a manual process of running unison-gtk once per day on my laptop, which does a two-way sync with my server. My server handles backing up to an external disk, which I rotate offsite.

    I don’t directly back up my laptop, since everything on it is easily restorable (install custom meta package to pull in apps, sync $HOME)

  5. @Zippy: No we won’t. No snapshot can rescue from stollen computer, damaged hardrive or formated partition.

    Don’t get me wrong. Snapshots are wonderful features and they do have its uses. They help with backups but they do not replace them.

  6. Palin says:

    I would point you to unison in case 2 & 3, some other work has to be done via scripting to ensure that it starts automatically on connect (I’m thinking of distro-based ways of telling if the network is up).

    For case #1, I would rather snapshot the volume with ext3/4 snapshotting and backing up the whole stuff.

  7. David says:

    You can add another point:

    * Handle external backup devices correctly (disconnected HD, etc.)!
    * Easy to use and configure

    Ok,it is similar to network based storage, but I did not find any program that handle the case correctly…

  8. Fri13 says:

    People should learn that RAID systems or filesystems snapshot features are not backup systems. The only real backup is to copy the data to other media and store it to different location than the original one. And even then should be remembered that when the original data is destroyed/lost, the backup is the original data. Thats why there is always a need to have two duplicated backups from the data.

  9. Luca says:

    Hi,

    if on linux, brfs could be an answer (especially for virtual machines). For less exotic configurations, a combination of BackupPC, rsync and bash/python scripting should do the job. Running dhcp / dns on your server should simplify things.

    Finally, a NAS in a remote location with enough bandwidth (parent’s house? remote storage service?) should handle the disaster recovery situation.

Comments are closed.