1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[Announce] rdiff-backup for NAS200

Discussion in 'Cisco/Linksys Network Storage Devices' started by dsc68, May 3, 2008.

  1. dsc68

    dsc68 Addicted to LI Member

    rdiff-backup is a reverse incremental backup system that works like rsync by only sending changes to files. The benefit of rdiff-backup is that it keeps the current copy of each file plus reverse incrementals that allow a previous version of the file to be recreated. It also preserves all file permissions and ownerships.


    Installing
    To install rdiff-backup:

    1. Install Jac0's alternative firmware and set up ssh as per the instructions
    2. Download and install Python
    3. Download rdiff-backup.tar and copy it to your NAS200
    4. Log into your NAS200 and untar the file in the root directory


    Setting Up

    There is no server side configuration required for rdiff-backup but it does require some trickery pokery to get around the idiosyncrasies of Linksys's firmware.

    The basic problem is that a user can be configured for access by ssh or ftp/web but not both at the same time. This is an issue because it means that backups made with rdiff-backup could only be accessed by the rdiff-backup client which defeats one of the benefits of rdiff-backup.

    There is a workaround though. The explanation is best by example.

    I have a server "holly" that I want to back up.

    First create a user in the Linksys web admin called "holly" with a personal folder.

    Now ssh into your NAS200 as root. Go to /etc and edit the passwd file with vi. Copy the line for the user "holly" and paste in a duplicate. Edit the second entry for "holly" and change the user name to "holly-backup". Leave the uid and gid numbers the same. Change the home directory to /harddisk/volume_1/data/holly and the login shell to /bin/sh. It should look something like this:

    Code:
    holly:sc/YJyfrdbEkA:2000:501::/home/user/holly:/dev/null1
    holly-backup:sc/YJyfrdbEkA:2000:501::/harddisk/volume_1/data/holly:/bin/sh
    To make it possible to do unattended backups we need to configure dropbear to use public key authentication instead of passwords. You will need to have a public/private key pair already generated on your server. Go to /harddisk/volume_1/data/holly and create a directory called '.ssh'. Copy your public key to a new file called 'authorized_keys' in the .ssh directory. Correct file ownership and permissions are essential for this to work properly. Change the owner and group of authorized_keys, .ssh and holly files/directories to holly:holly. Change the permissions to remove group and other write access from those three files/directories as well.

    It should now be possible to ssh into the NAS200 box as holly-backup without having to enter a password.

    Using

    Backups are made with the command:

    rdiff-backup --remote-schema 'ssh %s /harddisk/volume_1/conf/bin/rdiff-backup --server' --create-full-path --print-statistics directory holly-backup@192.168.1.2::holly

    where directory is the directory to backup from and 192.168.1.2 is the address of your NAS200 box. The backup will be stored in a directory 'holly' under harddisk/volume_1/data/holly.

    See the rdiff-backup documentation for more details on excluding files.

    The beauty of rdiff-backup is that the latest version of every file is easily available by ftp'ing into the NAS200 using 'holly' as the username. Files can be also accessed using the Linksys web interface but an idiosyncrasy of the interface is that files must be group readable to accessible. Rdiff-backup preserves file permissions and therefore files that are not group readable on the source will not be accessible via the web.

    Previous versions of a file can be retrieved using the rdiff-backup recovery options.
     
  2. jac_goudsmit

    jac_goudsmit Super Moderator Staff Member Member

    Great stuff!

    Great stuff, dsc68!

    I might want to use this for network backups on my own NAS200.

    Thanks!

    ===Jac

    (PS if you want to save some bandwidth, you can use tar.gz instead of tar. Then, on the NAS200 you can use "tar xvz -f rdiff-backup.tar.gz" to unpack. This is slightly different from the usual command form because this version of Busybox doesn't accept "f" as part of the first parameter).
     
  3. dsc68

    dsc68 Addicted to LI Member

    Just bear in mind it is slow. Calculating all those MD5 checksums is hard work for the tiny processor, especially with SSH thrown in as well. My initial backup is running at 400KB/s (approx 1GB/hr).

    I did experiment with running rdiff-backup using netcat->inetd which was about 4x faster but it kept failing after a couple of hours. And I couldn't get it to work on a smb drive mounted from the NAS200.

    Subsequent incremental backups won't be an issue as my system usually only has less than 100MB of changes per day and I am intending to relocate my NAS200 to an offsite location at the end of a long piece of damp string (aka residential broadband in Australia:thumbdown:)
     
  4. dsc68

    dsc68 Addicted to LI Member

    I've been successfully running backups to my remotely located NAS200 using rdiff-backup for a couple of months now. The NAS200 is located in another city about 300km away and I connect to it via an IPsec VPN configured between two routers. The connection is over a residential ADSL2+ service so it is limited to around 700Mb/s. I typically backup 100Gb in around 3-4 hours.

    A couple of ssh tuning tricks were required to get the system stable.

    First, set the cipher protocol to Blowfish as it has the lowest processing overhead. Encryption isn't strictly necessary over the VPN connection but ssh provides the authentication and connection management services. I have seen a patch to allow dropbear to work with no encryption but it would also require the client side to be modified as well.

    Second, the NAS200 spends a lot of time thinking about all those MD5 hashes it needs to produce. As a result both the VPN connection and the ssh connection were timing out, causing the backup to abort. This was avoided by setting the VPN connection to be always on and using the keep alive settings in ssh.

    The ssh tuning settings are located in /etc/ssh_config on your client (assuming Linux). Just add a section like:

    Code:
    Host nas-rdiff-backup
        ForwardX11 no
        BatchMode yes
        Cipher blowfish
        Compression no
        ConnectionAttempts 10
        KbdInteractiveAuthentication no
        PasswordAuthentication no
        ServerAliveInterval 20
        ServerAliveCountMax 5
        TCPKeepAlive yes
        HostName [I]your.nas.hostname[/I]

    Then use nas-rdiff-backup as the hostname in the rdiff-backup command.
     
  5. adama

    adama Addicted to LI Member

    Hi,

    Thanks for getting this working! I got python installed but the link to the rdiff-backup tarball is not working for me :frown: 404
    I would really like to get this set up at home!

    Thanks again,
    Adam
     
  6. dsc68

    dsc68 Addicted to LI Member

    Link is now fixed.

    I am currently working a custom firmware that will make the NAS200 a dedicated backup device supporting the latest rsync and rdiff-backup. Unlike Jac's firmware, it will not attempt to keep all of the existing Linksys functionality.
     

Share This Page