Personal Backup

It is against DreamHost policy for you to use your DreamHost file storage space on a web server for personal file storage that is not related to one of your websites.

However, you get a free 50 GB (and more for a fee) of space to backup your personal files on DreamHost's dedicated "backup" servers, with the following limitations:

• it's a separate ftp (or sftp) user on a separate server
• it can't serve any web pages
• There are no backups kept of these backups (they should already be your backups, not your only copy). It is kept in a RAID array, but they are always online and vulnerable to a virus or disaster that physically destroys the server. (Ref)

If you go over 50GB, extra space is 10 cents a GB a month.

If you have backup files on your DreamHost storage that you need to move to the backup server (say, you got a scary mail saying "Please move your files off the main servers, like NOW, or we will have no choice but to VAPORIZE them! Because they're taking up space and stuff."), please see: moving to the backup servers.

Technical notes

Technical notes:

• To enable this service, you need to set up a Backups User Account in the DreamHost Web Panel
• This user does not have a shell on the server – instead, it uses rssh to allow limited commands
• The following commands can be invoked on the server: scp sftp rsync

  (Bazaar, maybe others, can push changes over sftp w/out a client on the server. I haven't tried it yet, but is there any reason why it wouldn't work for this?)

• This backup feature was introduced in October 2008.
• Per DreamHost support, "gigabyte (GB)" means "gibibyte (GiB)" = 2³⁰ bytes. In detail, 50GB = 429,496,729,600 bits / 53,687,091,200 bytes / 52,428,800 kilobytes / 51,200 megabytes / 50 gigabytes. This is over 7% more than the "50 GB = 50 × 10⁹ = 50,000,000,000" using the so-called "GB" you may see on hard drives!

If you have any other questions or concerns, please do not hesitate to contact us.

Automated Backups

This section is for advanced users and requires some technical knowledge!

Password-less login

See: Password-less login

To really automate backups, you’ll need to set up password-less login – this is similar to usual setup, but with a few wrinkles, primarily due to lacking a shell (and instead using sftp). Due to the restricted shell, you can’t just use ssh-copy-id, and instead have to do it manually.

As usual, the goal is simply to have a file ~/.ssh/authorized_keys containing your public key (id_dsa.pub or id_rsa.pub).

• First, you’ll need a .ssh directory.

Most easily, you can sftp into your backup account and do: mkdir .ssh. Your session should look something like this:

sftp bXXXXX@hanjin.dreamhost.com
Connecting to hanjin.dreamhost.com...
bXXXXX@hanjin.dreamhost.com's password:
sftp> mkdir .ssh

• Now you need to upload an authorized_keys file.

Take a copy of your public key, rename it to authorized_keys and upload it into the .ssh directory you created on your backup account. If you want to backup from multiple machines, you’ll need to create an authorized_keys file that contains all of the public keys, and then upload that. You won’t be able to cat new keys onto the existing file at the backup host, because you don’t have a shell and can’t run cat, much less any more dangerous commands!

Most simply, your session can look like this (from your home computer):

cd ~/.ssh
scp id_dsa.pub bXXXXX@hanjin.dreamhost.com:.ssh/authorized_keys

Alternatively, if you have password-less login running on your regular web hosting at DreamHost, just copy that file over (from DreamHost):

cd ~/.ssh
scp authorized_keys bXXXXX@hanjin.dreamhost.com:.ssh/authorized_keys

Change appropriately if you have an RSA key or want to use a different authorized_keys file.

Next, check permissions – most basically ~/.ssh must have permissions 700 (user read/write/execute) and ~/.ssh/authorized_keys must have permissions 600 (user read/write). You can set these using the chmod command in sftp.

More subtly ~ (your home directory at the backup host) must not be writable by anyone other than you (in letters go-w makes it not (-) Writable by Group or Other; in numbers permissions 751 works). This can get messed up if you use rsync with the target the home directory (if your rsync target is something like bXXXXX@hanjin.dreamhost.com: or bXXXXX@hanjin.dreamhost.com:~), and that’s a little tricky to fix b/c you can’t use ~ as the argument for chmod, you might not know the full name of your home directory and can’t actually list the parent directory (b/c it has other peoples’ accounts in it!), so you can’t easily check the permissions. You can fix this by using pwd to get your home directory name, and then blindly set the permissions.

A sample session looks like this:

sftp bXXXXX@hanjin.dreamhost.com
Connecting to hanjin.dreamhost.com...
bXXXXX@hanjin.dreamhost.com's password:
sftp> chmod 700 .ssh
sftp> chmod 600 .ssh/authorized_keys
sftp> pwd
/vol/shelf6/customized/bXXXXX    (or whatever)
sftp> chmod 751 /vol/shelf6/customized/bXXXXX

If the file is correct and the permissions on all these files and directories are correct, it should work!

Programs

Personal backups can be automated using Rsync Backup, a Unix program that supports differential backups (backing up only data that has changed since the previous backup). Windows users can install Cygwin, a Unix environment and bash shell for PCs. Another option is to modify DeltaCopy, which makes Cygwin and Rsync configurable in Windows, to run on a schedule (daily, weekly, etc.).

Also try a fantastic program called Backup-Manager for automated backup. Once configure, it will tar, zip, encrypt and upload on a schedule, automatically. See the web site for its many, many great features.

The backup directory can also be mounted locally with SSH Filesystem and accessed with the same flexibility as your own file system. Another trick to work on top of sshfs: mount an encrypted file system inside the sshfs mount point with EncFS, which will provide a transparent layer of encryption.

How to efficiently back up multiple incremental snapshots

Rsync is easy to use for creating a single mirror of your personal files. However all local changes (including accidental file deletions or modifications) will be propagated to the remote server overwriting your old files on each backup. That defeats the purpose of having a backup in the first place. Better would be a series of snapshots of your data taken at regular intervals, but storing multiple full copies of your data is inefficient since most of the data won't have changed significantly and you'll have to transfer more than just the "deltas" which defeats the purpose of using rsync in the first place (you might as well just use sftp). Fortunately, using hard links it's possible to create multiple snapshots that take up essentially no space except for the files that have changed. Furthermore, after the initial transfer you won't have to transfer more than the "deltas" so backups will be quick and painless. This ingenious method is described in excellent detail by Mike Rubel on his rsync snapshots page. Unfortunately Dreamhost Backup users do not have shell access to execute the "mv" and "rm" commands required to implement this solution. Fortunately it's possible to duplicate the "mv" and "rm" functionality required by combining some rsync and sftp capabilities. This clever work-around is described here.

• There is also a blog post "RSnapshot-style backups on Dreamhost" and a script called "dhsnapshot" to set this up. --Carlos.lima 13:04, 2 March 2010 (UTC)

CyberDuck access

By default the Mac FTP client Cyberduck has a default Path of / - to access your files you'll need to remove that slash (in your bookmarks choose More Options and clear the Path field).

Software

Software that can be used with Personal Backup

Windows

• Cobian Backup (Open Source)
• CWRSYNC Is a packaging of rsync, ssh client and the required cygwin libraries for Windows. It runs on all versions of Windows.
• SyncBackSE (Not Free) works with FTP but not directly with secure FTP or SCP. However, you can do this by combining with Tunnelier and using Tunnelier's FTP host option that provides a local FTP service that will forward to a secure remote server.
• rdiff-backup (Open Source) uses rsync and ssh, use with cwrsync on Windows. It is able to keep unlimited versions of backup so that you can restore from any point in time. rdiff-backup cannot be used with dreamhost backup server. It must be installed on both the client and server (and it isn't).

GNU/Linux

• Backup-Manager
• rdiff-backup (Open Source) uses rsync and ssh. It is able to keep unlimited versions of backup so that you can restore from any point in time. rdiff-backup cannot be used with dreamhost backup server. It must be installed on both the client and server (and it isn't).
• duplicity (Open Source) uses rsync, sftp, or scp. Encrypted differential backup. Similar to rdiff-backup, but works quite nicely with this offering.

Moving to the backup servers

If you wish to move files from the main DreamHost servers to the backup servers (and also update your backup scripts) – say, you've been using DH for backup or as your own file locker, you can follow the following easy steps:

Set up and move data:

• Set up backup user
• Copy your data over (from DH to Backup)
• Verify it (from DH)
• Verify it (from your home copy)

  (you DO have a home copy, don't you?)

Fix backup scripts:

• Change automatic backup to point to Backup
• Test automatic backup

Finally:

• Check everything one more time
• Delete DH copy

Icing:

• Check for any other big data

du -chs * | grep G

...and move or remove it

In detail

• Set up a Backups User Account in the DreamHost Web Panel

  You do not get to choose your username; it’s b??? where ??? is a number (your user number?), and “b” is for “backup”.

• rsync the files from DH to backup

If you’ve been storing files in ~/backup then from DreamHost, the following command should work (with the proper user and host names):

rsync --archive --verbose --progress /home/DHUSER/backup/ BUSER@BHOST:

HOWEVER, this command will change the permissions on your home directory on the backup user (easily fixed) to whatever the permissions on ~/backup were, and you will need to fix this to have password-less login work.

• rsync again to see that the files have been moved

(Same command again, should show no files move.)

Fix your backup scripts (if any, say backup up from home):

• first, change rsync to "--dry-run", so you don’t accidentally nuke the files by messing up the path
• change the backup target (remote computer) to point to the backup servers, where they’re supposed to be, instead of DreamHost web hosting
• try running it – it should check all the files, but predict no behavior (b/c the backup is ok, right?)
• turn off dry-run, and run rsync again just to check

Finally:

• when you’re really, really sure your files have been safely moved to the backup servers and you have your local copies, delete the files at DH

You can also set up password-less login, as discussed in section above, or fix it if it’s gotten messed up.