Snapshotting

Filesystem snapshotting is a process that keeps a "copy" of a filesystem as it stood at a certain period of time. It's especially helpful with backups, so that a specific backup will contain consistent data. It was implemented in later versions of Solaris 8, and we're starting to use it on our Solaris 9 systems.

snapshot.daily, snapshot.1, snapshot.0

I have written three scripts to make the snapshotting code easier:

• snapshot.daily - runs level 4/2 backups and writes them to disk, keeping several days worth of old ones, using ufsdump. Should be run nightly by cron.
• snapshot.1 - as with snapshot.daily, but runs level 1 backups. Should be run only occasionally, such as when a new filesystem is put in place and you just want to have a copy of the data somewhere before you write it all to tape later in the month.
• snapshot.0 - runs level 0 backups, writing them to either tape or a remote filesystem (or remote tape drive) using SSH. Should be run monthly by hand.

These are evolutions of our old backup scripts, and are probably reasonably close to releasable. I haven't made any concrete plans along these lines yet, however.

Most of our systems will run these scripts as the user 'backup'. This, unfortunately, makes the process a bit trickier. Local configuration that is necessary to make this work:

• `/etc/ufsdump.config' - defines the basic variables that a system has to work with to do its backups. Should be world-readable, writable only by root (and perhaps user 'backup'). A sample is included with the encap.
  • Sample Client ufsdump.config
  • Sample Server ufsdump.config
• We need a 'backup' user, which has sudoers permission to run /usr/local/sbin/fssnap, /usr/sbin/fssnap, and /etc/init.d/xntpd, as well as a 'backup' group which the user is a member of. Sample code:

    Host_Alias      BACKUP = [...]
    User_Alias      BACKUP = backup
  
    BACKUP  BACKUP = NOPASSWD: /usr/sbin/fssnap, /etc/init.d/xntpd, \
                    /usr/local/sbin/fssnap
    

• A file server must share the filesystem `/Backup/snapshots', which is writable by 'backup' and should have root-write priveleges. It doesn't have to be very big (a few dozen gigs), but it's necessary for fssnap to write its files to.
• Permissions have to be set to group read/write for `/dev/fssnap.ctl', `/etc/dumpdates', and any disks that are being backed up. The group should also be set to 'backup'.
• The NTP start/stop script in /etc/init.d/xntpd has to properly work.

snapshot-gtar

Run as 'root' and not as 'backup', this script (in /home/backup/scripts) makes a snapshot, mounts it, and then uses tar to backup the filesystem over SSH to one of the file servers, where it is written off to tape.

• `/etc/tardump.config' - like ufsdump.backup, tells what filesystems to mount and how to deal with it. Should be world-readable, writable only by root. A sample is in /home/backup/etc.
  • The SSH configuration, set up in tardump.config, should be able to get to the appropriate file server without requiring a password.
  • Sample Client tardump.config
  • Sample Server tardump.config
• Remember, this runs as root, and not as 'backup'! We couldn't get sufficient file permissions otherwise.
• The biggest advantage: because the script is using the image of the filesystem and not the filesystem itself, the tar backups do *not* modify the file access times. Convenient.
• Unless /mnt/gtar_backup exists, the script may have some problems the first time it's run. This is a bug that I haven't worked out yet (the directory is being created).
• One other major change: old tar backups were performed in one big tape block. Now one tape block (the 'mt fsf' things) is used per file system.

Gotchas

• NTP has to be turned off to use 'fssnap' on the root disk, because it locks the disk in order to do its "real-time" work. It can be turned on again after we're done.
• When run as 'backup', ufsdump can't create the proper files in /tmp on a snapshot image. This isn't the end of the world - I haven't seen any problems with it yet - but it does force an annoying "Unable to create temporary directory..." error mesage.

Current Setup

Currently, nightly backups are being run with snapshot on edmonton. More systems will be added over time.

As of our last tar backups (2003-12), we used the snapshotting code for all filesystems except those on rio, roma, riga, and reno (and those systems just because they're running old versions of Solaris 8).

Contacts

Tim Skirvin put this together.

References

• 'ufsdump' and 'fssnap' man pages