\documentstyle{article} \title{The OSU-CIS Backup and Restore System} \author{Steve Romig} \date{March 15, 1994} \pagestyle{headings} \begin{document} \maketitle \tableofcontents \newcommand{\be}{\begin{enumerate}} \newcommand{\ee}{\end{enumerate}} \newcommand{\bi}{\begin{itemize}} \newcommand{\ei}{\end{itemize}} \newcommand{\bd}{\begin{description}} \newcommand{\ed}{\end{description}} \newcommand{\bc}{\begin{center}} \newcommand{\ec}{\end{center}} \section{Introduction} This document describes the The Ohio State University Computer and Information Science (OSU-CIS) Department's backup and restore system. The system still doesn't have a name so I'll just refer to it here as ``the backup system''. This document describes version 2.6 of the backup system. The backup system consists of a set of Perl scripts (and 1 small C program) that provide what is intended to be a comprehensive and configurable set of tools for doing incremental backups and restores. The main design constraints were that it should be easy to use, robust in handling errors, complementary to human skills and shortcomings, and that it should use native backup programs. ``Easy to use'' means that the programs that are used most, {\tt all-backups} and {\tt getback}, are user friendly in that they convey a consistent and simple interface to handling backups on many and diverse systems. If you fail to provide necessary information on the command line, they'll ask for what they want. An on-line database is used to make tape management and restores simple. ``Robust'' means that the system tries really, really hard to deal with errors. Programs that might damage or change the tapes or the online database check to ensure that the right tapes are being used in the right drives. There are all sorts of internal consistency checks intended to find bugs and problems in the database. It ain't perfect, but its better than a crufty set of shell scripts with little or no error checking. ``Complementary to human skills'' means that it tries to make up for what people are bad at, and tries to avoid what people are good at. It seems hard to write a program that could schedule backups and correctly take into account tape drive use, drive status, and operator schedules, so that is left to the users (if you have ideas about how to automatically schedule backups, assign tape drives and so on, please tell me about them!) People are forgetful and sometimes forget to do backups, and its easy to tell when one is overdue, so there's a {\tt check-backups} program that will complain about overdue backups. ``Uses native backup programs'' means that it should be easy to use these tools with just about any sort of archiving program and media. There are a plethora of file archiving programs and file system types. You've got short and long name file systems, BSD Fast File Systems, file systems that support strange objects like context dependent files or access control lists, and strange proprietary database file systems that have their own weird backup programs. You probably already have archiving programs (variants of {\tt dump}, {\tt tar} or {\tt cpio}) that can ``correctly'' save and restore these things. What you are probably missing is a framework that handles tape management and error handling and all that stuff. This backup system supplies that framework, allowing you to use the most appropriate archiving program for each file system without rewriting the rest of the backup system. This version supports incremental backups in multiple chains on multiple hosts, tape types, fairly easy restores, backups to tape or disk or both, parallel backups, event logs, offsite tape storage, and even more error checking than the previous versions. No, it still doesn't allow backups to span multiple tapes. Sorry. See section~\ref{future} for details about this and other future plans. This manual describes the backup system (section~\ref{overview}), how to install it (section~\ref{install}), how to do backups (section~\ref{backups}) and how to do restores (section~\ref{restores}). I'm aiming this document at describing the overall system and the philosophy behind it. More detailed information about the various programs is available in their man pages. \section{Overview of the Backup System} \label{overview} The backup system consists of backup, restore, and database management software. The programs that are intended for human use are {\tt all-backups}, {\tt getback}, {\tt add-tape}, {\tt move-tape}, {\tt show} and {\tt reuse}. In addition, {\tt run-clean-db}, {\tt offsite}, {\tt check-backups}, {\tt backup-report} and {\tt list-backups} are utility programs that are usually set up to be run by {\tt cron}. Everything else is used by those programs in some fashion or another. \subsection{Tapes} Let's start by talking about tapes. Each tape to be used by the backup system is registered in the tape library through the {\tt add-tape} program. {\tt Add-tape} writes a tape label at the start of the tape that includes a unique tape identification name. This name is used to identify tapes so it should be written onto the physical tape label in some fashion - I suggest writing the tape ID in ink. The tape label also contains information about what the tape was most recently used for so that if the physical tape label falls off you should still be able to identify the tape by reading the first file (it is a text file written in a simple format). The {\tt add-tape} program also enters the tape into the tape library database so that it can be used for backups. Normally all you will need to do to maintain the tape database and tape library is to add new tapes as required and replace tapes that have worn out. The {\tt add-tape} program can be used to add new tapes and to relabel old tapes whose tape labels are bad. The {\tt reuse} program is used to tell the tape management system that a particular tape can be reused right away rather than waiting for it to be recycled when the backups on it expire. This is mostly used when a backup run fails and you want the tape to be available for other backups. Each tape has a tape ID. The ID consists of a prefix of some sort and a number, such as ``exb-10'', ``rack1-12'', or ``test-tk50-123''. The prefix defaults to the type of the tape, but can be any arbitrary string (you can use the {\tt -p} option to {\tt add-tape} to set the prefix for a tape that you are adding.) The number is used to make the IDs with the same prefix unique. The prefix defaults to the tape type so that you can store tapes of the same type in a rack in sequential order (otherwise tapes 1 through 9 might be 8mm, but tape 10 is a tk50 and 11 is a 9 track reel - how would you rack them?) The system allows you to change the prefix so that you can make up your own racking system. Each tape also has a location and a home. The home is where the tape is supposed to reside when it isn't offsite and the location is where the tape currently is. If you have tapes that were added with pre-2.6 software, the location and home fields for those tapes might not be set. The location and home are mainly used for offsite tape storage and to ensure that {\tt all-backups} will use tapes from the free tape pool in the same location as the tape drive. \subsection{Database} The backup system maintains an on-line database that covers the tape library and all of the current backups. The databases are implemented as {\tt dbm} files accessed as associative arrays from within Perl. If you are backing up more than one host, you can share the database through NFS. Unfortunately the {\tt dbm} library that is normally used with Perl is byte order dependent and will not work correctly if you try to share the database files between hosts that don't use the same byte ordering. The BSD {\tt db} library provides a {\tt ndbm} library interface but is written to support byte order independent files that can be shared through NFS. All you need to do to use it is to construct a version of Perl that uses that library instead of the usual {\tt dbm} or {\tt ndbm} library and convert your existing database files from {\tt dbm} format to the new format. There are a few things you'll need to change in the configuration files - see the installation instructions (section~\ref{install}) for details on dealing with this. If you can't use NFS, then it should be possible to rewrite the database module (all the database routines are in one file) to allow access through some network connection or something like that. Contact me for more information - some people have started working on something like that already. More information about the internal structure of the databases can be found in appendix~\ref{database-internals}. The on-line database records information about: \bi \item each tape, how old it is, how often it's been used, what it is currently being used for, what its type and current format is, what drive it was last written in. \item each backup run (a collection of one or more backups that were made through a single invocation of {\tt all-backups}). What level of backup this is, what backup chain this is, which hosts were included in the run, when the run was started and when it finished. \item each backup. What run it was part of, what file system on which host, what span of time that the backup covers, what tape and which file on that tape, where it can be found online (if at all), what the status is, and what type of archive it is (e.g. {\tt dump} or GNU {\tt tar}). \ei The database information and tape labels are used all over the place to make things easy to use and relatively safe from accidental mistakes. For example the tape labels are checked prior to using the tape to make sure that the correct tape has been inserted in the tape drive to keep you from accidently overwriting the wrong tape. The backup information is used to make the process of restoring files easier by finding the appropriate tapes for the task for you. The databases are maintained automatically through the {\tt run-clean-db} program that is (supposed to be) run nightly by {\tt cron} from one (1) host. The {\tt run-clean-db} program expires old backups and runs from the backup and run databases. Once it knows which backups and runs are still current it frees any tapes that no longer contain current backups so that they can be reused for new backup runs. \subsection{Programs} Finally we come to the backup and restore software. Backups are initiated with the {\tt all-backups} script. {\tt All-backups} can be used to start backups on one host or many hosts. Like most of the rest of the software it is set up to ask you questions about what you want to do if you don't supply the necessary information on the command line, so it should be pretty self-explanatory to use. The collection of backups initiated through one invocation of {\tt all-backups} constitutes a backup run and is identified by a unique run ID number. The {\tt show} program can be used to view the contents of the databases associated with a particular run or to view the entries in the tape library. The restore program is called {\tt getback}. {\tt Getback} is intended to be very easy to use. You tell it what host and file system you want to restore the files from and it tells you what backups to use and walks you through the process of extracting the files. It supports three ``modes'': interactive, full and pick. In interactive and full restore modes {\tt getback} selects what it thinks is the best sequence of backups to use to restore files from and prompts you to insert the corresponding tapes, one by one, starting with the lowest level and working toward higher levels. The only difference between the two is that full mode extracts everything on the backup, and interactive mode allows you to use {\tt restore i} to selectively add and extract files from each backup (this only works with backups made with {\tt dump}). Pick mode shows you a list of all of the backups for the host and file system that you are interested in, and allows you to select which backups to extract files from. The {\tt check-backups} script can be used to check to see what backups are due to be done on a particular host. I suggest setting it up to run from {\tt crontab} nightly on each of the hosts that is to be backup up where it can serve as a reminder of missing backups for those who are supposed to do backups (and their supervisors :-). The {\tt list-backups} program prints a summary of all of the backups contained in the database. You can print this list out periodically and use it to do restores if the machine that hosts the backup database is down or if the backup database is damaged. If you store some of your tapes offsite, you can use the {\tt offsite} program to generate a report for each offsite storage location that lists the tapes that should be in storage there, what tapes should be checked out to that site (and then taken there), and what tapes should be brought back and check in. The {\tt move-tape} program is used to update tape locations as you move them to offsite storage and bring them back again. \subsection{Configuration Files} You're probably wondering how this whole thing is configured and controlled. There are four sources of configuration information: \bi \item Per-host installation information in the {\tt global.defs} and {\tt local.defs} files, which controls the locations and general operations of the programs. \item The global {\tt backup.config} file in the database directory, which controls overall backup operations (what levels, how often they should be done, etc.) \item The {\tt \$db\_dir/backups} file which controls which file systems get backed up, for which levels, and how. \item \begin{sloppypar} The command line on the {\tt all-backups} command, which determines which hosts will be included in each backup run, what chain and level to use, what tape drive, and how the run should be executed. \end{sloppypar} \ei See the section on configuration (section~\ref{config}) for more details. \subsection{Features} \label{features} \bd \item[Centralized Control] This system makes it easy to initiate backups on multiple hosts in a consistent fashion while allowing for individual variations among the hosts. ``Consistent fashion'' means that the same chains and levels are run on the various hosts with the same frequency, using a common tape library. ``Individual variation'' means that its still easy to determine which file systems should be backed up, what should be used to back them up, and what levels should be allowed for each chain for that file system. \item[Easy To Configure] \begin{sloppypar} The system is controlled by 3 global configuration files ({\tt global.defs}, {\tt backups} and {\tt backup.config}), and by one host-specific file ({\tt local.defs}). \end{sloppypar} \item[Tape Aliases] You can define convenient aliases for tape host and device specifications, which makes it more convenient for both backups and restores. \item[Tape Type Support] The system can deal with information about tape types and formats to make it easier to store your tapes neatly and to correctly ascertain that it is using the right kind of drive for a given tape. \item[Host Groups] You can also give groups of hosts a name, which makes it easy to start a backup run on all of them. \item[Tape Management] The system deals with tape allocation and recycling automatically, and will expire tapes as they become too old or overused. You just need to add more tapes as required. \item[Online Record of Backups] \begin{sloppypar} The backup system maintains an online database that records what each tape has been used for, what backup runs have been made, which hosts were involved in each run, what file systems on those hosts were backed up, when the backup finished, what the state of the backup was, what errors occurred, and what tape and file number the backup can be found on (and/or disk file). This is used wherever possible to make it easy to find and restore from backups. \end{sloppypar} \item[Error Checking] Each tape has a tape label at the start that is used wherever necessary to ensure that you are using the correct tape in the correct tape drive. This helps prevent accidental overwriting of tapes with {\tt add-tape} or {\tt all-backups}, or incorrect restores through {\tt getback}. \item[Backup Checker] The system provides a program that can be used to automatically warn you when a backup is due on any of the file systems that are supposed to be backed up. \item[User Friendly] The most commonly used tools will ask you for needed information if you don't supply it on the command line. All of the prompts give help if you type ``?''. The restore system helps you select backups to restore from and skips to the correct file on the tapes after you load them. \item[Multiple Independent Backup Chains] \begin{sloppypar} This system provides the ability to make several independent backup chains, to help avert the problems caused by missing or damaged backup tapes. \end{sloppypar} \item[Multiple Archive Programs] This system currently supports the use of both GNU {\tt tar} and BSD {\tt dump}, and it is fairly easy to add support for other programs. This means that you should be able to coordinate backups of special file systems (such as database partitions) along with the backups of your normal file systems on the same tapes. If you are really paranoid about backups, you can also do parallel chains with different archive programs to effect more complete independence among your backups. \item[Flexible File System Configurations] You can choose what chains and levels apply to each file system, and what archive program to use. This means that you can do only chain 0 full saves on one file system using {\tt dump}, and all chain 1 backups on another file system using GNU {\tt tar}. \item[Concurrent Backups] You can do multiple backups in parallel on many hosts and have the results copied sequentially onto the tape as they are completed to save time. \item[Backup During Boot] You can schedule backups to be done automatically while the file systems are quiescent during the early part of the boot process. \item[Reschedulable Backups] You can schedule backups to start at a later time. This means that at the end of your working day you could set up a full save to run at 5:00 A.M. the next morning in single user mode, and sleep through the whole thing. \item[Incremental Safety Latches] The system tries to detect situations where a full save should be done instead of an incremental, and warn you about them, for example if a full restore was done more recently than the latest full backup, or if it thinks a lower level backup has never been done. \item[Fine Control Over Backup Runs] You can use ``tags'' to control whether to include or exclude certain file systems from a backup run, so that you can split the file systems of several hosts onto separate tapes to ensure that they will fit. \item[Online Backups] You can do backups to tape, disk or both. If a backup is found online somewhere, the restore facilities will use that instead of the tape copy. \item[Multiple Configs on Same Hosts] You can use the {\tt config-name} entry in the {\tt backups.config} file to give names to separate incarnations of the backup system, so that you can run a production version of the system (config-name set to ``'') and a test version (config-name set to ``test'') at the same time without interference between them. \item[Online Log File] The tools in the backup system maintain a online log of important events. Entries are added when tapes are added to the database, when backup runs are started, and when restores are done. \item[Offsite Tape Storage Management] The backup system provides support for offsite tape storage. A simple program tells you what tapes to take offsite and to bring back. The backup and restore programs take a tapes location into account to avoid using tapes that are offsite unless you specify otherwise. \item[Cute Graphs] If you use BSD {\tt dump} for your backups you can use the graph making stuff to make interesting graphs that display run sizes and durations, backup sizes for each file system, tape use and so on. This is somewhat useful for planning - you can find out how much data you are putting on your level 2 backup tapes, for example. \ed \section{Installation} \label{install} This system is intended to be used in an environment where multiple hosts share a common database of tape and backup information. So there's one database that has to be set up in such a way that it will be available to all of the hosts in question. And each host needs its own copy of the programs and scripts. I suggest that you read carefully through all of these instructions before starting to follow them. \subsection{Things to do once...} The following things should only need to be done once. \be \item CD to the {\tt src} directory. \item If you have RCS, you might want to make sure that everything is checked out of the RCS subdirectory by running {\tt co RCS/*}. \item Edit the {\tt global.defs} file. This defines the names and locations of the backup database and other global information. The things that you will need to change are: \bd \item[\$our\_timezone] the local time zone. \item[\$default\_tape\_type] the default type for tapes. \item[\$default\_tape\_format] the default format for tapes. \item[\$backup\_dir] this is the default location for on-line backups. \item[\@online\_repositories] This is a list of directories where online backups are kept. This is used in {\tt run-clean-db} so that it can find them and delete any expired backups when it cleans house. This should include {\tt \$backup\_dir} and any other online backup directories that you use. \item[\$db\_dir] where the database files are. \item[\$db\_server] the name of the host where the database files are really located. \item[\$db\_ext] this should be the extension that your {\tt dbm} routines puts on the {\tt dbm} files. If you are using {\tt dbm} or {\tt ndbm}, this should be ``dir'', if you are using the BSD {\tt db} library use ``db''. \item[\$db\_ext2] this should be the other extension that your {\tt dbm} routines puts on the {\tt dbm} files. If you are using {\tt dbm} or {\tt ndbm}, this should be ``pag'', if you are using the BSD {\tt db} library use ``''. \item[\$db\_server, \$db\_remote, \$db\_local] are the host name, export path and local mount point for the file system that contains the backup database. These are used within {\tt single-begin} and {\tt single-end} to mount the file system so that the programs can access the database when doing backups in single user mode. \item[\$bdir] where the backup programs are (same as BDIR in the Makefile). \ed You shouldn't need to change anything else, but I do recommend that you read through the file and see whether the default settings are what you want. You should also check the names of the database files in {\tt global.defs} to see if they match your current names. If you are running on a short name file system (where file names are limited to 14 characters or less), the default database names will probably cause problems for you. You might be able to work around the limitations by making the database names shorter. Note that in the case of the per-host run and backup databases, the names of the hosts will be added to each file name. Note also that I have changed the names from the values used in release 2.5, so if you haven't upgraded since then, you'll either need to change the values in {\tt global.defs} or rename your database files to match the new names. \item Take a look at the {\tt local.defs} file also. I believe that I have it set up in such a way now that you can use the same {\tt local.defs} file on just about any version of UNIX. You may need to make minor adjustments for different types of hosts. If this is a System V Release 4 host then set {\tt \$svr4} to 1. Check the {\tt @fs\_types} array to see if your systems file system types are listed (these are the ``type'' values that you find in the {\tt /etc/fstab} file. You will need to change {\tt \$bitch\_prog} to something suitable. This is executed when the tape checker finds that the wrong tape is in the drive. You should read through the rest of the file and make sure that the defaults apply to your host. If you are using GNU {\tt tar} version 1.10 or greater you shouldn't have to change anything, otherwise you will have to adjust the various GNU tar flags. If the local version of {\tt dump} can handle remote device specs like {\tt user@host:device} then change {\tt \$dump\_does\_remote\_users} to 1. If you are using Ultrix but you are NOT using the default dump that comes with the Ultrix distribution, you need to change {\tt \$ultrix\_dump} to {\tt 0}. If you don't have GNU tar installed then comment out the definition for {\tt \$gtar\_prog}. \item \begin{sloppypar} Set up {\tt clean-db} to run nightly on the host that holds the backup database. It should be run on the backup database since it needs to have guaranteed access to the database files if it runs. This should be set up to run at a time when it is unlikely that any backups will be run to avoid database contention since it locks several databases, possibly for a longish time. There shouldn't be any problems if it is run while a backup is being done however. You might want to use the {\tt -v} flag in addition to the {\tt -y} flag and pipe the output into {\tt mail} to send it to a list of interested parties. This can produce lots of output when run on a large database, however. There's a program named {\tt run-clean-db} that you can use to run {\tt clean-db} once a day, logging the output to a subdirectory of {\tt \$db\_dir}, and sending all the interesting bits on through mail. We run it like this: \begin{verbatim} 0 1 * * * /usr/local/utils/nbu/run-clean-db \end{verbatim} \end{sloppypar} \item Set up {\tt list-backups} to run nightly on some host. {\tt list-backups} prints a report showing the known backups for each file system. The idea is that you print this daily and keep it in a log book for reference in an emergency when the online backup database isn't available. Since the report can get rather large I suggest that you print the whole thing once a week and then print incremental updates once a day. We do something like the following: \begin{verbatim} 0 6 * * 0 list-backups | lpr 0 6 * * 1 list-backups -N 1 | lpr 0 6 * * 2 list-backups -N 2 | lpr 0 6 * * 3 list-backups -N 3 | lpr 0 6 * * 4 list-backups -N 4 | lpr 0 6 * * 5 list-backups -N 5 | lpr 0 6 * * 6 list-backups -N 6 | lpr \end{verbatim} which prints the whole thing on Sunday, then prints 1 days worth on Monday, 2 days worth on Tuesday, etc. We keep Sunday's and today's listings in the notebook and throw the rest away. If you use off site tape storage, you should make a copy of the {\tt list-backups} reports and store them at each offsite tape storage location for use in disaster recovery. \item You can use {\tt backup-report} to print a table that shows the latest backups in table form, where you can easily recognize ``gaps'' in the backup coverage and take appropriate actions. We run it once a day as: \begin{verbatim} 30 6 * * * backup-report -N 14 | lpr \end{verbatim} \item If you are doing offsite tape storage, then you should set up the {\tt offsite} program to run daily on some host (probably the same one that runs {\tt run-clean-db}). The {\tt offsite} program generates a report that lists the offsite storage locations, the tapes that should be stored at each, a list of tapes to take to each location, and a list of tapes to be brought back from each one. You could set up the {\tt crontab} entry to mail the report to interested parties, as in: \begin{verbatim} 0 14 * * * offsite | mail -s "Offsite Storage Report"\ romig fine \end{verbatim} \ee \subsection{If you are creating a new database...} If you are creating a new database directory for this installation then please follow the following instructions. \be \item Create the \$db\_dir (the database directory) on a suitable server. You'll also need to create \$clean\_db\_log and \$error\_dir (these are subdirectories of \$db\_dir by default). The backup programs run as user root, group backup. The database files and directories are set to be owned by group backup, mode 770 (660 for files) so that the programs can still write them but so other users can't read them. Change \$db\_dir, \$clean\_db\_dir and \$error\_dir to be owned by group BACKUP (or whatever), mode 770. You could set things up to allow read access to OTHER but if I ever add the ability to maintain an on-line list of what files are in which backups then such open access to the database might be inappropriate, since making them world readable would allow everyone to see the names of everyone else's files. Note that the \$db\_dir must be available on each of the hosts that will use this backup system, under the name \$db\_dir. You could do this through NFS, for instance. If the database directory cannot have the same name on all hosts then you will have to adjust your {\tt global.defs} file on a host-by-host basis. The hosts that access the database directory through some sort of network file system will be running programs as root, group backup. These programs must be able to write the database files, and you may need to work some magic to get that to work right. If you are using NFS to share your database directory, the server that holds the database must meet one of these requirements: \be \item It has a clean implementation of NFS that allows programs to access files according to group write permissions even when running as root. SunOS 4.1 has a bug wherein root cannot access files even if it is running with the group that has specific write access to the file. To test, create a directory on the server in question owned by group backup, mode 770, and a file within that directory, group backup, mode 660. Make a setgid version of cat, group backup on some client that mounts that file system. Run that cat with the pathname of the file you just made. If it works, you're set. If it doesn't you'll have to either find a server that works or revert to method b. \item It allows root access for all of the hosts that will be backed up. You can do that either by setting a {\tt root=server1:server2:server3} line in your exports file (if supported) or by adding an entry like ``anon=0'' to the appropriate entry in your {\tt /etc/exports} file. \ee \item Copy {\tt samples/backup.config} to \$db\_dir and edit it. Refer to the man page on {\tt backup.config} for details. See the section on configuration (section~\ref{config}) for more details and ideas. \item Copy {\tt samples/exclude} to \$db\_dir. This lists files that should be excluded from the backups. The distributed exclude file keeps the system from backing up files named buid-NNN. This is actually used only by the GNU {\tt tar} backup code so if you don't plan to use that you don't need the exclude file. \ee \subsection{If you already have a database...} If you have an existing database directory and you are just updating the software then there are still some things that you need to do. \be \item The entries in the {\tt backup.config} file have changed somewhat. These are the changes in version 2.5: \bi \item The {\tt level} entries now mention the chain number. \item There are {\tt tape-type} entries, and the {\tt expire-uses} and {\tt expire-age} entries have gone away. \item The {\tt tape} definitions now include the type, read and write formats. \item There's an optional {\tt config-name} entry that you can use to set up a test version of the software. \ei In version 2.6, the following changes were added: \bi \item There's a {\tt location} entry which must be used to declare each tape drive/storage location. \item The {\tt tape} entries now include {\tt status} (``yes'' if ``mt status'' reports the correct file and block number, ``no'' otherwise) and {\tt location} (the location of this tape drive). \item The {\tt default-tape} entries are now specific for particular tape types and formats. \ei \item If you are running version 2.4 software or earlier, then you will need to use the 2.5 database conversion programs (contact Steve Romig, romig@cis.ohio-state.edu, for details). If you are running version 2.5 software, you don't have to do much. I've changed the names of the sequence files to match the corresponding backup database files - this will be handled automatically by {\tt run-clean-db}, though you will have to delete the old {\tt \*-id\*} files by hand. I've also changed the names of the database files in {\tt global.defs} to eliminate the superfluous ``.db'' - you can either change the names in {\tt global.defs} to match the old names, or rename your database files to match the new names. \ee I suggest that you make a copy of your existing database, install the new software, edit {\tt global.defs} and {\tt local.defs}, rename your backup database files, delete the old {\tt \*-id\*} files, and run {\tt clean-db -v} to create the new sequence files. If there are any mismatches between the real file names and the names in {\tt global.defs}, it should become painfully apparent, and you can interrupt {\tt clean-db} and correct the problem. Repeat until everything seems to work smoothly. \subsection{Things to do on each host...} \be \item Install the man pages somewhere. \item \begin{sloppypar} Edit the Makefile - you might want to change the definitions of PERL, SDIR (where the Perl scripts go), BDIR (where the binaries go), PRINTER (where you print your sources) and the ``make print'' entry itself. Oh, you'll need to update the BACKUP variable also if you decide to change the name of the ``backup'' group. \end{sloppypar} \item Create SDIR and BDIR. I'd suggest making them the same for simplicity, its up to you. You probably want a separate copy on each server that you are backing up so you don't have to depend on some other server being up to do backups on the others. \item Create the backup group using whatever name you want. The value of the GID doesn't matter - we use 31. \item Type {\tt make} to compile the C programs (this only needs to be done once for each group of machines with the same binary format and OS release) and then {\tt make install} to install the binaries, scripts and symlinks as needed. The Makefile uses {\tt maketime.\*} to try to avoid doing unnecessary work. This may get in your way - if so just delete it before running {\tt make install}. If you don't have {\tt yagrip.pl} installed where Perl can find it you'll need to take the copy from the {\tt misc} subdirectory and either install it in your Perl library directory or in SDIR. \item Create a {\tt backups} file that lists the file systems to be backed up. See the backups(5) man page for details. In previous versions, this would have been {\tt /etc/backups}, but the default now is to use {\tt \$db\_dir/backups} so that there is one place from which you can control what file systems are backed up and so on. If you were using {\tt /etc/backups} you might want to pull them all together into {\tt \$db\_dir/backups} (don't forget to prepend the proper hostname to each line). A minimal {\tt backups} file might look like this: \begin{verbatim} host:/ dump host:/usr dump host:/home dump \end{verbatim} which means ``do all backups (all chains, all levels) on those three file systems when this host is backed up, and use {\tt dump} to do all of them''. See the section on configuration for more suggestions (section~\ref{config}). \item If you are going to try to do backups in single user mode you may need to modify {\tt single-begin} and {\tt single-end} to suite your particular setup. {\tt single-begin} should set things up so that the backup database is available (e.g. mount its NFS file system or whatever), and {\tt single-end} just unsets everything so that the rest of the boot process can proceed normally. You will also need to modify the {\tt /etc/rc} file to look for and run the {\tt /backup} file if it exists. Here's a code fragment that will do the trick: \begin{verbatim} if [ -r /backup ]; then (echo 'Backing everything up...') >/dev/console mv /backup /backup.delete /backup.delete >/dev/console 2>&1 rm -f /backup.delete fi \end{verbatim} This should be inserted just after it becomes possible to mount NFS file systems (so you can get to the backup database) but before you start the NFS mount daemon (so that your file systems will be quiescent when the backups are run, which is the whole point of this exercise). Wait! Don't save that file yet. The backup commands in {\tt /backup} will save their output to {\tt /tmp/backup-output}. If it weren't for the fact that we're in single user mode we'd try to mail it. If you want the output mailed to you add this code somewhere near the end of your rc file (after the {\tt sendmail} daemon is started): \begin{verbatim} if [ -f /tmp/backup-output ]; then (echo 'mailing backup results...') >/dev/console /usr/lib/sendmail -t < /tmp/backup-output \ >/dev/console 2>\&1 fi \end{verbatim} \item If you want to use this to backup hosts with local disks that are also part of an HP cluster, then you will need to create a CDF (Context Dependent File) named /backup so that when you try to do single user backups on several hosts at once they won't have nasty name conflicts. Of course, you won't be doing a single user backup of the server at the same time (since the whole cluster will be down), but you might want to do single user backups of several clients at the same time (we have some clients in clusters which have local disks that need to be backed up, in other words, diskless HP clients with local disks.) \item Set up {\tt check-backups} to run by {\tt cron} nightly on each backup host. I would suggest using {\tt check-backups -m -n}, which will cause it to mail the results to the addresses defined in the {\tt backup.config} file and to send mail ONLY if there's a problem. \item Edit the {\tt .rhosts} files. Each of the hosts that will be backed up must trust the host from which the master {\tt all-backups} program is started. This is because {\tt all-backups} uses {\tt rsh} to start do-backups on each host in turn. In addition all of the backup hosts must be in the {\tt .rhosts} file on each of the tape hosts since GNU {\tt tar} and {\tt dump} use {\tt rsh} as root to allow for network access to the tape drive. You can test your root rsh access from each host by running something like {\tt rsh host hostname} from a host pointed at some other host. For example, run this from the host where you will start the backup runs from pointed at each of the hosts to be backed up and then from each of those to each of the tape drive hosts. If your version of {\tt dump} and {\tt GNU tar} support it, you can specify a remote user in the tape definitions in {\tt backup.config} and tell the system to use that user by changing {\tt \$gtar\_does\_remote\_users} and {\tt \$dump\_does\_remote\_users} to 1. You can try to get around the need to have the backup hosts trust root on the host you start the backups from. Try changing {\tt all-backups} to run {\tt do-backups} with an {\tt rsh host -l user ...'}. If that user has execute access to the backup programs, and the wrapper for them is running setgid to group backup, then that part of things should work. If you are using {\tt dump}, make sure that group backup can read the disk devices and edit the dumpdates files, or you might try making a separate copy of the wrapper that is setuid root, setgid backup, and point the {\tt backup-dump} and {\tt backup-gtar} links in {\tt \$bdir} at it so that the backups will run as root. \ee \section{Testing} \label{testing} Your backup and restore system is one of the most critical systems in your installation. If you've ever lost a disk or file system without a good backup to restore files from, you can appreciate the importance of a backup system in a personal way. If you've never lost a file system, never had a disk crash, never lost days or weeks (or years) of work on a computer with no way to restore the missing data, then believe me, you want to avoid the experience. I have added some limited regression tests for some of the low level routines in the system. There's a test bed for testing the database routines, the file locking system, the ``get\_recent\_backups'' routine and various other pieces of the software. My goal is to add regression test routines that can be run more or less automatically after changing the code to help ensure that future changes don't break the system. If you make significant changes to the internals of the system, you can do a sort of general regression testing by copying your current database and comparing the results of running the old software on it against the results of the new software. For example, I recently did some extensive testing by copying our (large) database to two test directories. I set up 2 copies of the backup software, one with the old software pointing at one database, the other with the new software pointing at the other database directory. Then I ran the same programs (at the same times) in the 2 installations and compared the results. Try {\tt list-backups}, {\tt show tape} and {\tt show run}, and {\tt getback}. When you are fairly certain that the software is working right, try {\tt clean-db} (which might be destructive), and then run through the others again to make sure that {\tt clean-db} didn't damage your database. Part of the problem with debugging a backup system is that certain events can take a long time. For example, checking to see that a backup expires after 7 days, or that {\tt check-backups} complains about the right things after several days of skipped backups, or that the {\tt offsite} program moves tapes back and forth correctly. One partial solution is to change the length of a ``day'' from 24 hours to 30 minutes or so. You can do this by adjusting the value of {\tt \$DAYS} in {\tt global.defs}. That way you can cycle through 2 ``days'' per hour of real time, and determine whether the various programs that use days as the basic unit of time are working correctly. I've also tried to reduce the pain associated with testing backup systems by making it possible to install several copies of the backup system on the same hosts and have them cooperate without interfering with each other so that you can use a production version of this system while testing a new release. See section~\ref{configname} for more details. Finally, I've created a null backup type that goes through the motions of making a backup, without actually doing a backup. This is useful in testing the infrastructure. Just use type ``null'' in your {\tt \$db\_dir/backups} file instead of {\tt dump} or {\tt gtar}. \section{Configuration} \label{config} Some of this information is repeated in the installation instructions (section~\ref{install}), but I'll include a brief description of everything here just to be complete. There are several sources of configuration information in this backup system: the {\tt global.defs} and {\tt local.defs} files, the (single) {\tt backup.config} file, and the {\tt \$db\_dir/backups} file. Once you've set things up, you should think long and hard before changing anything in the config files. Some things are safe to change at will. Some things are dangerous. I've tried to make the difference clear in the comments. For example, if you want to move the backup database, make the appropriate changes to {\tt \$db\_dir} in {\tt global.defs} and move the directory. You can add host groups, tape types and aliases, and change the list of mail recipients in {\tt backup.config} without major repurcussions. Changing the level definitions may significantly change the daily operations and general utility of the backup system. Changing the configuration name after you have already set up the database is a definite no-no. \subsection{global.defs, local.defs} The {\tt global.defs} and {\tt local.defs} file contains installation information that isn't likely to change once you've installed the system and gotten it working. Some of the information may actually change from host to host, but most should be the same. The installation instructions cover setting this file up in some detail, so I won't rehash that here. Suffice it to say that this is where the directories that contain the binaries, Perl scripts and databases are named, among other things. The comments in these files should indicate what's safe to change and what's not so safe. \subsection{backup.config} The {\tt backup.config} file is more interesting. It should be installed in the database directory, and it contains global definitions that affect all of the hosts that use that database directory. These definitions affect more of the operational concerns of the system: what backups to do, how often to do them, how long to keep them before reusing the tapes, when to expire tapes, and who to send reports to. It can also be used to give aliases or names to tape devices and groups of hosts to make life a little easier. You can refer to the sample {\tt backup.config} file provided with the distribution and to the man page {\tt backup.config(5)} for more specific details. Note that once you change the file, the changes take effect right away. It is important to carefully think out the changes that you are about to make, and to check the file when you are done to ensure that you didn't accidently change more than you planned. \subsubsection{chains} \begin{verbatim} chains 2 \end{verbatim} The {\tt chains} entry is used to determine the number of backup chains that you want to support. The default is to support 1 backup chain, which is analogous to what BSD {\tt dump} maintains in its usual usage. What is a backup chain? A backup chain consists of a set of related backups: a full save and the incrementals that record the files that have changed since that full save. The best way to visualize it is to consider BSD {\tt dump}. It uses the {\tt /etc/dumpdates} file to save information about when each level was done on each file system. When you do a level 1 backup, the {\tt dump} program uses the {\tt dumpdates} file to determine when the level 0 backup was done, and then it saves all of the files that changed since that time as the level 1 backup. So a sequence of backups: a level 0, 1 and 2, respectively contain all of the files, all of the files which changed since the level 0 back, and all of the files which changed since the level 1 backup. Note that the level 1 and 2 backups depend on the level 0 backup - if the tape that contains that backup is lost or damaged, you can't do a full restore of the file system using just the level 1 and 2 backups. The idea of multiple Independent backup chains is that we can use several {\tt dumpdates} files (one for each chain) to create several sets of full saves and their dependent incremental backups that do not depend on each other. This way if a tape is lost or damaged we can fall back on one of the other (hopefully intact) backup chains and restore the file system to a reasonably up to date state. You might want to start off with 1 chain and move to 2 chains at some later point - it is easy to do. Adding chains should never cause problems. If you reduce the number of chains, the backups associated with the old chains will have the default keep values applied, which might make them expire sooner or later than they would have with their original keep values. \subsubsection{level} The {\tt level} entries are used to define backup levels that you want to use in your system. Each level definition applies to a particular chain, and defines the level number (similar to {\tt dump's} notion of a level), a name, and values for {\tt every} and {\tt keep}. The values of {\tt keep} and {\tt every} determine how long the backups stick around on tape before the tapes can be reused and how often the backups of that level should be done. For example, a daily backup would have an every of 1 (day), a weekly would have an every of 7 (days), and a full save that was to be done monthly might have an every of 28 (days). You would then set the keep values for each of these levels to indicate how long the backups should remain valid. You might use 14 (days) for the dailies (to keep the last 2 weeks worth of daily backups), 28 (days) for the weeklies (keep a month's worth), and 365 (days) for the full saves (save them for a year). In general, {\tt keep} should be at least twice the value of {\tt every} to make it less likely that you will be using the latest daily tape to do today's daily (if you did that, there'd be a small window of time while the backup was in progress where you'd have neither today's daily nor yesterday's). Here's an example: \begin{verbatim} # chain level name every keep level 0 0 full 28 365 level 0 2 weekly 7 28 level 0 9 daily 1 7 level 1 0 full 7 14 level 1 2 daily 1 2 \end{verbatim} This defines 5 levels in 2 chains. Note that you can reuse the names in each chain, and that they do not have to have the same values for {\tt every} and {\tt keep}. In this example chain 0 has full saves that are done every 28 days, weeklies that are done every 7 days, and dailies that are done every day. Chain 1 has full saves that are done every week, and dailies that are done every day. Note that the chain 1 backups are kept for a minimal amount of time - this reduces the total number of tapes needed to support 2 chains but still provides the security of having a second chain. You should set the {\tt keep} values to retain backups that you want to refer to for restores. We often find it convenient to be able to retrieve my .login file from up to a week ago, so we set the {\tt keep} for our chain 0 dailies to 7. \subsubsection{location} \begin{verbatim} # name location bermuda location hi308 location hi308d \end{verbatim} The {\tt location} entries declare tape drive and tape storage locations for use in the {\tt offsite} and {\tt tape} entries. \subsubsection{offsite} \begin{verbatim} # location chain wait warn offsite bermuda 1 2 1 \end{verbatim} The {\tt offsite} entry is used to define an offsite tape storage location. Each backup chain that is involved in offsite tape storage is associated with a specific location, so that all of the tapes for that chain end up in one place - an important consideration in the event of a disaster. You can associate more than one chain with each location. You also define the number of days to wait before sending newly written tapes to the offsite storage site (you might want to retain them locally for a few days for simple restores), and the number of days advance warning to give before tapes will be freed (so that you can bring them back to the local site on the day that they will become free for general use again). I suggest using at least 2 backup chains and storing the tapes from one chain offsite (for disaster recovery) and one chain locally (for general restores). \subsubsection{group} \begin{verbatim} group a foo bar baz \end{verbatim} The {\tt group} entry defines a group of hosts. In the example above we define a group named ``a'' that includes hosts ``foo'', ``bar'' and ``baz''. You don't have to define groups, but they can be convenient. A host can belong to as many groups as you want. \subsubsection{tape-type} The {\tt tape-type} entry defines a type of tape. It looks like this: \begin{verbatim} # type age uses formats tape-type exb 2555 50 8200 8500 tape-type stone 99999 1 granite \end{verbatim} This defines two tape types: ``exb'' and ``stone''. Type ``exb'' (for Exabyte) tapes can be up to 2555 days old and used 50 times (note that these are numbers pulled out of thin air!), and it supports 8200 and 8500 formats. Type ``stone'' lasts a looong time, can only be used once, and comes in one format named granite. \subsubsection{tape} The {\tt tape} entries are used to declare names and definitions for tape drives. Some sample entries: \begin{verbatim} # name user@host:dev type reads writes status location tape 2 archive:/dev/nrmt8 hit 1600|6250 6250 yes hi308d tape 3 abyss:/dev/nrmt0 hit 1600|6250 1600 yes hi308d tape 4 abyss:/dev/nrmt8 hit 1600|6250 6250 yes hi308d tape 5 brachiosaur:/dev/nrst0 qic 11|24|150 150 no bermuda tape 6 favorite:/dev/nrst0 exb 8200 8200 yes hi308 \end{verbatim} This defines several tape drives. Each drive has a name (which can be any string of non-whitespace characters) and a type, and includes info about which host and device the drive is. It also declares the formats that this device can read and write. If the drive will automatically read several formats, you can give the list separated by ``{\tt |}'' characters. The ``status'' should be {\tt yes} if that host has a {\tt mt status} command that works properly (e.g. it is a Sun running SunOS 4.x or Solaris 2.x), and {\tt no} otherwise. \subsubsection{config-name} \label{configname} \begin{sloppypar} You can give a name to this incarnation of the backup system using the {\tt config-name} entry. The default configuration name is ``'', in which case everything acts like it always has been. You can set the config name to something like ``test'' to set up a test installation on a host that is running a production version of the backup system, so that your test version won't interfere with it. \end{sloppypar} The configuration name is used in various places to ensure that multiple incarnations of the backup system don't step on each other. For example, if the config name was ``test'', tape IDs would start with ``test-'' to avoid confusion with normal tape IDs. The {\tt .chain} files would include ``test'' in their names to distinguish them from the others. The {\tt dumpdates} file would include ``test'' in their names. Lock files created in {\tt /tmp} would include ``test'' in their names. This makes it possible to install a test installation of the system in conjunction with a production version to do more comprehensive testing before switching to it. It also makes it easy to set up several distinct systems on different sets of hosts and not confuse the tapes with each other. To set up a test installation, go through the installation procedures but set things up in separate directories. The directory for the scripts should be different than your production version, and you should certainly make the database directory different. Make sure that you change the {\tt Makefile}, {\tt global.defs} and {\tt local.defs} to reflect the correct locations of things for your test installation. Then edit the {\tt backup.config} file and use the {\tt config-name} entry to give your test installation a name, like ``test''. You might also want to change the value of {\tt \$backup\_file} in {\tt local.defs} so that you can backup a different set of file systems in the test configuration on each host. Once you've tested a new copy of the system to your satisfaction, you should be able to simply copy the new scripts and binaries to your production directory, update your production {\tt backup.config} file and continue working with your existing production database. Once you have added tapes to the database, you should not change the configuration name. This is because the name gets incorporated into the tape IDs that are used in the tape and backup databases, and in the tape labels on each tape. \subsubsection{Miscellanea} You can set up a default tape drive with the {\tt default-tape} entry. You can also define a list of addresses to mail reports to with the {\tt mail-to} entry. \subsection{\$db\_dir/backups} The {\tt \$db\_dir/backups} file determines which filesystems should be backed up and what program to use to back each of them up. You can also use this to control which backups to do on this file system, by chain and by level. For example, I could set up a file system to use {\tt dump} to do all chain 0 backups and GNU {\tt tar} to do only chain 1, level 0 backups with the following setup: \begin{verbatim} host:/export/0 dump chain=0 host:/export/1 gtar chain=1,0 \end{verbatim} You can also define where online backups should go, and you can assign one or more tags to each file system. See the {\tt backups(5)} man page for more details. I've added the ability to use {\tt host:directory} rather than just {\tt directory}, so that you can use a single {\tt backups} file (usually located in your database directory). This will probably be required in future releases of the backup system. You can also add entries for file systems that you do not want to backup by using {\tt none} as the backup method. This allows you to add entries to keep {\tt check-backups} from complaining about file systems that are in {\tt /etc/fstab} but not in {\tt \$db\_dir/backups}. \section{Doing Backups} \label{backups} Once you've installed the software and set up the configuration files, you should be ready to do backups. The first thing that you need to do is add tapes to the tape library. After that, backups consist simply of using the {\tt all-backups} program to initiate backup runs. \subsection{Adding Tapes to the Tape Database} Tapes must be added to the tape library before they can be used in the backup system. The {\tt add-tape} program is used to add tapes to the library. This program writes a tape label at the start of the tape, which is used to identify the tape automatically, and creates an entry for the tape in the tape database. Each tape gets a tape identification ``number'' (tape ID). By default, the tape ID for a tape is its type followed by a number, such as ``exb-25'' or ``tk50-2''. If you used the {\tt config-name} entry in the {\tt backups.config} file to give a name to this backup installation, then that name will be prepended to the tape IDs for that installation (for example, ``test-exb-25'' if {\tt config\_name} is ``test''). You can override this naming scheme by telling {\tt add-tape} to use a chosen prefix string with the {\tt -p prefix} option. If I were to run {\tt add-tape -t 5 -p foo} the tape ID of this tape would be ``foo-5''. This might be handy if you have your own tape labeling and storage scheme. I'd suggest marking the label of each tape in ink with the tape number, and using pencil to write other identifying info (hosts, chain, level, date, tape drive, run ID number). That way you can reuse the tapes easily by erasing the penciled in stuff, and you won't have to worry about relabeling the tape with the tape number every time. I also suggest that you keep a paper log book that records who did which backup, what the Run ID number and tape numbers were, which hosts, what date, chain and level, and so on. Normally {\tt add-tape} chooses the tape number for the tape, but you can specify a tape number using the {\tt -t} option. This can be useful if you want to replace a lost or damaged tape, or if the tape label on a tape was damaged somehow. You can also specify a range of tape numbers with {\tt -t n:m}, which would cause {\tt add-tape} to add tapes with numbers {\tt n} through {\tt m}. {\tt add-tape} is fairly safe to run - it will warn you if you try to add a tape with an ID that already exists in the database, or if you try to add a tape that already appears to have a tape label. \subsection{Starting a Backup Run} {\tt all-backups} is the program you want. If you start it with no command line arguments, it'll ask you for the information it needs. As you get more used to it, you can use command line arguments to get things done more quickly. I'd suggest starting with a full save on one machine. Heck, do a few full saves on different machines. Then try an incremental on those machines. If you tell it to do the backup ``now'', the backups will be done in the foreground (so to speak), where you can watch them. If you specify a time, {\tt all-backups} will reschedule the backups to be run at a later time with the {\tt at} command, and you won't see the output until the backup is over and the results are mailed to you. As you get used to how things work, you can start exploring the power of the system. Try using ``group aliases'' to do incrementals on the hosts that you did the full saves on. Use the rescheduling abilities to run backups in the background automatically. When you feel really confident, try out the backup during boot option. Careful - your system will go down at the time that you scheduled the backup for! If you are really daring, try using the {\tt -p} option to do an incremental on several hosts in parallel mode. This will cause the backups on those hosts to run concurrently, writing their backup files to disk rather than to tape. As they finish, {\tt all-backups} will copy the backup files to tape and update the database. This allows one to do backups on lots of hosts somewhat more quickly than you would normally be able to do. This is especially useful if you are also using the {\tt -m disk} or {\tt -m both} modes, since they have to do the backups to disk anyway. In a large installation where you cannot fit all of your daily backups on one tape, you'll probably want to do something like what we do: create several groups of hosts, each group will be done together for incrementals in the same backup run. We have 6 tape drives, so we created 6 groups with about 6 hosts in each, and when we do our daily or weekly backups, we just start 6 copies of {\tt all-backups}, one for each group. Works great. We made it simple to set up by writing a shell scripts named {\tt weekly} and {\tt daily} that start up 6 windows running each of the all-backups, with all the arguments set. Each window pops up and tells you what tape to mount in which drive. That way you can get the tapes, label them, pop them in the drives, and hit return in each window and poof, your backups have been set up. You can find a copy of our scripts in the {\tt cis} subdirectory. The current version is limited to 1 backup run per tape - if you hit the end of tape the run stops. This is because none of the currently supported archiving programs really deal with end of tape correctly, in that there's no way to coordinate tape use with the backup scripts and the archive programs without doing really absurd things. Eventually I hope to change this, but in the meantime, the limitation exists and you have to live around or with it. Unfortunately, we recently set up 3 servers that have about 4 gigabytes of disk space each, which certainly won't fit on out Exabyte 8200 tapes, so I've added some stuff to help you work around the one tape limit. If you need to backup a host whose file systems won't fit on a single tape, you can use the {\tt tag} facility in the {\tt \$db\_dir/backups} file, and the {\tt -i tag} and {\tt -x tag} options in {\tt all-backups} as a cheap workaround. Suppose I want to break up the backups on a host so that file systems A, B and C are on one backup run, and D, E and F are on another. Create a {\tt \$db\_dir/backups} file that looks something like this: \begin{verbatim} host:/A dump tag=first host:/B dump tag=first host:/C dump tag=first host:/D dump tag=second host:/E dump tag=second host:/F dump tag=second \end{verbatim} If you start a backup run with {\tt all-backups -i first}, then the run will cover every file system on every listed host that has a {\tt tag=first} entry. We often ran into a problem where we'd schedule a backup for 6 A.M., and someone would remove the tapes from the drive before then and forget to replace them. This made the backups abort, and that was a drag. We've added a feature that allows you to schedule a tape check for some time while there are still people around to fix the problem. Check out the {\tt -C} option to {\tt all-backups}. If you are concerned about losing a backup chain if someone trashes a full save tape or something, then use multiple chains. If you're worried about relying on {\tt dump} or GNU {\tt tar} for all of your backups, use {\tt dump} to do your backups in one backup chain and GNU {\tt tar} to do backups in the other. \subsection{Checking a Backup Run} Using the rescheduling option can be somewhat confusing. The {\tt atq} command is useful for viewing the {\tt at} queue - run it on the host that you ran {\tt all-backups} from to see whether the {\tt at} job got scheduled and what time it was scheduled for. A common mistake is to schedule a job for 5:00 A.M. when you meant to schedule it for 17:00. Note also that some versions of {\tt at} only run their queues every 15 minutes, so if you schedule something for 10:10, it might actually run at 10:15. Note also that some versions of {\tt at} will only run 2 or 3 jobs at a time. If you have such as system and schedule 6 jobs for the same time, several of them will wait until the running jobs finish. There are sometimes ways to change the limits - consult your man pages and reference manuals. You can also use the {\tt show run} command to list the database entries corresponding to a given run identification number. This will list all of the pertinent entries from the online databases and might give you some insight about what's finished and what's not. The {\tt show scheduled} command will give you a list of the backup runs that have been scheduled or started in the last 24 hours but which have not finished yet. See the section on Problems (section~\ref{problems}) for more suggestions about problem solving. \section{Doing Restores} \label{restores} The easiest way to restore files from the backup tapes is to use the {\tt getback} program. {\tt Getback} provides you with 3 modes of operation: {\b full} restore, {\b interactive} restore, and {\b pick} restore. The full and interactive restore modes are similar: you specify the host, directory and date that you want to restore as of, and {\tt get-back} will walk you through restoring the files from what it thinks is the best sequence of backups (starting with a full save, working toward the most recent incremental). Full restore mode extracts the whole contents of each backup into the current directory, whereas interactive restore mode allows you to select which files to extract ala the BSD {\tt restore i} option (and so it only works for {\tt dump} archives, and not for GNU {\tt tar} archives). Pick restore mode gives you a list of all of the relevant backups from the database and allows you to select a backup and extract files in interactive fashion from each backup. Again, this only works for {\tt dump} archives. {\tt getback} will check each tape that is loaded to ensure that it is the correct tape, and will skip to the correct file on the tape before running the appropriate restore software to extract the contents. So all you need to do is tell it the host, directory and time, and load tapes when it asks. If the backups are also located online, {\tt getback} won't bother with the tapes but will instead just retrieve the data from the online files. You can also restore files by hand, of course. Use {\tt show} to list the entries for the backup run that you want to restore from. Find the tape number and file number of the backup you are interested, and mount that tape in a tape drive. Use {\tt mt fsf} to skip to the appropriate file (tape files are numbered starting at 0, which is the tape label). Then run the restore command of your choice to extract the contents of that backup. For example, we might see something like this from {\tt show run}: \begin{verbatim} 33 0,0 done yes 660944047 660950541 solar solar buid 14 done tape 6.1 / solar buid 15 done tape 6.2 /export/0 \end{verbatim} This is an entry for backup run number 33, which was a chain 0, level 0 backup on host ``solar''. Two file systems were backed up: ``/'' which is on tape 6, file 1, and ``/export/0'' which is on tape 6 file 2. Mount the tape, skip to the file, and go. You can do similar things with the output from {\tt list-backups}. \section{Tape Management} \label{tapes} As previously mentioned, the {\tt add-tape} program is used to add tapes in the tape library and label them with a ``magnetic'' tape label. You may find it necessary to relabel tapes whose magnetic labels have been destroyed - you can use {\tt add-tape -t \#} to do that. The {\tt run-clean-db} program should be set up to run nightly. It will automatically take care of the various on-line database, and it is what controls the recycling of tapes as the backups that they contain expire. If you want to force a tape to be reused more quickly than {\tt run-clean-db} would free it for use, you can use the {\tt reuse} command. Be careful, you could easily ``reuse'' the wrong tape and lose all of the backups that are stored on it. After a long while, the tapes themselves will expire, either because they are older than the age set for that type type of tape or because they have been used more than the maximum number of times. At that point you should replace them with fresh tapes. If you are using offsite tape storage, you will be using the {\tt offsite} program to generate a report that describes what tapes to take to each offsite location and which tapes to bring back. The report will describe, for each offsite storage location, which tapes should be there, which tapes should be taken there, and which should be brought back from there. The tape database keeps track of the location of the tapes. Each tape has a home location, which defaults to the location of the tape drive used when the tape was added to the tape library with {\tt add-tape}. You can change the current location of a tape with the {\tt move-tape} program. \section{The Graph Making Stuff} This backup system includes some scripts that can be used to automatically generate graphs using {\tt gnuplot}. These graphs include general overviews of run size by rank and run ID, run duration by rank and ID, tape uses, and graphs of backup sizes by time, per file system. This is an optional portion of the system, so you don't need to install it. You might find it interesting, however. You can find a brief description and installation instructions in the {\tt graph} subdirectory. Here are 3 examples of the sorts of graphs that you can produce. \begin{figure} \vspace{4.5in} \special{psfile=backup-fish.ps hoffset=-108 voffset=-36 hscale=0.65 vscale=0.65} \caption{Backup sizes ranked by time for fish:/export/0.} \label{fish-graph} \end{figure} Figure~\ref{fish-graph} shows the sizes of the backups for one file system plotted by time. The different levels are distinguished with different line styles. This is from the {\tt Merged} directory, so it shows all of the backups that have been done since I created the graph system (and got it working right). \begin{figure} \vspace{4.5in} \special{psfile=backup-size.ps hoffset=-108 voffset=-36 hscale=0.65 vscale=0.65} \caption{Run sizes ranked by size and divided by level.} \label{size-graph} \end{figure} Figure~\ref{size-graph} plots the total size of a backup run (the sum of the sizes of the backups in the run) sorted in rank order from largest to smallest. The data is segregated by level, and each level is plotted with its own line style on top of each other for comparison. This graph also represents a summary of all of the data that we have collected. This shows that most of our runs are less than 500 megabytes in size, well within the limits of the 2.3 gigabyte tapes that we are using. \begin{figure} \vspace{4.5in} \special{psfile=backup-duration.ps hoffset=-108 voffset=-36 hscale=0.65 vscale=0.65} \caption{Run duration ranked by time.} \label{duration-graph} \end{figure} Finally, figure~\ref{duration-graph} shows the duration of each run plotted against its rank. In this example, the levels are plotted with their own line styles, but ranked together. This graph is from the {\tt Recent} directory, and it shows the data collected from runs in the last month. \section{Daily Procedures} Just to keep things in perspective, here's a list of the various procedures that should be performed on a regular basis. Most of these will probably be set up to run via {\tt cron} or some such method, though the backups will probably be initiated by hand. Most of these will probably be done daily. \bi \item Sometime in the wee hours of the morning {\tt run-clean-db} will run and delete old backups from the database and free the associated tapes for reuse. \item Collect the tapes from the previous night's backups, write protect them, rack them, and update the backup log book. \item If you are bringing tapes back from off-site, use {\tt move-tape} to check them back into local storage. Use {\tt move-tape} to move the tapes that you took off site yesterday to their off site locations also. \item Print the {\tt list-backups} report, and add the printout to the backup log book. If you are using off site tape storage, make a copy and send it off site for safe keeping. \item Read the {\tt check-backups} warnings from all of the hosts (if any) and schedule backups to correct the problems found. \item Review the {\tt backup-report} chart, investigate any gaps. Use this to check whether you are getting the {\tt check-backups} warnings that you should. \item Check the report given by {\tt offsite}, gather the tapes to be taken offsite. \item Schedule and set up backups that are to be run later in the day. \ei \section{Disaster Recovery} There are disasters, and then there are disasters. If a host has died and needs to be restored, then there are two ways that you can go about doing this. If you can get the host to boot to the point where the backup system software is available, then you should just be able to use the {\tt getback} program to restore each file system. In our case, its usually possible to boot a host as a diskless client of another machine, from which point we can use the {\tt getback} program to restore everything. If you cannot access the backup system software from the broken machine, then you can use {\tt getback} or {\tt list-backups} on another machine to get the list of backups that you will need to restore from, and then run the restores ``by hand'' by mounting the tapes, skipping to the right files and recovering the needed backups. If the database host is dead, or the backup database is otherwise unavailable, then you can use the printed copies of the {\tt list-backups} output (you DID make printed copies, didn't you?) to find the crucial backups for each of the file systems that you want to recover. You should also be able to use your backup log book, though that might entail a lot of extra work since the backups won't be sorted by file system. Failing that, you could always read the physical labels that you stuck on each tape. The worst disaster would be if your entire site were destroyed, including the backup tapes, log book, list-backups output and everything. If you were using offsite tape storage, then you can use them to recover your files. I would duplicate the tapes, keeping a copy in safe keeping at the offsite tape storage location. Take the copies to your new site and use your off site copy of the list-backups output to determine which backups to use to restore which file systems. You probably want to start with the backup database, since once that's restored you can use it to do the rest. In summary, to help ensure complete disaster recovery, you should: \bi \item Use offsite tape storage. \item Maintain a copy of the list-backups printouts at each offsite storage location. \item Maintain a backup journal that describes each backup run, the tape used, the chain and level numbers, the hosts, the date and the status. \item Label each tape to indicate its current contents, including chain and level number, hosts, and date. \ei \section{Problems that Crop Up} \label{problems} There are (of course) zillions of things that could go wrong when you try to do your backups. This is a description of the most common problems and some suggested solutions. Sometimes things seem to lock up - you schedule a backup, but it doesn't. Or it starts, but doesn't finish when you expect (or at all). You may have a problem with locking. Sometimes stale locks are left in the backup database directory, named \*.lock. If there are no other backups running now, and there are lock files present, you could just delete the existing lock files and see whether that frees things up. The lock files contain information about the process and host that created them - you can use that to find out whether the lock owner is still running. I've noticed at times that the server that hosts the backup database gets confused and won't allow other hosts to make the locks, even though the lock files don't exist. Sometimes rebooting will help that. There's also a funky problem with {\tt rsh}. If you {\tt rsh} a command to a remote host and that host crashes, the {\tt rsh} will hang for a long time (forever?) and {\tt all-backups} will hang waiting for the {\tt rsh} to exit. Use {\tt kill} to nuke the offending rsh's and {\tt all-backups} should continue. You might need to clean out stale locks that were left behind by the crashed host. Lock files for the {\tt dumpdates} files are in {\tt /tmp} on each host. If you use the ``reschedule with at'' option to start a backup in the background at some future time, make sure that the future time is far enough in the future that {\tt all-backups} has time to check the tape label and stuff before that time arrives. Otherwise {\tt at} will either reschedule your job for the next day or tell you that its too late to schedule the job. Use {\tt atq} to check the {\tt at} queue. Check the time AND the date that the jobs are scheduled for - its easy to get confused and schedule a backup for tomorrow at 5 A.M. when you really meant for it to run today at 5 P.M. (use 24 hour time). If you schedule a job and it disappears from the queue but didn't send mail about the results, there are several things you can look for to find more information. The output is collected in a file named {\tt /tmp/backup-output.NNN} where {\tt NNN} is the run ID for that job. You can read that to see what had been done so far. If that isn't there, its possible that the job didn't even start correctly, or that it ran but that the mail system is screwed up. Check wherever it is that mail for mailer-daemon and root go to - see if there's any informative notes there. Make sure that your mail system is working right. If you set {\tt \$use\_deadman\_stuff} to 1, then you can use the ``deadman switches'' to kill backups that are taking too long. If this variable is set, {\tt all-backups}, {\tt do-backups}, {\tt backup-dump} and {\tt backup-gtar} start up background processes that periodically check for one of several files in {\tt \$db\_dir/deadman}. If one of the files appears, then the process sends a hangup signal to the program that started it and exits. All of these programs look for a file named after the run ID for that run. In addition, {\tt do-backups} will die if a file named {\tt hostname.rXXXX} is created (where XXXX is the run ID) and {\tt backup-dump} exits if a file named {\tt hostname.bXXXX} is created (where XXXX is the backup ID number. The deadman processes for {\tt backup-dump} and {\tt backup-gtar} can also be used to kill the backup if a certain amount of time has elapsed. You can set a timeout for backups on a per filesystem basis by using the {\tt timeout=NNN} option in the {\tt backups} file (NNN is the number of seconds allowed for backups on that directory). See {\tt backups(5)} for details. Don't rely on {\tt check-backups} to schedule your backups - it is meant to be a safety measure, not a primary backup scheduler. Make a schedule, write it on a calendar, and try to stick to it. You might want to check periodically that {\tt check-backups} is running correctly on each host. Look over the {\tt \$db\_dir/backups} files while you are at it. Remember that you can't use the ``backup during boot'' option when you have specified several hosts, and that you have to schedule the backup on the host that you are rebooting. Be careful with {\tt reuse} - don't reuse the wrong tape! \section{Archive Programs} \label{archive} One of the advantages of this backup system is that it doesn't restrict your choice of archive programs too much. The archive programs should support full and incremental backups in some form, and there should be a way to pass information about the tape device or archive file to use, and the directory to backup on the command line. The system comes with support for BSD {\tt dump} and GNU {\tt tar} built in. You should be able to easily add support for other programs, like normal {\tt tar}, {\tt cpio} or database backup programs. Both {\tt dump} and GNU {\tt tar} are good general purpose archive programs. They each have their shortcomings, however. Neither of them will work completely correctly on active file systems. {\tt dump} is subject to more problems that GNU {\tt tar}, since it is reading the file system ``directly'' rather than going through the OS. There are patches available from Purdue that make {\tt dump} somewhat more safe to use on active file systems. Both {\tt dump} and GNU {\tt tar} will skip files that are being modified when they come to them, meaning that those files will not appear on the backup. It is also possible for files to move from places that the programs have not scanned yet to places that have already been scanned, in which case they will also not make it into the backup. For a more detailed analysis of the problems of backing up active file systems, see Steve Shumway's excellent paper ``Issues in On-line Backup'' (Proceedings of the Fifth Large Installation Systems Administration Conference, 1991). There are also other problems with various archiving programs - see Elizabeth Zwicky's paper ``Torture-testing Backup and Archive Programs: Things You Ought to Know But Probably Would Rather Not'' (also found in the LISA V proceedings). Sun has a product named Backup Copilot that solves this problem using a special set of locks that programs can invoke to freeze certain types of file system activity while they are scanning for changed files or reading files for a backup. You could either get that and use it for backups of your Suns (its a very nice backup system) or get it and use its version of {\tt dump} with this backup system - it should be easy to incorporate. If you don't have a means of safely backing up active file systems, you can either risk having bad backups, or do your backups in single user mode, or compromise. We compromise by doing all of our full saves in single user to ensure that we have at least one good backup a month, and then do many, many incrementals (2 per day for each file system) with the systems live. GNU {\tt tar} is nice in that the archives are more portable, and you can backup by directory rather than by file system. It doesn't support all of the strange file system objects that you can find, such as HP context dependent files or ACLs. It does support backup and restore of sparsely allocated files (such as core files), but does it by reading the file in 512 byte blocks (which is painfully slow for large sparse files), and may not get the sparse blocks in exactly the right places. If you decide to use GNU tar, I strongly suggest that you get at least version 1.11, since previous versions didn't correctly support deleting restores (a full restore where deleted files are deleted, if that makes sense. Its hard to explain). GNU {\tt tar} does not have a nice interactive restore program like the BSD {\tt restore -i}. You are restricted to extracting the whole archive or the files that you name on the command line. In that respect {\tt dump} is extremely handy - you can run {\tt getback} and use it to mount and check the tapes, and skip to the right files. At that point it will run {\tt restore -i} and you can poke around to see if this backup contains the files that you want to extract. If not, quit and {\tt getback} will go on to the next backup. \subsection{Adding Support for New Archive Programs} It should be fairly easy to add support for new archive programs. Each archive program {\tt foo} should have a {\tt backup-foo} and a {\tt restore-foo} program that the backup system uses to do backups and restores. In some cases (e.g. {\tt dump}) these programs are little more than wrappers to call the appropriate backup or restore program with suitable arguments. In other cases (GNU {\tt tar}) these programs have to ``map'' the full save/incremental structure onto a program that doesn't directly support the same ideas. The {\tt backup-*} program is given the name of a directory to back up, the chain and level number of this backup, the name of the tape device or file to write the archive to, the backup ID number, and (if applicable) a flag that tells it that the archive is being written to disk rather than tape. It needs to convert that information into whatever it needs to run the given archiving program. In the case of dump, that means that it uses the directory to find the device name, it uses the chain to select one of several {\tt dumpdates} files to use, and it passes the level straight to dump. In the case of GNU {\tt tar}, it uses the information available in the backup database. In both cases it updates the database entry for this backup with the dates of the previous backup and the date of this backup, sets up the output file, makes a command line, and runs the archiving program. I am planning on changing {\tt backup-dump} to work in a fashion similar to {\tt backup-gtar}, in that it will gather information from the backup database and create a fake {\tt dumpdates} entry to control the actions of {\tt dump}, which will eliminate the possibility of the backup database and the dumpdates file getting out of synch with one another. Similar approaches could be made to incorporate other archive programs. For example, a {\tt backup-cpio} program could use the dates to run find, printing a list of files modified since that date and pipe that into cpio with appropriate arguments set. I would use {\tt backup-dump} as a template and work from there. The {\tt restore-*} programs are simpler. They are passed command line arguments that tell whether to do an ``interactive'' style restore (ala BSD {\tt restore -i}) or a full restore, the name of the tape device or archive file to read, and optionally a list of files to extract from the archive. The only complication is that not all programs support an interactive restore mode. In the case of {\tt tar}, we just do a normal full restore for both modes, and assume that the user will give a list of files on the command line to get the specific files that they want. \begin{sloppypar} Once you've made your backup and restore programs, you will need to add the name of that archive type to the list {\tt \%valid\_archive\_types} in {\tt global.defs}. Then you can set up entries in your {\tt \$db\_dir/backups} files to use the new archive method. \end{sloppypar} \section{Future Plans} \label{future} I have zillions of ideas for things to add. Multiple tape support. Tape rollover. Copy the database to the end of each backup run for database disaster recovery. Provide support for ``strange'' media like WORMs or jukeboxes. Keep information about the contents (files and directories) of each backup on-line so that you can easily find the most recent backup that contains file X, and so that you can do user-initiated restores. \subsection{Multiple Tape Support} Multiple tape support is difficult. There are several potential problems that one can run into. The first is that not all archive programs support multiple tapes. This isn't a huge problem though, since the backup system could still support multiple tapes and leave it to the user to ensure that they don't use a bogus archiving program with multiple tapes. The second problem is that programs like {\tt dump} don't do end of tape (EOT) detection, but do EOT prediction. This means that {\tt dump} needs to be told how much data it can write to the current tape. This worked to some degree in an environment where you were writing one backup onto several tapes, but fails in the case where you are writing multiple backups to multiple tapes. It is difficult for the backup scripts to keep track of home much space is left on the tape, since there's no standard way for the archive programs to report that information or to get it from the tape device. And even if you did tell {\tt dump} how much space was left on this tape, it would assume that that applied to successive tapes as well, so if {\tt dump} started on a tape that had only 10 megabytes of space left, it would assume that successive tapes could only hold 10 megabytes of data. The third problem is that even if a program supports real EOT detection and so avoids the space prediction problem, it may not accept more than one tape device on the command line, meaning that it will have to interact with something through /dev/tty or stdin to get the name of the next drive to use, or to request that the next tape be mounted. One could ``solve'' this problem by interpreting the output from the program and supplying appropriate responses, but this complicates the handling of stdout and stderr from the archive program. Not all programs will allow you to switch to a different drive, let alone to a different drive on a different host, and so you might be stuck with human interaction anyway to mount the next tape, unless you have a tape stacker or jukebox. In other words, it is a mess. There are several approaches that I could take in addressing this problem: \be \item Ignore programs that can't handle proper EOT detection, and either make use of the special features in {\tt GNU tar} that allow you to call a program when it needs a new tape, or have backup-gtar interpret the output from the program and do the right things in response. There are versions of dump that support proper EOT detection (the latest BSD version, for instance). I could leave it to people to install (and test) that if they want, and solve the problem in a case-by-case fashion (interpret the output of {\tt dump}, use the program option with {\tt GNU tar}). \item Fake the programs into thinking that they have an infinite tape to write to. This is the approach used in DeeJay, a backup system developed at Columbia University (see ``DeeJay - The Dump Jockey: A Heterogeneous Network Backup System'', by Melissa Metz and Howie Kaye, Proceedings of the Sixth System Administration Conference, 1992 (LISA VI)). Each archive program writes its output to a proxy somewhere, which handles EOT detection, multiple tape handling, database updates and so on. It seems like a nice solution that would work transparently for almost any archive program. It has the added advantage that you should be able to improve the performance of writing the backup over the net since you should be able to avoid the synchronous nature of the normal rmt protocol. You can also have the proxy handle switching between tape drives, even on different hosts, so that you can use tape rollover instead of requiring a stacker or jukebox. \ee I suspect that I will pursue the second option. There's also the question of providing support for tape and disk stackers and jukeboxes. I don't have any of the hardware to play with, so my understanding of these devices is all based on hearsay and I don't really have a broad understanding of how people use these devices. We'll need to be able to identify tape drives as stackers or jukeboxes. For backups, {\tt all-backups} will have to change to choose free tapes that are in the queues of the drives named for that run. That's easy, we can create special tape locations for each queue. The vaporware that deals with multiple tape support will have to know how to change tapes on these devices, and I don't imagine that there's a standard command interface for these devices (that would be too simple...) so there'll have to be a level of information hiding in the code somewhere. Dealing with cases where the tapes needed for a backup or restore aren't in the queue for a drive will be more complicated. I imagine we'll need a program that will make suggestions about what to load in the queues and what to remove (akin to the playlist program mentioned in the DeeJay paper). I suspect that there are some differences in how people use different kinds of autochangers - whether to keep it mostly full of free tapes so that you can do a weeks worth of backups unattended, or whether to keep it mostly full of recent backups so that you can do restores more readily. I have more questions than answers at this point. \subsection{Control of Incrementals} \begin{sloppypar} The idea behind incremental backups is simple: save everything that has changed since some time $X$ in the past, which is is when the more recent lower level backup has been done. This is obviously useful, since it usually greatly reduces the amount of data that you have to save to make a complete copy of the current file system contents in backups. BSD {\tt dump} implements this through the {\tt /etc/dumpdates} file. This contains an entry for the most recent backup of each disk partition at each incremental level, giving the date of the backup. When I tell {\tt dump} that I want to do a level 1 backup, it consults the {\tt dumpdates} file to find the date of the most recent level 0 backup for that file system, and saves everything that has changed since that date. \end{sloppypar} There are two problems with this approach. The first is that to implement multiple independent backup chains, you need to use multiple copies of the {\tt dumpdates} file. This is somewhat messy, and adds an extra bit of complexity since the backup system needs to juggle the various versions around. The second is more insidious. The information in the {\tt dumpdates} file is a duplicate of stuff that can already be found in the backup database. Its possible for the two to get out of sync, since {\tt dumpdates} is maintained by {\tt dump} and the backup database by the backup system. For example, if I were to do a level 2 backup, the {\tt dumpdates} file would contain the date of that level 2 backup (call that ``Thursday''). If I were to accidently reuse or otherwise destroy that tape, the backup database would correctly adjust itself by removing that backup (since it no longer exists and cannot be restored from). Subsequent level 9 dumps will still be done as though the Thursday level 2 backup had been done, and won't save anything that changed before Thursday. I have already change {\tt backup-gtar} to fix this problem. I will be doing something similar with {\tt backup-dump} in the future. It would be nice if archive programs could be told a date to act on, like GNU {\tt tar}, since that makes it much easier to control them. \subsection{Online Catalogue of Files} The backup system makes restores easier by helping you find the backups that have been done, and to select the most appropriate backups. It also deals with requesting the tapes, checking to see that they are correct, and skipping to the right files on them and running the extraction programs for you. It would be nicer still if you could identify the backups that a specific file appears on so that you could avoid mounting and searching unnecessary backups. This could be done by storing a catalogue of files saved in each backup online somewhere, and writing something that would allow you to browse that and select versions of files to restore. Shoot, you could get fancy and provide for user initiated restores. The problem is that you need a standard way of representing the online catalogue that would work and be supported by all of the archive programs that you are using. \subsection{Backup of Active File Systems} It is impossible for an archive program to make a complete and correct backup of an active file system without some sort of support from the operating system. This is why people recommend that you do backups while the file systems are quiescent, like when the system is in single user mode. I consider this to be an important problem, but one that is outside the scope of the backup system, since I can't and won't be providing any operating system patches to support safe backups of active file systems. Sorry. If you are using Suns, consider getting a copy of Backup Copilot and using their dump program, which uses special locks to ensure that it gets a correct snapshot of the file system. If you are using System V, you can talk to Veritas about getting a copy of their file system that provides copy on write features that you can use to make safe backups of active file systems. AFS from Transarc also provides a copy on write facility, with their own incremental backup facility. \subsection{Interoperability} I think everyone would agree that it would be nice if archive programs could create archives that can be extracted onto other (possibly very different) systems, and have reasonable things happen. All too often it is difficult or impossible to extract an archive written on one system onto another system (for example, try reading a Pyramid {\tt dump} onto a Sun). Programs that use a ``standard'' format that can be read on many systems don't always support system dependent file types. For example, {\tt tar} doesn't support sparsely allocated file or HP context dependent files or access control lists. An ideal archive program would allow one to store any type of file in some standard format which could be read on all platforms. The extraction process could either ignore parts that don't correspond exactly to objects supported in the local file system (extracting context dependent files on Suns, for instance), or to do the best job possible (extracting a sparsely allocated file onto a system that doesn't support them would just create them with the holes filled in). Oh, that would be nice. You could consider doing this with a dump-like program that writes a {\tt dump header} for each file into the archive, as well as a generic ``pax'' style header. The restore program would have to be smart enough to deal with a wide variety of file system objects, and one would have to find a way to distribute and incorporate new extraction routines when they become available. I suppose the default handler could just ignore unknown file types or something. \appendix \section{The Backup Database} \label{database-internals} The backup database seems complicated due especially to the number of files involved, but it isn't really that bad. Conceptually, think of it as 2 separate pieces: an indexed list of tapes (the tape library) and a hierarchical graph of backup runs and backups. \subsection{Tape Database} The tape database has an entry for each of the tapes in the library. Each entry is tape ID number, which looks like ``exb-123'' (tape type and number) or ``test-exb-123'' (config name, type and number). Each entry has a number of fields. \bd \item[flags] The status of this tape. Valid values are ``free'' (free for use in a backup run), ``used'' (already used in a backup run), ``promised'' (promised for use in a later backup run) and ``expired'' (tape is tool old or has been used too many times). Tapes that are promised for backup runs are freed again after 24 hours - if you need to schedule backups more than 24 hours in advance, you might want to futz the timeout in {\tt run-clean-db} so that backup runs later in the week won't try to use the same tape. \item[type] The type of each tape is set when the tape is added to the database, and is always set to the type of the drive that is used to write the initial tape label. This field should never change. \item[format] The format is more specific than the type. Type refers to the general type of tape - what sort of drive it will physically fit into (e.g. exabyte, half-inch, quarter-inch, tk50). The format might be the tape format (qic-11, qic-24) or density (800 bpi, 1600 bpi) or whatever. This is set according to the format of the last drive that was used to write this tape. \item[add\_date] The date that the tape was added to the database. Should never change. This is used to determine when a tape should expire by age. \item[run\_id] The ID of the backup run that this tape contains (if any). \item[drive] The host and device name of the drive that was last used to write this tape. Useful to find the drive if you need to read the tape and the drives heads were out of alignment or something. \item[location] The location of the tape. If this field doesn't exist, or if its value is ``'', the tape is local. Otherwise the value is the name of the offsite storage location where this tape is stored at. \item[used\_date] This is the date that the tape was last used, whether it was promised for a backup or actually used in one. \item[uses] A count of the number of backup runs that this tape has been used for. Used to expire tapes by usage count. \item[comments] Not currently used for anything. \item[location] The current location of this tape. \item[home] The home location of this tape. \ed Each tape also contains a tape label written as the first file on the tape. The tape labels look something like this: \begin{verbatim} xyZZy T# exb-123 chain 0 level 2 drive allosaur:/dev/nrst0 run 2189 pun,saqqara,archive,shape,disney,laverne,mad,iron \end{verbatim} The first line is the most important - the magic ``number'' is used to see whether this is a tape label, and the tape number should correspond to the tape's ID on its physical label. The other lines are informational, written so that you can identify a tape's use if the physical label is erased or falls off. \subsection{The Run Database} Each backup run that you start with {\tt all-backups} creates a unique entry in the run database. These are indexed by run ID numbers, which should be monotonically increasing, starting at 1. The complete information for a backup run is actually stored in several files: the run database file (typically named ``run.db'' in the database directory - see {\tt global.defs}), the per-host run database files (typically named things like ``run.db.mammal.db''), and the per-host backup database files (named things like ``backup.db.mammal.db''). These are separated into different files to make the hosts somewhat more independent of each other and especially to reduce lock contention (since locking is done per file, rather than per record). The run database contains an entry for each backup run that is started with {\tt all-backups} (well, an entry for the backup runs that haven't expired yet, to be more accurate). These are indexed by run ID number. The per-host run database files are also indexed by run ID number, and simply contain a list of the backup ID numbers that were associated with each run. The per-host backup database files contain information about each backup in a run - what file system, what date, where the backup is located on disk and/or on tape, what the status was, etc. Let's examine these more closely. The run database contains entries for each run, which include the following fields: \bd \item[chain] The chain number of this run. \item[level] The level number of this run. \item[hosts] The names of the hosts that are included in this backup run (comma separated list, no whitespace). \item[flags] The status of this run. Valid entries are ``started'', meaning that the run has started, and ``done'', meaning that the run is done. Entries are not created until the run actually starts, so if you schedule a run to start in the future its run database entry will not appear until the run actually starts. There's no specific value for ``failed'' - you have to consult the individual backups on the run to see how they fared. If the run actually completed in the sense that it made it to the end of the program, the status is set to ``done''. \item[active] This isn't used for anything currently. It is set to ``yes'' if the backups were done in multi-user mode, and ``no'' if they were done in single user mode. \item[start\_date] When the backup started (timestamp). \item[done\_date] When the backup finished (timestamp). \item[scheduled\_for] This is the string passed to the {\tt at} program for scheduling. No attempt is made to interpret it and convert it into a real date. \ed Each host named in the ``hosts'' field may (but might not) have a corresponding entry in its per-host run database file. Its entry will be missing if the backup run could execute {\tt do-backups} on that host for some reason. The per-host run database entries are simply lists of the backup IDs associated with the backups of that host in this run. There's one field for each entry: {\b buid\_list}, which is a comma separated list of the backup IDs from that hosts per-host backup database that are part of this run. This might be empty if the backups couldn't be done. The entries in the per-host backup database are a bit meatier. They are indexed by backup ID, and contain information about a single backup (of a directory or file system). The fields are: \bd \item[run] The run ID that this backup is a part of. \item[fs] The name of the file system or directory that this is a backup of (should be the mount point if this is a file system, rather than the raw device file or whatever). \item[fromdate] When this backup's data spans. If this is a full save, this is set to 0. Otherwise it is set to the time of the most recent next lower level backup, ala dump's levels. \item[todate] The time that this backup started. \item[flags] The status of the tape copy of this backup, if it exists. Valid values are ``started'', ``done'', ``failed'' and ``aborted''. ``failed'' means that the archive program exited with non-zero status, ``aborted'' means that the program noted that the tape had been rewound and so it gives up for fear of overwriting a tape. \item[d-flags] The status of the disk copy of this backup, if it exits. Valid values are ``started'', ``done'' and ``failed''. \item[tflist] The list of tape/file numbers that this backup can be found on. The list should look like this: ``exb-123.3,exb-145.1'' which means that this backup is on tape exb-123, file 3 (counting from 0) and tape exb-145 file 1. Since this backup system doesn't support multiple tapes at all, your tflist's should be one element long. \item[comments] Not used for anything yet. \item[afile] Where this backup can be found on line, if anywhere. \item[volhdr] If you used GNU tar for this backup, this is a copy of the volume header. \item[type] The type of archive program used to create this backup (e.g., ``dump'' or ``gtar''). \ed In addition to the information in the database proper, the backup system also uses the ``files'', ``errors'' and ``tar-db-dir'' directories to store additional information. The {\tt tar-db-dir} directory is used to store GNU tar's incremental listings if you are using a version that supports {\tt +listed\_incremental}. The {\tt files} directory isn't used for anything - eventually I intend to store indices of what files are in which backups to make it even easier to do restores. The {\tt errors} directory contains a copy of the stdout and stderr from the archive program for each active backup. These are stored with names like ``fish.23'', which would be the output from backup ID 23 on host fish. These are used by ``show -v run'' to list the verbose run information. \begin{figure} \vspace{8in} \special{psfile=backup-1.ps hoffset=-72 hscale=0.7 vscale=0.7} \caption{Relationship between entries in the various database files.} \end{figure} \end{document}