\documentstyle{article}
\title{The OSU-CIS Backup and Restore System}
\author{Steve Romig}
\date{March 15, 1994}
\pagestyle{headings}
\begin{document}
\maketitle
\tableofcontents

\newcommand{\be}{\begin{enumerate}}
\newcommand{\ee}{\end{enumerate}}
\newcommand{\bi}{\begin{itemize}}
\newcommand{\ei}{\end{itemize}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
\newcommand{\bc}{\begin{center}}
\newcommand{\ec}{\end{center}}

\section{Introduction}

This document describes the The Ohio State University Computer and
Information Science (OSU-CIS) Department's backup and restore system.
The system still doesn't have a name so I'll just refer to it here as
``the backup system''.  This document describes version 2.6 of the
backup system. 

The backup system consists of a set of Perl scripts (and 1 small C
program) that provide what is intended to be a comprehensive and
configurable set of tools for doing incremental backups and restores.
The main design constraints were that it should be easy to use, robust
in handling errors, complementary to human skills and shortcomings,
and that it should use native backup programs.

``Easy to use'' means that the programs that are used most, {\tt
all-backups} and {\tt getback}, are user friendly in that they convey
a consistent and simple interface to handling backups on many and
diverse systems.  If you fail to provide necessary information on the
command line, they'll ask for what they want.  An on-line database is
used to make tape management and restores simple.

``Robust'' means that the system tries really, really hard to deal
with errors.  Programs that might damage or change the tapes or the
online database check to ensure that the right tapes are being used in
the right drives.  There are all sorts of internal consistency checks
intended to find bugs and problems in the database.  It ain't perfect,
but its better than a crufty set of shell scripts with little or no
error checking.

``Complementary to human skills'' means that it tries to make up for
what people are bad at, and tries to avoid what people are good at.
It seems hard to write a program that could schedule backups and
correctly take into account tape drive use, drive status, and operator
schedules, so that is left to the users (if you have ideas
about how to automatically schedule backups, assign tape drives and so
on, please tell me about them!)  People are forgetful and sometimes
forget to do backups, and its easy to tell when one is overdue, so
there's a {\tt check-backups} program that will complain about overdue
backups.

``Uses native backup programs'' means that it should be easy to use
these tools with just about any sort of archiving program and media.
There are a plethora of file archiving programs and file system types.
You've got short and long name file systems, BSD Fast File Systems,
file systems that support strange objects like context dependent files
or access control lists, and strange proprietary database file systems
that have their own weird backup programs.  You probably
already have archiving programs (variants of {\tt dump}, {\tt tar} or
{\tt cpio}) that can ``correctly'' save and restore these things.
What you are probably missing is a framework that handles tape
management and error handling and all that stuff.  This backup system
supplies that framework, allowing you to use the most appropriate
archiving program for each file system without rewriting the rest of
the backup system.

This version supports incremental backups in multiple chains on
multiple hosts, tape types, fairly easy restores, backups to tape
or disk or both, parallel backups, event logs, offsite tape storage,
and even more error checking than the previous versions.

No, it still doesn't allow backups to span multiple tapes.  Sorry.
See section~\ref{future} for details about this and other future
plans. 

This manual describes the backup system (section~\ref{overview}),
how to install it (section~\ref{install}), how to do backups
(section~\ref{backups}) and how to do restores
(section~\ref{restores}).  I'm aiming this document at describing the
overall system and the philosophy behind it.  More detailed
information about the various programs is available in their man
pages. 

\section{Overview of the Backup System}
\label{overview}

The backup system consists of backup, restore, and database management
software.  The programs that are intended for human use are {\tt
all-backups}, {\tt getback}, {\tt add-tape}, {\tt move-tape}, {\tt
show} and {\tt reuse}.  In addition, {\tt run-clean-db}, {\tt offsite},
{\tt check-backups}, {\tt backup-report} and {\tt list-backups} are
utility programs that 
are usually set up to be run by {\tt cron}.  Everything else is used
by those programs in some fashion or another.

\subsection{Tapes}

Let's start by talking about tapes.  Each tape to be used by the
backup system is registered in the tape library through the {\tt
add-tape} program.  {\tt Add-tape} writes a tape label at the start of
the tape that includes a unique tape identification name.  This
name is used to identify tapes so it should be written onto the
physical tape label in some fashion - I suggest writing the tape ID
in ink.  The tape label also contains information about what the tape
was most recently used for so that if the physical tape label falls
off you should still be able to identify the tape by reading the first
file (it is a text file written in a simple format).  The {\tt
add-tape} program also enters the tape into the tape library database
so that it can be used for backups.

Normally all you will need to do to maintain the tape database and
tape library is to add new tapes as required and replace tapes that
have worn out.  The {\tt add-tape} program can be used to add new
tapes and to relabel old tapes whose tape labels are bad.  The {\tt
reuse} program is used to tell the tape management system that a
particular tape can be reused right away rather than waiting for it
to be recycled when the backups on it expire.  This is mostly used
when a backup run fails and you want the tape to be available for
other backups.

Each tape has a tape ID.  The ID consists of a prefix of some sort and
a number, such as ``exb-10'', ``rack1-12'', or ``test-tk50-123''.  The
prefix defaults to the type of the tape, but can be any arbitrary
string (you can use the {\tt -p} option to {\tt add-tape} to set the
prefix for a tape that you are adding.)  The number is used to make
the IDs with the same prefix unique.  The prefix defaults to
the tape type so that you can store tapes of the same type in a rack
in sequential order (otherwise tapes 1 through 9 might be 8mm, but
tape 10 is a tk50 and 11 is a 9 track reel - how would you rack them?)
The system allows you to change the prefix so that you can make up
your own racking system. 

Each tape also has a location and a home.  The home is where the tape
is supposed to reside when it isn't offsite and the location is
where the tape currently is.  If you have tapes that were added with
pre-2.6 software, the location and home fields for those tapes might
not be set.  The location and home are mainly used for offsite tape
storage and to ensure that {\tt all-backups} will use tapes from the
free tape pool in the same location as the tape drive.

\subsection{Database}

The backup system maintains an on-line database that covers the tape
library and all of the current backups.  The databases are implemented
as {\tt dbm} files accessed as associative arrays from within Perl.
If you are backing up more than one host, you can share the database
through NFS.
Unfortunately the {\tt dbm} library that is normally used with Perl is
byte order dependent and will not work correctly if you try to share
the database files between hosts that don't use the same byte
ordering.  The BSD {\tt db} library provides a {\tt ndbm} library
interface but is written to support byte order independent files that
can be shared through NFS.  All you need to do to use it is to construct a
version of Perl that uses that library instead of the usual {\tt dbm}
or {\tt ndbm} library and convert your existing database files from
{\tt dbm} format to the new format.  There are a few things you'll
need to change in the configuration files - see the installation
instructions (section~\ref{install}) for details on dealing with this.
If you can't use NFS, then it should be possible to rewrite the
database module (all the database routines are in one file) to allow
access through some network connection or something like that.
Contact me for more information - some people have started working on
something like that already.

More information about the internal structure of the databases can be
found in appendix~\ref{database-internals}.

The on-line database records information about:
\bi
\item each tape, how old it is, how often it's been used, what it is
currently being used for, what its type and current format is, what
drive it was last written in.

\item each backup run (a collection of one or more backups that were
made through a single invocation of {\tt all-backups}).  What level of
backup this is, what backup chain this is, which hosts
were included in the run, when the run was started and when it
finished.

\item each backup.  What run it was part of, what file system on which host,
what span of time that the backup covers, what tape and which file on
that tape, where it can be found online (if at all), what the status
is, and what type of archive it is (e.g. {\tt dump} or GNU {\tt tar}). 
\ei

The database information and tape labels are used all over the place
to make things easy to use and relatively safe from accidental
mistakes.  For example the tape labels are checked prior to using the
tape to make sure that the correct tape has been inserted in the tape
drive to keep you from accidently overwriting the wrong tape.  The
backup information is used to make the process of restoring files
easier by finding the appropriate tapes for the task for you.

The databases are maintained automatically through the {\tt run-clean-db}
program that is (supposed to be) run nightly by {\tt cron} from one
(1) host.  The {\tt run-clean-db} program expires old backups and runs
from the backup and run databases.  Once it knows which backups and
runs are still current it frees any tapes that no longer contain
current backups so that they can be reused for new backup runs.

\subsection{Programs}

Finally we come to the backup and restore software.  Backups are
initiated with the {\tt all-backups} script.  {\tt All-backups} can be
used to start backups on one host or many hosts.  Like most of the
rest of the software it is set up to ask you questions about what you
want to do if you don't supply the necessary information on the
command line, so it should be pretty self-explanatory to use.  The
collection of backups initiated through one invocation of {\tt
all-backups} constitutes a backup run and is identified by a unique
run ID number.  The {\tt show} program can be used to view the
contents of the databases associated with a particular run or to view
the entries in the tape library.

The restore program is called {\tt getback}.  {\tt Getback} is
intended to be very easy to use.  You tell it what host and file
system you want to restore the files from and it tells you what
backups to use and walks you through the process of extracting the
files.  It supports three ``modes'': interactive, full and pick.  In
interactive and full restore modes {\tt getback} selects what it
thinks is the best sequence of backups to use to restore files from
and prompts you to insert the corresponding tapes, one by one,
starting with the lowest level and working toward higher levels.  The
only difference between the two is that full mode extracts everything
on the backup, and interactive mode allows you to use {\tt restore i}
to selectively add and extract files from each backup (this only works
with backups made with {\tt dump}).  Pick mode shows you a list of all of
the backups for the host and file system that you are interested in,
and allows you to select which backups to extract files from.

The {\tt check-backups} script can be used to check to see what backups are
due to be done on a particular host.  I suggest setting it up to run
from {\tt crontab} nightly on each of the hosts that is to be backup up
where it can serve as a reminder of missing backups for those who are
supposed to do backups (and their supervisors :-).

The {\tt list-backups} program prints a summary of all of the backups
contained in the database.  You can print this list out periodically
and use it to do restores if the machine that hosts the backup
database is down or if the backup database is damaged.

If you store some of your tapes offsite, you can use the {\tt offsite}
program to generate a report for each offsite storage location that
lists the tapes that should be in storage there, what tapes should be
checked out to that site (and then taken there), and what tapes should
be brought back and check in.  The {\tt move-tape} program is used to
update tape locations as you move them to offsite storage and bring
them back again.

\subsection{Configuration Files}

You're probably wondering how this whole thing is
configured and controlled.  There are four sources of configuration
information: 
\bi
\item Per-host  installation information in the {\tt global.defs} and 
    {\tt local.defs} files, which controls the locations and general
    operations of the programs.
\item The global {\tt backup.config} file in the database directory, which
    controls overall backup operations (what levels, how often they
    should be done, etc.)
\item The {\tt \$db\_dir/backups} file which controls which
    file systems get backed up, for which levels, and how.  
\item \begin{sloppypar} The command line on the {\tt all-backups} command, which
    determines which hosts will be included in each backup run, what
    chain and level to use, what tape drive, and how the run should be
    executed. \end{sloppypar}
\ei
See the section on configuration (section~\ref{config}) for more
details. 

\subsection{Features}
\label{features}

\bd
\item[Centralized Control] This system makes it easy to initiate
backups on multiple hosts in a consistent fashion while allowing for
individual variations among the hosts.  ``Consistent fashion''
means that the same chains and levels are run on the various hosts
with the same frequency, using a common tape library.  ``Individual
variation'' means that its still easy to determine which file systems
should be backed up, what should be used to back them up, and what
levels should be allowed for each chain for that file system.

\item[Easy To Configure] 
\begin{sloppypar}
The system is controlled by 3 global
configuration files ({\tt global.defs}, {\tt backups} and {\tt backup.config}), and
by one host-specific file ({\tt local.defs}).
\end{sloppypar}

\item[Tape Aliases] You can define convenient aliases for tape host
and device specifications, which makes it more convenient for both
backups and restores.

\item[Tape Type Support] The system can deal with information about
tape types and formats to make it easier to store your tapes neatly
and to correctly ascertain that it is using the right kind of drive
for a given tape.

\item[Host Groups]  You can also give groups of hosts a name, which
makes it easy to start a backup run on all of them.

\item[Tape Management] The system deals with tape allocation and
recycling automatically, and will expire tapes as they become too old
or overused.  You just need to add more tapes as required.

\item[Online Record of Backups] 
\begin{sloppypar}
The backup system maintains an online
database that records what each tape has been used for, what backup
runs have been made, which hosts were involved in each run, what file
systems on those hosts were backed up, when the backup finished, what
the state of the backup was, what errors occurred, and what tape and
file number the backup can be found on (and/or disk file).  This
is used wherever possible to make it easy to find and restore from
backups.
\end{sloppypar}

\item[Error Checking] Each tape has a tape label at the start that is
used wherever necessary to ensure that you are using the correct tape
in the correct tape drive.  This helps prevent accidental overwriting
of tapes with {\tt add-tape} or {\tt all-backups}, or incorrect
restores through {\tt getback}.

\item[Backup Checker] The system provides a 
program that can be used to automatically warn you when a backup is
due on any of the file systems that are supposed to be backed up.

\item[User Friendly] The most commonly used tools will ask you for
needed information if you don't supply it on the command line.  All of
the prompts give help if you type ``?''.  The restore system helps you
select backups to restore from and skips to the correct file on the
tapes after you load them.

\item[Multiple Independent Backup Chains] \begin{sloppypar}
This system provides the
ability to make several independent backup chains, to help avert the
problems caused by missing or damaged backup tapes.
\end{sloppypar}

\item[Multiple Archive Programs] This system currently supports the
use of both GNU {\tt tar} and BSD {\tt dump}, and it is fairly easy to
add support for other programs.  This means that you should be able to
coordinate backups of special file systems (such as database
partitions) along with the backups of your normal file systems on the
same tapes.  If you are really paranoid about backups, you can also do
parallel chains with different archive programs to effect more
complete independence among your backups.

\item[Flexible File System Configurations] You can choose what chains
and levels apply to each file system, and what archive program to use.
This means that you can do only chain 0 full saves on one file system
using {\tt dump}, and all chain 1 backups on another file system using GNU
{\tt tar}. 

\item[Concurrent Backups] You can do multiple backups in parallel on
many hosts and have the results copied sequentially onto the tape as
they are completed to save time.

\item[Backup During Boot] You can schedule backups to be done
automatically while the file systems are quiescent during the early
part of the boot process.

\item[Reschedulable Backups] You can schedule backups to start at a
later time.  This means that at the end of your working day you could
set up a full save to run at 5:00 A.M. the next morning in single user
mode, and sleep through the whole thing.

\item[Incremental Safety Latches] The system tries to detect situations
where a full save should be done instead of an incremental, and warn
you about them, for example if a full restore was done more recently
than the latest full backup, or if it thinks a lower level backup has
never been done.

\item[Fine Control Over Backup Runs] You can use ``tags'' to control
whether to include or exclude certain file systems from a backup run,
so that you can split the file systems of several hosts onto separate
tapes to ensure that they will fit.

\item[Online Backups] You can do backups to tape, disk or both.  If a
backup is found online somewhere, the restore facilities will use that
instead of the tape copy.

\item[Multiple Configs on Same Hosts] You can use the {\tt
config-name} entry in the {\tt backups.config} file to give names to
separate incarnations of the backup system, so that you can run a
production version of the system (config-name set to ``'') and a test
version (config-name set to ``test'') at the same time without
interference between them.

\item[Online Log File] The tools in the backup system maintain a
online log of important events.  Entries are added when tapes are
added to the database, when backup runs are started, and when restores
are done.

\item[Offsite Tape Storage Management] The backup system provides
support for offsite tape storage.  A simple program tells you what
tapes to take offsite and to bring back.  The backup and restore
programs take a tapes location into account to avoid using tapes that
are offsite unless you specify otherwise.

\item[Cute Graphs] If you use BSD {\tt dump} for your backups you can
use the graph making stuff to make interesting graphs that display run
sizes and durations, backup sizes for each file system, tape use and
so on.  This is somewhat useful for planning - you can find out how
much data you are putting on your level 2 backup tapes, for example. 
\ed

\section{Installation}
\label{install}

This system is intended to be used in an environment where multiple
hosts share a common database of tape and backup information.  So
there's one database that has to be set up in such a way that it will
be available to all of the hosts in question.  And each host needs its
own copy of the programs and scripts.  I suggest that you read
carefully through all of these instructions before starting to follow
them. 

\subsection{Things to do once...}

The following things should only need to be done once.

\be
\item CD to the {\tt src} directory.

\item If you have RCS, you might want to make sure that everything is
checked out of the RCS subdirectory by running {\tt co RCS/*}.

\item Edit the {\tt global.defs} file.  This defines the names
and locations of the backup database and other global
information.  The things that you will need to change are:
\bd
    \item[\$our\_timezone] the local time zone.
    \item[\$default\_tape\_type] the default type for tapes.
    \item[\$default\_tape\_format] the default format for tapes.
    \item[\$backup\_dir] this is the default location for on-line backups.
    \item[\@online\_repositories] This is a list of directories where
        online backups are kept.  This is used in {\tt run-clean-db} so
        that it can find them and delete any expired backups when it
	cleans house.  This should include {\tt \$backup\_dir} and any
	other online backup directories that you use.
    \item[\$db\_dir] where the database files are.
    \item[\$db\_server] the name of the host where the database files
	are really located.  
    \item[\$db\_ext] this should be the extension that your {\tt dbm}
        routines puts on the {\tt dbm} files.  If you are using {\tt
        dbm} or {\tt ndbm}, this should be ``dir'', if you are using
	the BSD {\tt db} library use ``db''. 
    \item[\$db\_ext2] this should be the other extension that your
	{\tt dbm} routines puts on the {\tt dbm} files.
	If you are using {\tt
        dbm} or {\tt ndbm}, this should be ``pag'', if you are using
	the BSD {\tt db} library use ``''. 
    \item[\$db\_server, \$db\_remote, \$db\_local] are the host name,
	export path and local mount point for the file system that
	contains the backup database.  These are used within {\tt
	single-begin} and {\tt single-end} to mount the file system so
	that the programs can access the database when doing backups
	in single user mode.
    \item[\$bdir] where the backup programs are (same as BDIR in
        the Makefile).
\ed
You shouldn't need to change anything else, but I do recommend that
you read through the file and see whether the default settings are
what you want. You should also check the names of the database files
in {\tt global.defs} to see if they match your current names.  If you
are running on a short name file system (where file names are limited
to 14 characters or less), the default database names will probably
cause problems for you.  You might be able to work around the
limitations by making the database names shorter.  Note that in the
case of the per-host run and backup databases, the names of the hosts
will be added to each file name.  Note also that I have changed the
names from the values used in release 2.5, so if you haven't upgraded
since then, you'll either need to change the values in {\tt
global.defs} or rename your database files to match the new names. 

\item Take a look at the {\tt local.defs} file also.  I
believe that I have it set up in such a way now that you can use the
same {\tt local.defs} file on just about any version of UNIX.  You may
need to make minor adjustments for different types of hosts.  If this
is a System V Release 4 host then set {\tt \$svr4} to 1.  Check the
{\tt @fs\_types} array to see if your systems file system types are
listed (these are the ``type'' values that you find in the {\tt
/etc/fstab} file.  You will need to change {\tt \$bitch\_prog} to
something suitable.  This is executed when the tape checker finds that
the wrong tape is in the drive.  You should read through the rest of
the file and make sure that the defaults apply to your host.  If you
are using GNU {\tt tar} version 1.10 or greater you shouldn't have to
change anything, otherwise you will have to adjust the various GNU tar
flags.  If the local version of {\tt dump} can handle remote device
specs like {\tt user@host:device} then change {\tt
\$dump\_does\_remote\_users} to 1.  If you are using Ultrix but you
are NOT using the default dump that comes with the Ultrix
distribution, you need to change {\tt \$ultrix\_dump} to {\tt
0}.  If you don't have GNU tar installed then comment out the definition
for {\tt \$gtar\_prog}.

\item \begin{sloppypar} 
Set up {\tt clean-db} to run nightly on the host that holds the
backup database.  It should be run on the backup database since it
needs to have guaranteed access to the database files if it runs.
This should be set up to run at a time when it is unlikely that any
backups will be run to avoid database contention since it locks
several databases, possibly for a longish time.  There shouldn't be
any problems if it is run while a backup is being done however.  You
might want to use the {\tt -v} flag in addition to the {\tt -y} flag
and pipe the output into {\tt mail} to send it to a list of interested
parties.  This can produce lots of output when run on a large
database, however.  There's a program named {\tt run-clean-db} that
you can use to run {\tt clean-db} once a day, logging the output to a
subdirectory of {\tt \$db\_dir}, and sending all the interesting bits
on through mail.  We run it like this: 
\begin{verbatim}
0 1 * * * /usr/local/utils/nbu/run-clean-db
\end{verbatim}
\end{sloppypar}

\item Set up {\tt list-backups} to run nightly on some host.  {\tt
list-backups} prints a report showing the known backups for each file
system.  The idea is that you print this daily and keep it in a log
book for reference in an emergency when the online backup database
isn't available.  Since the report can get rather large I suggest
that you print the whole thing once a week and then print incremental
updates once a day.  We do something like the following:
\begin{verbatim}
0 6 * * 0  list-backups | lpr 
0 6 * * 1  list-backups -N 1 | lpr
0 6 * * 2  list-backups -N 2 | lpr
0 6 * * 3  list-backups -N 3 | lpr
0 6 * * 4  list-backups -N 4 | lpr
0 6 * * 5  list-backups -N 5 | lpr
0 6 * * 6  list-backups -N 6 | lpr
\end{verbatim}
which prints the whole thing on Sunday, then prints 1 days worth on
Monday, 2 days worth on Tuesday, etc.  We keep Sunday's and
today's listings in the notebook and throw the rest away.  If you use
off site tape storage, you should make a copy of the {\tt
list-backups} reports and store them at each offsite tape storage
location for use in disaster recovery. 

\item You can use {\tt backup-report} to print a table that shows the
latest backups in table form, where you can easily recognize ``gaps''
in the backup coverage and take appropriate actions.  We run it once a
day as:
\begin{verbatim}
30 6 * * * backup-report -N 14 | lpr
\end{verbatim}

\item If you are doing offsite tape storage, then you should set up
the {\tt offsite} program to run daily on some host (probably the same
one that runs {\tt run-clean-db}).  The {\tt offsite} program generates a
report that lists the offsite storage locations, the tapes that should
be stored at each, a list of tapes to take to each location, and a
list of tapes to be brought back from each one.  You could set up the
{\tt crontab} entry to mail the report to interested parties, as in:
\begin{verbatim}
0 14 * * *  offsite | mail -s "Offsite Storage Report"\
        romig fine
\end{verbatim}
\ee

\subsection{If you are creating a new database...}

If you are creating a new database directory for this installation
then please follow the following instructions.

\be
\item Create the \$db\_dir (the database directory) on a suitable server.
You'll also need to create \$clean\_db\_log and \$error\_dir (these are
subdirectories of \$db\_dir by default).  The backup programs run as
user root, group backup.  The database files and directories are set to be
owned by group backup, mode 770 (660 for files) so that the programs
can still write them but so other users can't read them.  Change
\$db\_dir, \$clean\_db\_dir and \$error\_dir to be owned by group BACKUP
(or whatever), mode 770.

You could set things up to allow read access to OTHER but if I ever
add the ability to maintain an on-line list of what files are in which
backups then such open access to the database might be inappropriate,
since making them world readable would allow everyone to see the names
of everyone else's files.

Note that the \$db\_dir must be available on each of the hosts that
will use this backup system, under the name \$db\_dir.  You could do
this through NFS, for instance.  If the database directory cannot have
the same name on all hosts then you will have to adjust your {\tt
global.defs} file on a host-by-host basis.

The hosts that access the database directory through some sort of
network file system will be running programs as root, group backup.
These programs must be able to write the database files, and you may
need to work some magic to get that to work right.  If
you are using NFS to share your database directory, the server that
holds the database must meet one of these requirements:
\be
\item It has a clean implementation of NFS that allows programs to
      access files according to group write permissions even when
      running as root. SunOS 4.1 has a bug wherein root cannot access
      files even if it is running with the group that has specific
      write access to the file.  To test, create a directory on the
      server in question owned by group backup, mode 770, and a file
      within that directory, group backup, mode 660.  Make a setgid
      version of cat, group backup on some client that mounts that
      file system.  Run that cat with the pathname of the file you
      just made.  If it works, you're set.  If it doesn't you'll have
      to either find a server that works or revert to method b.

\item It allows root access for all of the hosts that will be backed
      up.  You can do that either by setting a
      {\tt root=server1:server2:server3} line in your exports file (if
      supported) or by adding an entry like ``anon=0'' to the
      appropriate entry in your {\tt /etc/exports} file.
\ee

\item Copy {\tt samples/backup.config} to \$db\_dir and edit it.
Refer to the man page on {\tt backup.config} for details.  See the
section on configuration (section~\ref{config}) for more details and
ideas.  

\item Copy {\tt samples/exclude} to \$db\_dir.  This lists files that
should be excluded from the backups.  The distributed exclude file
keeps the system from backing up files named buid-NNN.  This is
actually used only by the GNU {\tt tar} backup code so if you don't
plan to use that you don't need the exclude file.
\ee

\subsection{If you already have a database...}

If you have an existing database directory and you are just updating
the software then there are still some things that you need to do.

\be
\item The entries in the {\tt backup.config} file have changed
somewhat.  These are the changes in version 2.5:
\bi
    \item The {\tt level} entries now mention the chain number.
    \item There are {\tt tape-type} entries, and the {\tt expire-uses}
          and {\tt expire-age} entries have gone away.
    \item The {\tt tape} definitions now include the type, read and
          write formats. 
    \item There's an optional {\tt config-name} entry that you can use
          to set up a test version of the software.
\ei
In version 2.6, the following changes were added:
\bi
    \item There's a {\tt location} entry which must be used to declare
	  each tape drive/storage location.
    \item The {\tt tape} entries now include {\tt status} (``yes'' if
	  ``mt status'' reports the correct file and block number,
	  ``no'' otherwise) and {\tt location} (the location of this
	  tape drive).
    \item The {\tt default-tape} entries are now specific for
	  particular tape types and formats.
\ei

\item If you are running version 2.4 software or earlier, then you
will need to use the 2.5 database conversion programs (contact Steve
Romig, romig@cis.ohio-state.edu, for details).
If you are running version 2.5 software, you don't
have to do much.  I've changed the names of the sequence files to
match the corresponding backup database files - this will be handled
automatically by {\tt run-clean-db}, though you will have to delete the
old {\tt \*-id\*} files by hand.  I've also changed the names of the
database files in {\tt global.defs} to eliminate the superfluous
``.db'' - you can either change the names in {\tt global.defs} to
match the old names, or rename your database files to match the new
names. 
\ee

I suggest that you make a copy of your existing database, install the
new software, edit {\tt global.defs} and {\tt local.defs}, rename your
backup database files, delete the old {\tt \*-id\*} files, and run {\tt
clean-db -v} to create the new sequence files.  If there are
any mismatches between the real file names and the names in {\tt
global.defs}, it should become painfully apparent, and you can interrupt
{\tt clean-db} and correct the problem.  Repeat until everything seems
to work smoothly.

\subsection{Things to do on each host...}
\be
\item Install the man pages somewhere.  

\item \begin{sloppypar} 
Edit the Makefile - you might want to change the definitions of
PERL, SDIR (where the Perl scripts go), BDIR (where the binaries go),
PRINTER (where you print your sources) and the ``make print'' entry
itself.   Oh, you'll need to update the BACKUP variable also if you
decide to change the name of the ``backup'' group.
\end{sloppypar}

\item Create SDIR and BDIR.  I'd suggest making them the same for
simplicity, its up to you.  You probably want a separate copy on
each server that you are backing up so you don't have to depend on
some other server being up to do backups on the others.

\item Create the backup group using whatever name you want.  The value
of the GID doesn't matter - we use 31.

\item Type {\tt make} to compile the C programs (this only needs to be done
once for each group of machines with the same binary format and OS
release) and then {\tt make install} to install the binaries, scripts
and symlinks as needed.  The Makefile uses {\tt maketime.\*} to try to avoid
doing unnecessary work.  This may get in your way - if so just delete
it before running {\tt make install}.

If you don't have {\tt yagrip.pl} installed where Perl can find it you'll
need to take the copy from the {\tt misc} subdirectory and either install it
in your Perl library directory or in SDIR.

\item Create a {\tt backups} file that lists the file systems to be backed
up.  See the backups(5) man page for details.  In previous versions,
this would have been {\tt /etc/backups}, but the default now is to use
{\tt \$db\_dir/backups} so that there is one place from which you can
control what file systems are backed up and so on.  If you were using
{\tt /etc/backups} you might want to pull them all together into {\tt
\$db\_dir/backups} (don't forget to prepend the proper hostname to
each line).  A minimal {\tt backups} file might look like this:
\begin{verbatim}
host:/              dump
host:/usr           dump
host:/home          dump
\end{verbatim}
which means ``do all backups (all chains, all levels) on those three
file systems when this host is backed up, and use {\tt dump} to do all of
them''.  See the section on configuration for more suggestions
(section~\ref{config}).

\item If you are going to try to do backups in single user mode you may
need to modify {\tt single-begin} and {\tt single-end} to suite your
particular setup.  {\tt single-begin} should set things up so that the
backup database is available (e.g. mount its NFS file system or
whatever), and {\tt single-end} just unsets everything so that the
rest of the boot process can proceed normally.  You will also need
to modify the {\tt /etc/rc} file to look for and run the {\tt /backup} file if
it exists.  Here's a code fragment that will do the trick:
\begin{verbatim}
if [ -r /backup ]; then
        (echo 'Backing everything up...') >/dev/console
        mv /backup /backup.delete
        /backup.delete                    >/dev/console 2>&1
        rm -f /backup.delete
fi
\end{verbatim}

This should be inserted just after it becomes possible to mount NFS
file systems (so you can get to the backup database) but before you
start the NFS mount daemon (so that your file systems will be
quiescent when the backups are run, which is the whole point of
this exercise).

Wait!  Don't save that file yet.  The backup commands in {\tt /backup}
will save their output to {\tt /tmp/backup-output}.  If it weren't for
the fact that we're in single user mode we'd try to mail it.  If
you want the output mailed to you add this code somewhere near the
end of your rc file (after the {\tt sendmail} daemon is started): 
\begin{verbatim}
if [ -f /tmp/backup-output ]; then
        (echo 'mailing backup results...') >/dev/console
        /usr/lib/sendmail -t < /tmp/backup-output \
                >/dev/console 2>\&1
fi
\end{verbatim}

\item If you want to use this to backup hosts with local disks that
are also part of an HP cluster, then you will need to create a CDF
(Context Dependent File) named /backup so that when you try to do
single user backups on several hosts at once they won't have nasty
name conflicts.  Of course, you won't be doing a single user backup of
the server at the same time (since the whole cluster will be down),
but you might want to do single user backups of several clients at the
same time (we have some clients in clusters which have local disks
that need to be backed up, in other words, diskless HP clients with
local disks.)

\item Set up {\tt check-backups} to run by {\tt cron} nightly on each backup
host.  I would suggest using {\tt check-backups -m -n}, which will
cause it to mail the results to the addresses defined in the {\tt
backup.config} file and to send mail ONLY if there's a problem.

\item Edit the {\tt .rhosts} files.  Each of the hosts that will be backed up
must trust the host from which the master {\tt all-backups} program is
started.  This is because {\tt all-backups} uses {\tt rsh} to start
do-backups on each host in turn.  In addition all of the backup hosts
must be in the {\tt .rhosts} file on each of the tape hosts since GNU
{\tt tar} and {\tt dump} use {\tt rsh} as root to allow for network access to the
tape drive.  You can test your root rsh access from each host by
running something like {\tt rsh host hostname} from a host pointed at
some other host.  For example, run this from the host where you will
start the backup runs from pointed at each of the hosts to be backed
up and then from each of those to each of the tape drive hosts.

If your version of {\tt dump} and {\tt GNU tar} support it, you can
specify a remote user in the tape definitions in {\tt backup.config}
and tell the system to use that user by changing {\tt
\$gtar\_does\_remote\_users} and {\tt \$dump\_does\_remote\_users} to
1.

You can try to get around the need to have the backup hosts trust root
on the host you start the backups from.  Try
changing {\tt all-backups} to run {\tt do-backups} with an {\tt rsh
host -l user ...'}.  If that user has execute access to the backup
programs, and the wrapper for them is running setgid to group backup,
then that part of things should work.  If you are using {\tt dump},
make sure that group backup can read the disk devices and edit the
dumpdates files, or you might try making a separate copy of the
wrapper that is setuid root, setgid backup, and point the {\tt
backup-dump} and {\tt backup-gtar} links in {\tt \$bdir} at it so that
the backups will run as root.  
\ee

\section{Testing}
\label{testing}

Your backup and restore system is one of the most critical systems in
your installation.  If you've ever lost a disk or file system without
a good backup to restore files from, you can appreciate the importance
of a backup system in a personal way.  If you've never lost a file
system, never had a disk crash, never lost days or weeks (or years) of
work on a computer with no way to restore the missing data, then
believe me, you want to avoid the experience.

I have added some limited regression tests for some of the low level
routines in the system.  There's a test bed for testing the database
routines, the file locking system, the ``get\_recent\_backups'' routine
and various other pieces of the software.  My goal is to add
regression test routines that can be run more or less automatically
after changing the code to help ensure that future changes don't break
the system.

If you make significant changes to the internals of the system, you
can do a sort of general regression testing by copying your current
database and comparing the results of running the old software on it
against the results of the new software.  For example, I recently did
some extensive testing by copying our (large) database to two test
directories.  I set up 2 copies of the backup software, one with the
old software pointing at one database, the other with the new software
pointing at the other database directory.  Then I ran the same
programs (at the same times) in the 2 installations and compared the
results.  Try {\tt list-backups}, {\tt show tape} and {\tt show run},
and {\tt getback}.  When you are fairly certain that the software is
working right, try {\tt clean-db} (which might be destructive), and
then run through the others again to make sure that {\tt clean-db}
didn't damage your database.

Part of the problem with debugging a backup system is that certain
events can take a long time.  For example, checking to see that a
backup expires after 7 days, or that {\tt check-backups} complains
about the right things after several days of skipped backups, or that
the {\tt offsite} program moves tapes back and forth correctly.  One
partial solution is to change the length of a ``day'' from 24 hours to
30 minutes or so.  You can do this by adjusting the value of {\tt
\$DAYS} in {\tt global.defs}.  That way you can cycle through 2
``days'' per hour of real time, and determine whether the various
programs that use days as the basic unit of time are working
correctly.

I've also tried to reduce the pain associated with testing backup
systems by making it possible to install several copies of the backup
system on the same hosts and have them cooperate without interfering
with each other so that you can use a production version of this
system while testing a new release.  See section~\ref{configname} for
more details.

Finally, I've created a null backup type that goes through the motions
of making a backup, without actually doing a backup.  This is useful
in testing the infrastructure.  Just use type ``null'' in your {\tt
\$db\_dir/backups} file instead of {\tt dump} or {\tt gtar}.

\section{Configuration}
\label{config}

Some of this information is repeated in the installation instructions
(section~\ref{install}), but I'll include a brief description of
everything here just to be complete.  There are several sources of
configuration information in this backup system: the {\tt global.defs}
and {\tt local.defs} files, the (single) {\tt backup.config} file, and
the {\tt \$db\_dir/backups} file.

Once you've set things up, you should think long and hard before
changing anything in the config files.  Some things are safe to change
at will.  Some things are dangerous.  I've tried to make the
difference clear in the comments.

For example, if you want to move the backup database, make the
appropriate changes to {\tt \$db\_dir} in {\tt global.defs} and move
the directory.  You can add host groups, tape types and aliases, and
change the list of mail recipients in {\tt backup.config} without
major repurcussions.  Changing
the level definitions may significantly change the daily operations
and general utility of the backup system.  Changing the configuration
name after you have already set up the database is a definite no-no.

\subsection{global.defs, local.defs}

The {\tt global.defs} and {\tt local.defs} file contains installation
information that isn't likely to change once you've installed the
system and gotten it working.  Some of the information may actually
change from host to host, but most should be the same.  The
installation instructions cover setting this file up in some detail,
so I won't rehash that here.  Suffice it to say that this is where the
directories that contain the binaries, Perl scripts and databases are
named, among other things.

The comments in these files should indicate what's safe to change and
what's not so safe.

\subsection{backup.config}

The {\tt backup.config} file is more interesting.  It should be
installed in the database directory, and it contains global
definitions that affect all of the hosts that use that database
directory.  These definitions affect more of the operational concerns
of the system: what backups to do, how often to do them, how long to
keep them before reusing the tapes, when to expire tapes, and who to
send reports to.  It can also be used to give aliases or names to tape
devices and groups of hosts to make life a little easier.  You can
refer to the sample {\tt backup.config} file provided with the
distribution and to the man page {\tt backup.config(5)} for more
specific details.  Note that once you change the file, the changes
take effect right away.  It is important to carefully think out the
changes that you are about to make, and to check the file when you are
done to ensure that you didn't accidently change more than you
planned.

\subsubsection{chains}

\begin{verbatim}
chains 2
\end{verbatim}

The {\tt chains} entry is used to determine the number of backup
chains that you want to support.  The default is to support 1 backup
chain, which is analogous to what BSD {\tt dump} maintains in its usual
usage.  What is a backup chain?  A backup chain consists of a set of
related backups: a full save and the incrementals that record the
files that have changed since that full save.  The best way to
visualize it is to consider BSD {\tt dump}.  It uses the {\tt
/etc/dumpdates} file to save information about when each level was
done on each file system.  When you do a level 1 backup, the {\tt
dump} program uses the {\tt dumpdates} file to determine when the
level 0 backup was done, and then it saves all of the files that
changed since that time as the level 1 backup.  So a sequence of
backups: a level 0, 1 and 2, respectively contain all of the files,
all of the files which changed since the level 0 back, and all of the
files which changed since the level 1 backup.  Note that the level 1
and 2 backups depend on the level 0 backup - if the tape that contains
that backup is lost or damaged, you can't do a full restore of the
file system using just the level 1 and 2 backups.

The idea of multiple Independent backup chains is that we can use
several {\tt dumpdates} files (one for each chain) to create several sets of
full saves and their dependent incremental backups that do not depend
on each other.  This way if a tape is lost or damaged we can fall
back on one of the other (hopefully intact) backup chains and restore
the file system to a reasonably up to date state.

You might want to start off with 1 chain and move to 2 chains at some
later point - it is easy to do.  Adding chains should never cause
problems.  If you reduce the number of chains, the backups associated
with the old chains will have the default keep values applied, which
might make them expire sooner or later than they would have with their
original keep values.

\subsubsection{level}

The {\tt level} entries are used to define backup levels that you want
to use in your system.  Each level definition applies to a particular
chain, and defines the level number (similar to {\tt dump's} notion of a
level), a name, and values for {\tt every} and {\tt keep}.  The values
of {\tt keep} and {\tt every} determine how long the backups stick
around on tape before the tapes can be reused and how often the
backups of that level should be done.  For example, a daily backup
would have an every of 1 (day), a weekly would have an every of 7
(days), and a full save that was to be done monthly might have an
every of 28 (days).  You would then set the keep values for each of
these levels to indicate how long the backups should remain valid.
You might use 14 (days) for the dailies (to keep the last 2 weeks
worth of daily backups), 28 (days) for the weeklies (keep a month's
worth), and 365 (days) for the full saves (save them for a year).  In
general, {\tt keep} should be at least twice the value of {\tt every}
to make it less likely that you will be using the latest daily tape to
do today's daily (if you did that, there'd be a small window of time
while the backup was in progress where you'd have neither today's
daily nor yesterday's).

Here's an example:
\begin{verbatim}
#     chain level name   every keep
level 0     0     full   28    365
level 0     2     weekly 7     28
level 0     9     daily  1     7

level 1     0     full   7     14
level 1     2     daily  1     2
\end{verbatim}
This defines 5 levels in 2 chains.  Note that you can reuse the names
in each chain, and that they do not have to have the same values for
{\tt every} and {\tt keep}.  In this example chain 0 has full saves
that are done every 28 days, weeklies that are done every 7 days, and
dailies that are done every day.  Chain 1 has full saves that are done
every week, and dailies that are done every day.  Note that the chain
1 backups are kept for a minimal amount of time - this reduces the
total number of tapes needed to support 2 chains but still provides
the security of having a second chain.

You should set the {\tt keep} values to retain backups that you want
to refer to for restores.  We often find it convenient to be able to
retrieve my .login file from up to a week ago, so we set the {\tt
keep} for our chain 0 dailies to 7.

\subsubsection{location}

\begin{verbatim}
#               name
location        bermuda
location        hi308
location        hi308d
\end{verbatim}

The {\tt location} entries declare tape drive and tape storage
locations for use in the {\tt offsite} and {\tt tape} entries.

\subsubsection{offsite}

\begin{verbatim}
#       location        chain   wait    warn
offsite bermuda         1       2       1
\end{verbatim}

The {\tt offsite} entry is used to define an offsite tape storage
location.  Each backup chain that is involved in offsite tape storage
is associated with a specific location, so that all of the tapes for
that chain end up in one place - an important consideration in the
event of a disaster.  You can associate more than one chain with each
location.  You also define the number of days to wait before sending
newly written tapes to the offsite storage site (you might want to
retain them locally for a few days for simple restores), and the
number of days advance warning to give before tapes will be freed (so
that you can bring them back to the local site on the day that they
will become free for general use again).

I suggest using at least 2 backup chains and storing the tapes from
one chain offsite (for disaster recovery) and one chain locally (for
general restores).  
 
\subsubsection{group}

\begin{verbatim}
group a   foo bar baz
\end{verbatim}

The {\tt group} entry defines a group of hosts.  In the example above
we define a group named ``a'' that includes hosts ``foo'', ``bar'' and
``baz''.  You don't have to define groups, but they can be convenient.
A host can belong to as many groups as you want.  

\subsubsection{tape-type}

The {\tt tape-type} entry defines a type of tape.  It looks like this:
\begin{verbatim} 
#         type  age   uses formats
tape-type exb   2555  50   8200 8500
tape-type stone 99999 1    granite
\end{verbatim}
This defines two tape types: ``exb'' and ``stone''.  Type ``exb'' (for
Exabyte) tapes can be up to 2555 days old and used 50 times (note that
these are numbers pulled out of thin air!), and it supports 8200 and
8500 formats.  Type ``stone'' lasts a looong time, can only be used
once, and comes in one format named granite.

\subsubsection{tape}

The {\tt tape} entries are used to declare names and definitions for
tape drives.  Some sample entries:
\begin{verbatim}
#    name user@host:dev          type reads     writes status location
tape 2    archive:/dev/nrmt8     hit  1600|6250 6250   yes    hi308d
tape 3    abyss:/dev/nrmt0       hit  1600|6250 1600   yes    hi308d
tape 4    abyss:/dev/nrmt8       hit  1600|6250 6250   yes    hi308d
tape 5    brachiosaur:/dev/nrst0 qic  11|24|150 150    no     bermuda
tape 6    favorite:/dev/nrst0    exb  8200      8200   yes    hi308
\end{verbatim}
This defines several tape drives.  Each drive has a name (which can be
any string of non-whitespace characters) and a type, and includes info
about which host and device the drive is.  It also declares the
formats that this device can read and write.  If the drive will
automatically read several formats, you can give the list separated by
``{\tt |}'' characters.  The ``status'' should be {\tt yes} if that host has a
{\tt mt status} command that works properly (e.g. it is a Sun running
SunOS 4.x or Solaris 2.x), and {\tt no} otherwise.

\subsubsection{config-name}
\label{configname}

\begin{sloppypar}
You can give a name to this incarnation of the backup system using the
{\tt config-name} entry.  The default configuration name is ``'', in
which case everything acts like it always has been.  You can set the
config name to something like ``test'' to set up a test installation
on a host that is running a production version of the backup system,
so that your test version won't interfere with it.  
\end{sloppypar}

The configuration name is used in various places to ensure that
multiple incarnations of the backup system don't step on each other.
For example, if the config name was ``test'', tape IDs would start
with ``test-'' to avoid confusion with normal tape IDs.  The {\tt
.chain} files would include ``test'' in their names to distinguish
them from the others.  The {\tt dumpdates} file would
include ``test'' in their names.  Lock files created in {\tt /tmp}
would include ``test'' in their names.  

This makes it possible to install a test installation of the system in
conjunction with a production version to do more comprehensive testing
before switching to it.  It also makes it easy to set up several
distinct systems on different sets of hosts and not confuse the tapes
with each other.  

To set up a test installation, go through the installation procedures
but set things up in separate directories.  The directory for the
scripts should be different than your production version, and you
should certainly make the database directory different.  Make sure
that you change the {\tt Makefile}, {\tt global.defs} and {\tt
local.defs} to reflect the correct locations of things for your test
installation.  Then edit the {\tt backup.config} file and use the {\tt
config-name} entry to give your test installation a name, like
``test''.  You might also want to change the value of {\tt
\$backup\_file} in {\tt local.defs} so that you can backup a different
set of file systems in the test configuration on each host.

Once you've tested a new copy of the system to your satisfaction, you
should be able to simply copy the new scripts and binaries to your
production directory, update your production {\tt backup.config} file
and continue working with your existing production database.

Once you have added tapes to the database, you should not change the
configuration name.  This is because the name gets incorporated into
the tape IDs that are used in the tape and backup databases, and in
the tape labels on each tape.

\subsubsection{Miscellanea}

You can set up a default tape drive with the {\tt default-tape} entry.
You can also define a list of addresses to mail reports to with the
{\tt mail-to} entry.

\subsection{\$db\_dir/backups}

The {\tt \$db\_dir/backups} file determines which
filesystems should be backed up and what program to use to back each of
them up.  You can also use this to control which backups to do on this
file system, by chain and by level.  For example, I could set up a
file system to use {\tt dump} to do all chain 0 backups and GNU {\tt tar} to do
only chain 1, level 0 backups with the following setup:
\begin{verbatim}
host:/export/0	dump	chain=0
host:/export/1	gtar	chain=1,0
\end{verbatim}
You can also define where online backups should go, and you can assign
one or more tags to each file system.  See the {\tt backups(5)} man
page for more details. 

I've added the ability to use {\tt host:directory} rather than just
{\tt directory}, so that you can use a single {\tt backups} file
(usually located in your database directory).  This will probably be
required in future releases of the backup system.

You can also add entries for file systems that you do not want to
backup by using {\tt none} as the backup method.  This allows you to
add entries to keep {\tt check-backups} from complaining about file
systems that are in {\tt /etc/fstab} but not in {\tt \$db\_dir/backups}.

\section{Doing Backups}
\label{backups}

Once you've installed the software and set up the configuration files,
you should be ready to do backups.  The first thing that you need to
do is add tapes to the tape library.  After that, backups consist
simply of using the {\tt all-backups} program to initiate backup runs.

\subsection{Adding Tapes to the Tape Database}

Tapes must be added to the tape library before they can be used in
the backup system.  The {\tt add-tape} program is used to add tapes to
the library.  This program writes a tape label at the start of the
tape, which is used to identify the tape automatically, and creates an
entry for the tape in the tape database.  

Each tape gets a tape identification ``number'' (tape ID).  By
default, the tape ID for a tape is its type followed by a number, such
as ``exb-25'' or ``tk50-2''.  If you used the {\tt config-name} entry
in the {\tt backups.config} file to give a name to this backup
installation, then that name will be prepended to the tape IDs for
that installation (for example, ``test-exb-25'' if {\tt config\_name}
is ``test'').  

You can override this naming scheme by telling {\tt add-tape} to use a
chosen prefix string with the {\tt -p prefix} option.  If I were to
run {\tt add-tape -t 5 -p foo} the tape ID of this tape would be
``foo-5''.  This might be handy if you have your own tape labeling and
storage scheme.  

I'd suggest marking the label of each tape in ink with the tape
number, and using pencil to write other identifying info (hosts,
chain, level, date, tape drive, run ID number).  That way you can
reuse the tapes easily by erasing the penciled in stuff, and you
won't have to worry about relabeling the tape with the tape number
every time.

I also suggest that you keep a paper log book that records who did
which backup, what the Run ID number and tape numbers were, which
hosts, what date, chain and level, and so on.

Normally {\tt add-tape} chooses the tape number for the tape, but you
can specify a tape number using the {\tt -t} option.  This can be
useful if you want to replace a lost or damaged tape, or if the tape
label on a tape was damaged somehow.  You can also specify a range of
tape numbers with {\tt -t n:m}, which would cause {\tt add-tape} to
add tapes with numbers {\tt n} through {\tt m}.

{\tt add-tape} is fairly safe to run - it will warn you if you try to
add a tape with an ID that already exists in the database, or if you
try to add a tape that already appears to have a tape label.

\subsection{Starting a Backup Run}

{\tt all-backups} is the program you want.  If you start it with no
command line arguments, it'll ask you for the information it needs.
As you get more used to it, you can use command line arguments to get
things done more quickly.

I'd suggest starting with a full save on one machine.  Heck, do a
few full saves on different machines.  Then try an incremental on
those machines.  If you tell it to do the backup ``now'', the backups
will be done in the foreground (so to speak), where you can watch
them.  If you specify a time, {\tt all-backups} will reschedule the
backups to be run at a later time with the {\tt at} command, and you
won't see the output until the backup is over and the results are
mailed to you.

As you get used to how things work, you can start exploring the power
of the system.  Try using ``group aliases'' to do incrementals on the
hosts that you did the full saves on.  Use the rescheduling abilities to
run backups in the background automatically.

When you feel really confident, try out the backup during boot option.
Careful - your system will go down at the time that you scheduled the
backup for!  

If you are really daring, try using the {\tt -p} option to do an
incremental on several hosts in parallel mode.  This will cause the
backups on those hosts to run concurrently, writing their backup files
to disk rather than to tape.  As they finish, {\tt all-backups} will
copy the backup files to tape and update the database.  This allows
one to do backups on lots of hosts somewhat more quickly than you
would normally be able to do.  This is especially useful if you are
also using the {\tt -m disk} or {\tt -m both} modes, since they have
to do the backups to disk anyway.

In a large installation where you cannot fit all of your daily backups
on one tape, you'll probably want to do something like what we do:
create several groups of hosts, each group will be done together for
incrementals in the same backup run.  We have 6 tape drives, so we
created 6 groups with about 6 hosts in each, and when we do our daily
or weekly backups, we just start 6 copies of {\tt all-backups}, one
for each group.  Works great.  We made it simple to set up by writing
a shell scripts named {\tt weekly} and {\tt daily} that start up 6
windows running each of the all-backups, with all the arguments set.
Each window pops up and tells you what tape to mount in which drive.
That way you can get the tapes, label them, pop them in the drives,
and hit return in each window and poof, your backups have been set up.
You can find a copy of our scripts in the {\tt cis} subdirectory.

The current version is limited to 1 backup run per tape - if you hit
the end of tape the run stops.  This is because none of the currently
supported archiving programs really deal with end of tape correctly,
in that there's no way to coordinate tape use with the backup scripts
and the archive programs without doing really absurd things.
Eventually I hope to change this, but in the meantime, the limitation
exists and you have to live around or with it.  Unfortunately, we
recently set up 3 servers that have about 4 gigabytes of disk space
each, which certainly won't fit on out Exabyte 8200 tapes, so I've
added some stuff to help you work around the one tape limit.

If you need to backup a host whose file systems won't fit on a single
tape, you can use the {\tt tag} facility in the {\tt \$db\_dir/backups}
file, and the {\tt -i tag} and {\tt -x tag} options in {\tt
all-backups} as a cheap 
workaround.  Suppose I want to break up the backups on a host so that
file systems A, B and C are on one backup run, and D, E and F are on
another.  Create a {\tt \$db\_dir/backups} file that looks something like this:
\begin{verbatim}
    host:/A  dump tag=first
    host:/B  dump tag=first
    host:/C  dump tag=first
    host:/D  dump tag=second
    host:/E  dump tag=second
    host:/F  dump tag=second
\end{verbatim}
If you start a backup run with {\tt all-backups -i first}, then the
run will cover every file system on every listed host that has a {\tt
tag=first} entry.

We often ran into a problem where we'd schedule a backup for 6 A.M.,
and someone would remove the tapes from the drive before then and
forget to replace them.  This made the backups abort, and that was a
drag.  We've added a feature that allows you to schedule a tape check
for some time while there are still people around to fix the problem.
Check out the {\tt -C} option to {\tt all-backups}.  

If you are concerned about losing a backup chain if someone trashes a
full save tape or something, then use multiple chains.  If you're
worried about relying on {\tt dump} or GNU {\tt tar} for all of your
backups, use {\tt dump} to do your backups in one backup chain and
GNU {\tt tar} to do backups in the other.  

\subsection{Checking a Backup Run}

Using the rescheduling option can be somewhat confusing.  The {\tt
atq} command is useful for viewing the {\tt at} queue - run it on the
host that you ran {\tt all-backups} from to see whether the {\tt at}
job got scheduled and what time it was scheduled for.  A common
mistake is to schedule a job for 5:00 A.M. when you meant to schedule
it for 17:00.  Note also that some versions of {\tt at} only run their
queues every 15 minutes, so if you schedule something for 10:10, it
might actually run at 10:15.  Note also that some versions of {\tt at}
will only run 2 or 3 jobs at a time.  If you have such as system and
schedule 6 jobs for the same time, several of them will wait until the
running jobs finish.  There are sometimes ways to change the limits -
consult your man pages and reference manuals.

You can also use the {\tt show run} command to list the database
entries corresponding to a given run identification number.  This will
list all of the pertinent entries from the online databases and might
give you some insight about what's finished and what's not.  The {\tt
show scheduled} command will give you a list of the backup runs that
have been scheduled or started in the last 24 hours but which have not
finished yet.

See the section on Problems (section~\ref{problems}) for more
suggestions about problem solving.

\section{Doing Restores}
\label{restores}

The easiest way to restore files from the backup tapes is to use the
{\tt getback} program.  {\tt Getback} provides you with 3 modes of
operation: {\b full} restore, {\b interactive} restore, and {\b pick}
restore.  The full and interactive restore modes are similar: you
specify the host, directory and date that you want to restore as of,
and {\tt get-back} will walk you through restoring the files from what
it thinks is the best sequence of backups (starting with a full save,
working toward the most recent incremental).  Full restore mode
extracts the whole contents of each backup into the current directory,
whereas interactive restore mode allows you to select which files to
extract ala the BSD {\tt restore i} option (and so it only works for
{\tt dump} archives, and not for GNU {\tt tar} archives).  Pick restore mode gives
you a list of all of the relevant backups from the database and allows
you to select a backup and extract files in interactive fashion from
each backup.  Again, this only works for {\tt dump} archives.

{\tt getback} will check each tape that is loaded to ensure that it is
the correct tape, and will skip to the correct file on the tape before
running the appropriate restore software to extract the contents.  So
all you need to do is tell it the host, directory and time, and load
tapes when it asks.  If the backups are also located online, {\tt
getback} won't bother with the tapes but will instead just retrieve
the data from the online files.

You can also restore files by hand, of course.  Use {\tt show}
to list the entries for the backup run that you want to restore from.
Find the tape number and file number of the backup you are interested,
and mount that tape in a tape drive.  Use {\tt mt fsf} to skip to the
appropriate file (tape files are numbered starting at 0, which is the
tape label).  Then run the restore command of your choice to extract
the contents of that backup.

For example, we might see something like this from {\tt show run}:
\begin{verbatim}
    33	  0,0     done  yes    660944047    660950541 solar
	  solar      buid 14   done      tape   6.1 /
	  solar      buid 15   done      tape   6.2 /export/0
\end{verbatim}
This is an entry for backup run number 33, which was a chain 0, level
0 backup on host ``solar''.  Two file systems were backed up: ``/''
which is on tape 6, file 1, and ``/export/0'' which is on tape 6 file
2.  Mount the tape, skip to the file, and go.

You can do similar things with the output from {\tt list-backups}.

\section{Tape Management}
\label{tapes}

As previously mentioned, the {\tt add-tape} program is used to add
tapes in the tape library and label them with a ``magnetic'' tape
label.  You may find it necessary to relabel tapes whose magnetic
labels have been destroyed  - you can use {\tt add-tape -t \#} to do
that. 

The {\tt run-clean-db} program should be set up to run nightly.  It will
automatically take care of the various on-line database, and it is
what controls the recycling of tapes as the backups that they contain
expire.  If you want to force a tape to be reused more quickly than
{\tt run-clean-db} would free it for use, you can use the {\tt reuse}
command.  Be careful, you could easily ``reuse'' the wrong tape and
lose all of the backups that are stored on it.

After a long while, the tapes themselves will expire, either because
they are older than the age set for that type type of tape or because
they have been used more than the maximum number of times.  At that
point you should replace them with fresh tapes.

If you are using offsite tape storage, you will be using the {\tt
offsite} program to generate a report that describes what tapes to
take to each offsite location and which tapes to bring back.  The
report will describe, for each offsite storage location, which tapes
should be there, which tapes should be taken there, and which should
be brought back from there.

The tape database keeps track of the location of the tapes.  Each tape
has a home location, which defaults to the location of the tape drive
used when the tape was added to the tape library with {\tt add-tape}.
You can change the current location of a tape with the {\tt move-tape}
program.  

\section{The Graph Making Stuff}

This backup system includes some scripts that can be used to
automatically generate graphs using {\tt gnuplot}.  These graphs
include general overviews of run size by rank and run ID, run duration
by rank and ID, tape uses,  and graphs of backup sizes by time, per
file system.  This is an optional portion of the system, so you don't
need to install it.  You might find it interesting, however.

You can find a brief description and installation instructions in the
{\tt graph} subdirectory.

Here are 3 examples of the sorts of graphs that you can produce.  

\begin{figure}
\vspace{4.5in}
\special{psfile=backup-fish.ps hoffset=-108 voffset=-36 hscale=0.65 vscale=0.65}
\caption{Backup sizes ranked by time for fish:/export/0.}
\label{fish-graph}
\end{figure}

Figure~\ref{fish-graph} shows the sizes of the backups for one file
system plotted by time.  The different levels are distinguished with
different line styles.  This is from the {\tt Merged} directory, so it
shows all of the backups that have been done since I created the graph
system (and got it working right).

\begin{figure}
\vspace{4.5in}
\special{psfile=backup-size.ps hoffset=-108 voffset=-36 hscale=0.65 vscale=0.65}
\caption{Run sizes ranked by size and divided by level.}
\label{size-graph}
\end{figure}

Figure~\ref{size-graph} plots the total size of a backup run (the sum
of the sizes of the backups in the run) sorted in rank order from
largest to smallest.  The data is segregated by level, and each level
is plotted with its own line style on top of each other for
comparison.  This graph also represents a summary of all of the
data that we have collected.  This shows that most of our runs are
less than 500 megabytes in size, well within the limits of the 2.3
gigabyte tapes that we are using.

\begin{figure}
\vspace{4.5in}
\special{psfile=backup-duration.ps hoffset=-108 voffset=-36 hscale=0.65 vscale=0.65}
\caption{Run duration ranked by time.}
\label{duration-graph}
\end{figure}

Finally, figure~\ref{duration-graph} shows the duration of each run
plotted against its rank.  In this example, the levels are plotted
with their own line styles, but ranked together.  This graph is from
the {\tt Recent} directory, and it shows the data collected from runs
in the last month.

\section{Daily Procedures}

Just to keep things in perspective, here's a list of the various
procedures that should be performed on a regular basis.  Most of these
will probably be set up to run via {\tt cron} or some such method,
though the backups will probably be initiated by hand.  Most of these
will probably be done daily.

\bi
\item Sometime in the wee hours of the morning {\tt run-clean-db}
will run and delete old backups from the database and free the
associated tapes for reuse.

\item Collect the tapes from the previous night's backups, write
protect them, rack them, and update the backup log book.

\item If you are bringing tapes back from off-site, use {\tt
move-tape} to check them back into local storage.  Use {\tt move-tape}
to move the tapes that you took off site yesterday to their off site
locations also.

\item Print the {\tt list-backups} report, and add the printout to the 
backup log book.  If you are using off site tape storage, make a copy
and send it off site for safe keeping.

\item Read the {\tt check-backups} warnings from all of the hosts (if
any) and schedule backups to correct the problems found.

\item Review the {\tt backup-report} chart, investigate any gaps.  Use
this to check whether you are getting the {\tt check-backups} warnings
that you should.

\item Check the report given by {\tt offsite}, gather the tapes to be
taken offsite.

\item Schedule and set up backups that are to be run later in the day.
\ei

\section{Disaster Recovery}

There are disasters, and then there are disasters.  

If a host has died and needs to be restored, then there are two ways
that you can go about doing this.  If you can get the host to boot to
the point where the backup system software is available, then you
should just be able to use the {\tt getback} program to restore each
file system.  In our case, its usually possible to boot a host as a
diskless client of another machine, from which point we can use the
{\tt getback} program to restore everything.  If you cannot access the
backup system software from the broken machine, then you can use {\tt
getback} or {\tt list-backups} on another machine to get the list of
backups that you will need to restore from, and then run the restores
``by hand'' by mounting the tapes, skipping to the right files and
recovering the needed backups.

If the database host is dead, or the backup database is otherwise
unavailable, then you can use the printed copies of the {\tt
list-backups} output (you DID make printed copies, didn't you?) to
find the crucial backups for each of the file systems that you want to
recover.  You should also be able to use your backup log book, though
that might entail a lot of extra work since the backups won't be
sorted by file system.  Failing that, you could always read the
physical labels that you stuck on each tape.

The worst disaster would be if your entire site were destroyed,
including the backup tapes, log book, list-backups output and
everything.  If you were using offsite tape storage, then you can use
them to recover your files.  I would duplicate the tapes, keeping a
copy in safe keeping at the offsite tape storage location.  Take the
copies to your new site and use your off site copy of the list-backups
output to determine which backups to use to restore which file
systems.  You probably want to start with the backup database, since
once that's restored you can use it to do the rest.

In summary, to help ensure complete disaster recovery, you should:
\bi
\item Use offsite tape storage.
\item Maintain a copy of the list-backups printouts at each offsite
storage location.
\item Maintain a backup journal that describes each backup run, the
tape used, the chain and level numbers, the hosts, the date and the
status. 
\item Label each tape to indicate its current contents, including
chain and level number, hosts, and date.
\ei

\section{Problems that Crop Up}
\label{problems}

There are (of course) zillions of things that could go wrong when you
try to do your backups.  This is a description of the most common
problems and some suggested solutions.

Sometimes things seem to lock up - you schedule a backup, but it
doesn't.  Or it starts, but doesn't finish when you expect (or at
all).  You may have a problem with locking.  Sometimes stale locks are
left in the backup database directory, named \*.lock.  If there are no
other backups running now, and there are lock files present, you could
just delete the existing lock files and see whether that frees things
up.  The lock files contain information about the process and host
that created them - you can use that to find out whether the lock
owner is still running.  I've noticed at times that the server that
hosts the backup database gets confused and won't allow other hosts to
make the locks, even though the lock files don't exist.  Sometimes
rebooting will help that.  There's also a funky problem with {\tt
rsh}.  If you {\tt rsh} a command to a remote host and that host
crashes, the {\tt rsh} will hang for a long time (forever?) and {\tt
all-backups} will hang waiting for the {\tt rsh} to exit.  Use {\tt
kill} to nuke the offending rsh's and {\tt all-backups} should
continue.  You might need to clean out stale locks that were left
behind by the crashed host.  Lock files for the {\tt dumpdates} files
are in {\tt /tmp} on each host.

If you use the ``reschedule with at'' option to start a backup in the
background at some future time, make sure that the future time is far
enough in the future that {\tt all-backups} has time to check the tape
label and stuff before that time arrives.  Otherwise {\tt at} will
either reschedule your job for the next day or tell you that its too
late to schedule the job.  Use {\tt atq} to check the {\tt at} queue.  Check
the time AND the date that the jobs are scheduled for - its easy to
get confused and schedule a backup for tomorrow at 5 A.M. when you
really meant for it to run today at 5 P.M. (use 24 hour time).

If you schedule a job and it disappears from the queue but didn't send
mail about the results, there are several things you can look for to
find more information.  The output is collected in a file named {\tt
/tmp/backup-output.NNN} where {\tt NNN} is the run ID for that job.
You can read that to see what had been done so far.  If that isn't
there, its possible that the job didn't even start correctly, or that
it ran but that the mail system is screwed up.  Check wherever it is
that mail for mailer-daemon and root go to - see if there's any
informative notes there.  Make sure that your mail system is working
right. 

If you set {\tt \$use\_deadman\_stuff} to 1, then you can use the
``deadman switches'' to kill backups that are taking too long.  If
this variable is set, {\tt all-backups}, {\tt do-backups}, {\tt
backup-dump} and {\tt backup-gtar} start up background processes that
periodically check for one of several files in {\tt
\$db\_dir/deadman}.  If one of the files appears, then the process
sends a hangup signal to the program that started it and exits.  All
of these programs look for a file named after the run ID for that run.
In addition, {\tt do-backups} will die if a file named {\tt
hostname.rXXXX} is created (where XXXX is the run ID) and {\tt
backup-dump} exits if a file named {\tt hostname.bXXXX} is created
(where XXXX is the backup ID number.  The deadman processes for {\tt
backup-dump} and {\tt backup-gtar} can also be used to kill the backup
if a certain amount of time has elapsed.  You can set a timeout for
backups on a per filesystem basis by using the {\tt timeout=NNN}
option in the {\tt backups} file (NNN is the number of seconds allowed
for backups on that directory).  See {\tt backups(5)} for details.

Don't rely on {\tt check-backups} to schedule your backups - it is
meant to be a safety measure, not a primary backup scheduler.  Make a
schedule, write it on a calendar, and try to stick to it.

You might want to check periodically that {\tt check-backups} is
running correctly on each host.  Look over the {\tt \$db\_dir/backups}
files while you are at it.

Remember that you can't use the ``backup during boot'' option when you
have specified several hosts, and that you have to schedule the backup
on the host that you are rebooting.

Be careful with {\tt reuse} - don't reuse the wrong tape!

\section{Archive Programs}
\label{archive}

One of the advantages of this backup system is that it doesn't
restrict your choice of archive programs too much.  The archive
programs should support full and incremental backups in some form, and
there should be a way to pass information about the tape device or
archive file to use, and the directory to backup on the command
line.  The system comes with support for BSD {\tt dump} and GNU {\tt
tar} built in.  You should be able to easily add support for other
programs, like normal {\tt tar}, {\tt cpio} or database backup
programs.

Both {\tt dump} and GNU {\tt tar} are good general purpose archive
programs.  They each have their shortcomings, however.  

Neither of them will work completely correctly on active file systems.
{\tt dump} is subject to more problems that GNU {\tt tar}, since it is
reading the file system ``directly'' rather than going through the OS.
There are patches available from Purdue that make {\tt dump} somewhat
more safe to use on active file systems.  Both {\tt dump} and GNU
{\tt tar} will skip files that are being modified when they come to
them, meaning that those files will not appear on the backup.  It is
also possible for files to move from places that the programs have not
scanned yet to places that have already been scanned, in which case
they will also not make it into the backup.  For a more detailed
analysis of the problems of backing up active file systems, see Steve
Shumway's excellent paper ``Issues in On-line Backup'' (Proceedings of
the Fifth Large Installation Systems Administration Conference, 1991).

There are also other problems with various archiving programs - see
Elizabeth Zwicky's paper ``Torture-testing Backup and Archive
Programs: Things You Ought to Know But Probably Would Rather Not''
(also found in the LISA V proceedings).

Sun has a product named Backup Copilot that solves this problem using
a special set of locks that programs can invoke to freeze certain
types of file system activity while they are scanning for changed
files or reading files for a backup.  You could either get that and
use it for backups of your Suns (its a very nice backup system) or get
it and use its version of {\tt dump} with this backup system - it
should be easy to incorporate.  

If you don't have a means of safely backing up active file systems,
you can either risk having bad backups, or do your backups in single
user mode, or compromise.  We compromise by doing all of our full
saves in single user to ensure that we have at least one good backup a
month, and then do many, many incrementals (2 per day for each file
system) with the systems live.

GNU {\tt tar} is nice in that the archives are more portable, and you
can backup by directory rather than by file system.  It doesn't
support all of the strange file system objects that you can find, such
as HP context dependent files or ACLs.  It does support backup and
restore of sparsely allocated files (such as core files), but does it
by reading the file in 512 byte blocks (which is painfully slow for
large sparse files), and may not get the sparse blocks in exactly the
right places.  If you decide to use GNU tar, I strongly suggest that
you get at least version 1.11, since previous versions didn't
correctly support deleting restores (a full restore where deleted
files are deleted, if that makes sense.  Its hard to explain).  

GNU {\tt tar} does not have a nice interactive restore program like
the BSD {\tt restore -i}.  You are restricted to extracting the whole
archive or the files that you name on the command line.  In that
respect {\tt dump} is extremely handy - you can run {\tt getback} and
use it to mount and check the tapes, and skip to the right files.  At
that point it will run {\tt restore -i} and you can poke around to see
if this backup contains the files that you want to extract.  If not,
quit and {\tt getback} will go on to the next backup.

\subsection{Adding Support for New Archive Programs}

It should be fairly easy to add support for new archive programs.
Each archive program {\tt foo} should have a {\tt backup-foo} and a
{\tt restore-foo} program that the backup system uses to do backups
and restores.  In some cases (e.g. {\tt dump}) these programs are
little more than wrappers to call the appropriate backup or restore
program with suitable arguments.  In other cases (GNU {\tt tar}) these
programs have to ``map'' the full save/incremental structure onto a
program that doesn't directly support the same ideas.  

The {\tt backup-*} program is given the name of a directory to back
up, the chain and level number of this backup, the name of the tape
device or file to write the archive to, the backup ID number, and (if
applicable) a flag that tells it that the archive is being written to
disk rather than tape.  It needs to convert that information into
whatever it needs to run the given archiving program.  In the case of
dump, that means that it uses the directory to find the device name,
it uses the chain to select one of several {\tt dumpdates} files to
use, and it passes the level straight to dump.  In the case of GNU
{\tt tar}, it uses the information available in the backup database.
In both cases it updates the database entry for this backup with the
dates of the previous backup and the date of this backup, sets up the
output file, makes a command line, and runs the archiving program.

I am planning on changing {\tt backup-dump} to work in a fashion
similar to {\tt backup-gtar}, in that it will gather information from
the backup database and create a fake {\tt dumpdates} entry to control
the actions of {\tt dump}, which will eliminate the possibility of the
backup database and the dumpdates file getting out of synch with one
another.

Similar approaches could be made to incorporate other archive
programs.  For example, a {\tt backup-cpio} program could use the
dates to run find, printing a list of files modified since that date
and pipe that into cpio with appropriate arguments set.  I would use
{\tt backup-dump} as a template and work from there.

The {\tt restore-*} programs are simpler.  They are passed command
line arguments that tell whether to do an ``interactive'' style
restore (ala BSD {\tt restore -i}) or a full restore, the name of the
tape device or archive file to read, and optionally a list of files to
extract from the archive.  The only complication is that not all
programs support an interactive restore mode.  In the case of {\tt
tar}, we just do a normal full restore for both modes, and assume that
the user will give a list of files on the command line to get the
specific files that they want.

\begin{sloppypar}
Once you've made your backup and restore programs, you will need to
add the name of that archive type to the list {\tt
\%valid\_archive\_types} in {\tt global.defs}.  Then you can set up
entries in your {\tt \$db\_dir/backups} files to use the new archive method.
\end{sloppypar}

\section{Future Plans}
\label{future}

I have zillions of ideas for things to add.  Multiple tape support.
Tape rollover.  Copy the database to the end of each backup run for
database disaster recovery.  Provide support for ``strange'' media
like WORMs or jukeboxes.  Keep information about the contents (files
and directories) of each backup on-line so that you can easily find
the most recent backup that contains file X, and so that you can do
user-initiated restores.

\subsection{Multiple Tape Support}

Multiple tape support is difficult.  There are several potential
problems that one can run into.  

The first is that not all archive
programs support multiple tapes.  This isn't a huge problem though,
since the backup system could still support multiple tapes and leave
it to the user to ensure that they don't use a bogus archiving program
with multiple tapes.  

The second problem is that programs like {\tt dump} don't do end of
tape (EOT) detection, but do EOT prediction. 
This means that {\tt dump} needs to be told how much data it can write
to the current tape.  This worked to some degree in an environment
where you were writing one backup onto several tapes, but fails in the
case where you are writing multiple backups to multiple tapes.  It is
difficult for the backup scripts to keep track of home much space is
left on the tape, since there's no standard way for the archive
programs to report that information or to get it from the tape device.
And even if you did tell {\tt dump} how much space was left on this
tape, it would assume that that applied to successive tapes as well,
so if {\tt dump} started on a tape that had only 10 megabytes of space
left, it would assume that successive tapes could only hold 10
megabytes of data.

The third problem is that even if a program supports real EOT
detection and so avoids the space prediction problem, it may not
accept more than one tape device on the command line, meaning that it
will have to interact with something through /dev/tty or stdin to get
the name of the next drive to use, or to request that the next tape be
mounted.  One could ``solve'' this problem by interpreting the output
from the program and supplying appropriate responses, but this 
complicates the handling of stdout and stderr from the archive
program.  Not all programs will allow you to switch to a different
drive, let alone to a different drive on a different host, and so you
might be stuck with human interaction anyway to mount the next tape,
unless you have a tape stacker or jukebox.

In other words, it is a mess.

There are several approaches that I could take in addressing this
problem:
\be
\item Ignore programs that can't handle proper EOT detection, and
either make use of the special features in {\tt GNU tar} that allow
you to call a program when it needs a new tape, or have backup-gtar
interpret the output from the program and do the right things in
response. 

There are versions of dump that support proper EOT detection
(the latest BSD version, for instance).  I could leave it to people to
install (and test) that if they want, and solve the problem in a
case-by-case fashion (interpret the output of {\tt dump}, use the program
option with {\tt GNU tar}).  

\item Fake the programs into thinking that they have an infinite tape
to write to.  This is the approach used in DeeJay, a backup system
developed at Columbia University (see ``DeeJay - The Dump Jockey: A
Heterogeneous Network Backup System'', by Melissa Metz and Howie Kaye,
Proceedings of the Sixth System Administration Conference, 1992 (LISA
VI)).  Each archive program writes its output to a proxy somewhere,
which handles EOT detection, multiple tape handling, database updates
and so on.  It seems like a nice solution that would work
transparently for almost any archive program.  It has the added
advantage that you should be able to improve the performance of
writing the backup over the net since you should be able to avoid the
synchronous nature of the normal rmt protocol.  You can also have the
proxy handle switching between tape drives, even on different hosts,
so that you can use tape rollover instead of requiring a stacker or
jukebox. 
\ee

I suspect that I will pursue the second option.

There's also the question of providing support for tape and disk
stackers and jukeboxes.  I don't have any of the hardware to play
with, so my understanding of these devices is all based on hearsay and
I don't really have a broad understanding of how people use these
devices.  We'll need to be able to identify tape drives as stackers or
jukeboxes.  For backups, {\tt all-backups} will have to change to
choose free tapes that are in the queues of the drives named for that
run.  That's easy, we can create special tape locations for each
queue.  The vaporware that deals with multiple tape support will have
to know how to change tapes on these devices, and I don't imagine that
there's a standard command interface for these devices (that would be
too simple...) so there'll have to be a level of information hiding in
the code somewhere.  Dealing with cases where the tapes needed for a
backup or restore aren't in the queue for a drive will be more
complicated.  I imagine we'll need a program that will make
suggestions about what to load in the queues and what to remove (akin
to the playlist program mentioned in the DeeJay paper).  I suspect
that there are some differences in how people use different kinds of
autochangers - whether to keep it mostly full of free tapes so that
you can do a weeks worth of backups unattended, or whether to keep it
mostly full of recent backups so that you can do restores more
readily.  I have more questions than answers at this point.

\subsection{Control of Incrementals}

\begin{sloppypar}
The idea behind incremental backups is simple: save everything that
has changed since some time $X$ in the past, which is is when the more
recent lower level backup has been done.  This is obviously useful,
since it usually greatly reduces the amount of data that you have to
save to make a complete copy of the current file system contents in
backups.  BSD {\tt dump} implements this through the {\tt
/etc/dumpdates} file.  This contains an entry for the most recent
backup of each disk partition at each incremental level, giving the
date of the backup.  When I tell {\tt dump} that I want to do a level
1 backup, it consults the {\tt dumpdates} file to find the date of the
most recent level 0 backup for that file system, and saves everything
that has changed since that date.
\end{sloppypar}

There are two problems with this approach.  The first is that to
implement multiple independent backup chains, you need to use multiple
copies of the {\tt dumpdates} file.  This is somewhat messy, and adds
an extra bit of complexity since the backup system needs to juggle the
various versions around.

The second is more insidious.  The information in the {\tt dumpdates}
file is a duplicate of stuff that can already be found in the backup
database.  Its possible for the two to get out of sync, since {\tt
dumpdates} is maintained by {\tt dump} and the backup database by the
backup system.  For example, if I were to do a level 2 backup, the
{\tt dumpdates} file would contain the date of that level 2 backup
(call that ``Thursday'').  If I were to accidently reuse or otherwise
destroy that tape, the backup database would correctly adjust itself
by removing that backup (since it no longer exists and cannot be
restored from).  Subsequent level 9 dumps will still be done as though
the Thursday level 2 backup had been done, and won't save anything
that changed before Thursday.

I have already change {\tt backup-gtar} to fix this problem.  I will
be doing something similar with {\tt backup-dump} in the future.

It would be nice if archive programs could be told a date to act on,
like GNU {\tt tar}, since that makes it much easier to control them.

\subsection{Online Catalogue of Files}

The backup system makes restores easier by helping you find the
backups that have been done, and to select the most appropriate
backups.  It also deals with requesting the tapes, checking to see
that they are correct, and skipping to the right files on them and
running the extraction programs for you.

It would be nicer still if you could identify the
backups that a specific file appears on so that you could avoid
mounting and searching unnecessary backups.  This could be done by
storing a catalogue of files saved in each backup online somewhere,
and writing something that would allow you to browse that and select
versions of files to restore.  Shoot, you could get fancy and provide
for user initiated restores.  The problem is that you need a
standard way of representing the online catalogue that would work and
be supported by all of the archive programs that you are using.  

\subsection{Backup of Active File Systems}

It is impossible for an archive program to make a complete and correct
backup of an active file system without some sort of support from the
operating system.  This is why people recommend that you do backups
while the file systems are quiescent, like when the system is in
single user mode.

I consider this to be an important problem, but one that is outside
the scope of the backup system, since I can't and won't be providing
any operating system patches to support safe backups of active file
systems.  Sorry.  If you are using Suns, consider getting a copy of
Backup Copilot and using their dump program, which uses special locks
to ensure that it gets a correct snapshot of the file system.  If you
are using System V, you can talk to Veritas about getting a copy of
their file system that provides copy on write features that you can
use to make safe backups of active file systems.  AFS from Transarc
also provides a copy on write facility, with their own incremental
backup facility.  

\subsection{Interoperability}

I think everyone would agree that it would be nice if archive programs
could create archives that can be extracted onto other (possibly very
different) systems, and have reasonable things happen.  All too often
it is difficult or impossible to extract an archive written on one
system onto another system (for example, try reading a Pyramid {\tt
dump} onto a Sun).  Programs that use a ``standard'' format that can
be read on many systems don't always support system dependent file
types.  For example, {\tt tar} doesn't support sparsely allocated file
or HP context dependent files or access control lists.

An ideal archive program would allow one to store any type of file in
some standard format which could be read on all platforms.  The
extraction process could either ignore parts that don't correspond
exactly to objects supported in the local file system (extracting
context dependent files on Suns, for instance), or to do the best job
possible (extracting a sparsely allocated file onto a system that
doesn't support them would just create them with the holes filled in).

Oh, that would be nice.  You could consider doing this with a
dump-like program that writes a {\tt dump header} for each file into
the archive, as well as a generic ``pax'' style header.  The restore
program would have to be smart enough to deal with a wide variety of
file system objects, and one would have to find a way to distribute
and incorporate new extraction routines when they become available.  I
suppose the default handler could just ignore unknown file types or
something. 

\appendix
\section{The Backup Database}
\label{database-internals}

The backup database seems complicated due especially to the number of
files involved, but it isn't really that bad.  Conceptually, think of
it as 2 separate pieces: an indexed list of tapes (the tape library)
and a hierarchical graph of backup runs and backups.

\subsection{Tape Database}

The tape database has an entry for each of the tapes in the library.
Each entry is tape ID number, which looks like ``exb-123'' (tape type
and number) or ``test-exb-123'' (config name, type and number).  Each
entry has a number of fields.  
\bd
\item[flags] The status of this tape.  Valid values are ``free'' (free
for use in a backup run), ``used'' (already used in a backup run),
``promised'' (promised for use in a later backup run) and ``expired''
(tape is tool old or has been used too many times).  Tapes that are
promised for backup runs are freed again after 24 hours - if you need
to schedule backups more than 24 hours in advance, you might want to futz
the timeout in {\tt run-clean-db} so that backup runs later in the week
won't try to use the same tape.

\item[type] The type of each tape is set when the tape is added to the
database, and is always set to the type of the drive that is used to
write the initial tape label.  This field should never change.

\item[format] The format is more specific than the type.  Type refers
to the general type of tape - what sort of drive it will physically
fit into (e.g. exabyte, half-inch, quarter-inch, tk50).
The format might be the tape format (qic-11, qic-24) or density (800
bpi, 1600 bpi) or whatever.  This is set according to the format of
the last drive that was used to write this tape.

\item[add\_date] The date that the tape was added to the database.
Should never change.  This is used to determine when a tape should
expire by age.

\item[run\_id] The ID of the backup run that this tape contains (if
any).

\item[drive] The host and device name of the drive that was last used
to write this tape.  Useful to find the drive if you need to read the
tape and the drives heads were out of alignment or something.

\item[location] The location of the tape.  If this field doesn't
exist, or if its value is ``'', the tape is local.  Otherwise the
value is the name of the offsite storage location where this tape is
stored at.

\item[used\_date] This is the date that the tape was last used, whether
it was promised for a backup or actually used in one.

\item[uses] A count of the number of backup runs that this tape has
been used for.  Used to expire tapes by usage count.

\item[comments] Not currently used for anything.

\item[location] The current location of this tape.

\item[home] The home location of this tape.
\ed

Each tape also contains a tape label written as the first file on the
tape.  The tape labels look something like this: 
\begin{verbatim}
xyZZy T# exb-123
chain 0 level 2 drive allosaur:/dev/nrst0 run 2189
pun,saqqara,archive,shape,disney,laverne,mad,iron
\end{verbatim}
The first line is the most important - the magic ``number'' is used to
see whether this is a tape label, and the tape number should
correspond to the tape's ID on its physical label.  The other lines
are informational, written so that you can identify a tape's use if
the physical label is erased or falls off.

\subsection{The Run Database}

Each backup run that you start with {\tt all-backups} creates a unique
entry in the run database.  These are indexed by run ID numbers, which
should be monotonically increasing, starting at 1.  The complete
information for a backup run is actually stored in several files: the
run database file (typically named ``run.db'' in the database
directory - see {\tt global.defs}), the
per-host run database files (typically named things like
``run.db.mammal.db''), and the per-host backup database files (named
things like ``backup.db.mammal.db'').  These are separated into
different files to make the hosts somewhat more independent of each
other and especially to reduce lock contention (since locking is done
per file, rather than per record).

The run database contains an entry for each backup run that is started
with {\tt all-backups} (well, an entry for the backup runs that
haven't expired yet, to be more accurate).  These are indexed by run
ID number.  The per-host run database files are also indexed by run ID
number, and simply contain a list of the backup ID numbers that were
associated with each run.  The per-host backup database files contain
information about each backup in a run - what file system, what date,
where the backup is located on disk and/or on tape, what the status
was, etc.

Let's examine these more closely.  The run database contains entries
for each run, which include the following fields:
\bd
\item[chain] The chain number of this run.

\item[level] The level number of this run.

\item[hosts] The names of the hosts that are included in this backup
run (comma separated list, no whitespace).

\item[flags] The status of this run.  Valid entries are ``started'',
meaning that the run has started, and ``done'', meaning that the run
is done.  Entries are not created until the run actually starts, so if
you schedule a run to start in the future its run database entry will
not appear until the run actually starts.  There's no specific value
for ``failed'' - you have to consult the individual backups on the run
to see how they fared.  If the run actually completed in the sense
that it made it to the end of the program, the status is set to
``done''. 

\item[active] This isn't used for anything currently.  It is set to
``yes'' if the backups were done in multi-user mode, and ``no'' if
they were done in single user mode.

\item[start\_date] When the backup started (timestamp).

\item[done\_date] When the backup finished (timestamp).

\item[scheduled\_for] This is the string passed to the {\tt at}
program for scheduling.  No attempt is made to interpret it and
convert it into a real date.
\ed

Each host named in the ``hosts'' field may (but might not) have a
corresponding entry in its per-host run database file.  Its entry will
be missing if the backup run could execute {\tt do-backups} on that
host for some reason.  The per-host run database entries are simply
lists of the backup IDs associated with the backups of that host in
this run.  There's one field for each entry: {\b buid\_list}, which is
a comma separated list of the backup IDs from that hosts per-host
backup database that are part of this run.  This might be empty if the
backups couldn't be done.

The entries in the per-host backup database are a bit meatier.  They
are indexed by backup ID, and contain information about a single
backup (of a directory or file system).  The fields are:
\bd
\item[run] The run ID that this backup is a part of.

\item[fs] The name of the file system or directory that this is a
backup of (should be the mount point if this is a file system, rather
than the raw device file or whatever).

\item[fromdate] When this backup's data spans.  If this is a full
save, this is set to 0.  Otherwise it is set to the time of the most
recent next lower level backup, ala dump's levels.

\item[todate] The time that this backup started.

\item[flags] The status of the tape copy of this backup, if it exists.
Valid values are ``started'', ``done'', ``failed'' and ``aborted''.
``failed'' means that the archive program exited with non-zero status,
``aborted'' means that the program noted that the tape had been
rewound and so it gives up for fear of overwriting a tape.

\item[d-flags] The status of the disk copy of this backup, if it
exits.  Valid values are ``started'', ``done'' and ``failed''.

\item[tflist] The list of tape/file numbers that this backup can be
found on.  The list should look like this: ``exb-123.3,exb-145.1'' which means
that this backup is on tape exb-123, file 3 (counting from 0) and tape
exb-145 file 1.  Since this backup system doesn't support multiple
tapes at all, your tflist's should be one element long.

\item[comments] Not used for anything yet.

\item[afile] Where this backup can be found on line, if anywhere.

\item[volhdr] If you used GNU tar for this backup, this is a copy of
the volume header.

\item[type] The type of archive program used to create this backup
(e.g., ``dump'' or ``gtar'').
\ed

In addition to the information in the database proper, the backup
system also uses the ``files'', ``errors'' and ``tar-db-dir''
directories to store additional information.  The {\tt tar-db-dir}
directory is used to store GNU tar's incremental listings if you are
using a version that supports {\tt +listed\_incremental}.  The {\tt
files} directory isn't used for anything - eventually I intend to
store indices of what files are in which backups to make it even
easier to do restores.  The {\tt errors} directory contains a copy of
the stdout and stderr from the archive program for each active backup.
These are stored with names like ``fish.23'', which would be the
output from backup ID 23 on host fish.  These are used by ``show -v
run'' to list the verbose run information.

\begin{figure}
\vspace{8in}
\special{psfile=backup-1.ps hoffset=-72 hscale=0.7 vscale=0.7}
\caption{Relationship between entries in the various database files.}
\end{figure}

\end{document}
