\begin{bf}
\begin{slide}{}
\begin{center}
{\bf Backup at Ohio State, Take 2}

Steve Romig

romig@cis.ohio-state.edu\\
614-292-0915\\
Computer and Information Science Department\\
The Ohio State University\\
Columbus, Ohio, 43210
\end{center}
\end{slide}

% Motivation
\begin{slide}{}
Motivation:
\begin{itemize}
    \item High density tape media
    \item More automated == more trust in software
    \item ``Fix'' human error problems
    \item Fix other problems
\end{itemize}
\end{slide}

% Old scheme
\begin{slide}{}
Doing a daily backup, the old way:
\begin{itemize}
    \item 29 1/2 inch tapes, 5 drives, 29 servers.
    \item Operator starts script on each server, 5 at a time.
    \item Mount tapes, confirm with dump, unmount tapes, relabel
	  tapes, update log book.
    \item Re-rack 29 tapes.
\end{itemize}
Took 2.5 hours, fairly labor intensive.
\end{slide}

% New scheme
\begin{slide}{}
Doing a daily, the new way:
\begin{itemize}
    \item An 8 mm. tape, an Exabyte drive, 29 servers.
    \item Operator starts 1 backup program.
    \item Program asks for tape to be mounted, checks tape, starts
	  backups on all servers.
    \item Operator re-racks tape, updates paper log.
\end{itemize}
Takes about 2.5 hours, but only 15 minutes of operator time.
\end{slide}

% Segue
\begin{slide}{}
Not going to talk about programs, databases, implementation - read the
paper, get other documents (see end of talk).

Want to talk instead about tricks employed to solve various problems,
in hopes that others will find them useful.
\end{slide}

% Problem 1 - labels
\begin{slide}{}
{\center Labels and Logs}

Problem: human error.  Mislabeled tapes, overwritten tapes.

Solution: labels and logs, both for people and computers.
\begin{itemize}
    \item Physical tape label, paper log book
    \item Magnetic tape label, archive labels, on-line database.
\end{itemize}
\end{slide}

\begin{slide}{}
{\center Labels and Logs}

Uses:
\begin{itemize}
    \item Identify contents of physically unlabeled or Mislabeled tape.
    \item Verify that correct tape is mounted.
    \item Identify archives on tape if database is unavailable.
    \item Reconstruct database if necessary from tape information.
\end{itemize}
\end{slide}

% Problem 2 - multiple chains
\begin{slide}{}
{\center Multiple Chains}

Chain - minimal set of backups needed to restore most recent saved
copy of a file system.  Most recent full, weekly and daily saves.

Problem with chain - lose tape, break chain, can't restore everything.
\end{slide}

% problem 2 cont'd - multiple chains
\begin{slide}{}
{\center Multiple Chains}

``Simple'' solution - use multiple, independent chains.  

If you lose a tape from one chain, you can restore from the other
chain.

Can make as many chains as you want - how many are enough?

In our case, 2 seem fine - no significant extra cost, already doing 2
dailies a day, extra full save just takes a tape and 15 minutes per
server. 
\end{slide}

% Parallel Stuff
\begin{slide}{}
{\center Parallel Backups}

First test run of dailies on 18 servers took 10 hours!  Ouch.  Single
tape bottleneck.

Fixed problem by using disk as a staging device.  Run backups in
parallel on servers, archive to disk, copy to tape as they finish.
\end{slide}

% Parallel stuff
\begin{slide}{}
{\center Parallel Backups}
\begin{itemize}
    \item Each server searches for a file system with enough free
	  space to store backup archive.
    \item Preference given to locally mounted file systems, and
	  especially to file systems mounted on disks other than the
	  one being backed up.
    \item Backup archives copied to tape as they finish.
    \item Archives deleted from disk when copy is done.
\end{itemize}
Can now do daily backups on 22 servers in 2.5 hours.
\end{slide}


% Large Incrementals
\begin{slide}{}
{\center Large Incrementals}

If a full restore is done, inode ctime is updated, next incremental
will include {\bf everything} on the file system.  

\begin{itemize}
    \item Fake timestamp in dumpdates.
    \item Reset system clock to time of backup you are restoring from.
    \item Use flag to warn not to do incremental.
\end{itemize}
\end{slide}

% Large Incrementals
\begin{slide}{}
{\center Large Incrementals}

Using flag to avoid incrementals:
\begin{itemize}
    \item Create .chainN just before doing the full save for chain N.
    \item When doing an incremental backup, compare the ctime of the
	  .chainN file against the time of the last full save for that
	  chain. 
    \item If .chainN newer, a full restore was done, would rather not
	  do incremental, warn user to do full save.
\end{itemize}
Also checked by the program that checks to see if all backups were
done.  If it notes that a full save should be done, it bitches about
it. 
\end{slide}

% Scheduling Backups
\begin{slide}{}
{\center Scheduling Backups}

Best time for backups are often the worst for people to have to work.
Early morning, late at night.  Yech.

\begin{itemize}
    \item Backup asks user for time to start backup.
    \item User mounts tape as normal, script checks tape.
    \item If all ok, script uses ``at'' to reschedule itself.
    \item At appointed time, script starts again, rechecks tape, does
	  backup. 
\end{itemize}
\end{slide}

% Backups During Reboot
\begin{slide}{}
{\center Backups During Reboot}

Don't want to have to be here to babysit system while it does a backup
in single user mode.

\begin{itemize}
    \item Added code to /etc/rc to run /etc/backup early on, if it
	  exists. 
    \item If you tell backup script to run during reboot, it creates
	  /etc/backup and inserts the commands for doing the backup.
    \item Schedules /etc/fastboot with ``at'' for the appointed time.
    \item System boots, executes /etc/boot, does backup, /etc/rc
	  continues, system comes back up again.
\end{itemize}
\end{slide}

% Availability of software
\begin{slide}{}
{\center Availability}

Welcome to use these ``tricks''.  Have a blast.

Software is more or less working.  No restore software yet - have to
do that by hand.  

Interested in alpha testing?  Contact Steve.  

Once it seems stable, we'll release it somehow.
\end{slide}

\end{bf}