\parindent=0pt
\parskip=15pt
\baselineskip=40pt
\hsize=9in
\vsize=7in
\hoffset=.2in
\nopagenumbers
\font\capsmcap=t-romsc at 24pt
\font\helvb=h-bol at 24pt
\font\bull=cmbsy10 scaled \magstep 3
\helvb
\centerline{Backup at Ohio State, Take 2}
\vskip 20pt
\centerline{Steve Romig}
\vskip 20pt
\centerline{romig@cis.ohio-state.edu}
\centerline{614-292-0915}
\centerline{Computer and Information Science Department}
\centerline{The Ohio State University}
\centerline{Columbus, Ohio, 43210}
\vfill\eject

% Motivation
Motivation:

\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} High density tape media
\item{\bull\raise5pt\hbox{\char'017}} More automated, more trust in
software, need more trustworthy software
\item{\bull\raise5pt\hbox{\char'017}} ``Fix'' human error problems
\item{\bull\raise5pt\hbox{\char'017}} Fix other problems
\parindent=0pt

\vfill\eject

% Old scheme
Doing a daily backup, the old way:

\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} 29 1/2 inch tapes, 5 drives, 29 servers
\item{\bull\raise5pt\hbox{\char'017}} Operator starts script on each server, 5 at a time
\item{\bull\raise5pt\hbox{\char'017}} Mount tapes, confirm with dump, unmount tapes, relabel tapes, update log book
\item{\bull\raise5pt\hbox{\char'017}} Re-rack 29 tapes
\parindent=0pt

Took 2.5 hours, fairly labor intensive.
\vfill\eject

% New scheme
Doing a daily, the new way:

\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} An 8 mm. tape, an Exabyte drive, 29 servers
\item{\bull\raise5pt\hbox{\char'017}} Operator starts 1 backup program
\item{\bull\raise5pt\hbox{\char'017}} Program asks for tape to be mounted, checks tape, starts backups on all servers
\item{\bull\raise5pt\hbox{\char'017}} Operator re-racks tape, updates paper log
\parindent=0pt

Takes about 2.5 hours, but only 15 minutes of operator time.
\vfill\eject

% Segue
Not going to talk about programs, databases, implementation - read the
paper.

Want to talk instead about tricks employed to solve various problems,
in hopes that others will find them useful.
\vfill\eject

% Problem 1 - labels
\centerline{Labels and Logs}

Problem: human error: mislabeled tapes, overwritten tapes.

Solution: labels and logs, both for people and computers.

\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} Physical tape label, paper log book
\item{\bull\raise5pt\hbox{\char'017}} Magnetic tape label, archive labels, on-line database
\parindent=0pt

\vfill\eject

\centerline{Labels and Logs}

Uses:

\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} Identify contents of physically unlabeled or mislabeled tape
\item{\bull\raise5pt\hbox{\char'017}} Verify that correct tape is mounted
\item{\bull\raise5pt\hbox{\char'017}} Identify archives on tape if database is unavailable
\item{\bull\raise5pt\hbox{\char'017}} Reconstruct database if necessary from tape information
\parindent=0pt

\vfill\eject

% Problem 2 - multiple chains
\centerline{Multiple Chains}

Chain - minimal set of backups needed to restore most recent saved
copy of a file system.  Most recent full, weekly and daily saves.

Problem with chain - lose tape, break chain, can't restore everything.
\vfill\eject

% problem 2 cont'd - multiple chains
\centerline{Multiple Chains}

``Simple'' solution - use multiple, independent chains.  

If you lose a tape from one chain, you can restore from the other
chain.

Can make as many chains as you want - how many are enough?

In our case, 2 seem fine - no significant extra cost, already doing 2
dailies a day, extra full save just takes a tape and 15 minutes per
server. 
\vfill\eject

% Parallel Stuff
\centerline{Parallel Backups}

First test run of dailies on 18 servers took 10 hours!  Ouch.  Single
tape bottleneck.

Fixed problem by using disk as a staging device.  Run backups in
parallel on servers, archive to disk, copy to tape as they finish.
\vfill\eject

% Parallel stuff
\centerline{Parallel Backups}

On each server:
\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} Search for a file system with
enough free space for backup
\item{\bull\raise5pt\hbox{\char'017}} Preference given to locally mounted file systems, and especially to file systems mounted on disks other than the one being backed up
\item{\bull\raise5pt\hbox{\char'017}} Copy backups to tape when they
are done
\item{\bull\raise5pt\hbox{\char'017}} Delete from disk when tape copy
is done
\parindent=0pt

Can now do daily backups on 22 servers in 2.5 hours.
\vfill\eject


% Large Incrementals
\centerline{Large Incrementals}

If a full restore is done, inode ctime is updated, next incremental
will include everything on the file system.  

\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} Fake timestamp in dumpdates
\item{\bull\raise5pt\hbox{\char'017}} Reset system clock to time of backup you are restoring from
\item{\bull\raise5pt\hbox{\char'017}} Use flag to warn not to do incremental
\parindent=0pt

\vfill\eject

% Large Incrementals
\centerline{Large Incrementals}

Using flag to avoid incrementals:

\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} Create .chainN just before doing the full save for chain N
\item{\bull\raise5pt\hbox{\char'017}} For incremental: compare the ctime of the .chainN file against the time of the last full save for that chain 
\item{\bull\raise5pt\hbox{\char'017}} If .chainN newer a full restore was done, warn user to do full save
\parindent=0pt

Also checked by the program that checks to see if all backups were
done.  If it notes that a full save should be done, it bitches about
it. 

\vfill\eject

% Scheduling Backups
\centerline{Scheduling Backups}

Best time for backups are often the worst for people to have to work.
Early morning, late at night.  Yech.

\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} Backup asks user for time to start backup
\item{\bull\raise5pt\hbox{\char'017}} User mounts tape as normal, script checks tape
\item{\bull\raise5pt\hbox{\char'017}} If all ok, script uses ``at'' to reschedule itself
\item{\bull\raise5pt\hbox{\char'017}} At appointed time, script starts again, rechecks tape, does backup 
\parindent=0pt

\vfill\eject

% Backups During Reboot
\centerline{Backups During Reboot}

Don't want to have to be here to babysit system while it does a backup
in single user mode.

\parindent=.4in
\item{\bull\raise5pt\hbox{\char'017}} Added code to /etc/rc to run /etc/backup if it exists 
\item{\bull\raise5pt\hbox{\char'017}} Backup script creates /etc/backup
\item{\bull\raise5pt\hbox{\char'017}} Schedules /etc/fastboot with ``at''
\item{\bull\raise5pt\hbox{\char'017}} System boots, executes /etc/boot, does backup
\item{\bull\raise5pt\hbox{\char'017}} /etc/rc then continues, system comes up
\parindent=0pt

\vfill\eject

% Availability of software
\centerline{Availability}

You're welcome to use these ``tricks''.  Have a blast.

Software is more or less working.  No restore software yet - have to
do that by hand.  

Interested in alpha testing?  Contact Steve.  

Once it seems stable, we'll release it somehow.
\vfill\eject

\centerline{Backup at Ohio State, Take 2}
\vskip 20pt
\centerline{Steve Romig}
\vskip 20pt
\centerline{romig@cis.ohio-state.edu}
\centerline{614-292-0915}
\centerline{Computer and Information Science Department}
\centerline{The Ohio State University}
\centerline{Columbus, Ohio, 43210}
\vfill\eject

\end