Motivation

    * high density media, reduce tape flipping
    * more auto, less human - more trust in SW, need better SW
    * fix some human error problems
    * fix other scaling problems

Old Scheme

    29 1/2 tapes for dailies.  5 1/2 inch tape drives.  opr starts
    backup script on each server in turn, mounts/unmounts tapes as
    needed.  relabels tapes, updates log.  reracks tapes.  takes about
    2.5 hours.

New Scheme

    1 Exabyte tape.  1 drive.  opr starts backup script once, adds
    info about chain, level, list of servers, location of tape drive.
    script selects tape from free tape pool.  opr mounts tape, script
    checks to see that its the right tape, then handles backups on all
    servers.  when done, tape ejected, opr relabels, updates log,
    reracks.  takes about 2.5 hours of real time, 15 minutes of people
    time. 

Not going to talk much about specifics in system - read paper, etc.

Concentrate on tricks used to solve various problems.

Problem 1: human error

    mislabeled tapes (phys label wrong or damaged)
    misused tapes (wrong tape for wrong backup)

    labels - phys label on tape, tape ID, on-line database, magnetic
    tape label, volume labels.

    can identify what's on a unlabeled tape.  verify that correct tape
    is mounted in drive.

    can identify volumes on tape even if on-line database is
    corrupted.  

    can reconstruct necessary parts of on-line database from tape
    info.  

Problem 2: missing tapes

    concept of a chain - latest full, weekly, daily
    lose a tape, break the chain.

    "easy" solution: multiple, independant chains.
    easily implemented with multiple dumpdates files - choose
    dumpdates.0 or dumpdates.1.  

    if chain 0 is broken, restore from chain 1.

    cost: extra backups.  in our case, already doing 2 dailies a day,
    just make 1 set in a different chain.  cost of extra full saves
    isn't that high - easy to set up, only costs a tape per server.
    avoid downtime by running it while system is active.

Problem 3: single tape bottleneck

    found that daily backup of 18 servers onto 1 tape took 10 hours,
    used to take 2.5 for 29 servers.  

    problems is that the single tape is a bottleneck.

    solution is to use any available disk space as a staging area for
    backups.  daily backups started in parallel on all servers, done
    onto available local disk space, preference given to space on
    other disk devices for performance.  copied to tape when done.  

    now takes about 2.5 hours for dailes on 29 servers again.

Problem 4: large incrementals

    do full restore, all ctimes change, do daily, everything gets
    backed up.  ouch.

    solutions: there are several.  can hack date in tardates/dumpdates
    file when you do a full restore, so that subsequent incrementals
    work correctly.  This won't work if you are using multiple chains.

    set the time to the time of the backup when restoring from it.
    then ctime's will have the 'right' value.  orf.

    our solution: when we do a full save, create a .chainN file at the
    root of the file system being saved.  if we do a full restore, the
    ctime on the .chainN file will change.  when we do incrementals
    check the ctime on the .chain file - if more recent than time of
    last full save, skip this file system and bitch about the need for
    a full save here.  

    there's a program that runs nightly on each server which checks to
    see that the backups have been done that also bitches about file
    systems that need full saves.

Problem 6: scheduling backups

    best times for backups aren't convenient for people.  backups
    don't require much human intervention now, don't need to be around
    but to start it.  backup script prompts you for start time, checks
    tape after you've mounted it to make sure its right, then
    reschedules itself with 'at' to start at the appointed time.  when
    it restarts, rechecks tape, does backup, sends mail to folks when
    done. 

Problem 7: backups during reboot

    Some folks like running backup with file systems quiet.  Best
    time, late at night, or over weekend.  Hate having to be here to
    take system down, run backup, etc.

    Add code to /etc/rc to run /etc/backup if it exists.  backup
    script changed to ask whether to run backup in singleuser mode -
    if yes, creates /etc/backup with appropriate commands to set
    system up for backups and to run backup script.  then schedules
    fastboot with at for appointed time.  time rolls around, system
    boots, runs /etc/backup, backup done, tape ejected, mail sent,
    /etc/backup nuked, boot finishes, system back up again.

future

    on-line backups from staging areas for easy restores

Availability

