Checklists

Situation

  Sometimes when a backup run is finished, some of the file systems
  that should have been backed up weren't, either because the host was
  down or for some other error.  

  When you run several runs in a night (as we do), you can accumulate
  quite a pile of file systems that need to be backed up.  Scheduling
  backups to "catch up" with the backlog is a pain in the neck.

  It would be nice if there were a way to tell all-backups to catch up
  the undone backups from one or more previous runs.

Implementation Thoughts

  I have at least two different schemes for implementing this, which
  I'll describe separately.

Implementation One - checklist files

  Currently all-backups uses the host list (from the command line) to
  determine where to run do-backups.  The do-backups program reads
  backup.hosts (which can be global, or local to that host) to
  determine what file systems should be backed up.

  I propose changing this so that all-backups reads backup.hosts
  directly, creating a "checklist" of backups that need to be done,
  containing at least the following information:

	proxyhost  realhost  filesystem  chain  level  status

  The checklist might be stored in a file (say,
  $db_dir/checklists/run#), or they might be stored as a special
  database. 

  The fields are:

	proxyhost	the host that do-backups should run on, this is
			where database updates will be made from.

	realhost	the real host that is being backed up.  would
			normally be the same as proxyhost, except when
			we are doing proxy backups.  This is where the
			backup command will be run, and would normally
			be the host that "owns" the file system being
			backed up.

	filesystem	name of the file system to back up

	chain, level	the chain, level of this backup - don't know
			if we need them, but we do want to make sure
			that if this is a 0,1 run that we don't do
			items from a 2,9 checklist. 

	status		the status of this particular backup -
			notstarted, started, done, failed

  If we create each line with a fixed size (with the status field
  padded to hold the largest possible value) then we can update the
  status field in place without rewriting the whole file for each
  update.

  all-backups would create the checklist from the list of hosts (-g
  switch), the backup.hosts file (which would have to be global), the
  -i and -x switches and so on.  Just the way that things are done
  now, except that the decision making is spread between all-backups
  and do-backups currently.

  all-backups would call do-backups on each proxyhost.  do-backups on
  the proxyhost will reread the checklist, pull out entries for that
  proxyhost, and run backup-* for each file system (urk, guess that we
  either have to put some/all of the info from backup.hosts into the
  checklist file, or have do-backups read it and merge the info with
  the checklist...)  it should ignore entries that are marked "done".

  as backups are started, fail, or finish their status in the
  checklist file would be updated.

  the key here is that when a run finishes, you're left with a
  checklist that describes what was done and what was left undone.  If
  we change all-backups to allow it to process a list of existing
  checklists instead of creating one, then we have a convenient way to
  use all-backups to finish the loose ends from any number of previous
  backup runs simply by telling it to run their checklists.

  If we implement this using separate files, then clean-db should be
  modified to delete them when their associated runs expire.

  In other words:

  all-backups (normal operation)

	read backup.hosts
	for each host in -g
	    for each file system in backup.hosts for that host
		if no backup needs done (by -i, -x and chain, level) 
		  continue

		add to checklist

	...

	for each proxy host
	    call do-backups, don't pass -i, -x and junk

  do-backups (all operations)

	read backup.hosts
	read checklist
	for each entry listing me as proxyhost
	    do backup for realhost:filesystem, using backup.hosts info
	      as needed
	    update status in the checklist file if the backup worked

  all-backups (catch-up operation, given a list of runs on the command
  line) 

	read backup.hosts
	merge checklists
	  by eliminating done backups and duplicates, and ensure that
	  they were all for the same chain,level that we are dealing
	  with now

	...

	rest procedes as above

  There are some other issues about how the -g, -i and -x switches
  should be used (or should they be disallowed?) when in catch-up
  mode.

Implementation Two - database

  Do we need a separate file?  Couldn't we just tell all-backups to
  catch up what was left undone in run 213, 214 and 217?

  Good point.  We could write something that would traverse the
  database, looking for backups that were left undone in those runs,
  and generate an artificial -i option to specifically do just those
  backups. 

  Hmmm...

	read the backup.hosts file 
	for each run that we are redoing
	    lookup the run database entry
	    get the list of hosts
	    for each host that was supposed to be done in that run
		get the list of backups for that host from
		  backup.hosts and the chain,level and other info
		* this is the flaw - we don't know whether the
		  previous run was given special switches like -i or
		  -x, so we can't duplicate the exact list of backups
		  it was supposed to have done.
		for each backup, check and see whether it was done
		  * this is icky - the host run entry might not
		    exist, in which case all need to be done; the
		    backup entry for one or more might not exist...
		  if the backup entry isn't found or status isn't
		    done, add it to the list to do

	blech.

  Maybe the first idea was better...

Depends On

  It would be nice if we had a routine that read the backup.hosts file
  into some reasonable database, though I don't know what that would
  look like exactly.  You'd want to be able to find a specific
  host:filesystem entry, so maybe that should be the index.  See
  "backup.hosts". 

