Checklists Situation Sometimes when a backup run is finished, some of the file systems that should have been backed up weren't, either because the host was down or for some other error. When you run several runs in a night (as we do), you can accumulate quite a pile of file systems that need to be backed up. Scheduling backups to "catch up" with the backlog is a pain in the neck. It would be nice if there were a way to tell all-backups to catch up the undone backups from one or more previous runs. Implementation Thoughts I have at least two different schemes for implementing this, which I'll describe separately. Implementation One - checklist files Currently all-backups uses the host list (from the command line) to determine where to run do-backups. The do-backups program reads backup.hosts (which can be global, or local to that host) to determine what file systems should be backed up. I propose changing this so that all-backups reads backup.hosts directly, creating a "checklist" of backups that need to be done, containing at least the following information: proxyhost realhost filesystem chain level status The checklist might be stored in a file (say, $db_dir/checklists/run#), or they might be stored as a special database. The fields are: proxyhost the host that do-backups should run on, this is where database updates will be made from. realhost the real host that is being backed up. would normally be the same as proxyhost, except when we are doing proxy backups. This is where the backup command will be run, and would normally be the host that "owns" the file system being backed up. filesystem name of the file system to back up chain, level the chain, level of this backup - don't know if we need them, but we do want to make sure that if this is a 0,1 run that we don't do items from a 2,9 checklist. status the status of this particular backup - notstarted, started, done, failed If we create each line with a fixed size (with the status field padded to hold the largest possible value) then we can update the status field in place without rewriting the whole file for each update. all-backups would create the checklist from the list of hosts (-g switch), the backup.hosts file (which would have to be global), the -i and -x switches and so on. Just the way that things are done now, except that the decision making is spread between all-backups and do-backups currently. all-backups would call do-backups on each proxyhost. do-backups on the proxyhost will reread the checklist, pull out entries for that proxyhost, and run backup-* for each file system (urk, guess that we either have to put some/all of the info from backup.hosts into the checklist file, or have do-backups read it and merge the info with the checklist...) it should ignore entries that are marked "done". as backups are started, fail, or finish their status in the checklist file would be updated. the key here is that when a run finishes, you're left with a checklist that describes what was done and what was left undone. If we change all-backups to allow it to process a list of existing checklists instead of creating one, then we have a convenient way to use all-backups to finish the loose ends from any number of previous backup runs simply by telling it to run their checklists. If we implement this using separate files, then clean-db should be modified to delete them when their associated runs expire. In other words: all-backups (normal operation) read backup.hosts for each host in -g for each file system in backup.hosts for that host if no backup needs done (by -i, -x and chain, level) continue add to checklist ... for each proxy host call do-backups, don't pass -i, -x and junk do-backups (all operations) read backup.hosts read checklist for each entry listing me as proxyhost do backup for realhost:filesystem, using backup.hosts info as needed update status in the checklist file if the backup worked all-backups (catch-up operation, given a list of runs on the command line) read backup.hosts merge checklists by eliminating done backups and duplicates, and ensure that they were all for the same chain,level that we are dealing with now ... rest procedes as above There are some other issues about how the -g, -i and -x switches should be used (or should they be disallowed?) when in catch-up mode. Implementation Two - database Do we need a separate file? Couldn't we just tell all-backups to catch up what was left undone in run 213, 214 and 217? Good point. We could write something that would traverse the database, looking for backups that were left undone in those runs, and generate an artificial -i option to specifically do just those backups. Hmmm... read the backup.hosts file for each run that we are redoing lookup the run database entry get the list of hosts for each host that was supposed to be done in that run get the list of backups for that host from backup.hosts and the chain,level and other info * this is the flaw - we don't know whether the previous run was given special switches like -i or -x, so we can't duplicate the exact list of backups it was supposed to have done. for each backup, check and see whether it was done * this is icky - the host run entry might not exist, in which case all need to be done; the backup entry for one or more might not exist... if the backup entry isn't found or status isn't done, add it to the list to do blech. Maybe the first idea was better... Depends On It would be nice if we had a routine that read the backup.hosts file into some reasonable database, though I don't know what that would look like exactly. You'd want to be able to find a specific host:filesystem entry, so maybe that should be the index. See "backup.hosts".