How Incremental Backups Work
----------------------------

Between releases 0.6.0 and 0.6.1, I finally decided to investigate why
the Backup conduit would occasionally decide to back up everything,
even though much of it clearly hadn't been modified.  At the same
time, I decided to investigate the application behavior required for
BackupBuddy (www.backupbuddy.com) certification, to see if it could be
exploited to make Pyrite backups more efficient.

Each database header contains several fields which are relevant when
deciding whether to back it up.

The Backup Flag:

  The Palm Desktop backs up all databases that have this flag set,
  apparently with the idea that it would be unset on databases which
  are synchronized, and set otherwise.  In practice, trusting the
  backup flag alone does not produce a full backup, because it is not
  used consistently by all applications.

  Pyrite's default behavior is to ignore the backup flag, on the
  theory that it is better to back up too much than to back up too
  little.  The problem is that if the backup flag is not set, there is
  no way to easily tell why -- it could be unset because of programmer
  oversight, it could be unset because the data is normally handled by
  a conduit on Windows, or it could be unset because of user choice.

  In the future, the Backup conduit will probably become more
  intelligent about the backup flag, basing its behavior on what other
  conduits are active, heuristics about application support for the
  backup flag (eg. Doc readers usually support it correctly), and so
  on.  For now, it's probably safest to leave the "ignore_backup_bit"
  option on.
  
The Modification Date:

  The OS changes this date any time the database is modified,
  INCLUDING whenever the blocks in the database are moved.  Since OS
  memory management issues are uninteresting to the desktop, the
  modification date is not a reliable indicator of when a backup is
  actually necessary because the OS will often rearrange the contents
  of memory whenever a database is added or removed.

  On the other hand, BackupBuddy Certified applications deliberately
  reset the modification date to its former value, whenever they make
  an unimportant change to the database.  (The most obvious example of
  this is when a Doc reader records the current viewing position;
  this changes the database, but isn't important enough to warrant a
  complete backup.)

The Modification Number:

  The database header contains an integer which is incremented any
  time a record is added, deleted, or modified.  By itself, it is the
  most accurate means of determining when a database has been
  changed.

  BackupBuddy Certified applications, however, don't reset the
  modification number.  If such an application changes a database, the
  modifications will be counted in the modification number, but if the
  change is unimportant the modification date will be reset.  (Note: I
  definitely do NOT think this is improper behavior -- even when the
  desktop doesn't need to back something up, there should always be a
  way for other applications on the handheld to tell that a database
  has been changed.)


As you can see, it is not adequate to use either the modification
number or modification date alone to determine whether to back up a
database.  Using the date alone will cause unwanted backups whenever
memory is rearranged by the OS; using the number alone will cause
unwanted backups of data from BackupBuddy Certified applications.  As
a result, the Pyrite incremental backup only copies a database if BOTH
the number and date have been changed.

Incidentally, I don't really have any idea how BackupBuddy and
BackupBuddy Certified apps really work -- the above comments are based
on observation of what happens to database headers on my PalmPilot as
I manipulate its contents with various apps and utilities...